A control method, device, medium and product

Abstract

The application discloses a control method, device, medium and product. The method comprises the following steps: inputting a real-time collected cutting piece image and a voice instruction into a target control model to obtain a plurality of groups of 7-dimensional control parameters, wherein the 7-dimensional control parameters comprise a three-dimensional position, a three-dimensional posture and a gripper opening degree; and controlling a mechanical arm based on the plurality of groups of 7-dimensional control parameters, so that the mechanical arm drives the cutting piece to move to a target position. Through the technical scheme, the cutting piece alignment accuracy can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of textile and apparel manufacturing technology, and in particular to a control method, equipment, medium and product. Background Technology

[0002] Currently, in the textile and apparel manufacturing industry, cutting piece alignment is an important process before sewing, requiring robots to accurately place the cutting pieces at specific positions and angles.

[0003] Therefore, how to precisely control the robot to place the cut pieces at specific positions and angles has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a control method, device, medium, and product that can improve the accuracy of cut piece alignment.

[0005] According to one aspect of the present invention, a control method is provided, comprising: The real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: three-dimensional position, three-dimensional posture and gripper opening degree. The robotic arm is controlled based on multiple sets of 7-dimensional control parameters, so that the robotic arm can move the cut piece to the target position.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method described in any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the control method described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the control method as described in any of the embodiments of the present invention.

[0009] This invention provides an embodiment of the invention that inputs real-time acquired images of the cut pieces and voice commands into a target control model to obtain multiple sets of 7-dimensional control parameters. These 7-dimensional control parameters include: three-dimensional position, three-dimensional posture, and gripper opening degree. Based on these multiple sets of 7-dimensional control parameters, the robotic arm is controlled to move the cut pieces to the target position. This allows for accurate control parameters, enabling the robot to precisely place the cut pieces at specific positions and angles to the target location, thereby improving the accuracy of cut piece alignment.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a control method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] Example 1 Figure 1 This is a flowchart illustrating a control method provided in an embodiment of the present invention. This embodiment is applicable to the control of a robotic arm. The method can be executed by a control device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps: S110 inputs real-time collected cut images and voice commands into the target control model to obtain multiple sets of 7-dimensional control parameters.

[0017] The seven control parameters include: three-dimensional position, three-dimensional attitude, and gripper opening degree.

[0018] In this embodiment, the target control model can be a vision-language-action (VLA) model.

[0019] In this embodiment, the three-dimensional position includes the X, Y, and Z coordinates of the gripper located at the end of the robotic arm. The three-dimensional attitude includes the roll angle, pitch angle, and yaw angle of the gripper located at the end of the robotic arm.

[0020] In this embodiment, the cut piece image can be captured by an industrial camera, and the voice command can be input by the operator or the host system.

[0021] Optionally, the real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: The real-time captured cut images and voice commands are input into the target control model, and the joint angles are output. Multiple sets of 7-dimensional control parameters are obtained based on the joint angle mapping.

[0022] Optionally, the target control model includes, from input to output, a visual encoder, an LLM backbone network, and an action prediction head.

[0023] The real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: The real-time captured images of the cut pieces are input into a visual encoder to obtain visual features.

[0024] In this embodiment, the visual encoder is a visual encoder with adjusted bit count and weight matrix for each network layer.

[0025] Voice commands are input into the LLM backbone network to obtain semantic features.

[0026] In this embodiment, the LLM backbone network is a quantized LLM backbone network.

[0027] It should be noted that the LLM backbone network is the component with the largest number of parameters in the target control model. INT4 quantization is performed on the LLM backbone network, which is a core method to reduce its memory footprint. The quantized LLM backbone network can accurately adapt to the high-precision visual feature token sequence transmitted from the visual front end while maintaining general semantic understanding capabilities. During the inference phase, the INT4 weights are dequantized to FP16 precision in real time, and a mixed precision calculation mode of 4-bit weight, 16-bit activation (W4A16) is executed. This maximizes the model's multimodal instruction inference capability while extremely compressing GPU memory usage.

[0028] Visual and semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters.

[0029] In this embodiment, the method of inputting visual features and semantic features into the motion prediction head to obtain multiple sets of 7-dimensional control parameters can be as follows: the visual features are pre-screened, the screened visual features and semantic features are input into the motion prediction head, the joint angles are output, and multiple sets of 7-dimensional control parameters are obtained based on the joint angle mapping.

[0030] Optionally, before inputting the real-time acquired cut images and voice commands into the target control model to obtain multiple sets of 7-dimensional control parameters, the following steps are also included: Obtain the training sample set.

[0031] The training sample set includes training samples and the corresponding cropping coordinates of the training samples. The training samples include cropping image samples and voice command samples.

[0032] In this embodiment, the cropped image samples in the training sample set represent the cropped alignment accuracy. The corresponding cut image.

[0033] The training samples are input into the model to be trained to obtain the predicted 7-dimensional control parameters.

[0034] In this embodiment, the training samples are input into the model to be trained, and the predicted joint angles are output. Based on the predicted joint angle mapping, the predicted 7-dimensional control parameters are obtained.

[0035] Based on the predicted 7-dimensional control parameters and the cut piece position coordinates corresponding to the training samples, the alignment task loss function is determined.

[0036] In this embodiment, the alignment task loss function is determined based on the predicted 7-dimensional control parameters and the corresponding piece position coordinates of the training samples by performing a coordinate system transformation on the predicted 7-dimensional control parameters to obtain the predicted 3D position coordinates of the piece. It should be noted that the original predicted action output from the bottom layer of the action prediction head is a joint space sequence of the robotic arm, and errors in joint space cannot directly reflect the actual physical space errors. Therefore, before quantitative evaluation, the joint angles must be accurately mapped to the gripper's 3D position and 3D posture in Cartesian space. in, is the homogeneous transformation matrix of the end effector gripper of the robotic arm relative to the robotic arm base; n is the number of joints of the robotic arm; Let be the joint angle of the i-th joint predicted by the model; It is a homogeneous transformation matrix from the (i-1)th link coordinate system to the ith link coordinate system, defined by conventions such as DH parameters, and the multiplication must be performed on the right strictly in the joint order from the base to the end flange. It is a fixed tool coordinate system transformation matrix used to compensate for the geometric dimensional offset between the flange mounting surface and the actual clamping center of the jaws.

[0037] Subsequently, from the matrix Extract the three-dimensional position and three-dimensional orientation of the gripper in the robot arm coordinate system (i.e. The virtual coordinate system is then transformed into the three-dimensional position coordinates of the predicted pattern piece based on the affine transformation mapping between the virtual coordinate system and the robotic arm coordinate system. The affine transformation mapping formula between the virtual coordinate system and the robotic arm coordinate system is as follows: ; in, Let these be the coordinates of the three-dimensional position of the cut piece in the virtual coordinate system. The gripper position coordinates in the robotic arm coordinate system output by the model; Let be the global affine matrix representing the rotation, scaling, and shearing transformations between the two coordinate systems; b is the translation vector representing the relative spatial offset between the origins of the two coordinate systems. Wherein, the affine matrix... During the system deployment phase, the translation vector b is obtained by manually collecting more than four sets of known three-dimensional visual-physical correspondence point data within the workspace of the robotic arm, and then using the least squares method to solve for the global optimal solution.

[0038] The number of bits and the target weight matrix for each network layer of the visual encoder are determined based on the alignment task loss function.

[0039] In this embodiment, the number of bits in the network layer can be either INT4 or INT8.

[0040] In this embodiment, the method for determining the number of bits in each network layer of the visual encoder based on the alignment task loss function can be as follows: determine the cropping position error function based on the alignment task loss function; determine the cropping alignment accuracy of each network layer of the visual encoder based on the cropping position error function; determine the cropping alignment sensitivity of each network layer of the visual encoder based on the cropping alignment accuracy of each network layer of the visual encoder and the cropping alignment accuracy corresponding to the training sample set; and determine the number of bits in each network layer of the visual encoder based on the cropping alignment sensitivity of each network layer of the visual encoder.

[0041] In this embodiment, the method for determining the target weight matrix of each network layer of the visual encoder based on the alignment task loss function can be as follows: determine the scaling factor of the J-th channel of the initial weight matrix of each network layer of the visual encoder according to the alignment task loss function and the initial weight matrix of each network layer of the visual encoder; determine the target weight matrix of each network layer of the visual encoder according to the initial weight matrix of each network layer of the visual encoder, the scaling factor of the J-th channel and the quantization step size of the J-th channel.

[0042] The target control model is obtained by updating the training model based on the number of bits in each network layer of the visual encoder and the target weight matrix.

[0043] Optionally, determining the number of bits for each network layer of the visual encoder based on the alignment task loss function includes: Based on the alignment task loss function, determine the piece position error function.

[0044] In this embodiment, the cut piece position error function is determined based on the alignment task loss function and the first formula, wherein the first formula is: ,in, Let be the loss value corresponding to the k-th training sample, where K is the total number of training samples in the training sample set. E is the piece position error function.

[0045] Based on the piece position error function, the piece alignment accuracy of each network layer of the visual encoder is determined.

[0046] In this embodiment, the clipping alignment accuracy of the l-th network layer of the visual encoder is determined by quantizing only the l-th network layer to b bits, while all other layers of the model maintain FP16 accuracy. After the model is fully run, the final alignment accuracy is determined based on the clipping position error function.

[0047] The crop alignment sensitivity of each network layer of the visual encoder is determined based on the crop alignment accuracy of each network layer and the crop alignment accuracy of the training sample set.

[0048] In this embodiment, the crop alignment sensitivity of each network layer of the visual encoder is determined based on the crop alignment accuracy of each network layer, the crop alignment accuracy corresponding to the training sample set, and a second formula. The second formula is: , The crop alignment sensitivity is quantized to b bits for the l-th network layer of the visual encoder. When using the controlled variable method, only the l-th network layer is quantized to b bits, while all other layers of the model maintain FP16 precision. The final alignment accuracy output after running the model completely is the metric that directly reflects the impact of each layer's quantization on the accuracy of the cut piece alignment terminal task, rather than the intermediate feature quality loss. To determine the alignment accuracy of the cut pieces corresponding to the training sample set.

[0049] The number of bits for each network layer of the visual encoder is determined based on the clip alignment sensitivity of each network layer.

[0050] In this embodiment, the number of bits for each network layer of the visual encoder is determined based on the clipping alignment sensitivity of each network layer. A pre-set alignment accuracy incremental loss threshold is also included. (Considering both position and angle equivalent errors), for each network layer: like This layer uses INT4 quantization, and the loss of quantization accuracy is within the tolerance range of the cut piece alignment. like and This layer uses INT8 quantization, which results in excessive loss for INT4 but is acceptable for INT8. like This layer maintains FP16, and the quantization error exceeds the tolerance range of the cut piece alignment task.

[0051] This invention's embodiments complete bit allocation based on measured accuracy loss, departing from the inherent paradigm of general hierarchical quantization; it accurately determines highly sensitive deep network layers by forming differentiated sensitivity rules for network layers corresponding to three types of features: corner points, contours, and textures of the cut pieces, thus avoiding the problem of over-quantization of key layers caused by general prior rules and adapting to the alignment conditions of the cut piece robot.

[0052] Optionally, determining the target weight matrix for each network layer of the visual encoder based on the alignment task loss function includes: Based on the alignment task loss function and the initial weight matrix of each network layer of the visual encoder, determine the scaling factor of the J-th channel of the initial weight matrix of each network layer of the visual encoder.

[0053] In this embodiment, to protect the representation accuracy of key geometric features such as corner points and contours of the cut pieces at extremely low bit widths, the average gradient of the J-th channel of the weight matrix W of each layer of the visual encoder with respect to the loss of the cut piece alignment task is calculated, and this gradient is used as the scaling factor of the J-th channel of the initial weight matrix of each network layer of the visual encoder. The scaling factor of the J-th channel of the initial weight matrix of each network layer of the visual encoder is calculated based on the following formula: ; in, It was obtained through backpropagation calculation of the complete model. is the scaling factor for the J-th channel.

[0054] The target weight matrix of each network layer of the visual encoder is determined based on the initial weight matrix of each network layer, the scaling factor of the J-th channel, and the quantization step size of the J-th channel.

[0055] In this embodiment, the target weight matrix of the l-th network layer of the visual encoder is determined based on the following formula: ; in, Let be the initial weight matrix of the l-th network layer. Let l be the target weight matrix of the l-th network layer. is the quantization step size of the J-th channel of the l-th network layer.

[0056] It should be noted that for the critical channel of pattern alignment ( High), scaling factor This allows the channel weight to occupy a finer discrete level within the quantization range, resulting in smaller quantization errors; for non-critical channels in piece alignment ( (Low), allowing for larger quantization steps and more aggressive compression.

[0057] Optionally, the visual features include a sequence of visual tokens; Visual and semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters, including: Obtain the cropped area mask of the cropped image sample.

[0058] In this embodiment, the method for obtaining the cut area mask of the cut image sample can be as follows: convert the cut image sample into a grayscale image; determine the foreground pixels and background pixels in the cut image sample based on the grayscale image; and determine the cut area mask of the cut image sample based on the foreground pixels and background pixels in the cut image sample.

[0059] The cropped region mask of the cropped image sample is mapped to the visual token space to obtain the region type of each visual token in the visual token sequence.

[0060] The types of regions include: cut area and background area.

[0061] In this embodiment, the method of mapping the cropped region mask of the cropped image sample to the visual token space to obtain the region type of each visual token in the visual token sequence can be as follows: determine the proportion of cropped mask pixels corresponding to each visual token based on the cropped region mask of the cropped image sample and the pixel set of each visual token in the visual token sequence; determine the region type of each visual token in the visual token sequence based on the proportion of cropped mask pixels corresponding to each visual token.

[0062] The visual tokens of the cropped area, the visual tokens of the filtered background area, and the semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters.

[0063] In this embodiment, the method of inputting the visual tokens of the cut area, the visual tokens of the filtered background area, and semantic features into the action prediction head to obtain multiple sets of 7-dimensional control parameters can be as follows: The visual tokens of the background area are filtered to obtain filtered background area visual tokens. The visual tokens of the cut area, the filtered background area visual tokens, and semantic features are input into the action prediction head to obtain joint angles. Multiple sets of 7-dimensional control parameters are obtained based on the joint angle mapping. It should be noted that the method of filtering the visual tokens of the background area can be as follows: A score for the visual token of each background area is determined, and visual tokens of background areas with scores below a threshold are discarded. The score of the visual token of each background area can be determined based on attention weights.

[0064] Optionally, the cropped region mask of the cropped image sample is mapped to the visual token space to obtain the region type of each visual token in the visual token sequence, including: Based on the cropped area mask of the cropped image sample and the pixel set in each visual token in the visual token sequence, the proportion of cropped mask pixels corresponding to each visual token is determined.

[0065] In this embodiment, the pixel ratio of the cropped mask corresponding to the i-th visual token is determined based on the following formula: ; in, The pixel percentage of the cropped mask corresponding to the i-th visual token. Let i be the set of pixels in the i-th visual token. Let be the total number of pixels in the i-th visual token. This is the mask for the cropped area of ​​the cropped image sample.

[0066] Based on the pixel proportion of the crop mask corresponding to each visual token, determine the region type of each visual token in the visual token sequence.

[0067] In this embodiment, a threshold for determining the cut area is preset. ,when When the i-th visual token is selected, it is marked as the cropped area; otherwise, it is marked as the background area.

[0068] Optionally, obtaining the cropped region mask of the cropped image sample includes: The cropped image sample is converted into a grayscale image.

[0069] Based on the grayscale image, determine the foreground pixels and background pixels in the cropped image sample.

[0070] The cropping region mask of the cropping image sample is determined based on the foreground and background pixels in the cropping image sample.

[0071] In this embodiment, the foreground pixels in the cut image sample are marked as 1 and the background pixels are marked as 0 to obtain the cut area mask of the cut image sample.

[0072] S120 controls the robotic arm based on multiple sets of 7-dimensional control parameters, so that the robotic arm can move the cut piece to the target position.

[0073] This invention abandons the common model lightweighting paradigm. Instead, it performs layer-by-layer quantization configuration of the visual encoder through piece alignment accuracy analysis and guides AWQ calibration by replacing activation amplitude with task loss gradient. Simultaneously, it introduces physical masks in conjunction with attention scoring to perform dual-stream pruning, ensuring absolute preservation of key geometric features. Finally, based on the quantized model, multiple sets of 7-dimensional control parameters are obtained. These parameters are used to control a robotic arm, enabling it to move the piece to the target position, thus achieving both high real-time performance and high precision piece alignment. It should be noted that the control method provided in this embodiment can be deployed on a server or in an intelligent robot. If deployed on a server, it needs to cooperate with an intelligent robot to move the piece to the target position via the robotic arm. If deployed in an intelligent robot, the model can be pre-quantized on the server, and the quantized model can then be deployed in the intelligent robot. This invention does not impose any limitations on this approach.

[0074] The technical solution of this embodiment obtains multiple sets of 7-dimensional control parameters by inputting real-time acquired cut piece images and voice commands into the target control model. The 7-dimensional control parameters include: three-dimensional position, three-dimensional posture, and gripper opening degree. Based on the multiple sets of 7-dimensional control parameters, the robotic arm is controlled so that the robotic arm moves the cut piece to the target position. Accurate control parameters can be obtained to precisely control the robot to place the cut piece at a specific position and angle to the target position, thereby improving the accuracy of cut piece alignment.

[0075] Example 2 Figure 2A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0076] like Figure 2 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.

[0077] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0078] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as control methods.

[0079] In some embodiments, the control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via read-only memory 12 and / or communication unit 19. When the computer program is loaded into random access memory 13 and executed by processor 11, one or more steps of the control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the control method by any other suitable means (e.g., by means of firmware).

[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0085] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0086] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0087] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the control method according to any embodiment of the invention.

[0088] In the implementation of a computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A control method, characterized in that, include: The real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: three-dimensional position, three-dimensional posture and gripper opening degree. The robotic arm is controlled based on multiple sets of 7-dimensional control parameters, so that the robotic arm can move the cut piece to the target position.

2. The method according to claim 1, characterized in that, The target control model, from input to output, includes: a visual encoder, an LLM backbone network, and an action prediction head. The real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: The real-time captured images of the cut pieces are input into a visual encoder to obtain visual features; Voice commands are input into the LLM backbone network to obtain semantic features; Visual and semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters.

3. The method according to claim 2, characterized in that, Before inputting the real-time acquired cut images and voice commands into the target control model to obtain multiple sets of 7-dimensional control parameters, the following steps are also included: Obtain a training sample set, wherein the training sample set includes training samples and the corresponding cropping position coordinates of the training samples, and the training samples include: cropping image samples and voice command samples; The training samples are input into the model to be trained to obtain the predicted 7-dimensional control parameters; Based on the predicted 7-dimensional control parameters and the cut piece position coordinates corresponding to the training samples, the alignment task loss function is determined. The number of bits and the target weight matrix of each network layer of the visual encoder are determined based on the alignment task loss function. The target control model is obtained by updating the training model based on the number of bits in each network layer of the visual encoder and the target weight matrix.

4. The method according to claim 3, characterized in that, Determining the number of bits for each network layer of the visual encoder based on the alignment task loss function includes: Based on the alignment task loss function, determine the piece position error function; Based on the piece position error function, the piece alignment accuracy of each network layer of the visual encoder is determined; Based on the crop alignment accuracy of each network layer of the visual encoder and the crop alignment accuracy corresponding to the training sample set, the crop alignment sensitivity of each network layer of the visual encoder is determined. The number of bits for each network layer of the visual encoder is determined based on the clip alignment sensitivity of each network layer.

5. The method according to claim 3, characterized in that, The target weight matrix for each network layer of the visual encoder is determined based on the alignment task loss function, including: Based on the alignment task loss function and the initial weight matrix of each network layer of the visual encoder, determine the scaling factor of the J-th channel of the initial weight matrix of each network layer of the visual encoder. The target weight matrix of each network layer of the visual encoder is determined based on the initial weight matrix of each network layer, the scaling factor of the J-th channel, and the quantization step size of the J-th channel.

6. The method according to claim 3, characterized in that, The visual features include a visual token sequence; Visual and semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters, including: Obtain the cropped area mask of the cropped image sample; The cropped region mask of the cropped image sample is mapped to the visual token space to obtain the region type of each visual token in the visual token sequence, wherein the region type includes: cropped region and background region; The visual tokens of the cropped area, the visual tokens of the filtered background area, and the semantic features are input into the action prediction head to obtain multiple sets of 7-dimensional control parameters.

7. The method according to claim 6, characterized in that, The cropped region mask of the cropped image sample is mapped to the visual token space to obtain the region type of each visual token in the visual token sequence, including: Based on the cropped area mask of the cropped image sample and the pixel set in each visual token in the visual token sequence, determine the cropped mask pixel ratio corresponding to each visual token; Based on the pixel proportion of the crop mask corresponding to each visual token, determine the region type of each visual token in the visual token sequence.

8. The method according to claim 6, characterized in that, Obtaining the cropped region mask of the cropped image sample includes: Convert the cropped image sample into a grayscale image; Based on the grayscale image, determine the foreground pixels and background pixels in the cropped image sample; The cropping region mask of the cropping image sample is determined based on the foreground and background pixels in the cropping image sample.

9. The method according to claim 1, characterized in that, The real-time acquired cut images and voice commands are input into the target control model to obtain multiple sets of 7-dimensional control parameters, including: The real-time captured cut images and voice commands are input into the target control model, and the joint angles are output. Multiple sets of 7-dimensional control parameters are obtained based on the joint angle mapping.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the control method according to any one of claims 1-9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the control method according to any one of claims 1-9.

Images