Control method, device, system, robot and storage medium for robot

Abstract

Embodiments of the present application provide a control method, device and system for a robot, the robot and a storage medium. The control method comprises: obtaining a region image of a working region of the robot, the working region comprising at least one part; inputting the region image into a preset vision model to output a processed region image through the preset vision model, the processed region image comprising an unmarked region and a region where the part with added labels is located; determining a task type to be performed by the robot on the part based on a category of the part; inputting the processed region image and the task type to be performed into an action segmentation algorithm model to output a control strategy for the part through the action segmentation algorithm model, wherein the action segmentation algorithm model comprises a self-attention mechanism encoder, and the control strategy comprises at least one of a motion trajectory, a transportation speed and a transportation torque of a plurality of mechanical arms; and controlling the plurality of mechanical arms to perform corresponding control operations on the part based on the control strategy.

Description

Technical Field

[0001] This application relates to the field of engineering machinery technology, specifically to a control method, device, system, robot, and storage medium for robots. Background Technology

[0002] Due to the wide variety of types, shapes, and weights of construction machinery parts, the traditional single-arm robotic operation mode is insufficient to meet the needs of modern production lines. Two-handed collaborative operation technology for transporting complex parts is developed. This technology uses a model-trained coordination algorithm to adjust the movements of the robot's two arms in real time, ensuring efficient collaboration during the transport of complex parts. Two-handed collaborative operation can improve the production efficiency of construction machinery, reduce labor costs, and ensure production safety.

[0003] Existing technologies typically employ low-cost hardware for learning sophisticated two-handed manipulation and have proposed the ACT (Active Action Control) strategy. However, this approach often relies on pre-defined template matching or simple feature extraction, failing to accurately identify parts with complex and varied shapes. Furthermore, it cannot automatically adjust the manipulation strategy based on part characteristics and transportation tasks, thus limiting the transportation efficiency and accuracy of robotic two-handed collaborative manipulation. Summary of the Invention

[0004] The purpose of this application is to provide a control method, device, system, robot, and storage medium for robots, in order to solve the technical defects in the prior art, such as the inability to accurately identify parts with complex and varied shapes, and the inability to automatically adjust the operation strategy for parts according to the characteristics of the parts and the transportation task, resulting in low transportation efficiency and accuracy of the robot's two-handed coordination.

[0005] To achieve the above objectives, the first aspect of this application provides a control method for a robot, the robot comprising multiple cooperating robotic arms, the control method comprising:

[0006] Acquire a region image of the robot's working area, which includes at least one part;

[0007] The region image is input into a preset visual model to determine the category of the part and output the processed region image. The processed region image includes unmarked areas and areas where the marked parts are located.

[0008] The type of task the robot needs to perform on a part is determined based on the part's category;

[0009] The processed region image and the type of task to be executed are input into the action block algorithm model so that the action block algorithm model can output a control strategy for the part. The action block algorithm model includes a self-attention mechanism encoder and the control strategy includes at least one of the motion trajectory of multiple robotic arms, transport speed and transport torque.

[0010] Based on the control strategy, multiple robotic arms are controlled to perform corresponding control operations on the parts.

[0011] In embodiments of this application, determining the type of task to be performed by the robot for a part based on the part's category includes: determining task rules matching the part's category based on a preset task rule library, wherein each task rule in the preset task rule library defines the transportation method corresponding to each type of part; and determining the type of task to be performed by the robot for the part based on the task rules.

[0012] In the embodiments of this application, the preset visual model outputs the processed region image as follows: after inputting the region image into the preset visual model, the preset visual model extracts the shape, size and material of the parts in the region image for comprehensive analysis to identify the parts; the identified parts are marked, and the region image including the unmarked area and the area where the marked parts are located is output.

[0013] In the embodiments of this application, the action segmentation algorithm model outputs a control strategy for the part, which includes: after inputting the processed region image and the type of task to be executed into the action segmentation algorithm model, determining the target parameters that match the type of task to be executed, wherein the action segmentation algorithm model includes multiple model parameters, which are trained based on multiple types of parts and the transportation method of each type of part; adjusting the parameters of the control action segmentation algorithm model to the target parameters, so as to output a control strategy for the part based on the processed region image.

[0014] In embodiments of this application, the control method further includes: during the process of controlling multiple robotic arms to perform corresponding control operations on a part based on a control strategy, acquiring the operation duration and operation data of the multiple robotic arms in real time, wherein the operation data includes at least one of the position, posture, and force conditions of the multiple robotic arms; determining that the robot is in an abnormal state when the operation duration reaches a preset threshold; switching the control strategy for the part based on the operation data, and controlling the multiple robotic arms to perform corresponding control operations on the part based on the updated control strategy.

[0015] In embodiments of this application, the control method further includes: after acquiring the region image, performing noise reduction, contrast enhancement, and size cropping on the region image in sequence to improve the image quality of the region image.

[0016] A second aspect of this application provides a control device for a robot, comprising:

[0017] The memory is configured to store instructions;

[0018] The controller is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the aforementioned control method for the robot.

[0019] A third aspect of this application provides a control system for a robot, including the aforementioned control device for a robot.

[0020] A fourth aspect of this application provides a robot, comprising:

[0021] The aforementioned control system for robots;

[0022] Multiple robotic arms working together.

[0023] A fifth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned control method for a robot.

[0024] The above technical solution acquires a region image of the robot's working area, inputs the region image into a preset visual model, and outputs a processed region image through the preset visual model. The processed region image includes unmarked areas and areas where marked parts are located. Based on the part's category, the type of task the robot needs to perform on the part is determined. The processed region image and the task type are then input into an action segmentation algorithm model, which outputs a control strategy for the part. The action segmentation algorithm model includes a self-attention mechanism encoder, and the control strategy includes at least one of the motion trajectories, transport speeds, and transport torques of multiple robotic arms. Based on the control strategy, multiple robotic arms are controlled to perform corresponding control operations on the part. This solution introduces a preset visual model to adapt to parts with complex and varied shapes, and simultaneously introduces an action segmentation algorithm model to output control strategies for the parts, thereby achieving accurate identification of parts with complex and varied shapes and adjusting the control strategy for the parts according to the part's characteristics and the task type.

[0025] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0026] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0027] Figure 1 The illustration shows a flowchart of a control method for a robot according to an embodiment of this application;

[0028] Figure 2 This illustration schematically shows a flowchart of an image processing method according to an embodiment of the present application;

[0029] Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0031] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0032] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0033] Figure 1 The illustration schematically shows a flowchart of a control method for a robot according to an embodiment of this application. Figure 1 As shown in the figure, this application provides a control method for a robot, the robot including multiple robotic arms that cooperate with each other, and the control method may include the following steps.

[0034] Step 101: Obtain an image of the robot's working area, which includes at least one part.

[0035] In this embodiment, it should be noted that the bimanual cooperative operation technology refers to a coordination algorithm based on model training that adjusts the movements of the robot's two arms in real time to ensure efficient cooperation during transportation. A high-definition camera is installed on the robot, and the lens of this camera can cover the robot's working area. Therefore, the high-definition camera can capture real-time images of the working area. The acquired image should include at least one component.

[0036] Step 102: Input the region image into the preset visual model to determine the category of the part through the preset visual model and output the processed region image, wherein the processed region image includes unmarked areas and areas where marked parts are located.

[0037] In this embodiment, it should be noted that the preset visual model can refer to the CogVLM model. CogVLM is a visual language model, consisting of a Vision Transformer (ViT) encoder, an MLP adapter, a pre-trained Large Language Model (GPT), and a visual expert module. CogVLM achieves deep fusion of visual and language features without sacrificing the performance of any NLP task by utilizing a trainable visual expert module to deeply integrate language and visual encoding in the attention layer and feedforward network layer. In this technical solution, the CogVLM model is trained based on historical images of various engineering machinery parts; therefore, the CogVLM model already possesses the ability to recognize various engineering machinery parts. Specifically, after capturing real-time images of the robot's working area using a high-definition camera mounted on the robot, the regional images are input into the CogVLM model. Through its powerful visual understanding capabilities, the CogVLM model can accurately identify parts, mark their locations, and output the processed regional image. Therefore, the regional image output by the CogVLM model includes unmarked areas and the areas where marked parts are located. Figure 2 As shown, a flowchart of image processing is provided, such as Figure 2 As shown, after the region image is input into the CogVLM model, the region image output by the CogVLM model marks the area where the part is located with lines.

[0038] In this embodiment of the application, the control method further includes: after acquiring the region image, performing noise reduction, contrast enhancement, and size cropping on the region image in sequence to improve the image quality of the region image.

[0039] In this embodiment, it should be noted that the region image captured by the high-definition camera is the original image. Original images typically contain noise, which is unexplained data interference, usually manifested as isolated pixels or pixel blocks, affecting image quality and observable information. Therefore, to improve image quality for subsequent model processing, before outputting the region image to the CogVLM model, the original region image needs to undergo denoising, contrast enhancement, and cropping to improve image quality. Denoising effectively removes noise from the original image, restoring a realistic image while minimizing the loss of original features. Contrast enhancement effectively improves the visual effect of the image. Furthermore, since the high-definition camera's capture range covers the robot's working area, the original region image can be cropped to remove unnecessary and invalid areas.

[0040] In this embodiment of the application, the output of the processed region image by the preset visual model includes: after the region image is input into the preset visual model, the preset visual model extracts the shape, size and material of the parts in the region image for comprehensive analysis to identify the parts; the identified parts are marked, and the region image including the unmarked area and the area where the marked parts are located is output.

[0041] In this embodiment, it should be noted that the CogVLM model can comprehensively analyze the shape, size, and material of parts to achieve accurate part identification. Therefore, after acquiring the original area image of the robot's working area through a high-definition camera, the original area image undergoes preprocessing such as noise reduction, contrast enhancement, and size cropping. The preprocessed area image is then output to the CogVLM model. The CogVLM model can extract the shape, size, and material of the parts in the area image for comprehensive analysis to identify the parts. The identified parts are then labeled, outputting an area image that includes both unlabeled areas and areas containing labeled parts. Specifically, before deployment to the actual robot, images of all parts that need to be transported during operations can be collected and labeled according to part type and operation type, forming an engineering machinery dataset. Based on this private dataset, the CogVLM model is fine-tuned to learn how to identify part types and their corresponding operations, including but not limited to learning their shape, size, and material. After fine-tuning to achieve the required classification accuracy, it can be deployed to the robot for perception. Meanwhile, by fine-tuning the industrial dataset, if a new part is added, a series of images of that part are added to the dataset for fine-tuning, thus enabling the identification of part categories.

[0042] Step 103: Determine the type of task to be performed by the robot for the part based on the part category.

[0043] In this embodiment of the application, it should be noted that, in this technical solution, the robot is configured with a corresponding task type to be performed for each type of part. For example, taking a small part of a certain type as an example, the task type to be performed by the robot for that small part could be to grasp and transfer the small part. Taking a large part of a certain type as an example, the task type to be performed by the robot for that large part could be to hold and move the large part with both hands. Therefore, after determining the type of part based on the CogVLM model, the task type to be performed by the robot for that part can be further determined based on the type of part.

[0044] In this embodiment of the application, determining the type of task to be performed by the robot for the part based on the part category includes: determining the task rule that matches the part category based on a preset task rule library, wherein each task rule in the preset task rule library defines the transportation method corresponding to each type of part; and determining the type of task to be performed by the robot for the part based on the task rule.

[0045] In this embodiment, it should be noted that a preset task rule library exists in this technical solution. This rule library contains corresponding task rules for each category of parts. Each task rule defines the transportation method for the part. For example, for a certain type of small part, the transportation method could be grasping; for a certain type of large part, the transportation method could be handling. The task rules in the preset task rule library are bound to the robot's task type. Therefore, after determining the task rule matching the part, the robot can further determine the task type to be performed on the part based on the task rule.

[0046] Step 104: Input the processed region image and the type of task to be executed into the action block algorithm model so as to output the control strategy for the part through the action block algorithm model. The action block algorithm model includes a self-attention mechanism encoder, and the control strategy includes at least one of the motion trajectory of multiple robotic arms, transport speed and transport torque.

[0047] In this embodiment, it should be noted that the action segmentation algorithm model can refer to the ACT strategy of Mobile ALOHA, and the self-attention mechanism encoder can refer to the Transformer encoder. The Transformer encoder is one of the core components of the Transformer model, and its main task is to convert the input sequence into a context vector representation. Traditional ACT strategies are insufficient in capturing global information of the input image and cannot process the input sequence in parallel. This technical solution introduces a Transformer encoder to replace the original CNN structure at the image input layer of the decoder in the ACT strategy. By processing the input image using a self-attention mechanism, the ACT strategy can process the input sequence in parallel, achieving better capture of the global feature information of the parts. After outputting the region image (including unlabeled areas and areas where labeled parts are located) through the CogVLM model, and determining the robot's task based on the part category, the processed region image and the task type are input into the ACT strategy. The ACT strategy can then output a control strategy for the parts. The output control strategy includes the motion trajectory, transport speed, and transport torque of multiple robotic arms when performing the task type.

[0048] In this embodiment of the application, the control strategy for the part output by the action segmentation algorithm model includes: after inputting the processed region image and the type of task to be executed into the action segmentation algorithm model, determining the target parameters that match the type of task to be executed, wherein the action segmentation algorithm model includes multiple model parameters, which are trained according to multiple types of parts and the transportation method of each type of part; adjusting the parameters of the control action segmentation algorithm model to the target parameters, so as to output the control strategy for the part based on the processed region image.

[0049] In this embodiment, it should be noted that the ACT strategy includes multiple pre-trained model parameters, each corresponding to a specific part transportation method. Specifically, the ACT strategy's multiple model parameters can be trained based on various types of parts and their transportation methods. Furthermore, reinforcement learning algorithms can be used to optimize the ACT strategy in a simulation environment. The algorithm continuously interacts with the environment, tries different actions, and optimizes and adjusts the control strategy based on feedback from the reward function. Therefore, after inputting the processed region image and the task type into the ACT strategy, the ACT strategy adjusts the corresponding target model parameters according to the input task type, thereby outputting a control strategy for the part based on the processed region image using the target model parameters.

[0050] Step 105: Based on the control strategy, control multiple robotic arms to perform corresponding control operations on the parts.

[0051] In this embodiment of the application, after obtaining the control strategy output by the ACT strategy, the controller will further convert the control strategy into instructions that the robot can execute, and send them to the robot control system to control the robot's multiple robotic arms to perform corresponding control operations on the parts.

[0052] In embodiments of this application, the control method further includes: during the process of controlling multiple robotic arms to perform corresponding control operations on a part based on a control strategy, acquiring the operation duration and operation data of the multiple robotic arms in real time, wherein the operation data includes at least one of the position, posture, and force conditions of the multiple robotic arms; determining that the robot is in an abnormal state when the operation duration reaches a preset threshold; switching the control strategy for the part based on the operation data, and controlling the multiple robotic arms to perform corresponding control operations on the part based on the updated control strategy.

[0053] In this embodiment, it should be noted that during the operation of the robot's multiple robotic arms, to ensure the safety and accuracy of the operation, the position, posture, and force of the multiple robotic arms can be monitored in real time. Therefore, during the process of controlling multiple robotic arms to perform corresponding control operations on parts based on the control strategy, the operation duration, position, posture, and force of the multiple robotic arms are acquired in real time. The preset threshold can be set based on historical experience. The completion time of transporting each part should be within a certain time range. Therefore, by detecting the duration of the control operation performed by the robotic arms on the parts, it can be determined whether the robot is currently in an abnormal state. When the robot is in an abnormal state, different control strategies can be dynamically switched to ensure the smooth progress of the operation process. Specifically, if the operation duration reaches the preset threshold, it is determined that the robot is in an abnormal state. At this time, the control strategy for the parts can be switched based on the operation data, and the multiple robotic arms can be controlled to perform corresponding control operations on the parts based on the updated control strategy.

[0054] This solution introduces a pre-defined visual model to adapt to parts with complex and varied shapes, and introduces an action segmentation algorithm model to output control strategies for the parts. This enables accurate identification of parts with complex and varied shapes, and adjustment of control strategies for the parts based on the characteristics of the parts and the type of task.

[0055] This technical solution boasts high recognition accuracy and classification precision. The introduction of CogVLM significantly improves the accuracy of part recognition and task classification, and reduces the false positive rate.

[0056] This technical solution features a flexible operating strategy that can automatically adjust the operating strategy according to the characteristics of the parts and the transportation task, thereby improving transportation efficiency and accuracy.

[0057] This technical solution has powerful global information processing capabilities. The improved ACT strategy uses a Transformer encoder to better capture global information and improve decision-making accuracy.

[0058] This technical solution has good generalization ability. The improvement of the model architecture enhances its generalization ability, making the model applicable to more types of parts and transportation scenarios.

[0059] This application provides a control device for a robot, including:

[0060] The memory is configured to store instructions;

[0061] The controller is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the aforementioned control method for the robot.

[0062] This application provides a machine-readable storage medium storing a program that, when executed by a processor, implements the aforementioned control method for a robot.

[0063] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores data for robot control methods. The network interface A02 communicates with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a robot control method.

[0064] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] This application provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring a region image of the robot's working area, the working area including at least one part; inputting the region image into a preset vision model to determine the part's category and outputting a processed region image, wherein the processed region image includes unmarked areas and areas where marked parts are located; determining the type of task to be performed by the robot on the part based on the part's category; inputting the processed region image and the task type into an action segmentation algorithm model to output a control strategy for the part, wherein the action segmentation algorithm model includes a self-attention mechanism encoder, and the control strategy includes at least one of the motion trajectories, transport speeds, and transport torques of multiple robotic arms; and controlling the multiple robotic arms to perform corresponding control operations on the part based on the control strategy.

[0066] In one embodiment, determining the type of task to be performed by the robot for a part based on the part's category includes: determining task rules that match the part's category based on a preset task rule library, wherein each task rule in the preset task rule library defines the transportation method corresponding to each type of part; and determining the type of task to be performed by the robot for the part based on the task rules.

[0067] In one embodiment, the preset visual model outputs a processed region image, which includes: after inputting the region image into the preset visual model, the preset visual model extracts the shape, size, and material of the parts in the region image for comprehensive analysis to identify the parts; the identified parts are marked, and a region image including unmarked areas and the areas where the marked parts are located is output.

[0068] In one embodiment, the action segmentation algorithm model outputs a control strategy for a part by: after inputting the processed region image and the type of task to be executed into the action segmentation algorithm model, determining target parameters that match the type of task to be executed, wherein the action segmentation algorithm model includes multiple model parameters, which are trained based on multiple types of parts and the transportation method of each type of part; adjusting the parameters of the control action segmentation algorithm model to the target parameters, so as to output a control strategy for the part based on the processed region image.

[0069] In one embodiment, the control method further includes: during the process of controlling multiple robotic arms to perform corresponding control operations on a part based on a control strategy, acquiring the operation duration and operation data of the multiple robotic arms in real time, wherein the operation data includes at least one of the position, posture, and force conditions of the multiple robotic arms; determining that the robot is in an abnormal state when the operation duration reaches a preset threshold; switching the control strategy for the part based on the operation data, and controlling the multiple robotic arms to perform corresponding control operations on the part based on the updated control strategy.

[0070] In one embodiment, the control method further includes: after acquiring the region image, performing noise reduction, contrast enhancement, and size cropping on the region image in sequence to improve the image quality of the region image.

[0071] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring a region image of the robot's working area, the working area including at least one part; inputting the region image into a preset vision model to determine the category of the part through the preset vision model and outputting a processed region image, wherein the processed region image includes unmarked areas and areas where marked parts are located; determining the type of task to be performed by the robot for the part based on the category of the part; inputting the processed region image and the type of task to be performed into an action block algorithm model to output a control strategy for the part through the action block algorithm model, wherein the action block algorithm model includes a self-attention mechanism encoder, and the control strategy includes at least one of the motion trajectory of multiple robotic arms, transport speed, and transport torque; and controlling multiple robotic arms to perform corresponding control operations on the part based on the control strategy.

[0072] In one embodiment, determining the type of task to be performed by the robot for a part based on the part's category includes: determining task rules that match the part's category based on a preset task rule library, wherein each task rule in the preset task rule library defines the transportation method corresponding to each type of part; and determining the type of task to be performed by the robot for the part based on the task rules.

[0073] In one embodiment, the preset visual model outputs a processed region image, which includes: after inputting the region image into the preset visual model, the preset visual model extracts the shape, size, and material of the parts in the region image for comprehensive analysis to identify the parts; the identified parts are marked, and a region image including unmarked areas and the areas where the marked parts are located is output.

[0074] In one embodiment, the action segmentation algorithm model outputs a control strategy for a part by: after inputting the processed region image and the type of task to be executed into the action segmentation algorithm model, determining target parameters that match the type of task to be executed, wherein the action segmentation algorithm model includes multiple model parameters, which are trained based on multiple types of parts and the transportation method of each type of part; adjusting the parameters of the control action segmentation algorithm model to the target parameters, so as to output a control strategy for the part based on the processed region image.

[0075] In one embodiment, the control method further includes: during the process of controlling multiple robotic arms to perform corresponding control operations on a part based on a control strategy, acquiring the operation duration and operation data of the multiple robotic arms in real time, wherein the operation data includes at least one of the position, posture, and force conditions of the multiple robotic arms; determining that the robot is in an abnormal state when the operation duration reaches a preset threshold; switching the control strategy for the part based on the operation data, and controlling the multiple robotic arms to perform corresponding control operations on the part based on the updated control strategy.

[0076] In one embodiment, the control method further includes: after acquiring the region image, performing noise reduction, contrast enhancement, and size cropping on the region image in sequence to improve the image quality of the region image.

[0077] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0083] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A control method for a robot, characterized in that, The robot includes multiple robotic arms that cooperate with each other, and the control method includes: Acquire a region image of the robot's working area, the working area including at least one part; The region image is input into a preset visual model to determine the category of the part and output a processed region image, wherein the processed region image includes unmarked areas and areas where marked parts are located; The type of task to be performed by the robot for the part is determined based on the category of the part; The processed region image and the type of task to be executed are input into the action block algorithm model to output a control strategy for the part. The action block algorithm model includes a self-attention mechanism encoder, and the control strategy includes at least one of the motion trajectory, transport speed and transport torque of the multiple robotic arms. Based on the control strategy, the multiple robotic arms are controlled to perform corresponding control operations on the part; In the process of controlling the multiple robotic arms to perform corresponding control operations on the part based on the control strategy, the operation time and operation data of the multiple robotic arms are acquired in real time. The operation data includes at least one of the position, posture and force conditions of the multiple robotic arms. If the operation time reaches a preset threshold, the robot is determined to be in an abnormal state. Based on the operation data, the control strategy for the part is switched, and multiple robotic arms are controlled to perform corresponding control operations on the part based on the updated control strategy. The control strategy for the part output through the action block algorithm model includes: After inputting the processed region image and the type of task to be executed into the action segmentation algorithm model, target parameters matching the type of task to be executed are determined. The action segmentation algorithm model includes multiple model parameters, which are trained based on multiple categories of parts and the transportation method of each category of parts. The parameters of the control action block algorithm model are adjusted to the target parameters to output a control strategy for the part based on the processed region image.

2. The control method for a robot according to claim 1, characterized in that, Determining the type of task to be performed by the robot for the part based on the part's category includes: Based on a preset task rule library, task rules matching the category of the parts are determined, wherein each task rule in the preset task rule library defines the transportation method corresponding to each category of parts; The task type to be performed by the robot for the part is determined based on the task rules.

3. The control method for a robot according to claim 1, characterized in that, The processed region image output by the preset visual model includes: After the region image is input into a preset visual model, the preset visual model extracts the shape, size, and material of the parts in the region image for comprehensive analysis to identify the parts; The identified parts are marked, and a region image is output, including both unmarked areas and the areas where marked parts are located.

4. The control method for a robot according to claim 1, characterized in that, The control method further includes: After acquiring the region image, the region image is sequentially processed by denoising, contrast enhancement, and size cropping to improve the image quality of the region image.

5. A control device for a robot, characterized in that, include: The memory is configured to store instructions; The controller is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the control method for a robot according to any one of claims 1 to 4.

6. A control system for a robot, characterized in that, Includes the control device for a robot as described in claim 5.

7. A robot, characterized in that, include: The control system for a robot according to claim 6; Multiple robotic arms working together.

8. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by the controller, the instruction causes the controller to be configured to perform the control method for a robot according to any one of claims 1 to 4.

Images