Flat part sorting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a flat part sorting method and device, electronic equipment and a storage medium, and belongs to the technical field of logistics. The method comprises the steps that a flat part sorting instruction and the current running state of the robot are obtained; a preset task planner is called to perform task planning based on the flat part sorting instruction and the current running state, a flat part sorting plan is obtained, and the flat part sorting plan comprises a plurality of ordered target sorting sub-plans; obtaining a plan completion state of the robot, and generating a target task instruction based on the corresponding target sorting sub-plans in sequence according to the plan completion state; and calling a preset upper limb controller to control the upper limbs of the robot according to the target task instruction, or calling a preset lower limb controller to control the lower limbs of the robot according to the target task instruction, so that the robot executes the corresponding target sorting sub-plan. According to the embodiment of the invention, the flat part sorting flexibility of the robot can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of logistics, and in particular to a flat piece sorting method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In a logistics sorting scenario, a flat piece can refer to an envelope type express, which usually contains high value, small volume items such as documents, invoices, etc. In order to ensure the quality and safety of the transfer of such express, an independent manual sorting area is usually set up for flat piece sorting operations. However, this method consumes a lot of manpower.

[0003] With the development of humanoid robots and embodied intelligence technology, it is possible to use robots to replace manual sorting of flat pieces, but the current method of sorting flat pieces based on robots has the problem of poor flexibility. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a flat piece sorting method and device, an electronic device, and a storage medium, which can improve the flexibility of robot sorting of flat pieces.

[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a flat piece sorting method, the method comprising: obtaining a flat piece sorting instruction and a current running state of a robot; wherein the robot is an embodied robot, and the robot includes an upper limb and a lower limb; calling a preset task planner to perform task planning based on the flat piece sorting instruction and the current running state, to obtain a flat piece sorting plan, wherein the flat piece sorting plan includes a plurality of ordered target sorting sub-plans; obtaining a plan completion state of the robot, and generating a target task instruction based on the corresponding target sorting sub-plan in order according to the plan completion state; controlling the upper limb of the robot according to the target task instruction by calling a preset upper limb controller, or controlling the lower limb of the robot according to the target task instruction by calling a preset lower limb controller, so that the robot executes the corresponding target sorting sub-plan.

[0006] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a flat piece sorting device, the device comprising: a data acquisition unit configured to obtain a flat piece sorting instruction and a current running state of a robot; wherein the robot is an embodied robot, and the robot includes an upper limb and a lower limb; a task planning unit configured to call a preset task planner to perform task planning based on the flat piece sorting instruction and the current running state, to obtain a flat piece sorting plan, wherein the flat piece sorting plan includes a plurality of ordered target sorting sub-plans; An instruction generation unit is configured to acquire a plan completion state of the robot, and generate a target task instruction based on a corresponding target sorting sub-plan in sequence according to the plan completion state; A sorting control unit is configured to control an upper limb of the robot by calling a preset upper limb controller according to the target task instruction, or control a lower limb of the robot by calling a preset lower limb controller according to the target task instruction, so that the robot executes the corresponding target sorting sub-plan.

[0007] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of the first aspect embodiments when executing the computer program.

[0008] To achieve the above object, a fourth aspect of the embodiments of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of any one of the first aspect embodiments.

[0009] The flat part sorting method, device, electronic device and storage medium provided by the embodiments of the present application can dynamically decompose the flat part sorting instruction into an ordered sub-plan by using a task planner, trigger the corresponding target task instruction in sequence through plan completion state feedback, and finally execute the corresponding upper limb or lower limb controller based on the target task instruction. In this way, the full autonomy and intelligence of the robot from task understanding to action execution in the logistics sorting scene are realized, the adaptability of the robot in coping with different scene layouts and task contents is improved, and the problem of poor flexibility caused by insufficient generalization in related technologies is reduced. In addition, by using a humanoid robot as an execution end for executing the flat part sorting task, the moving ability of the lower limbs of the humanoid robot is utilized, so that the method provided by the present application can adapt to existing unmodified manual sorting sites and environmental facilities, and there is no need to modify the site to adapt to automatic equipment. At the same time, the humanoid upper limb structure of the humanoid robot is utilized, so that the robot can adapt to the operation action of different sorting plan requirements, so that the perception flexibility and operation adaptability close to manual operation can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is an embodiment flowchart of the flat part sorting method provided by the embodiments of the present application; Figure 2 is a schematic diagram of the sorting plan planning by the task planner provided by the embodiments of the present application; Figure 3 is an embodiment flowchart of calling the upper limb controller to control the robot provided by the embodiments of the present application; Figure 4is a structural schematic diagram of an upper limb controller provided by an embodiment of the present application; Figure 5 is an embodiment flowchart of a training method of the upper limb controller provided by an embodiment of the present application; Figure 6 is an embodiment flowchart of calling the lower limb controller to control the robot provided by an embodiment of the present application; Figure 7 is an embodiment flowchart of a training method of the lower limb controller provided by an embodiment of the present application; Figure 8 is a schematic diagram of the training method of the lower limb controller provided by an embodiment of the present application; Figure 9 is an overall implementation schematic diagram of the flat part sorting method provided by an embodiment of the present application; Figure 10 is a structural schematic diagram of a flat part sorting device provided by an embodiment of the present application; Figure 11 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0012] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms “first”, “second”, and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0013] First, the terms related to the embodiments of the present application are explained: Automated Guided Vehicle (AGV): an automatic transportation device based on fixed path navigation, usually relying on magnetic strips, guide rails or two-dimensional codes for movement. The characteristics of AGV are fixed running route, simple task logic, and are suitable for highly structured environment (such as assembly line, warehouse shelf).

[0014] Simultaneous Localization and Mapping (SLAM): A responsible algorithm framework designed to solve the problem of simultaneously achieving self-localization and environment map construction for mobile agents (such as robots, drones, etc.) in unknown environments. SLAM technology is essentially a highly coupled state estimation problem: the agent needs to use its own sensors (such as lidar, camera, etc.) to perceive the surrounding environment, and estimate its degrees of freedom pose (position and attitude) from noisy observation data in real time, and on this basis, incrementally generate a consistent environment spatial map.

[0015] Vision-Language Model (VLM): A multi-modal artificial intelligence architecture based on deep learning architecture. The core goal of VLM is to establish a deep understanding and connection between visual information and natural language. VLM is usually composed of a visual encoder, a large language model, and an alignment module.

[0016] Vision-Language-Action (VLA): An end-to-end intelligent framework, the core idea of VLA is to input vision and action into the same model, after training, the model can directly output the specific action sequence (such as joint torque or end pose) to control the robot to perform tasks.

[0017] Embodied robots (also known as humanoid robots): A kind of intelligent robot that simulates human body structure, through the coordinated operation of upper limbs, lower limbs and trunk, it realizes efficient interaction with the physical world. This kind of robot usually has similar human upper limb structure, including multi-degree-of-freedom shoulder, elbow, wrist joints and dexterous hands, which can complete fine tasks such as grasping, carrying and operating tools. At the same time, the lower limb part often simulates the structure of legs and feet, equipped with joint driving and balance control system, realizes the moving ability such as walking, climbing and obstacle avoidance. Through the deep integration of sensors, processors and mechanical structures, embodied robots can adapt to various complex environments.

[0018] In the logistics sorting scene, flat pieces can refer to envelope-like express items, usually containing high-value, small-volume items such as documents and invoices. In order to ensure the quality and safety of the transfer of such express items, an independent manual sorting area is usually set up for flat piece sorting operations. However, this method consumes a lot of manpower.

[0019] With the development of humanoid robots and embodied intelligent technology, it is possible to use robots to replace manual sorting of flat pieces, but the current method of sorting flat pieces based on robots has the following problems: 1. The current visual solution of the robot cannot effectively segment and identify various express items in the logistics scene; 2. The current robot control method based on rules and fixed trajectory planning has poor generalization and is difficult to adapt to changes in different workstations; 3. The current method is difficult to realize reliable and flexible human-computer interaction.

[0020] Therefore, based on the above, the embodiments of the present application provide a flat part sorting method and device, electronic equipment and storage medium, which can improve the flexibility of the robot in sorting flat parts.

[0021] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0022] The flat part sorting method provided by the embodiments of the present application relates to the field of logistics technology. The flat part sorting method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, or can be configured as a server cluster or a distributed system formed by multiple physical servers, or can be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms. Basic cloud computing services such as platform, and the like, but are not limited to the above forms.

[0023] The application is operable in a variety of general purpose or special purpose computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0024] Referring to Figure 1 , Figure 1 is an optional flowchart of the flat part sorting method provided by the embodiments of the application. In some embodiments of the application, the method in Figure 1 may specifically include but is not limited to steps S110 to S140.

[0025] In step S110, a flat part sorting instruction and a current running state of a robot are acquired; wherein the robot is a body-equipped robot, and the robot includes an upper limb and a lower limb; In step S120, a preset task planner is called to perform task planning based on the flat part sorting instruction and the current running state, to obtain a flat part sorting plan, wherein the flat part sorting plan includes a plurality of ordered target sorting sub-plans; In step S130, a plan completion state of the robot is acquired, and a target task instruction is generated based on the corresponding target sorting sub-plan according to the plan completion state in sequence; In step S140, a preset upper limb controller is called to control the upper limb of the robot according to the target task instruction, or a preset lower limb controller is called to control the lower limb of the robot according to the target task instruction, so that the robot executes the corresponding target sorting sub-plan.

[0026] In step S110 of some embodiments, the flat part sorting instruction can refer to a command requiring the robot to complete a specific flat part sorting task, and the flat part sorting instruction can be in a text format, a language format, etc., which is not specifically limited. For example, an operator can issue a flat part sorting instruction of "go to the No. 1 cabinet to sort flat parts" through voice or a text interface. The robot can refer to an intelligent robot capable of imitating human form and behavior, and capable of autonomous interaction with the environment, i.e., the robot can be a humanoid robot. The robot includes upper limbs and lower limbs. In the embodiments of the present application, the robot is an execution end for performing a flat part sorting task. The current running state of the robot can refer to the state variables of the robot, and the current running state can be obtained by a plurality of sensors integrated in the robot body. For example, the current running state usually includes but is not limited to: the pose information of the robot in the logistics field corresponding to the flat part sorting task (such as two-dimensional or three-dimensional coordinates and orientation which can be obtained by SLAM), the angle and speed data of each joint of the whole body obtained by the joint encoder of the robot, the body attitude angle and acceleration measured by the inertial sensor, the surrounding environment point cloud or video stream collected in real time by the vision sensor (such as a depth camera), etc.

[0027] In step S120 of some embodiments, the task planner can refer to an agent constructed based on a VLM model. The task planner can utilize the logistics knowledge and reasoning ability obtained by large-scale pre-training to decompose abstract instructions (such as flat part sorting instructions) into a series of specific, executable and logically rigorous target sorting sub-plans. Specifically, the working process of the task planner can include scene understanding and generation of a flat part sorting plan (the flat part sorting plan can refer to a collection sequence of a plurality of target sorting sub-plans): first, the task planner can analyze the current running state of the robot, identify the key objects (such as sorting cabinets, compartments, etc.) in the environment where the robot is currently located, the attributes (such as position, shape, etc.) of these key objects, and the spatial relationship of these key objects. Then, the information obtained by analysis can be combined with the flat part sorting instruction to generate a global task sequence (i.e., a flat part sorting plan). For example, as shown in FIG. 2, the task planner can first analyze the current running state of the robot, and then combine the analysis result with the flat part sorting instruction to generate a global task sequence (i.e., a flat part sorting plan) of "go to the No. 1 cabinet, pick up the flat part, and put it into the No. 1 compartment". Figure 2As shown, for the flat item sorting instruction "go to cabinet 1 to sort flat items," the task planner can understand the scenario based on the flat item sorting execution and the robot's current operating state (e.g., "there is a sorting cabinet and some letters in front of us, and we need to deliver these letters to the correct slots"), and generate the following four target sorting sub-plans: 1. Move to the front of the letters; 2. Pick up the letters; 3. Query the slots corresponding to the sorting plan (i.e., determine which slot to deliver the letters to); 4. Move to the slot; 5. Deliver the letters to the slot. This embodiment of the application determines the flat item sorting plan based on the task planner, which, compared to the fixed-rule-based planning in related technologies, has significant generalization capabilities and can adapt to different workstation layouts and dynamically changing scenarios. It is understood that the task planner can be deployed in the cloud.

[0028] In step S130 of some embodiments, the plan completion status can refer to determining that the robot has completed the target sorting sub-plan. The plan completion status can be obtained through sensor data within the robot body (e.g., determining the plan completion status based on the sensor feedback of the reached position, or determining the plan completion status through scene images collected by the sensors). After determining the plan completion status, the task planner can be invoked to sequentially determine the next target sorting sub-plan to be executed from the flat part sorting plan based on the plan completion status, and convert the target sorting sub-plan to be executed into a target task instruction. For example, as... Figure 2 As shown, when the plan completion status indicates "Completed: None" (i.e., no target sorting sub-plans have been completed), the target sorting sub-plan to be executed can be determined as "Move to the front of the letters first." Thus, an instruction to complete the target task of "Move to the front of the letters first" can be generated. It is understandable that the above operation can be repeated until it is determined, based on the plan completion status, that the robot has completed all target sorting sub-plans. This sequential triggering mechanism based on status feedback improves the reliability and safety of task execution and reduces the chaos in subsequent operations caused by the failure of a certain step.

[0029] In step S140 of some embodiments, an upper limb controller and a lower limb controller may be deployed within the robot body. The upper limb controller drives the robot's arms, dexterous hands, and other upper limb structures, enabling the robot to perform fine motor actions such as grasping and placing. The lower limb controller drives the robot's legs, waist, and other lower limb structures, enabling the robot to move and maintain balance. Based on the semantic content of the target task instruction, the controller can automatically determine and send the target task instruction to the corresponding controller (upper limb controller or lower limb controller) for execution. After receiving the target task instruction, the controller can determine the specific action sequence based on the robot's current operating state and control the robot to complete the corresponding actions based on the action sequence, thereby controlling the robot to execute the corresponding target sorting sub-plan.

[0030] The embodiment of the application dynamically decomposes the flat part sorting instruction into an ordered sub-plan by using a task planner, sequentially triggers the corresponding target task instruction through plan completion state feedback, and finally executes the corresponding upper limb or lower limb controller based on the target task instruction. In this way, the whole autonomy and intelligence of the robot from task understanding to action execution in the logistics sorting scene are realized, the adaptability of the robot in dealing with different scene layouts and task contents is improved, and the problem of poor flexibility caused by insufficient generalization in the related art is reduced, that is, the flexibility and efficiency of the flat part sorting operation are improved.

[0031] As follows, the specific control mode of the upper limb controller and the lower limb controller is described respectively.

[0032] First, the upper limb controller is described, that is, the situation of calling the upper limb controller to execute the target task instruction based on the semantic content of the target task instruction is described.

[0033] Referring to Figure 3 In some embodiments, the upper limb controller includes a feature extraction module and an action strategy module, and the control of the upper limb of the robot by the upper limb controller according to the target task instruction can include but is not limited to steps S310 to S330.

[0034] Step S310, acquiring first target visual data of the robot, calling the feature extraction module to perform feature extraction based on the first target visual data and the target task instruction, and obtaining a target feature vector; Step S320, acquiring a current running state of the robot, calling the action strategy module to perform cross-attention processing based on the target feature vector and the current running state, and obtaining a target action sequence; Step S330, controlling the upper limb of the robot according to the target action sequence.

[0035] In step S310 of some embodiments, as Figure 4As shown, the upper limb controller can include a feature extraction module, which can employ a VLM model architecture. The feature extraction module is configured to collaboratively encode and deeply understand the input visual data and the task instruction in text form, and obtain a corresponding feature vector. Specifically, the first target visual data and the target task instruction can be taken as input data of the feature extraction module, the feature extraction module can extract key features in the first target visual data, and perform semantic understanding on the target task instruction, and finally align and fuse the extracted visual information and semantic information to obtain a target feature vector. The target feature vector represents the joint information of "what is currently seen" and "what needs to be done", and provides an explicit intention guide for subsequent action generation. The first target visual data can refer to video, image and other visual data captured by the vision sensor carried by the robot in real time.

[0036] In step S320 of some embodiments, as Figure 4 As shown, the upper limb controller can also include an action policy module, which can refer to a model structure capable of deciding the optimal upper limb operation strategy required to achieve the task target. Specifically, the action policy module can include a state encoder, a cross-attention layer, and an action decoder. The state encoder can encode the current running state into a state feature vector. Then, the cross-attention layer can perform cross-attention processing on the target feature vector and the state feature vector, so that the physical state of the robot is finely aligned with the task intention, thereby deciding the optimal action change required to achieve the task intention. Finally, the action decoder can decode the fusion features output by the cross-attention layer into a target action sequence. The target action sequence defines the motion path of each joint of the robot upper limb in the future period of time for achieving the corresponding task.

[0037] It can be understood that the cross-attention layer can be a DiT diffusion model cross-attention layer, and the upper limb controller as a whole can be a VLA model structure.

[0038] In step S330 of some embodiments, the target action sequence can be issued to the joint drive of the robot upper limb, and the joint drive can accurately control the motion of each joint motor according to the target action sequence, thereby driving the arm, dexterous hand and other parts of the robot to execute the corresponding target sorting sub-plan (such as grasping the letter).

[0039] The embodiment of the present application constructs an upper limb controller capable of fully understanding task semantics and adapting to the real-time physical state of the robot. The upper limb controller can fuse natural language instructions and visual scene information into unified semantic features, and then dynamically adapt to the robot body state information through an attention mechanism, so as to generate a fine and physically constrained motion sequence. In this way, the success rate and adaptability of the robot in handling stacking and easily deformed flat objects are improved, and the problem of poor flexibility caused by robot control based on pre-programming and other fixed rules in the related art is solved.

[0040] In some embodiments, after step S330, the method provided by the embodiment of the present application can further include the following steps: determining balance influence data of the robot according to the target motion sequence; calling the lower limb controller to control the lower limbs of the robot based on the balance influence data and the current running state of the robot.

[0041] In the embodiment of the present application, as known from the foregoing description, the target motion sequence is the expected motion trajectory of each joint of the upper limb for a plurality of future time steps, which is planned by the upper limb controller. The balance influence data can refer to a quantitative prediction of the change in the overall posture stability of the robot that can be caused by the target motion sequence, such as the predicted offset of the overall center of gravity of the robot, the disturbance change to the balance of the robot due to the motion of the upper limb, etc. Specifically, the simplified dynamics model of the robot can be constructed for real-time simulation calculation. Alternatively, a pre-trained neural network prediction model is used to analyze the expected motion speed and mass distribution of each joint in the target motion sequence, and the disturbance direction and amplitude that will be caused by the execution of these actions to the overall balance are inferred, so as to provide an explicit and quantitative adjustment target for the compensation control of the lower limbs.

[0042] After determining the balance influence data, the lower limb controller can use a preset lower limb motion strategy to calculate a set of optimal lower limb joint adjustment motion sequences based on the balance influence data and the current motion state of the robot, and control the lower limbs of the robot according to the calculated motion sequences. For example, the angles of the lower limb structures such as the ankle joint and the knee joint can be fine-tuned to change the overall support polygon shape of the robot, or a step can be taken to re-establish a stable support, so as to actively generate a compensation moment opposite in direction and appropriately sized to a predicted disturbance direction, and finally realize dynamic adjustment of the center of gravity of the robot to ensure the stability of the overall robot during and after the motion of the upper limb.

[0043] Reference Figure 5 In some embodiments, the training method of the upper limb controller can include, but is not limited to, steps S510 to S550.

[0044] Step S510, obtaining a first sample sorting sub-plan; Step S520, obtaining first sample task instructions of the first sample sorting sub-plan and first sample visual data of the robot; Step S530, calling a pre-trained feature extraction module to perform feature extraction based on the first sample visual data and the first sample task instructions, to obtain a sample feature vector; Step S540, obtaining a reference action sequence of the first sample sorting sub-plan and a sample running state of the robot, and calling an action policy module to perform cross-attention processing based on the sample feature vector and the sample running state, to obtain a sample upper limb action sequence; Step S550, performing parameter freezing on the feature extraction module, and training an upper limb controller based on the sample upper limb action sequence and the reference action sequence.

[0045] In step S510 of some embodiments, a training task instance can be extracted from the constructed task library. The first sample sorting sub-plan can refer to the extracted training task instance. Alternatively, when the training task instance contains multiple operation steps, the first sample sorting sub-plan can refer to one operation step decomposed from the training task instance. For example, the first sample sorting sub-plan can refer to “grabbing a flat part from a workbench”, “delivering the flat part to an A5 grid opening”, etc.

[0046] In step S520 of some embodiments, the first sample task instructions can refer to specific operation commands expressed in natural language corresponding to the first sample sorting sub-plan. The first sample visual data can refer to visual data collected before the robot starts to perform the first sample sorting sub-plan in a simulation environment or a real environment. The first sample visual data describes the environmental state, object appearance, and relative position, etc., and provides the original visual context.

[0047] In steps S530 to S550 of some embodiments, the input multi-modal data (i.e., the first sample visual data and the first sample task instruction) is subjected to feature extraction by using the pre-trained feature extraction module (VLM) to obtain a sample feature vector. Then, the sample feature vector and the sample running state of the robot in the simulation environment or the real environment are taken as input data of the action policy module, and the action policy module performs cross-attention processing on the input data to obtain a sample upper limb action sequence. The sample upper limb action sequence can refer to the optimal action sequence predicted by the action policy module to execute the first sample sorting sub-plan. The reference action sequence can refer to the ideal upper limb movement trajectory sequence recorded by the expert operator through the teaching device or when executing the first sample sorting sub-plan in the simulation environment. In this way, the training loss can be determined based on the difference between the reference action sequence and the sample upper limb action sequence, and the parameters of the upper limb controller can be adjusted according to the training loss, thereby realizing the training of the upper limb controller. It can be understood that in the training in this stage, the parameters of the feature extraction module need to be set to a frozen state to ensure that the visual language recognition ability of the feature extraction module remains stable in this training stage. That is, the training loss is used to optimize and update only the parameters of the action policy module, thereby gradually improving the ability of the action policy module to generate accurate actions based on the task semantic intention and the robot ontology state.

[0048] The upper limb controller training method provided by the embodiments of the present application fixes the parameters of the pre-trained feature extraction module and optimizes the parameters of the action policy module, realizes accurate optimization of the action generation ability on the basis of stable semantic understanding, thereby improving the accuracy and robustness of the upper limb in performing operations such as grabbing and delivering in complex logistics scenarios, and overcoming the dependence on accurate dynamic models and the limitation of reprogramming when facing new tasks.

[0049] In some embodiments, the method for pre-training the feature extraction module can include but is not limited to the following steps: Obtaining an environment image of a logistics site where the robot is located and a description text of the environment image; Fine-tuning the pre-set feature extraction model according to the environment image and the description text.

[0050] In the embodiment of the present application, initially, the feature extraction module can adopt an open-source basic VLM model, at this time the feature extraction module has weak understanding ability for the logistics scene, therefore it is necessary to fine-tune the feature extraction module for the logistics scene understanding ability. Specifically, the environment image can refer to the image captured by the fixed camera device, the mobile acquisition equipment or the robot body in the working process deployed in the logistics site. The content of the environment image can include the scene of the logistics sorting operation (such as workbench, sorting cabinet, etc.), and various flat pieces in the stacking, scattering or transmission state. The description text can refer to the natural language text used to describe the visual content contained in the corresponding environment image. The description text can be generated in a labeled manner (such as "X blue envelopes are stacked on the left edge of No. 1 grid"), or is obtained by associating the operation in the robot operation log corresponding to the environment image acquisition timestamp (such as associating the operation log of "the robot arm performs the grabbing operation" with the environment image of "blue envelope", to obtain the description text of "the robot arm is preparing to grab the blue envelope"), which is not limited specifically. In this way, based on the environment image and the description text corresponding to the environment image, the image-text pair dataset can be constructed.

[0051] It can be understood that initially, the feature extraction module can be a VLM model that has been pre-trained on a large-scale general image-text pair. In this way, the feature extraction module already has basic visual concept recognition and language association ability. Fine-tuning training can refer to updating the parameters of the feature extraction module using the image-text pair dataset constructed for the logistics site. For example, the training target can be to optimize the feature extraction module so that the similarity of the matched image-text pair in the embedding space is much higher than that of the unmatched pair. Through fine-tuning training, the feature extraction module can more accurately understand the proper nouns in the logistics field (such as "grid", "flat piece", etc.), more acutely recognize the key objects and spatial relationships of the objects in the scene, thereby providing high-quality and robust semantic feature representation for upper limb control.

[0052] The embodiment of the present application fine-tunes the general VLM model with logistics scene data, so that the feature extraction module obtained after fine-tuning has logistics domain knowledge and can accurately understand the object, state and instruction semantics in the sorting environment, thereby improving the adaptability and task execution success rate of the entire system in complex logistics scenes.

[0053] Secondly, the lower limb controller is described, that is, the situation of calling the lower limb controller to execute the target task instruction based on the semantic content of the target task instruction is described.

[0054] Reference Figure 6In some embodiments, invoking the lower limb controller to control the lower limbs of the robot according to the target task instruction can include, but is not limited to, steps S610-S620.

[0055] At step S610, a target navigation path is generated according to the target task instruction and a pre-constructed navigation map, wherein the target navigation path includes a plurality of target path points, and each target path point includes a target position coordinate and a target orientation. At step S620, the second target visual data and the current running state of the robot are obtained, and the lower limb controller is invoked to control the lower limbs of the robot based on the second target visual data, the current running state, the target position coordinate and the target orientation corresponding to each target path point.

[0056] In step S610 of some embodiments, the target task instruction refers to a control instruction issued by an upper task planner, which contains an explicit spatial target, such as "move to the front of the No. 1 sorting cabinet" or "navigate to coordinates (X, Y, Z)", etc. The target task instruction specifies the final working position that the robot needs to reach. The navigation map can refer to a map of the logistics site constructed in advance based on SLAM technology. The navigation map can contain geometric structure information (such as obstacle boundaries, walkable areas) and key marker points (such as coordinates of key positions such as sorting cabinets and workbenches) of the logistics site. In this way, combined with the end position specified by the target task instruction, obstacle information and walkable areas in the navigation map, an optimized collision-free path (i.e., the target navigation path) from the current position of the robot to the end position is calculated through a path search algorithm. The target path point can refer to a series of intermediate points obtained by discretizing the target navigation path, and each target path point can contain a three-dimensional target position coordinate (i.e., X, Y, Z coordinate values) and an orientation (R), and these target path points collectively constitute a reference trajectory for the movement of the robot.

[0057] In step S620 of some embodiments, the second target visual data and the current motion state of the robot are obtained (the definition of the second target visual data is similar to that of the first target visual data, which will not be repeated here), and the second target visual data, the current motion state and the target navigation path are taken as input data of the lower limb controller. The lower limb controller can control the lower limbs of the robot to advance, retreat, turn left, turn right, etc. based on the input data and the lower limb motion strategy (such as strategies including robot advancing, retreating, turning, obstacle avoidance, stopping, maintaining balance, etc.). For example, the lower limb controller can perform real-time calculation in combination with the input data and the lower limb control strategy: first, the current nearest target path point is taken as a short-term motion target, and then the real-time environmental conditions perceived by the second target visual data (such as whether there are temporarily appearing obstacles) and the motion conditions of the robot reflected by the current running state are comprehensively considered, and finally specific lower limb joint control instructions are generated. For example, the lower limb controller can output 16 action steps each time, and each step has a delay of about 50ms or so, so as to ensure the smoothness of the robot motion. In this way, the robot can smoothly move along the target navigation path, while avoiding obstacles in real time when encountering dynamic obstacles, and always maintaining dynamic balance during movement.

[0058] With reference to Figure 7 and Figure 8 In some embodiments, the training method of the lower limb controller can include, but is not limited to, steps S710 to S750.

[0059] In step S710, a second sample sorting sub-plan is obtained, and a second sample task instruction is determined according to the second sample sorting sub-plan; In step S720, a sample navigation path is generated according to the second sample task instruction and the pre-constructed navigation map, wherein the sample navigation path includes a plurality of sample path points, and each sample path point includes a sample position coordinate and a sample orientation; In step S730, the second sample visual data and the current running state of the robot in the simulation environment are obtained, and the lower limb controller is called to generate a sample lower limb action sequence based on the second sample visual data, the current running state, the sample position coordinate and the sample orientation corresponding to each sample path point; In step S740, the lower limbs of the robot are controlled to move according to the sample lower limb action sequence, and an updated running state of the robot after the lower limb motion is obtained; In step S750, an action reward score is determined according to the updated running state and a preset reward function, and the lower limb controller is trained according to the action reward score.

[0060] The step S710 is similar to the step S510, and the steps S720 to S730 are similar to the steps S610 to S620. Therefore, the steps S710 to S730 will not be described again. The sample lower limb action sequence can refer to a set of controls for the robot lower limbs to advance, retreat, turn left, turn right, and the like.

[0061] In the step S740 of some embodiments, the effects of a series of actions in the sample lower limb action sequence on the robot can be determined in the simulation environment, and the robot lower limb joints are driven according to the effects to make the robot generate displacement and attitude changes in the simulation environment. After the movement is completed, the updated running state of the robot can be calculated based on the simulation environment, and the updated running state reflects the new pose of the robot after the action is performed.

[0062] In the step S750 of some embodiments, the lower limb controller can be trained through a reinforcement learning algorithm. Specifically, the reward function can be a function designed according to the target of the mobile navigation task, and the design principle of the reward function can be to encourage desired behavior and punish undesirable behavior. For example, the reward function can generally include multiple reward items: a progress reward for encouraging the robot to advance to the target point (such as the sample path point), a collision penalty for punishing collisions with obstacles, a balance reward for encouraging smooth gait, and a reach reward for encouraging successful arrival at the target point (such as the end position indicated by the sample navigation path), and the like. In this way, the updated running state of the robot after performing the sample lower limb action sequence can be used to calculate these reward items and aggregate the action reward score. The lower limb controller is trained according to the action reward score, and the goal of the training process (such as using the PPO reinforcement learning algorithm) is to find a lower limb movement strategy (such as including whole body balance strategy, standing strategy, getting up strategy, walking strategy, obstacle avoidance strategy, and the like) that can maximize the cumulative reward. Specifically, the action reward score can be used to update the parameters of the lower limb controller (i.e., update the lower limb movement strategy) through gradient ascent and the like, so that the lower limb controller can output a higher reward action sequence in the subsequent output.

[0063] The training method of the lower limb controller provided in the embodiments of the present application overcomes the dependence of the control strategy based on the model in the related art on the accurate physical model and the inherent limitations by training through reinforcement learning (RL). Specifically, the method of optimizing control by using algorithms such as PID, LQR and MPC in the related art highly depends on the accuracy of the robot dynamics model that can be established, and it is difficult to establish an accurate model containing all uncertainties and contact dynamics, and the calculation cost is also high, thereby causing the method in the related art to have insufficient generalization ability when facing unknown disturbances in the real environment. The embodiments of the present application guide the lower limb movement strategy to tend to obtain comprehensive performance such as balance, obstacle avoidance and efficient navigation through a reward function. This method enables the lower limb controller finally trained to naturally adapt to uncertainties in the scene and handle dynamic disturbances not seen during training, thereby improving the flexibility of the robot in the actual logistics scene.

[0064] The flat piece sorting method provided in the embodiments of the present application realizes the automation of flat piece sorting in the logistics site by applying the VLA model, the VLM model and the humanoid robot technology, and reduces the labor input. The hierarchical control architecture (upper task planner, upper limb controller, lower limb controller and navigation strategy) is adopted to effectively improve the overall system of the robot in understanding natural language instructions and planning and executing long sequence tasks. Specifically, the upper task planner uses the VLM technology to overcome the problem that the computer vision solution in the related art cannot effectively identify and segment various express items (such as stacked, wrinkled or never seen express items). The upper limb controller uses the VLA technology to enable the robot to convert semantic instructions into flexible action sequences, thereby solving the limitation problem that the related art can only rely on fixed programming and cannot adapt to multi-station requirements and flexible scheduling. In this way, the flat piece sorting provided in the embodiments of the present application improves the efficiency, reliability and flexibility of logistics sorting. In addition, by using the embodied robot as an execution end for executing the flat piece sorting task, the movement ability of the lower limbs of the embodied robot is used, so that the method proposed in the present application can adapt to the existing unmodified manual sorting site and environmental facilities, without the need for site modification to adapt to automated equipment. At the same time, the humanoid upper limb structure of the embodied robot is used, so that the robot can adapt to the operation actions of different sorting plan requirements, so that the perception flexibility and operation adaptability close to manual operation can be realized.

[0065] In a specific embodiment, as shown in Figure 9 The workflow of the upper limb flat piece sorting can include the following steps: 1. The task planner issues task instructions, delivery bin positions and other information; 2. The lower limb controller controls the robot to move to the delivery slot position according to the positioning (such as specific target path points) determined by the navigation strategy and the corresponding task instructions; 3. The lower limb controller controls the robot to adjust the center of gravity height according to the lower limb movement strategy; 4. The upper limb controller outputs the action sequence according to the upper limb movement strategy, the corresponding task instructions, the current running state, and the visual data; 5. The upper limb controller controls the robot upper limb according to the action sequence; 6. The lower limb controller controls the robot lower limb to follow the movement to keep the robot stable according to the lower limb movement strategy.

[0066] The lower limb control process can include the following steps: 1. Construct a navigation map based on the navigation strategy; 2. Generate a target navigation path according to the task instructions sent by the task planner and the navigation map; 3. The lower limb controller controls the lower limb movement according to the lower limb movement strategy and the target path points in the target navigation path; if an obstacle is encountered, stop and send a request to update the target path points; 4. Call the navigation strategy to re-plan the path based on the request; 5. When performing the upper limb action, the lower limb controller autonomously adjusts the robot's center of gravity according to the lower limb movement strategy.

[0067] Referring to Figure 10 The embodiment of the present application also provides a flat piece sorting device, which comprises: A data acquisition unit 1010 is configured to acquire flat piece sorting instructions and the current running state of a robot; wherein the robot is a body-equipped robot, and the robot comprises an upper limb and a lower limb; A task planning unit 1020 is configured to call a preset task planner to perform task planning based on the flat piece sorting instructions and the current running state to obtain a flat piece sorting plan, wherein the flat piece sorting plan comprises a plurality of ordered target sorting sub-plans; An instruction generation unit 1030 is configured to acquire the plan completion state of the robot, and generate target task instructions based on the corresponding target sorting sub-plans in sequence according to the plan completion state; A sorting control unit 1040 is configured to call a preset upper limb controller to control the upper limb of the robot according to the target task instructions, or call a preset or lower limb controller to control the lower limb of the robot according to the target task instructions, so that the robot executes the corresponding target sorting sub-plan.

[0068] It should be noted that the flat piece sorting device provided by the embodiments of the present application is used to implement the flat piece sorting method provided by the above embodiments, and the specific implementation process corresponds to the flat piece sorting method of the above embodiments. Please refer to the foregoing flat piece sorting method, which will not be described here.

[0069] The embodiments of the present application also provide an electronic device (i.e., a computer device), which includes a memory and a processor. The memory stores a computer program, and the processor can implement the flat piece sorting method of any one of the above embodiments when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0070] Please refer to Figure 11 , Figure 11 The hardware structure of an electronic device of another embodiment is illustrated, which includes: The processor 1110 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application. The memory 1120 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1120 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1120 and called and executed by the processor 1110 to implement the flat piece sorting method of the embodiments of the present application. The input / output interface 1130 is used to implement information input and output. The communication interface 1140 is used to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 1150 transmits information between various components (for example, the processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140) of the device. The processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140 are connected to each other through the bus 1150 for communication connection within the device.

[0071] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is used for causing a computer to execute the flat part sorting method in the above embodiment.

[0072] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0073] The embodiment of the present application further provides a computer program product, which stores program instructions, and the program instructions cause a computer to implement the flat part sorting method described in any of the above embodiments when executed by the computer.

[0074] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0075] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0076] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0077] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0078] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order unless otherwise stated below.

[0079] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0080] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0081] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0082] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0083] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0084] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for sorting flat parts, characterized in that, The method includes: Obtain flat part sorting instructions and the current operating status of the robot; wherein, the robot is a body-bound robot, and the robot includes upper limbs and lower limbs; The preset task planner is invoked to perform task planning based on the flat part sorting instructions and the current running status to obtain a flat part sorting plan, wherein the flat part sorting plan includes multiple ordered target sorting sub-plans; Obtain the robot's plan completion status, and generate target task instructions sequentially based on the corresponding target sorting sub-plan according to the plan completion status; The robot can execute the corresponding target sorting sub-plan by calling a preset upper limb controller to control the robot's upper limbs according to the target task instruction, or by calling a preset lower limb controller to control the robot's lower limbs according to the target task instruction.

2. The method according to claim 1, characterized in that, The upper limb controller includes a feature extraction module and a motion strategy module. Based on the target task instructions, the upper limb controller is invoked to control the robot's upper limbs, including: The robot acquires first target visual data, and the feature extraction module is invoked to perform feature extraction based on the first target visual data and the target task instruction to obtain a target feature vector. The current running state of the robot is obtained, and the action strategy module is invoked to perform cross-attention processing based on the target feature vector and the current running state to obtain the target action sequence; The robot's upper limbs are controlled according to the target action sequence.

3. The method according to claim 2, characterized in that, Before invoking the feature extraction module to extract features based on the first target visual data and the target task instruction to obtain the target feature vector, the method further includes training the upper limb controller, including: Obtain the first sample sorting sub-plan; Obtain the first sample task instruction of the first sample sorting sub-plan and the first sample visual data of the robot; The pre-trained feature extraction module is invoked to extract features based on the first sample visual data and the first sample task instruction, thereby obtaining the sample feature vector; The reference action sequence of the first sample sorting sub-plan and the sample running state of the robot are obtained. The action strategy module is called to perform cross-attention processing based on the sample feature vector and the sample running state to obtain the sample upper limb action sequence. The parameters of the feature extraction module are frozen, and the upper limb controller is trained based on the sample upper limb action sequence and the reference action sequence.

4. The method according to claim 3, characterized in that, The method for pre-training the feature extraction module includes: Obtain environmental images of the logistics site where the robot is located and descriptive text of the environmental images; The preset feature extraction model is fine-tuned and trained based on the environmental image and the descriptive text.

5. The method according to claim 2, characterized in that, After controlling the robot's upper limb according to the target action sequence, the method further includes: The balance impact data of the robot is determined based on the target action sequence; The lower limb controller is invoked to control the robot's lower limbs based on the balance impact data and the robot's current operating state.

6. The method according to claim 1, characterized in that, The lower limb controller is invoked to control the robot's lower limbs according to the target task instruction, including: A target navigation path is generated based on the target task instructions and a pre-built navigation map, wherein the target navigation path includes multiple target path points, and each target path point includes target location coordinates and target orientation; The robot acquires the second target visual data and current operating status, and then invokes the lower limb controller to control the robot's lower limbs based on the second target visual data, the current operating status, the target position coordinates corresponding to each target path point, and the target orientation.

7. The method according to claim 6, characterized in that, Before invoking the lower limb controller to control the robot's lower limbs based on the second target visual data, the current operating state, the target position coordinates corresponding to each target path point, and the orientation, the process further includes training the lower limb controller, including: Obtain the second sample sorting sub-plan, and determine the second sample task instruction based on the second sample sorting plan; A sample navigation path is generated based on the second sample task instruction and the pre-built navigation map, wherein the sample navigation path includes multiple sample path points, and each sample path point includes sample position coordinates and sample orientation; The robot acquires second sample visual data and current running state in a simulation environment, and calls the lower limb controller to generate a sample lower limb action sequence based on the second sample visual data, current running state, sample position coordinates and sample orientation corresponding to each sample path point; The robot's lower limbs are controlled to move according to the sample lower limb movement sequence, and the updated running status of the robot after the lower limb movement is obtained; The action reward score is determined based on the updated running status and the preset reward function, and the lower limb controller is trained based on the action reward score.

8. A flat parts sorting device, characterized in that, The device includes: A data acquisition unit is used to acquire flat part sorting instructions and the current operating status of the robot; wherein, the robot is a body-bound robot, and the robot includes upper limbs and lower limbs; The task planning unit is used to call a preset task planner to perform task planning based on the flat part sorting instructions and the current running status to obtain a flat part sorting plan, wherein the flat part sorting plan includes multiple ordered target sorting sub-plans; The instruction generation unit is used to obtain the robot's plan completion status and generate target task instructions in sequence based on the corresponding target sorting sub-plan according to the plan completion status. The sorting control unit is used to call a preset upper limb controller to control the upper limbs of the robot according to the target task instruction, or to call a preset lower limb controller to control the lower limbs of the robot according to the target task instruction, so that the robot can execute the corresponding target sorting sub-plan.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images