A method for retrieving and placing items and an embodied robot

Abstract

This application relates to a method for retrieving and placing items and an embodied robot. The method includes: acquiring video images of the work area using the embodied robot; identifying the target cabinet door containing the target container from the video images; determining the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door; and planning the motion path of the robotic arm; and controlling the robotic arm to retrieve and place items from the target container based on the motion path. This application achieves the sequential determination of the target cabinet door's state, the position of the target container inside the cabinet door, and the movement of the robotic arm to retrieve items such as flux based on video images acquired by the embodied robot. This reduces the risk of manual labor coming into contact with hazardous materials and improves work efficiency, solving the problem of how to apply embodied robots in high-risk and highly repetitive industrial manufacturing.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method for picking up and placing items and an embodied robot. Background Technology

[0002] In recent years, with the continuous development of artificial intelligence, sensor technology and materials science, embodied robots, which previously only existed in movies and TV shows, have begun to appear in people's real lives. These robots not only have human physical forms, but are also able to learn and improve through interaction with their environment.

[0003] In the industrial manufacturing sector, the handling and application of flux is a critical step in the welding process. Currently, the industry generally still relies on manual operation. However, the volatile chemicals in flux (such as rosin and acidic components) may have adverse effects on workers' health, and the high-frequency repetitive operation can easily lead to fatigue.

[0004] Currently, no effective solution has been proposed for the problem of how to apply embodied robots in high-risk and highly repetitive industrial manufacturing. Summary of the Invention

[0005] This application provides a method for picking up and placing items and a hymen robot, to at least solve the problem in the related art of how to realize the application of hymen robots in high-risk and highly repetitive industrial manufacturing.

[0006] In a first aspect, embodiments of this application provide a method for taking and placing items, the method comprising:

[0007] Video images of the work area are collected using an embodied robot;

[0008] Identify the target cabinet door where the target container is located from the video image, and determine the current state of the target cabinet door;

[0009] Based on the current state of the target cabinet door, determine the position coordinates of the target container inside the target cabinet door;

[0010] Based on the position coordinates of the target container, the motion path of the robotic arm is planned;

[0011] Under the constraints of the motion path, the robotic arm is controlled to pick up items from the target container.

[0012] In some embodiments, identifying the target cabinet door where the target container is located from the video image and determining the current state of the target cabinet door includes:

[0013] The target container is identified from the video image using a target detection model, and the embodied robot is controlled to move to the front of the target cabinet door.

[0014] The distance between the target cabinet door and the door frame is detected. If the distance is within a preset threshold range, the target cabinet door is currently in a half-open state; if the distance is greater than the preset threshold range, the target cabinet door is currently in a fully open state; if the distance is less than the preset threshold range, the target cabinet door is currently in a fully closed state.

[0015] In some embodiments, determining the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door includes:

[0016] If the target cabinet door is currently in a half-open state, and a target container is detected inside the target cabinet door, the position coordinates of the target container are directly determined; otherwise, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container are determined.

[0017] In some embodiments, determining the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door further includes:

[0018] If the target cabinet door is currently in a fully open state, the position coordinates of the target container inside the target cabinet door are directly determined.

[0019] In some embodiments, determining the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door further includes:

[0020] When the target cabinet door is currently in a completely closed state, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container inside the target cabinet door are determined.

[0021] In some embodiments, the motion path of the robotic arm is planned based on the position coordinates of the target container, including:

[0022] Based on the position coordinates of the target container and the current state of the robotic arm, the initial motion path of the robotic arm is planned through a preset path planning algorithm, wherein the preset path planning algorithm includes the potential field method and the probabilistic route map algorithm.

[0023] In some embodiments, under the constraints of the motion path, after the robotic arm is controlled by the motion control model to move to the target position, the end effector of the robotic arm is then controlled to pick up the item from the target container, including:

[0024] During the process of the robotic arm moving to the target position under the initial motion path constraint, the initial motion path is continuously optimized and adjusted by the motion control model based on the end-effector video images collected in real time by the end-effector camera until the robotic arm reaches the target position.

[0025] Based on the force sensors on the end effector of the robotic arm that collect force in real time, the motion control model controls the end effector of the robotic arm to pick up items from the target container.

[0026] In some embodiments, the method includes:

[0027] A target detection model is constructed based on a preset target detection algorithm, wherein the preset target detection algorithm includes the YOLO target detection algorithm and the SSD target detection algorithm;

[0028] The target detection model is trained using images of the target cabinet door and the target container to obtain a trained target detection model;

[0029] A motion control model is constructed based on a preset deep learning algorithm, and the motion control model is trained using robot joint data, pose data and video images generated by the robot in the process of picking up and placing items, so as to obtain a trained motion control model.

[0030] In some embodiments, the target detection model and the motion control model are deployed locally within the embodied robot.

[0031] Secondly, embodiments of this application provide a body-worn robot, which is used to perform the item picking and placing method described in the first aspect above;

[0032] The embodied robot collects video images of its work area;

[0033] The embodied robot identifies the target cabinet door where the target container is located from the video image and determines the current state of the target cabinet door;

[0034] The embodied robot determines the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door;

[0035] The embodied robot plans the motion path of the robotic arm based on the position coordinates of the target container;

[0036] The embodied robot, constrained by the motion path, controls the end effector of its robotic arm to pick up items from the target container.

[0037] Compared to related technologies, the present application provides a method for retrieving and placing items and an embodied robot. This method involves the embodied robot acquiring video images of the work area; identifying the target cabinet door containing the target container from the video images and determining the current state of the target cabinet door; determining the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door; planning the motion path of the robotic arm based on the position coordinates of the target container; and controlling the robotic arm to retrieve items from the target container under the constraints of the motion path. This method achieves the sequential determination of the target cabinet door's state, the position of the target container inside the cabinet door, and the movement of the robotic arm to retrieve items such as flux, based on the video images acquired by the embodied robot. This reduces the risk of manual labor coming into contact with hazardous materials and improves work efficiency, solving the problem of how to apply embodied robots in high-risk and highly repetitive industrial manufacturing. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a flowchart of the steps of the article retrieval and placement method according to an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0042] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0043] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0044] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0045] This application provides a method for taking and placing items. Figure 1 This is a flowchart of the steps of the article retrieval and placement method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0046] Step S102: Collect video images of the work area using the embodied robot;

[0047] Specifically, in step S102, the embodied robot's main camera is a depth camera, which is used to acquire video images of the target cabinet door where the item (such as flux) is stored and its surrounding environment. The depth camera not only provides ordinary two-dimensional image information but also depth information of objects in the scene, helping to accurately determine the position and state of the target cabinet door. Furthermore, the embodied robot can also be equipped with a global camera, which is an RGB color camera. This provides the robot with an overall view of its surroundings, preventing the main camera from failing to correct its orientation when no target information is visible, and enabling the robot to promptly handle image anomalies. Furthermore, the acquired video images are pre-processed (e.g., image denoising, enhancement, cropping, etc.) to improve image quality and facilitate subsequent processing.

[0048] Before step S102, the method further includes step S101, which involves self-checking and initializing the embodied robot. The robot control system and related equipment are turned on, and a system self-check is performed to check whether the robotic arm, end effector, sensors, and other hardware are functioning properly, ensuring that there are no fault alarms in any component; the robot's motion parameters and control parameters (such as the initial position of the robotic arm and the initial state of the gripper) are initialized.

[0049] Step S104: Identify the target cabinet door where the target container is stored from the video image and determine the current state of the target cabinet door;

[0050] Step S104 specifically includes the following steps:

[0051] Step S1041: Construct an object detection model based on a preset object detection algorithm (such as YOLO algorithm, SSD algorithm, etc.), and train the object detection model using the target cabinet door image and the target container image to obtain a trained object detection model, so that the model learns to recognize the target cabinet door and the target container.

[0052] Step S1042: Identify the target cabinet door where the target container is located from the video image using the target detection model, and control the embodied robot to move to the front of the target cabinet door.

[0053] Step S1043: Based on the depth information and image features of the video image, determine the current state of the identified target cabinet door. This current state includes a half-open state, a fully open state, and a fully closed state. Specifically, the method for determining the current state of the target cabinet door can be: ① Detecting the distance between the target cabinet door and the door frame. If the distance is within a preset threshold range, the target cabinet door is in a half-open state; if the distance is greater than the preset threshold range, the target cabinet door is in a fully open state; if the distance is less than the preset threshold range (including cases where the distance is zero, i.e., not detected), the target cabinet door is in a fully closed state; or ② Displaying the current state of the target cabinet door using LEDs in different colors, and acquiring the opening state through sensors. Furthermore, the opening direction of the target cabinet door (e.g., left, right, up, down) can be determined by detecting the features of the cabinet door edge and its relationship with the surrounding environment, providing a more accurate cabinet door state for the robotic arm's motion path planning in subsequent steps.

[0054] Step S106: Based on the current state of the target cabinet door, determine the position coordinates of the target container inside the target cabinet door;

[0055] Specifically, step S106 states that the current state of the target cabinet door includes three states: half-open, fully open, and fully closed.

[0056] If the target cabinet door is currently in a half-open state, and the target container inside the target cabinet door is detected, the position coordinates of the target container are directly determined; otherwise, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container are determined.

[0057] It should be noted that when the target cabinet door is half-open, if the target container inside cannot be detected, the robotic arm's end effector gripper will perform an opening action, using the gripper to pull the cabinet door open by grasping its edge. If the target container can be detected, its position coordinates are directly determined. This enables targeted item retrieval and placement. If the target container's position coordinates can be determined directly when the target cabinet door is half-open, these coordinates are used directly, rather than uniformly opening the cabinet door first to determine the coordinates, thus improving the efficiency of the embodied robot in retrieving and placing items.

[0058] If the target cabinet door is currently fully open, the position coordinates of the target container inside the target cabinet door can be determined directly.

[0059] When the target cabinet door is currently in a completely closed state, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container inside the target cabinet door are determined.

[0060] It should be noted that when the target cabinet door is closed, the motion path of the robotic arm's end effector gripper is planned based on the door's position and depth information. Specifically, the required position and orientation of the gripper are calculated to grasp the door handle or other operable parts. Considering the robot's kinematic constraints, the three-dimensional position information of the cabinet door is converted into the joint angles of the robotic arm to ensure that the robotic arm can smoothly reach the operating position without collision. After confirming that the gripper has reached the predetermined position, the gripper is controlled to perform the door-opening action against the door latch. Furthermore, appropriate force and torque are applied to the target cabinet door, rotating or pulling the gripper in the opening direction to open the door. Here, the resistance of the cabinet door needs to be considered; the required force can be determined experimentally to avoid damaging the door or gripper due to excessive force.

[0061] Step S108: Based on the position coordinates of the target container, the motion path of the robotic arm is planned;

[0062] Specifically, in step S108, based on the position coordinates of the target container and the current state of the robotic arm, the initial motion path of the robotic arm is planned using a preset path planning algorithm, which includes the potential field method and the probabilistic route map algorithm.

[0063] It should be noted that the position coordinates and orientation information of the target container holding the item (such as flux) are determined through visual recognition (or directly determined by preset coordinate information, which requires accurate measurement and input of the target container's position coordinates beforehand). A path planning algorithm is then used to plan the robotic arm's motion path from its current position to the target container's position. Path planning must consider factors such as the robotic arm's range of motion, joint limitations, and movement speed to ensure the safety and feasibility of the path.

[0064] Step S110: Under the constraints of the motion path, control the robotic arm to pick up the items in the target container.

[0065] Step S110 specifically includes the following steps:

[0066] Step S1101: During the process of the robotic arm moving to the target position under the initial motion path constraint, the initial motion path is continuously optimized and adjusted by the motion control model based on the end-effector video images collected in real time by the end-effector camera until the robotic arm reaches the target position.

[0067] It should be noted that the robotic arm moves towards the target container according to the initial motion path. During the movement, the robotic arm's motion status is monitored in real time by a vision sensor (camera at the end of the robotic arm). Fine adjustments are made based on the information fed back by the sensor to ensure that the robotic arm accurately reaches the predetermined position above the container, accurately detects the position and posture of the target container, and adjusts the angle and position of the gripper to accurately align it with the gripping part of the container.

[0068] Step S1102: Based on the force collected in real time by the force sensor on the end effector of the robotic arm, the end effector of the robotic arm is controlled by the motion control model to pick up the items in the target container.

[0069] It should be noted that after the clamp is aligned with the target container, it closes with appropriate force based on factors such as the container's material, shape, and weight, gripping the container firmly. Simultaneously, a force sensor monitors the clamp's gripping force in real time, ensuring that the gripping force is sufficient to secure the container without damaging it or causing flux leakage due to excessive force. Further, after successful gripping, the robotic arm pulls the flux container from its storage location at a smooth speed and appropriate angle. During the pull-out process, care must be taken to avoid collisions with surrounding equipment or objects, while maintaining the container's stability to prevent spillage of items (such as flux). Furthermore, in scenarios where items are stored, after the container is pulled to the appropriate replenishment position, appropriate operations are performed on the container according to the replenishment system's requirements (such as opening the container's outlet valve, adjusting the container's tilt angle, etc.) to facilitate smooth flux flow for replenishment.

[0070] Through the steps described in this application embodiment, based on the video images acquired by the embodied robot, the state of the target cabinet door is determined sequentially, the position of the target container inside the cabinet door is determined, and the robotic arm moves to pick up items such as flux. This reduces the risk of manual operation coming into contact with dangerous items and improves work efficiency, solving the problem of how to realize the application of embodied robots in high-risk and highly repetitive industrial manufacturing.

[0071] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0072] In some embodiments, for the motion control model used in the above embodiments, the method includes: constructing a motion control model based on a preset deep learning algorithm, and training the motion control model using robot joint data, pose data, and video images generated by the android during object retrieval and placement to obtain a trained motion control model. Specifically, this includes the collection of training data for the motion control model, data preprocessing, model selection and training, and model evaluation and optimization.

[0073] I. Robot Joint Data Collection: During normal operation of the production line, sensors are used to record joint data of the embodied robot, including joint angles, speeds, accelerations, torques, etc. This data can be collected through built-in sensors or externally mounted high-precision sensors. The collected data is stored in a database or file system for subsequent processing and analysis. II. Video Image Data Collection: Using the embodied robot's main camera and the end effector camera of the robotic arm, video images containing RGB color images and corresponding depth maps are acquired during the item handling process. The cameras should cover the robot's working area and key locations for flux replenishment. Image data can be saved in common image formats (such as JPEG, PNG, BMP), and video data can be stored in MP4 or similar formats. To better correlate with robot joint data, image and video data need to be timestamped to ensure that each image or video frame at any given time corresponds to the relevant robot joint data.

[0074] II. Robot Joint Data Preprocessing: The collected joint data is cleaned to remove noise and outliers. For example, statistical methods (such as mean filtering and median filtering) are used to handle data anomalies caused by potential sensor errors; the joint data is normalized by converting it from motor angles to radians (-π to π range) to improve the stability and convergence speed of model training; the joint data is organized according to time series so that the model can learn the time dependencies of robot movements. Image and Video Data Preprocessing: Image data is cropped, rotated, and scaled to adjust it to a uniform size and resolution to meet the model's input requirements; for video data, it can be decomposed into frame sequences, and each frame can undergo similar preprocessing operations as the image data. Image enhancement techniques from deep learning (such as random cropping, flipping, and color transformation) are used to enhance image and video frames to increase data diversity and improve the model's generalization ability. Convert image or video frame data into a suitable tensor format so that it can be input into a deep learning model (e.g., use the OpenCV library to convert the image into a NumPy array, and then convert it into a PyTorch or TensorFlow tensor).

[0075] III. Model Selection and Training: For control models that combine joint data and image / video data, deep learning architectures can be considered. For example, Deep Q-Networks (DQN) and its variants (Double DQN, Dueling DQN) or policy gradient-based methods (such as A2C, A3C, PPO) can be used. Alternatively, Convolutional Neural Networks (CNNs) can be used to process image / video data, combining the processed results with joint data through fully connected layers, and then processing time-series information through Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) to finally output control actions. Another option is to use Variational Autoencoders (VAEs) or variants of Generative Adversarial Networks (GANs) to encode image / video data and joint data into a common latent space, and then generate control actions through a decoder. The preprocessed joint data and image / video data are used as input, and the desired robot action is used as the output label to construct a training dataset. The dataset is divided into training, validation, and test sets. For deep learning models, appropriate optimizers (such as Adam, SGD) and loss functions (such as mean squared error, cross-entropy) are used for training. During training, model performance is monitored using a validation set to prevent overfitting. Hyperparameters, such as learning rate, batch size, number of network layers, and number of nodes, are adjusted based on the validation set performance.

[0076] IV. Model Evaluation and Optimization: Evaluate the trained model using a test set, calculating performance metrics such as accuracy, mean squared error, recall, and precision. Select appropriate metrics based on the specific requirements of the task. Observe the model's prediction results on the test set, analyze prediction errors, and identify potential problems with the model, such as inaccurate predictions of certain actions or poor performance in complex scenarios. Perform visualization analysis, such as comparing the robot's predicted actions with actual actions, and using visualization tools (such as matplotlib) to plot the predicted and actual values ​​of joint angles and positions as curves for intuitive error viewing. Optimize the model based on the evaluation results. If overfitting is the issue, regularization techniques (such as L1 and L2 regularization) and Dropout methods can be used. Adjust the model structure, such as increasing or decreasing the number of network layers, adjusting the number of neurons, and changing the activation function.

[0077] In some embodiments, the object detection model and motion control model are deployed locally within the embodied robot. It's important to note that local deployment of the models means that data does not need to be transmitted to a remote server for processing, reducing network latency and enabling the robot to react more quickly to environmental changes. Even in situations with poor network conditions or no network connection, the robot can independently complete tasks using the local model. Furthermore, by appropriately configuring local computing resources to adapt to different application scenarios and workloads, specific tasks can be executed more efficiently.

[0078] This application provides a body-worn robot for performing the article picking and placing method described in the first aspect above.

[0079] The embodied robot collects video images of its work area;

[0080] The embodied robot identifies the target cabinet door where the target container is located from video images and determines the current state of the target cabinet door;

[0081] The embodied robot determines the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door;

[0082] The embodied robot plans the motion path of its robotic arm based on the position coordinates of the target container;

[0083] The embodied robot controls its robotic arm to retrieve items from a target container, constrained by a motion path.

[0084] The embodied robot in this application embodiment enables the determination of the state of the target cabinet door, the position of the target container inside the cabinet door, and the movement of the robotic arm to pick up items such as flux based on the video images acquired by the embodied robot. This reduces the risk of manual operation coming into contact with dangerous items and improves work efficiency, solving the problem of how to apply embodied robots in high-risk and highly repetitive industrial manufacturing.

[0085] This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0086] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0087] Optionally, the electronic device may further include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for picking up and placing items. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0088] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0089] In addition, in conjunction with the item retrieval and placement methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the item retrieval and placement methods in the above embodiments.

[0090] In one embodiment, Figure 2 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 2 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 2 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a method for picking up and placing items, and the database stores data.

[0091] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0093] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for taking and placing items, characterized in that, The method includes: Video images of the work area are collected using an embodied robot; The target container is identified from the video image using a target detection model, and the embodied robot is controlled to move to the front of the target container. The distance between the target cabinet door and the door frame is detected. If the distance is within a preset threshold range, the current state of the target cabinet door is half open; if the distance is greater than the preset threshold range, the current state of the target cabinet door is fully open; if the distance is less than the preset threshold range, the current state of the target cabinet door is fully closed. If the target cabinet door is currently in a half-open state, and a target container is detected inside the target cabinet door, the position coordinates of the target container are directly determined; otherwise, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container are determined. If the target cabinet door is currently fully open, the position coordinates of the target container inside the target cabinet door are directly determined. When the target cabinet door is currently in a completely closed state, the robotic arm of the embodied robot is autonomously controlled by the motion control model to make the target cabinet door fully open, and then the position coordinates of the target container inside the target cabinet door are determined. Based on the position coordinates of the target container, the motion path of the robotic arm is planned; Under the constraints of the motion path, the robotic arm is controlled by the motion control model to move to the target position, and then the end effector of the robotic arm is controlled to pick up the item in the target container.

2. The method according to claim 1, characterized in that, Based on the position coordinates of the target container, the planned motion path of the robotic arm includes: Based on the position coordinates of the target container and the current state of the robotic arm, the initial motion path of the robotic arm is planned through a preset path planning algorithm, wherein the preset path planning algorithm includes the potential field method and the probabilistic route map algorithm.

3. The method according to claim 2, characterized in that, Under the constraints of the motion path, after the robotic arm is controlled to move to the target position by the motion control model, the end effector of the robotic arm is then controlled to pick up the item from the target container, including: During the process of the robotic arm moving to the target position under the initial motion path constraint, the initial motion path is continuously optimized and adjusted by the motion control model based on the end-effector video images collected in real time by the end-effector camera until the robotic arm reaches the target position. Based on the force sensors on the end effector of the robotic arm that collect force in real time, the motion control model controls the end effector of the robotic arm to pick up items from the target container.

4. The method according to claim 1, characterized in that, The method includes: A target detection model is constructed based on a preset target detection algorithm, wherein the preset target detection algorithm includes the YOLO target detection algorithm and the SSD target detection algorithm; The target detection model is trained using images of the target cabinet door and the target container to obtain a trained target detection model; A motion control model is constructed based on a preset deep learning algorithm, and the motion control model is trained using robot joint data, pose data and video images generated by the robot in the process of picking up and placing items, so as to obtain a trained motion control model.

5. The method according to claim 1, characterized in that, The target detection model and the motion control model are deployed locally in the embodied robot.

6. A embodied robot, characterized in that, The embodied robot is used to perform the article picking and placing method according to any one of claims 1 to 5; The embodied robot collects video images of its work area; The embodied robot identifies the target cabinet door where the target container is located from the video image and determines the current state of the target cabinet door; The embodied robot determines the position coordinates of the target container inside the target cabinet door based on the current state of the target cabinet door; The embodied robot plans the motion path of the robotic arm based on the position coordinates of the target container; The embodied robot, under the constraints of the motion path, controls its robotic arm to retrieve items from the target container.

Images