Pose planning method, robot, computer device and storage medium

Abstract

The application discloses a pose planning method, a robot, a computer device and a storage medium, relates to the technical field of robots, and aims to solve the problem that code scanning fails in actual cargo box stack scanning operations in the prior art. The method comprises the following steps: acquiring observation data corresponding to a current side of a cargo box stack; determining a target label to be scanned on the current side based on the observation data, and a position coordinate of the target label to be scanned in a robot body coordinate system; determining a plane orientation feature corresponding to the current side based on the observation data; and planning a target pose of the robot for scanning the side of the cargo box stack according to the position coordinate and the plane orientation feature.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a pose planning method, a robot, a computer device, and a storage medium. Background Technology

[0002] With the development of logistics automation technology, robots are increasingly used in warehousing operations. In the cargo information entry process, robots are often used to automatically scan the labels on stacks of cargo boxes to complete data collection.

[0003] In related technologies, robots can locate labels based on visual perception and move to the label's location to perform scanning operations. However, in practical applications, stacks of containers are usually composed of multiple containers stacked together, and their arrangement is uncertain. In this case, relying solely on the label's location information to guide the robot's operation can easily lead to scanning failures. Summary of the Invention

[0004] Based on this, a pose planning method, a robot, a computer device, and a storage medium are provided to solve the problem that scanning failures are common in actual cargo stacking barcode scanning operations.

[0005] In a first aspect, this application provides a pose planning method, the method comprising: Acquire observation data corresponding to the current side of the cargo stack; wherein, the current side is the side of the robot facing the cargo stack; Based on the observation data, the target barcode label to be scanned on the current side is determined, as well as the position coordinates of the target barcode label in the robot body coordinate system; Based on the observation data, the planar orientation feature corresponding to the current side is determined; Based on the position coordinates and the plane orientation features, the target pose of the robot is planned so that it is performing a barcode scanning operation on the current side.

[0006] By analyzing the observation data of the current side of the cargo container stack using the above method, the position coordinates of the target label to be scanned on the current side in the robot's body coordinate system and the planar orientation characteristics of the side can be accurately obtained. Based on the label position coordinates and the planar orientation characteristics of the side, the optimal scanning target pose of the robot facing the current side can be planned, thereby improving the scanning success rate.

[0007] In some embodiments, determining the target barcode label to be scanned on the current side and the position coordinates of the target barcode label in the robot body coordinate system based on the observation data includes: Based on the observation data, the first pixel region corresponding to the target label to be scanned is extracted; Calculate the center pixel coordinates of the first pixel region, and obtain the depth value corresponding to the center pixel coordinates from the observation data; Based on preset coordinate transformation parameters, the center point pixel coordinates and the depth value are transformed to the robot body coordinate system to obtain the position coordinates of the target label to be scanned in the robot body coordinate system.

[0008] Using the above method, based on the observation data, the first pixel region corresponding to the target label to be scanned is extracted and its center pixel coordinates are calculated. Then, combined with the depth value corresponding to the position coordinates of the target label to be scanned, the positioning of the target label to be scanned can be accurately obtained. Then, by using coordinate transformation parameters, the pixel coordinates and depth value are mapped to the robot body coordinate system, and the actual spatial position coordinates of the target label to be scanned relative to the robot can be accurately obtained, providing an accurate label position basis for subsequent planning of the robot's working pose facing the side.

[0009] In some embodiments, determining the planar orientation feature corresponding to the current side based on the observation data includes: Based on the observation data, the second pixel region corresponding to the current side is extracted, and the depth dataset corresponding to the second pixel region is obtained from the observation data; Based on the second pixel region and the depth dataset, a point cloud dataset of the current side surface in the robot body coordinate system is generated; The point cloud dataset is fitted with a plane, and the normal vector of the current side is determined based on the fitting result, which is used as the plane orientation feature.

[0010] By extracting the second pixel region and its corresponding depth dataset corresponding to the current side using the above method, a point cloud dataset of the current side in the robot body coordinate system can be generated. Then, plane fitting is performed on the point cloud dataset to calculate the normal vector of the current side, which can accurately obtain the planar orientation features of the current side, thereby providing an accurate orientation basis for planning the robot's working pose facing the side.

[0011] In some embodiments, the target pose includes at least a target position and a target orientation; the step of planning the target pose for the robot to perform barcode scanning operations on the current side based on the position coordinates and the planar orientation features includes: The position coordinates are projected onto a preset plane in the robot's body coordinate system to obtain the projection center point; The target position is determined based on the projection center point and the plane orientation feature, wherein the distance between the target position and the projection center point is a preset working distance, which is used to ensure that the robot's scanning device can perform scanning operations on the target label to be scanned within the effective working distance; Based on the planar orientation characteristics, the target orientation is determined so that the robot is facing the current side.

[0012] Using the above method, the position coordinates of the target label to be scanned are projected onto a preset plane to obtain the projection center point. Combined with the planar orientation characteristics of the current side, the target position of the robot is planned. At the same time, based on the planar orientation characteristics, the target orientation that makes the robot face the current side is planned, thereby accurately generating the optimal working pose that meets the scanning distance requirements and ensures the facing posture, laying the foundation for the robot to perform efficient and accurate scanning operations.

[0013] In some embodiments, before acquiring the observation data corresponding to the current side of the container stack, the method further includes: Acquire pre-collected image data corresponding to the stack of cargo containers from multiple sides; The area of ​​the label to be scanned on the current side is determined based on the pre-acquired image data; The step of determining the target tag to be scanned on the current side based on the observation data includes: Based on the observation data and the area of ​​the label to be scanned, the target label to be scanned on the current side is determined.

[0014] By analyzing the pre-collected image data from multiple sides of the cargo container stack using the above method, the area of ​​the label to be scanned on the current side can be determined in advance. In actual operation, based on the matching of real-time observation data with the predefined area, the target label to be scanned can be quickly and accurately locked, effectively improving the targeting and efficiency of the scanning operation.

[0015] In some embodiments, before acquiring the observation data corresponding to the current side view of the container stack, the method further includes: In response to the robot being positioned at a preset waiting position, monitoring data is acquired in a specified direction within the robot's body coordinate system; Multi-target tracking is performed based on the monitoring data, and the multi-target tracking includes at least tracking a forklift and a stack of containers, wherein the forklift is used to move the stack of containers. If, based on the tracking results, it is determined that the stack of containers meets the preset static condition and the forklift meets the preset departure condition, then the pose planning method is executed.

[0016] Using the above method, multi-target tracking is performed based on the monitoring data obtained by the robot at the preset waiting position to simultaneously monitor the status of the forklift and the stack of goods. It can determine in real time whether the stack of goods is stationary and whether the forklift has left the work area. The subsequent pose planning process is only triggered after the dual safety conditions are met, thereby effectively avoiding the execution of barcode scanning operations under dynamic interference and ensuring the accuracy and safety of robot operations.

[0017] In some embodiments, after planning the target pose for the robot to perform barcode scanning operation on the current side based on the position coordinates and the planar orientation features, the method further includes: In response to the scanning operation completion signal of the current side, obtain the current observation bit index; The next observation position is determined based on the current observation position index and the preset observation position queue; wherein, the preset observation position queue includes multiple observation positions, each of which is located at a different preset position facing the cargo container stack; Control the robot to move to the next observation position and execute the pose planning method; Once the scanning operation has been completed at all observation positions, the robot is controlled to return to the preset waiting position.

[0018] Using the above method, when the scanning operation on the current side is completed, the current observation position index is obtained and combined with the preset multi-observation position queue around the cargo stack. The robot can be automatically determined and guided to move to the next observation position to repeat the pose planning and scanning process until all preset observation positions are traversed and the robot automatically returns to the preset waiting position. This realizes the fully automatic cyclic scanning operation on multiple sides of the cargo stack, ensuring the integrity of task coverage and closed-loop management of the process.

[0019] Secondly, this application provides a robot, the robot comprising: An observation module is used to acquire observation data corresponding to the current side of the cargo stack; wherein, the current side is the side of the robot facing the cargo stack; The first determining module is used to determine, based on the observation data, the target barcode label to be scanned on the current side and the position coordinates of the target barcode label to be scanned in the robot body coordinate system; The second determining module is used to determine the planar orientation feature corresponding to the current side based on the observation data; The planning module is used to plan the target pose of the robot performing the barcode scanning operation on the current side based on the position coordinates and the plane orientation features.

[0020] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the pose planning method of the first aspect described above.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pose planning method of the first aspect described above.

[0022] The technical effects of each of the second to fourth aspects mentioned above, as well as the technical effects that each aspect may achieve, are described above with reference to the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the floor plan layout for an application scenario applicable to the embodiments of this application; Figure 2 A flowchart illustrating a pose planning method provided in an embodiment of this application; Figure 3 A flowchart illustrating a robot control model training method provided in an embodiment of this application; Figure 4 A flowchart illustrating a robot control method provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a robot control model optimization method provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing together, or B existing alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0025] Before introducing the pose planning method provided in the embodiments of this application, the technical background of this application will be described in detail below for ease of understanding.

[0026] In related technologies, robots can locate labels based on visual perception and move to the label's position to perform barcode scanning. However, in practical applications, stacks of containers are typically composed of multiple containers stacked together, and their arrangement is uncertain. In this case, relying solely on the label's positioning information to guide the robot's operation can easily lead to scanning failures. For example, when the robot is not facing the side of the container, the robot's camera optical axis is not perpendicular to the label plane, causing perspective distortion in the QR code image, thus reducing the scanning recognition rate or even causing scanning failure. Furthermore, since the actual position of the container stack usually differs from the preset position, the robot, after reaching the planned location, still needs to repeatedly move and adjust to find the appropriate scanning posture, resulting in low scanning efficiency.

[0027] In view of this, this application provides a pose planning method, a robot, a computer device, and a storage medium to solve the problems of scanning failure and low efficiency in actual cargo stacking barcode scanning operations.

[0028] The following is a brief introduction to the planar layout diagram of the application scenarios to which the technical solution of this application is applicable. It should be noted that the layout described below is for illustrative purposes only and not for limitation. In specific implementation, the technical solution provided by this application can be flexibly applied according to actual needs.

[0029] Figure 1 This is a schematic diagram of a planar layout for an application scenario to which the pose planning method provided in this application embodiment is applicable. Figure 1 As shown, a global coordinate system XYZ is established, where the XY plane is the horizontal ground, and the positive Z-axis is perpendicular to the ground and pointing upwards. Robot 10 initially resides at a preset waiting position, which is a fixed position pre-defined according to the scene's (e.g., warehouse) floor plan layout. Its projection onto the horizontal ground of the scene is (x...). r y r ), which is the coordinate of the center point of the chassis of robot 10.

[0030] The cargo container stack 20 is located at a fixed position in the scene, and the nominal coordinates of its center point are (x, y). c y c However, in actual operation, there may be a positional error of 10-40 cm. The container stack 20 is typically rectangular in shape, with a length L along the X-axis and a width W along the Y-axis. The container stack 20 has four sides, each of which may have several QR code labels affixed. These labels can be QR codes, barcodes, or other forms of identifiable codes. To scan all labels completely, the robot 10 needs to observe each side sequentially and perform the scanning operation.

[0031] Therefore, four observation positions 21, 22, 23, and 24 are pre-set on the four sides facing the container stack 20, corresponding to the -Y, +X, +Y, and -X sides of the container stack 20, respectively. The pre-set positions of each observation position are based on (x... c y c The dimensions (L and W) of the cargo container stack 20 were calculated.

[0032] For example, the coordinates of the first observation position 21 (facing the -Y side) are (x c y c -W / 2-S1, θ1), the coordinates of the second observation position 22 (facing the +X side) are (x c +L / 2+S1, y c The coordinates of the third observation position 23 (facing the +Y side) are (x, θ2), θ2). c y c +W / 2+S1,θ3), the coordinates of the fourth observation position 24 (facing the -X side) are (x c -L / 2-S1, y c ,θ4).

[0033] S1 is the preset observation distance, used to ensure that the robot 10 can obtain a complete side view. The preferred range is 1.5 to 2 m, and the specific value can be set according to the actual situation. It is not limited here. θ1, θ2, θ3, and θ4 are the initial orientation angles of the robot 10 corresponding to each observation position. They are preset according to the principle that the robot faces the same direction. These angles are usually consistent with the coordinate axis direction (for example, when facing the -Y side, θ1=-90° or 270°, when facing the +X side, θ2=0°, when facing the +Y side, θ3=90°, and when facing the -X side, θ4=180°).

[0034] Optionally, the number of sides of the cargo stack 20 is not limited to four, and can be set according to the actual shape of the cargo stack 20 and the label distribution; the number of observation positions and the preset positions can also be adjusted according to actual needs. For example, when only three sides of the cargo stack 20 are labeled, only three observation positions can be preset; when the cargo stack size is large, S1 can be appropriately increased, and vice versa. This application embodiment does not limit the specific number and coordinate form of the observation positions, as long as the robot 10 can effectively collect the observation data of the corresponding side of the cargo stack 20 at each observation position. S1 ensures that there is sufficient safety margin between the robot 10 and the cargo stack 20, and also ensures that its sensors can effectively collect complete side images.

[0035] In addition, such as Figure 1 As shown, a robot body coordinate system X was also established. b Y b Z bThe origin of this coordinate system is located at the center of the chassis of robot 10, X... b The axis points directly in front of the robot (i.e., the direction of movement when facing the stack of 20 boxes), Y b The axis points to the left side of the robot, Z. b The axis is perpendicular to the horizontal ground and points upwards, aligning with the Z-axis direction of the global coordinate system.

[0036] Based on the aforementioned coordinate system, the robot 10 moves sequentially to each observation position according to the preset observation position queue. When a label is detected on the side of the cargo stack 20 corresponding to a certain observation position, the robot 10's pose planning method is executed, causing it to perform the scanning operation in a target posture facing that side. The scanning operation refers to using the barcode scanner on the robot 10 to scan each label on that side and record the cargo information in the label. After scanning is completed, the robot 10 automatically moves to the next observation position in the preset observation position queue.

[0037] If no tag is detected on the side of the container stack 20 corresponding to the current observation position, the next observation position is determined from the preset observation position queue. The robot is then controlled to move to that observation position and continue to execute the pose planning method and barcode scanning operation. After traversing all observation positions, the robot 10 is controlled to return to the preset waiting position.

[0038] Optionally, the robot 10 can mark the sides of the stack of containers 20 where no tags were detected, for subsequent statistical analysis or as a reference for historical data. This embodiment of the application does not limit this.

[0039] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 2 This is a flowchart illustrating a pose planning method provided in an embodiment of the present application, used to plan a target pose facing the side of a stack of containers before the robot performs a barcode scanning operation; Figure 3 This is a flowchart illustrating a robot control model training method provided in an embodiment of this application, used to efficiently fine-tune the model parameters after acquiring robot operation data; Figure 4 This is a flowchart illustrating a robot control method provided in an embodiment of this application, which is used to generate a target action sequence using a trained model and control the robot to perform barcode scanning during actual operation; Figure 5 This is a flowchart illustrating a robot control model optimization method provided in an embodiment of this application. The above together constitute a complete technical solution from pose planning → model training → model inference → model optimization.

[0040] Figure 2 This application provides a flowchart illustrating a pose planning method, which is applied to... Figure 1 Robot 10 in the process mainly includes the following steps: S201, Obtain the observation data corresponding to the current side view of the cargo container stack; S202, Based on the observation data, determine the target label to be scanned on the current side, and the position coordinates of the target label to be scanned in the robot's body coordinate system; S203, Based on the observation data, determine the planar orientation characteristics corresponding to the current side; S204, based on the position coordinates and planar orientation characteristics, plan the target pose for the robot to perform barcode scanning operations on the current side.

[0041] In this embodiment, the robot 10 acquires observation data of the stack of containers 20 from its onboard vision sensors. The current side is the side of the robot 10 facing the stack of containers 20. The vision sensors include at least a depth camera deployed on the head of the robot 10. This head-mounted depth camera has a sampling frequency of 30 Hz and a depth measurement range covering 0.5 m to 10 m, meeting the needs of various indoor and outdoor warehousing environments.

[0042] Alternatively, the visual sensor can also be a combination of a binocular camera, a structured light camera, a lidar and an RGB camera, etc. The specific choice can be made according to the actual application scenario, and there is no limitation here.

[0043] Using the above method, based on the current side observation data obtained by the vision sensor on the robot 10, the position coordinates of the target label to be scanned and the current spatial orientation of the side can be accurately analyzed. Based on this key information, the optimal scanning pose of the robot facing the current side can be planned, which effectively improves the robot's adaptability to different warehouse environments and increases the success rate and automation level of scanning operations.

[0044] In some embodiments, it is exemplarily illustrated that, based on the observation data corresponding to the current side, the target barcode label to be scanned on the current side and the position coordinates of the target barcode label in the robot body coordinate system are determined, including but not limited to: First, based on the observation data corresponding to the current side view, the first pixel region corresponding to the target label to be scanned is extracted.

[0045] For example, the observation data corresponding to the current side view includes at least an RGB image. A target segmentation algorithm is used to perform semantic segmentation on the RGB image, outputting pixel-level classification results of the same size as the input image, to obtain the first pixel region corresponding to the label to be scanned and the second pixel region corresponding to the current side view.

[0046] The target segmentation algorithm primarily outputs segmentation masks for two types of targets: a first segmentation mask, used to identify the first pixel region of the target label to be scanned in the RGB image (if the first segmentation mask is empty, it means that there is no label on the current side); and a second segmentation mask, used to identify the second pixel region of the current side in the RGB image. The target segmentation algorithm can employ Segment Anything Model (SAM), Mask R-CNN (Mask Region-based Convolutional Neural Network), or other instance segmentation models. These models can accurately segment the pixel region of each label in the RGB image and the pixel region of the side of the cargo stack 20. Even with multiple labels, partial label occlusion, or textures, stains, or partial occlusion on the surface of the cargo stack 20, it can accurately distinguish different label instances and accurately identify the second pixel region of that side.

[0047] Then, the center pixel coordinates of the first pixel region are calculated, and the depth value corresponding to the center pixel coordinates is obtained from the observation data corresponding to the current side view. The methods for determining the center pixel coordinates include, but are not limited to: Obtain the smallest bounding rectangle of the first pixel region as the bounding box, and calculate the average value of all pixel coordinates within the bounding box as the center pixel coordinates; or, use the geometric center coordinates of the smallest bounding rectangle as the center pixel coordinates.

[0048] Alternatively, the center point pixel coordinates can be determined by using a keypoint detection algorithm to detect the four corner points of the first pixel region, and then using the geometric center coordinates of the four corner points as the center point pixel coordinates. The specific determination method depends on the situation and is not limited here.

[0049] After determining the center point pixel coordinates, the depth value corresponding to the center point pixel coordinates is obtained from the observation data corresponding to the current side view.

[0050] For example, the observation data corresponding to the current side view also includes the depth image corresponding to the RGB image. The depth value corresponding to the center point pixel coordinates is obtained from the depth image. If the depth value is invalid (e.g., 0, null value, etc.), a neighborhood interpolation method can be used to obtain the depth value: the mean or median of valid depth values ​​within a certain range (e.g., a 3×3 window, the specific range depends on the situation and is not limited here) around the center point pixel coordinates of the label is obtained as a replacement depth value. If there is still no valid value within the neighborhood, the depth value acquisition is marked as failed, and a re-acquisition or anomaly handling process is triggered.

[0051] Finally, based on preset coordinate transformation parameters, the center point pixel coordinates and depth values ​​are transformed to the robot body coordinate system to obtain the position coordinates of the target label to be scanned in the robot body coordinate system.

[0052] To facilitate coordinate transformation, this application embodiment also establishes a camera coordinate system X. v Y v Z v The origin of this coordinate system is located at the optical center of the vision sensor of robot 10, X. v The axis points to the right of the vision sensor, Y v The axis points below the vision sensor, Z. v The axis along the optical axis of the vision sensor points directly forward. The transformation relationship between the camera coordinate system and the robot body coordinate system can be represented by a pre-calibrated intrinsic and extrinsic parameter matrix, which is a specific implementation of the coordinate transformation parameters preset in this application.

[0053] For example, the intrinsic parameter matrix of the vision sensor is K, and the extrinsic parameter transformation matrix from the camera coordinate system to the robot body coordinate system is [R|t], where R is the rotation matrix and t is the translation vector. The calibration of the intrinsic and extrinsic parameters can be achieved using well-known camera calibration methods (such as Zhang Zhengyou's calibration method) and hand-eye calibration methods, which will not be elaborated here.

[0054] Then, based on K, the center pixel coordinates (u, v), and the corresponding depth value d, the three-dimensional coordinates (X, V, y) of the target label to be scanned in the camera coordinate system are calculated. v1 Y v1 Z v1 The calculation method for X is as follows: v1 Y v1 Z v1 ] T =d×K -1 ×[u, v, 1] T .

[0055] Based on [R|t], (X v1 Y v1 Z v1 Transform to the robot's body coordinate system to obtain the three-dimensional coordinates [X] of the target label to be scanned in the robot's body coordinate system. b1 Y b1 Z b1 The conversion method can be: [X] b1 Y b1 Z b1 ] T =R×[X v1 Y v1 Z v1 ] T +t.

[0056] Optionally, if there are other coordinate transformation relationships between the vision sensor and the robot 10, such as Euler angles or quaternions, the preset coordinate transformation parameters can be the corresponding transformation matrix or transformation parameters. The specific transformation parameters depend on the situation and are not limited here.

[0057] Optionally, when the observation data corresponding to the current side view also includes data from other modalities, the data from these other modalities can be combined to help determine the position coordinates of the target label to be scanned. For example, when the observation data corresponding to the current side view includes point cloud data, the first pixel region obtained from semantic segmentation can be directly mapped to the point cloud space to directly obtain the three-dimensional coordinates of the target label to be scanned from the point cloud space, thereby replacing the calculation method based on depth images; or, when poor lighting conditions cause a decrease in the quality of RGB images, infrared images can be combined to assist in the identification and segmentation of the target label to be scanned.

[0058] Optionally, to improve data processing efficiency, foreground filtering can be performed on the initial depth image after it is obtained.

[0059] For example, a depth threshold is set (determined based on the depth measurement range of the vision sensor, e.g., 3m). Pixels in the depth image with depth values ​​greater than this threshold are marked as background and removed, while foreground regions with depth values ​​less than or equal to the threshold are retained. Subsequent observation data processing will only target these foreground regions, effectively reducing computation and improving processing speed. The depth threshold value can also be adjusted according to the actual scenario. For example, in scenarios where the dimensions of the cargo container stack 20 are large or small, the depth threshold can be increased or decreased accordingly. The specific value depends on the situation and is not limited here.

[0060] Using the method described above, the first pixel region corresponding to the target label to be scanned is accurately extracted from the RGB image of the observation data corresponding to the current side view, based on the target segmentation algorithm. This accurate segmentation is achieved even in complex situations involving multiple labels, partial occlusion, or surfaces with textured dirt. Subsequently, the center pixel coordinates of this region are calculated, and the corresponding depth value is obtained. If the depth value is invalid, neighborhood interpolation is used for compensation, ensuring data reliability. Finally, based on preset coordinate transformation parameters, the center pixel coordinates and depth value are transformed to the robot's body coordinate system, accurately obtaining the three-dimensional coordinates of the target label in actual space. This overcomes environmental interference and data loss issues, improves the accuracy and robustness of label positioning, and lays a solid foundation for subsequent planning of precise scanning poses.

[0061] In some embodiments, it is illustrated, by way of example, that the planar orientation feature corresponding to the current side is determined based on the observation data corresponding to the current side, including but not limited to: First, based on the observation data corresponding to the current side, the second pixel region corresponding to the current side is extracted, and the depth dataset corresponding to the second pixel region is obtained from the observation data corresponding to the current side.

[0062] For example, semantic segmentation is performed on the RGB image in the observation data corresponding to the current side using a target segmentation algorithm to obtain the second pixel region corresponding to the current side (see the aforementioned embodiment for details, which will not be repeated here). Then, the depth dataset corresponding to the second pixel region is obtained from the depth image of the observation data corresponding to the current side.

[0063] Then, based on the second pixel region and depth dataset, a point cloud dataset of the current side of the cargo stack 20 in the robot body coordinate system is generated.

[0064] For example, for each pixel covered by the second pixel region, the three-dimensional coordinates of each pixel in the camera coordinate system are calculated in combination with the corresponding depth value, and then transformed to the robot body coordinate system to obtain the point cloud dataset of the current side of the cargo stack 20 in the robot body coordinate system (for the specific transformation method, please refer to the aforementioned embodiment, which will not be repeated here).

[0065] Optionally, to reduce computational load, the second pixel region can be downsampled, for example, by taking a point every other pixel or every few pixels to participate in point cloud generation, in order to reduce point cloud density and improve processing speed.

[0066] Optionally, to improve the quality of the point cloud, the depth image can be preprocessed, such as through filtering and noise reduction, and hole filling, to reduce the impact of noise points on subsequent plane fitting. The specific method for point cloud generation depends on the situation and is not limited here.

[0067] Finally, a plane fit is performed on the point cloud dataset, and the normal vector of the current side is determined based on the fitting result, which serves as the plane orientation feature.

[0068] For example, the RANSAC (Random Sample Consensus) algorithm can be used to fit a point cloud dataset to a plane. The RANSAC algorithm iteratively selects a minimum set of points (3 points for a plane) to fit a planar model, counts the number of interior points that fit the model, and finally selects the planar model with the most interior points as the optimal fit. The resulting plane equation is: Ax + By + Cz + D = 0, where D is a constant term, and the normal vector n = [A, B, C]. T By normalizing n, we can obtain the unit normal vector n. unit Thus, n or n unit As a planar orientation feature, it accurately characterizes the spatial orientation of the current side of the container stack 20 in the robot body coordinate system.

[0069] Alternatively, the least squares method can be used for plane fitting. The least squares method solves for the plane equation parameters by minimizing the sum of squared distances from all points to the fitted plane. This method is computationally efficient but sensitive to noise points, and is suitable for scenarios with good point cloud quality.

[0070] Optionally, when there are large pits or protrusions on the current side of the container stack 20, the robustness advantage of RANSAC can be combined to first remove outliers using RANSAC, and then the remaining inliers can be finely fitted using the least squares method to balance robustness and accuracy.

[0071] Optionally, the point cloud dataset can be preprocessed before plane fitting. For example, statistical filtering can be used to remove outliers, or voxel filtering can be used to reduce the point cloud density to improve fitting accuracy and efficiency. The specific preprocessing method depends on the situation and is not limited here.

[0072] Alternatively, the method for determining the planar orientation feature can also be: extracting the depth gradient information of the current side of the container stack 20 from the depth image, and deriving the planar orientation feature of the current side based on the principal direction of the depth gradient information; or, detecting the edge lines of the current side from the RGB image, and determining the planar orientation feature of the current side based on the direction vector of the edge lines. The specific determination method depends on the situation and is not limited here.

[0073] Using the above method, the second pixel region corresponding to the current side is extracted, and its corresponding depth dataset is obtained. Based on the second pixel region and the depth dataset, a point cloud dataset of the current side in the robot's body coordinate system is generated. Subsequently, a plane fitting algorithm is used to process the point cloud dataset, which can effectively remove noise points and fit the optimal plane equation, thereby accurately calculating the normal vector of the current side as a plane orientation feature. This overcomes the influence of uneven side surfaces or the presence of interference points, ensuring the accuracy and reliability of spatial orientation information, and providing a key orientation basis for subsequent planning of the robot's working pose facing the side.

[0074] In some embodiments, the target pose is exemplarily described as including at least the target position and the target orientation; based on the position coordinates and planar orientation features, the target pose for the robot to perform barcode scanning operations on the current side is planned, including but not limited to: First, the position coordinates of the target label to be scanned in the robot's body coordinate system are projected onto a preset plane in the robot's body coordinate system to obtain the coordinates of the projection center. The preset plane is... Figure 1 X in b Y b flat.

[0075] For example, there are n target labels to be scanned on the current side of the cargo stack 20, and the three-dimensional coordinates of the n target labels to be scanned in the robot's body coordinate system are {(X... bi Y bi Z bi Let the n targets be X, i = 1, 2, ..., n. Then the n target tags to be scanned are located in X. b Y b The projection center point (X) on the plane center Y center It can be calculated using the following formula: X center =(1 / n)×∑X bi Y center =(1 / n)×∑Y bi .

[0076] Then, according to (X) center Y center The target position of robot 10 is determined by the planar orientation features of the current side of the stack of containers 20 and (X) and the current side of the stack of containers 20, wherein the target position is related to (X) and ... center Y center The current side distances are all preset working distances. The preset working distances are used to ensure that the robot's scanning equipment can perform scanning operations on the target labels within the effective working distance.

[0077] For example, the current side's planar orientation feature is n. unit =(n unit_x n unit_y n unit_z ), which is in X b Y b The projection components on the plane are (n unit_x n unit_y The target location (X) is then determined. target Y target The method for determining ) can be: [X target Y target ] T =[X center Y center ] T -S2×[n unit_x n unit_y ] T The meaning of this formula is that from (X) center Y center Departing from X b Y b plane along n unitThe robot moves in the reverse direction by S2 to obtain the target position. S2 is a preset working distance used to ensure that after the robot 10 reaches the target position, its robotic arm can extend smoothly and its onboard scanning device can perform the scanning operation on the target label within the effective working distance. This ensures an effective scanning working distance without affecting the resolution of the label image due to excessive distance. The preferred value range for S2 is 0.5m to 0.8m. The specific value of S2 can be adjusted according to the actual application scenario and is not limited here. For example, when the QR code label size is large or the camera's field of view is small, S2 can be appropriately increased to ensure that the label falls completely in the image; when the label size is small or a higher resolution image is required, S2 can be appropriately decreased.

[0078] At the same time, based on the planar orientation characteristics of the current side, the target orientation of robot 10 is determined so that robot 10 faces the current side.

[0079] Wherein, the target orientation is θ target It can be set to n unit The opposite direction can be determined as follows: θ target =atan2(-n unit_y -n unit_x ), where atan2 is the four-quadrant arctangent function, and its return value is the robot 10 orientation relative to the X-axis. b The included angle along the positive axis. This included angle ensures that the center of the robot chassis is directly opposite the current side of the stack of containers 20, meaning that the optical axis of the robot's vision sensor is perpendicular to the current side of the stack of containers 20.

[0080] In practical applications, θ is not mandatory. target The vector is absolutely opposite to the normal vector of the current side, and a certain angular deviation (e.g., ±5°) is allowed, as long as the deviation is within the allowable range to ensure the success rate of scanning.

[0081] Using the method described above, the three-dimensional coordinates of the target label to be scanned in the robot's body coordinate system are projected onto the X-axis. b Y b The plane is used to obtain the projection center point, which can accurately determine the geometric center of the label group; then, combined with the planar orientation features of the current side, the X-axis is used to obtain the center point. b Y b The projection component on the plane can determine the target position that maintains the optimal working distance between the robot 10 and the stack of containers 20; at the same time, based on the orientation characteristics of this plane, the target orientation of the robot 10 facing the current side can be determined, ensuring that the optical axis of the vision sensor is perpendicular to the current side. This ensures both the reasonableness of the scanning distance to obtain a clear image and the precise facing posture of the robot 10, improving the success rate of the scanning operation.

[0082] In real-world scenarios, at the corners of the stack of containers 20, there is often a situation where a container is exposed on both sides simultaneously. If both sides of the container are labeled, when the robot 10 performs the barcode scanning operation on each side, it may repeatedly scan the label on the same container, resulting in data redundancy and reduced efficiency.

[0083] To address this issue, this embodiment of the application further includes, before acquiring the observation data corresponding to the current side view of the cargo container stack 20: First, acquire pre-collected image data of the cargo container stack 20 from multiple sides.

[0084] For example, such as Figure 1 As shown, before acquiring the observation data corresponding to the current side of the cargo stack, the robot 10 is controlled to move sequentially to each observation position according to a preset observation position queue (such as the first observation position 21, the second observation position 22, the third observation position 23, and the fourth observation position 24), and pre-collects image data of more than 20 sides of the cargo stack through its vision sensors. The robot 10 autonomously plans its path and moves according to the coordinates of the current observation position and the coordinates of the next observation position through its navigation system.

[0085] Then, the area of ​​the label to be scanned on the current side is determined based on the pre-acquired image data.

[0086] For example, joint analysis is performed on pre-collected image data from multiple sides to determine the side to which each tag belongs and the repetition status. Based on preset deduplication rules, a corresponding tag area to be scanned is assigned to each side. The joint analysis methods include, but are not limited to: Based on the image acquisition time sequence, the observation position to which each label belongs in the pre-acquired image is determined, and combined with the correspondence between the observation position and the side in the preset observation position queue, the initial correspondence between each label and the side is established.

[0087] Subsequently, feature extraction is performed on each label in the pre-acquired image to obtain the appearance features corresponding to each label. The appearance features include, but are not limited to, the size (pixel area) of the label, the texture features of the label (such as the distribution pattern of black and white blocks in a QR code), the border ratio and shape features of the label, the local binary pattern or histogram of oriented gradients of the label, etc.

[0088] Meanwhile, based on the pixel coordinates of each label in the image and its corresponding depth information, the spatial position coordinates of each label in the global coordinate system are calculated (for the specific calculation method, please refer to the aforementioned embodiment, which will not be repeated here).

[0089] Based on the apparent features corresponding to each label, the similarity between different labels is calculated, and based on the spatial coordinates of each label in the global coordinate system, the spatial distance between different labels in the global coordinate system is calculated. If the spatial distance between two labels in the global coordinate system is less than a preset distance threshold (such as the maximum size of a cargo box), and their corresponding similarity is greater than a preset similarity threshold, then these two labels are determined to be the same label; if only the spatial distance is less than the preset distance threshold and the similarity is less than or equal to the preset similarity threshold, or only the similarity is greater than the preset similarity threshold and the spatial distance is greater than or equal to the preset distance threshold, then they are determined to be different labels.

[0090] Preset deduplication rules include, but are not limited to: For labels that appear only on a single side, directly assign that side as the label to be scanned; For the same label that appears repeatedly on multiple sides, it is assigned to one of the sides according to a preset strategy. The preset strategy can be: assigned to the first observed side, or assigned to the side with the best label imaging quality. The specific strategy depends on the situation and is not limited here.

[0091] Through the above processing, each side obtains a unique label area to be scanned, thereby ensuring that the label on each carton is scanned only once during the entire multi-sided scanning process, avoiding data redundancy.

[0092] Therefore, based on the observation data corresponding to the current side, the target labels to be scanned on the current side are determined, including but not limited to: Based on the observation data corresponding to the current side and the area of ​​labels to be scanned, the target labels to be scanned on the current side are determined. For example, the labels detected in real time in the observation data corresponding to the current side are matched with the labels in the pre-allocated area of ​​labels to be scanned on that side: if the detected label exists in the area of ​​labels to be scanned, it is determined as the target label to be scanned; if the detected label does not exist in the area of ​​labels to be scanned (e.g., it is assigned to another side due to deduplication rules), the label is ignored and no scanning operation is performed.

[0093] Using the above method, pre-collected image data from more than 20 sides of the cargo container stack are jointly analyzed to extract the appearance features and global spatial location of each label. Pre-set distance and similarity thresholds are used to accurately identify the same label appearing across different sides. Furthermore, a unique label area to be scanned is assigned to each side based on a pre-set deduplication rule. In actual operation, real-time observation data is matched with this pre-assigned area, and scanning is performed only on labels belonging to the current side. This effectively avoids redundant scanning caused by repeated label exposure at cargo container corners and improves the efficiency and data cleanliness of multi-side scanning tasks.

[0094] In some embodiments, exemplarily illustrated, before acquiring the observation data corresponding to the cargo container stack below the current side, the method further includes: In response to the robot 10 being in a preset waiting position, monitoring data is acquired in a specified direction in the robot's body coordinate system. The specified direction in the robot's body coordinate system can be the orientation of the vision sensor, such as the front of the head depth camera, or other preset directions, as long as the area where the forklift and cargo stack appear can be effectively collected. The specific direction depends on the situation and is not limited here.

[0095] Specifically, after the robot autonomously moves to the preset waiting position through the navigation system, it activates its onboard vision sensors to continuously collect RGB image streams and depth image streams in the specified direction of the robot's body coordinate system as monitoring data.

[0096] Optionally, to improve processing efficiency, the depth image can also be subjected to foreground filtering, which is consistent with the foreground filtering method described above and will not be repeated here.

[0097] Then, multi-target tracking is performed based on the monitoring data. Multi-target tracking includes tracking at least forklifts and stacks of containers, with forklifts used to move stacks of containers.

[0098] For example, a multi-target tracking algorithm can be used to process the RGB image stream to simultaneously track two targets: a forklift and a stack of containers. The multi-target tracking algorithm can be DeepSORT, ByteTrack, or other deep learning-based multi-target tracking algorithms; the specific algorithm depends on the situation and is not limited here. The multi-target tracking algorithm assigns a unique ID to each detected target and maintains ID consistency across consecutive frames, thereby achieving continuous tracking of the forklift and the stack of containers' trajectories.

[0099] If the tracking results determine that the stack of containers meets the preset static condition and the forklift meets the preset departure condition, then the aforementioned pose planning method is executed.

[0100] The method for determining the preset static condition can be as follows: For the tracked target of the cargo stack 20, calculate the maximum displacement of its bounding box center point within M consecutive frames (preferably M=30, i.e., a 1-second time window; the specific value depends on the situation and is not limited here). Let the coordinates of the center point of the bounding box of the cargo stack 20 in the i-th frame be (x... i y i If the maximum displacement of the cargo stack 20 within the current window [n-M+1, n] is: , i, j∈[n-M+1, n], where n is the current frame number.

[0101] When d maxWhen the value is less than the preset static threshold (preferably 5 pixels, the specific threshold depends on the situation and is not limited here), the cargo container stack is determined to meet the preset static condition.

[0102] Alternatively, the method for determining the preset static condition can also be: calculating the intersection-over-union ratio (IoU) of the container stack 20 in consecutive frames, and determining that the container stack meets the preset static condition when the IoU is greater than a target threshold. The specific determination method depends on the situation and is not limited here.

[0103] The method for determining the preset departure condition can be as follows: for the tracked forklift, determine whether it has left the field of view of the vision sensor. For example, if the bounding box of the forklift completely exceeds the image boundary, or if the forklift is not detected within a continuous R frames (preferably R=15, i.e. 0.5 seconds, the specific value depends on the situation and is not limited here), then the forklift is determined to meet the preset departure condition.

[0104] Optionally, the determination of the preset departure condition can also be combined with the movement trajectory of the forklift. For example, when the forklift moves away from the stack of goods 20 and the distance exceeds the target distance threshold, it is determined that the forklift meets the preset departure condition. The specific determination method can be selected according to actual needs, and this application embodiment does not limit it.

[0105] In some embodiments, when both of the above conditions are met simultaneously, a time window (preferably 2 seconds, the specific value may vary depending on the situation and is not limited here) can be set for verification. Only when both conditions are continuously met within the time window is the aforementioned pose planning method finally confirmed to be executed, thereby effectively avoiding misjudgments caused by factors such as instantaneous occlusion and detection jitter.

[0106] Using the above method, when the robot is in the preset waiting position, continuous monitoring and multi-target tracking can accurately determine whether the cargo stack 20 meets the preset static condition and whether the forklift meets the preset departure condition. Thus, the pose planning method is automatically triggered at the appropriate time, which basically avoids accidental triggering of barcode scanning when the forklift is still working, and improves the reliability and automation level of the operation process.

[0107] In some embodiments, exemplarily illustrated, after planning the target pose of the robot performing a barcode scanning operation on the current side based on position coordinates and planar orientation features, the method further includes: In response to the completion signal of the current side's barcode scanning operation, the current observation position index is obtained, and the next observation position is determined based on the current observation position index and the preset observation position queue; wherein, the preset observation position queue includes multiple observation positions, each observation position is located at a different preset position facing the container stack (e.g., Figure 1 (As shown).

[0108] For example, maintain a pre-defined observation bit queue Queue = [first observation bit 21, second observation bit 22, third observation bit 23, fourth observation bit 24] and an observation bit index idx. Initially, idx = 1, indicating that the scanning operation will proceed to the first observation bit 21. When the scanning operation for the first observation bit 21 is completed, check if idx is less than 4. If so, set idx = idx + 1, and set the second observation bit 22 as the next observation bit.

[0109] Then, control robot 10 to move to the next observation position and execute the aforementioned pose planning method.

[0110] For example, Robot 10 uses its navigation system to autonomously plan a global path, starting from its current location and ending at the coordinates of the next observation position. During the movement, it uses visual sensors for real-time positioning to ensure that it accurately reaches the next observation position.

[0111] Finally, once all observation positions have completed the scanning operation, the robot is controlled to return to the preset waiting position. For example, when idx=4 and the scanning operation of observation position 24 is completed, it means that all observation positions have generated a scanning operation completion signal. At this time, robot 10 returns to the preset waiting position to wait for the next stack of containers to arrive.

[0112] Optionally, the preset observation position queue is not limited to the fixed order described above. It can also prioritize moving to the side with a higher probability of label appearance based on the historical distribution information of the side labels on the cargo container stack 20; or, select the observation position closest to the preset waiting position as the starting point to shorten the total movement path. The specific strategy can be set according to actual needs, and this application embodiment does not limit it.

[0113] Optionally, during the process of returning to the preset waiting position, robot 10 can simultaneously upload the result data of this multi-faceted observation and barcode scanning operation for the background system to record and analyze.

[0114] In passing Figure 2 After the method shown plans the target pose of robot 10, how to control the robotic arm and barcode scanner of robot 10 to accurately and efficiently complete the scanning operation of the target label becomes an urgent problem to be solved. To this end, this application embodiment also provides a robot control model training method, such as... Figure 3 As shown, the method mainly includes the following steps: S301, acquire robot operation data in the target work scenario; S302, Freeze the original parameters of the robot control model and add incremental parameters to the specified network layer of the robot control model; S303 optimizes incremental parameters based on operational data to obtain a trained robot control model.

[0115] In this embodiment of the application, the target operation scenario is as follows: Figure 1 The diagram shows a scenario where robot 10 performs a barcode scanning operation on a stack of containers 20. Figure 2 After planning and controlling the robot 10 to face the current side of the cargo stack 20, the robot 10 performs barcode scanning operations in the target operation scenario through remote control. The operation data includes at least the multimodal data and sample action sequences of the robot 10.

[0116] The dimensions of the sample motion sequence are related to the number of joints in robot 10 and the barcode scanner control signals. For example, if robot 10 has 14 arm joints and 4 waist joints, the sample motion sequence is a 19-dimensional vector: the first 14 dimensions correspond to the angle increments of the 14 arm joints, arranged in the order of 7 joints in the left arm and 7 joints in the right arm; dimensions 15-18 correspond to the angle increments of the 4 waist joints; and the 19th dimension corresponds to the barcode scanning trigger signal. When the value of this dimension is greater than a target threshold (e.g., 0.5, the specific threshold depends on the situation and is not limited here), it indicates that the barcode scanner is triggered to perform a scanning operation. The aforementioned dimension settings are only illustrative and can be adjusted according to the configuration of robot 10 and operational requirements in actual applications. No limitation is made here.

[0117] For example, the operator controls the robot 10 to perform a barcode scanning operation on the current side of the stack of containers 20 through a remote control device. The system simultaneously records the joint status data of the robot 10, the observed images, the working status data of the barcode scanner, and the operation signals.

[0118] For example, the operator controls the movement of the robot 10's two arm joints and the timing of the barcode scanner trigger via a game controller, enabling the robot 10 to scan the target barcode label on the current side of the stack of containers 20. During operation, the system synchronously records the robot 10's joint status data (14 arm joint angles and 4 waist joint angles), barcode scanner's working status data (whether scanning is triggered and the corresponding trigger time), RGB images collected by the robot 10's vision sensor (preferably with a resolution of 224×224 pixels, the specific resolution can be adjusted according to actual needs, and is not limited here), and the operator's operation signals (including joint control commands and barcode trigger commands).

[0119] The visual sensors include, but are not limited to, head cameras and dual-wrist cameras, as well as depth cameras, binocular cameras, structured light cameras, etc., deployed in other locations on the robot 10. The specific sensor type can be selected according to actual needs and is not limited here.

[0120] Alternatively, the remote control device can also be a master / slave joystick, wearable device, touch screen / tablet, augmented reality / virtual reality device, etc. The specific device selection can be determined according to the actual application scenario and operating habits, and is not limited here.

[0121] Alternatively, the operation data can also be acquired by the operator directly dragging the robotic arm of robot 10 for teaching, and robot 10 recording the joint trajectory and the time of barcode scanning trigger as operation data. The specific acquisition method can be selected according to actual needs and resource conditions, and is not limited here.

[0122] Then, the original parameters of the robot control model are frozen, and incremental parameters are added to specified network layers of the robot control model. The robot control model uses a large visual language model as its base model, which has been pre-trained on massive amounts of data and possesses powerful image and language understanding capabilities.

[0123] For example, based on the base model, a parameter-efficient fine-tuning technique is used to freeze all the original parameters of the model and add trainable incremental parameters to the specified network layers.

[0124] In this embodiment, a low-rank decomposition matrix is ​​added as an incremental parameter to the attention layer and MLP (Multi-Layer Perceptron) layer of the specified robot control model using the LoRA (Low-Rank Adaptation) technique.

[0125] In this setup, the LoRA parameter can be empirically set to Rank = 16 to balance the model's expressive power and parameter count; the Alpha parameter (α) = 32, used to scale the update magnitude of the LoRA branch, typically set to twice the Rank value to control the intensity of new knowledge injection during fine-tuning; and Dropout = 0.1, which randomly discards some neurons with a 10% probability to prevent overfitting and improve generalization ability. These parameter settings are merely illustrative; in practical applications, they can be adjusted based on factors such as task complexity and data size, and are not limited here.

[0126] Finally, based on the operational data, the incremental parameters are optimized to obtain a trained robot control model.

[0127] For example, based on operational data, a flow matching algorithm is used to optimize incremental parameters, resulting in a trained robot control model. The fine-tuning of hyperparameters in this model includes, but is not limited to: Optimizer: Adaptive Moment Estimation with Weight Decay Optimizer, which decouples weight decay from gradient update based on adaptive moment estimation, and can more effectively suppress overfitting; Learning rate: 5e-5 (an empirical value that ensures the model learns effectively on new tasks without destroying the original knowledge of the base model due to excessive step size), and adopts linear warm-up (e.g., 100 steps, which means that the learning rate is linearly increased from 0 to the initial value in the first 100 training steps to avoid gradient instability in the early stage of training) and cosine annealing (after the warm-up, the learning rate is decayed from the initial value to close to 0 according to the cosine function, so that the model can converge more finely in the later stage of training) learning rate scheduling strategy. Batch size: 32, refers to the number of samples used each time parameters are updated; Training epochs: 50, which refers to the number of times the training sample set is completely traversed; Gradient clipping: The maximum gradient norm is 1.0, which is used to limit the norm of the gradient vector to no more than 1.0, to prevent the model parameters from oscillating violently due to gradient explosion during training, and to ensure the stability of the training process; The weight decay is 0.01, which is used to suppress excessively large model parameters by adding a penalty term of the sum of squared parameters to the loss function, effectively preventing overfitting and improving the model's generalization ability.

[0128] The hyperparameters set above are empirical values ​​based on the task characteristics and model size of this embodiment. In practical applications, they can be adjusted according to factors such as task complexity, dataset size, and computing resources, and are not limited here. For example, the learning rate can be appropriately reduced for larger-scale models, and the number of training epochs can be reduced for simpler tasks.

[0129] Alternatively, the optimization method for incremental parameters can also employ a diffusion model: learning the mapping from random noise to real actions by predicting noise, such as a denoised diffusion probability model or a denoised diffusion implicit model; or a generative adversarial network: learning the distribution of real actions through adversarial training between the generator and the discriminator. The aforementioned algorithms are all known generative models or action learning algorithms in the art, and their specific implementation details can be found in relevant technologies, which will not be described further here. Those skilled in the art can choose appropriate optimization algorithms according to actual needs, as long as they can optimize the incremental parameters based on the operational data; the specific selection is not limited here.

[0130] By freezing the original parameters and introducing incremental parameters only at specified layers, the computational resource consumption and overfitting risk of model training are reduced. Subsequently, based on the robot's operation data in the target work scenario, the incremental parameters are optimized to obtain a trained robot control model. This model can accurately understand multimodal inputs and directly output complete action sequences, enabling the robot to adaptively complete fine operation tasks in complex work scenarios and improving the success rate of the operation.

[0131] In some embodiments, exemplarily described, acquiring robot operation data in a target work scenario includes, but is not limited to: In response to the operation signals that control the robot to perform tasks in the target work scenario, the system collects the corresponding status data and observation images of the robot.

[0132] The status data includes at least the robot's joint status data and the working status data of the actuators mounted on the robot.

[0133] In this embodiment, the robot is a dual-arm robot, and its joint state data includes 14 joint angles of the two arms and 4 joint angles of the waist; the actuator is a barcode scanner.

[0134] Optionally, the actuator can also be other types of end effectors such as welding torches or spray guns, and its working status data can include welding torch ignition status, spray gun on / off status, etc.

[0135] The observed image includes at least a current side view of the container stack 20, which includes a target label to be scanned. The label can be a QR code, barcode, or other form of identifiable code.

[0136] To ensure the diversity and coverage of the training samples, the above data collection process covers a variety of operational scenario variations. These operational scenario variations include at least one of the following: changes in container type, label layout, container stacking height, lighting conditions, and container position deviation.

[0137] For example, changes in cargo box type: collecting data on cargo boxes of different sizes, colors, and materials, such as cardboard boxes, plastic boxes, and wooden boxes.

[0138] Label layout variations: Collect label data with different quantities, positions, and arrangements, such as single labels, multiple labels, vertical arrangement, horizontal arrangement, etc.

[0139] Cargo container stacking height variation: Collect data on different stacking heights from single layer to multiple layers (e.g., 1-5 layers).

[0140] Lighting conditions change: Collect light change data in different areas (such as windowsills and corners) of the workshop at different times (such as daytime, evening, and nighttime).

[0141] Cargo container position deviation: Random position deviation data within the range of 10-40cm were collected to simulate the uncertainty of cargo container placement in actual operations.

[0142] This embodiment collected 150 complete sets of barcode scanning data. Each set of data includes the complete scanning process of all sides of the cargo container stack, containing approximately 800-1400 frames of continuous data. The data collection quantity, data length, and frame range are for illustrative purposes only and can be adjusted according to actual scenario requirements, model size, computing resources, etc., without limitation here. For example, for more complex operation scenarios or models requiring higher precision, the amount of data collected can be appropriately increased; for simpler scenarios or situations with limited resources, the amount of data can be reduced accordingly.

[0143] Simultaneously, the operation signal is converted into a sample action sequence.

[0144] For example, the continuous control signals sent by the operator via a remote control device are divided into action sequence segments of 50 steps each to obtain sample action sequences. The shape of the sample action sequence is (50, 19), where 50 represents the actions in the next 50 time steps, and 19 represents the 19 dimensions corresponding to the sample action sequence.

[0145] Then, the observed images are preprocessed, and the preprocessed observed images, status data, and task instructions corresponding to the target operation scenario are used as multimodal data. The preprocessing includes at least one of image resizing, normalization, and data augmentation.

[0146] For example, the observed image is adjusted to a resolution of 224×224, and then normalized (e.g., the pixel values ​​are scaled to the range of [0, 1] or [-1, 1]). Then, data augmentation is performed on the normalized image to obtain a preprocessed image. Data augmentation includes, but is not limited to, random flipping, random rotation (the rotation angle can range from -10° to 10°, and the specific range can be adjusted according to actual needs; no limit is imposed here), and color dithering, to expand the training samples.

[0147] The task instruction is a natural language description corresponding to the target operation scenario, which can vary according to the specific task. In this embodiment, the task instruction is a fixed task description text, such as "You are a logistics robot, performing barcode scanning and warehouse entry operations".

[0148] By using the above methods, robot state data and observation images covering various scene changes are collected. Based on the preprocessing of the observation images and the segmentation of the operator's continuous control signals into sample action sequences, the diversity and comprehensiveness of the operation data are ensured, thereby improving the model's generalization ability and robustness in complex dynamic environments.

[0149] In some embodiments, the method for determining the specified network layer is illustrated by way of example, including but not limited to: The sensitivity values ​​of each network layer in the robot control model to the target operation scenario are calculated, and network layers with sensitivity values ​​greater than a preset sensitivity threshold are designated as specified network layers. The specific value of the preset sensitivity threshold depends on the situation and is not limited here.

[0150] For example, the sensitivity value of each network layer can be calculated using the Fisher information matrix. The Fisher information matrix measures the influence of model parameters on the output, with its diagonal elements reflecting the importance of the corresponding parameters. For each network layer, the Fisher information values ​​of all its parameters can be summed or averaged to obtain the sensitivity value of that layer. A higher sensitivity value indicates a greater impact of the network layer on the task performance in the target scenario. Therefore, network layers with sensitivity values ​​greater than a preset sensitivity threshold can be designated as specified network layers for incremental parameter addition, maximizing the fine-tuning benefits with a limited number of parameters.

[0151] Optionally, the sensitivity value can also be calculated based on the gradient magnitude. For example, the average gradient norm of each network layer's parameters on the training data can be calculated as the sensitivity value of that network layer. The larger the average gradient norm, the greater the update magnitude of the layer's parameters during training, and the greater its impact on the task. The specific calculation method can be selected according to actual needs and is not limited here; or, Based on preset computational resource constraints, select the network layer that satisfies the computational resource constraints from each network layer of the robot control model as the designated network layer.

[0152] For example, computational resource constraints may include at least one of memory limitations, training time limitations, and parameter limitations. When memory resources are limited, a network layer with a smaller number of parameters (such as a low-dimensional projection layer in a feedforward network) can be selected as the designated network layer; when training time is limited, a network layer closer to the output layer can be selected for fine-tuning, because these layers usually converge faster and can achieve better fine-tuning results in a shorter time.

[0153] Optionally, a combination of various resource constraints can be considered when making a selection. For example, under the dual constraints of GPU memory and training time, network layers with fewer parameters but greater impact on the task can be prioritized. The specific selection strategy can be determined based on actual hardware conditions and task requirements, and is not limited here.

[0154] Alternatively, the two methods described above can be used in combination. For example, candidate network layers can be selected first through sensitivity evaluation, and then a specific network layer can be selected from the candidate network layers based on resource constraints, thereby satisfying resource limitations while ensuring fine-tuning effects.

[0155] The method for determining the specified network layer described above is only an example. In actual implementation, the method can be flexibly selected according to factors such as the actual application scenario, model architecture, and resource conditions. No limitation is made here.

[0156] The above methods achieve two main goals: First, by calculating the sensitivity of each network layer to the target task scenario, key network layers with high sensitivity are prioritized for incremental parameter addition, maximizing fine-tuning benefits with a limited number of parameters. Second, by selecting network layers that meet resource constraints, efficient fine-tuning is achieved in resource-constrained scenarios. These two approaches can also be combined: candidate network layers are selected based on sensitivity, and then specific network layers are determined by combining sensitivity with resource constraints. This ensures both the model's adaptability to the target task scenario and addresses resource limitations in actual deployment, achieving a balance between fine-tuning efficiency and resource consumption.

[0157] In some embodiments, the method for obtaining the robot control model is illustrated by way of example, including but not limited to: First, obtain a pre-training dataset, which includes at least demonstration data collected from multiple robot platforms performing various operational tasks.

[0158] This embodiment uses the Open X-Embodiment (OXE, an open-source cross-platform robot dataset) as the pre-training dataset. The OXE dataset contains large-scale demonstration data from multiple robot platforms (such as Fetch, PR2, Panda, UR, etc.) and various operational tasks (such as grasping, placing, pushing and pulling, opening doors, folding, etc.). Considering the limitations of computing resources, this embodiment uses 50% of the samples in the dataset for pre-training. The specific number of samples can be flexibly adjusted according to the actual computing resources and task complexity, and is not limited here.

[0159] Optionally, the pre-training dataset may also include other public datasets, such as RoboNet (a robot pushing dataset that includes pushing task data from various robots) and RH20T (a rich dataset of robot operation demonstrations). The specific selection and combination can be based on the actual task requirements and model size, and there are no restrictions here.

[0160] Then, the initial model is pre-trained based on the pre-trained dataset to obtain the robot control model. The initial model is a large-scale vision-language multimodal model, such as the Qianwen 3-visual language model, a large-scale language and vision assistant, or a flamingo visual language model. The specific model can be selected according to the task requirements, model size, and computing resources, and is not limited here.

[0161] Pre-training hyperparameter settings include, but are not limited to: Optimizer: Adaptive moment estimation with weight decay optimizer; Learning rate: 1e-4 (empirical value), with cosine annealing learning rate scheduling strategy; Batch size: 128; Training epochs: 10 epochs; Gradient clipping: Maximum gradient norm is 1.0.

[0162] The aforementioned pre-training hyperparameters can be adjusted based on the model size and dataset size, and are not limited here. For example, for larger-scale models, the learning rate can be appropriately reduced; for smaller datasets, the number of training epochs can be reduced to prevent overfitting.

[0163] During pre-training, model parameters are updated based on a flow matching algorithm for each training sample. Through large-scale pre-training, the model learns to understand task intent from visual and linguistic input and generate corresponding action sequences, laying the foundation for subsequent fine-tuning for specific tasks.

[0164] Using the above method, the initial model is pre-trained with a large-scale cross-platform robot dataset, enabling the model to learn a general vision-language-action mapping capability in diverse operational tasks. It can understand the task intent from multimodal inputs and generate corresponding action sequences, laying a solid knowledge foundation for efficient parameter fine-tuning for specific operational scenarios.

[0165] In some embodiments, it is illustrated by way of example that incremental parameters are optimized based on operational data to obtain a trained robot control model, including but not limited to: First, generate a noisy action sequence corresponding to the time step parameters.

[0166] For example, the shape of the sample action sequence (actions) is (50, 19). Then, a noise action sequence corresponding to the time step parameter (t) is randomly generated, where noise ~ N(0, I), with a shape of (50, 19), where I is the identity matrix, and N(0, I) represents a multidimensional normal distribution with a mean of 0 and a covariance matrix of I. Simultaneously, t ~ Beta(1.5, 1), where Beta(1.5, 1) is a Beta distribution. Non-uniform sampling using the Beta distribution is employed to increase the sampling density at intermediate time steps, allowing the model to focus more on the subtle changes during action transitions. Furthermore, t is normalized: t = t × 0.999 + 0.001, making the range of t (0.001, 1.0) to avoid boundary values.

[0167] Alternatively, t can also be sampled from a uniform distribution, an exponential distribution, or other similar distributions. The specific sampling method can be adjusted according to the characteristics of the training task, and is not limited here.

[0168] Then, based on noise, t, and actions, the target velocity field u migrating from noise to actions is calculated. t .

[0169] For example, constructing interpolated samples x based on noise, t, and actions. t =t×noise+(1-t)×actions, where x t This represents a linear interpolation from actions to noise. And for x... t By taking the derivative, we obtain the target velocity field u. t =noise-actions, where u t This represents the velocity of the interpolated sample migrating towards the noise direction, i.e., the target velocity field on the manifold.

[0170] Then, based on the multimodal data and t, the predicted velocity field u for the migration from noise to actions is inferred. tp .

[0171] For example, based on multimodal data and t, multimodal features are extracted by the encoder, combined with the current noisy action sequence and the parameter embedding of the current time step, and output u through model forward propagation. tp .

[0172] Next, according to u tp with u t Calculate the loss value. The loss can be calculated as follows: loss = mean_square(u tp -u t ), where mean_square is the mean square error function.

[0173] Alternatively, the loss function can also be root mean square error, L1 loss, or a combination of the above loss functions. The specific loss function can be determined based on the actual training effect, and is not limited here.

[0174] Finally, the incremental parameters are optimized based on the loss until a preset termination condition is met, resulting in a well-trained robot control model. By minimizing the loss, the model learns to predict the correct velocity field at any time step, enabling it to generate reasonable action sequences through iterative denoising during inference.

[0175] During training, the performance of the current model can be evaluated on the validation set every preset number of epochs (e.g., 5 epochs, depending on the situation, not limited here). The model with the smallest validation loss is saved as the final model. The validation set can be obtained from the collected operational data. For example, 80% of the collected data can be used as the training set and 20% as the validation set. The specific ratio can be adjusted according to actual needs, not limited here.

[0176] Preset termination conditions include, but are not limited to: reaching the maximum number of training epochs, the validation loss no longer decreasing after multiple consecutive epochs, and the validation loss being less than a preset loss threshold. These conditions can be adjusted according to specific training results and resource requirements, and are not limited here. When the preset termination conditions are met, training stops, and the trained robot control model is obtained.

[0177] By using the above method, the incremental parameters are optimized based on the flow matching algorithm, thereby transforming the motion generation task into a velocity field fitting problem. This enables the model to learn a continuous migration path from noise distribution to real motion distribution, improving the stability and accuracy of model training. Furthermore, by monitoring the validation set, overfitting can be effectively prevented, ultimately resulting in a robot control model that can accurately generate high-quality motion sequences in the target work scenario.

[0178] In some embodiments, it is exemplarily illustrated that a predicted velocity field for migration from a noisy action sequence to a sample action sequence is inferred based on multimodal data and time step parameters, including but not limited to: First, the observed images and task instructions in the multimodal data are encoded to obtain visual language features.

[0179] For example, the visual encoder of the visual-language multimodal large model encodes the observed image to obtain visual features. The language encoder of the visual-language multimodal large model encodes the task instructions to obtain linguistic features. The visual and linguistic features are concatenated along the sequence dimension to obtain visual-language features. These visual-language features are then input into the Large Language Model (LLM) module of the visual-language multimodal large model to generate and cache key-value pairs. These cached key-value pairs can be reused in subsequent iterative denoising processes, avoiding redundant computation and improving inference speed.

[0180] Then, the state data in the multimodal data is encoded to obtain state features, and the time step parameter (t) and noise action sequence are jointly encoded to obtain temporal action features.

[0181] For example, state data can be encoded using MLP in a vision-language multimodal large model to obtain state features.

[0182] Encode t using sine-cosine positional coding (posemb_sincos) to obtain the time code t. e =posemb_sincos(t, d, period_min=4e-3, period_max=4.0).

[0183] Where d represents the embedding dimension, which is the same as the hidden layer dimension of the model; posemb_sincos is defined as: generating a frequency sequence for d. Then t e =[sin(t×freq0), cos(t×freq0), sin(t×freq1), cos(t×freq1),...]. period min period max The `freq` parameter represents the minimum and maximum periods of the encoding, used to control the frequency range of the encoding. i denoted as the i-th frequency component, the frequency increases exponentially as i increases; sin / cos represents mapping t to periodic functions of different frequencies through sine and cosine functions, enabling the model to distinguish different time steps.

[0184] Alternatively, learnable position coding or other time-step coding methods can be used, depending on the actual needs, and are not limited here.

[0185] Motion features are obtained by linearly projecting noise using an MLP. (The last part, "t," appears to be an unrelated fragment and is left untranslated.) e Extending to the same sequence length as the action features, we obtain the time feature as repeat(t). e“bd→bsd”, s=50), represents the change of t e Repeat 50 times along the sequence dimension, where b is the batch size, s is the sequence length (50), and d is the embedding dimension.

[0186] The temporal features and action features are then concatenated to obtain preliminary temporal action features. These preliminary temporal action features are then sequentially input into the first MLP, the activation function, and the second MLP for enhancement processing to obtain the final temporal action features.

[0187] Next, the state features and temporal action features are concatenated to obtain the fused features.

[0188] Finally, based on visual language features, the fused features are inferred to obtain the predicted velocity field.

[0189] For example, cached key-value pairs (containing visual and language features) and fused features are input into the LLM of a large visual-language multimodal model, and mapped to u via an MLP. t .

[0190] The above method employs a visual-language multimodal large model to encode and cache key-value pairs of observed images and task instructions, enabling the reuse of visual-language features in subsequent iterations of denoising and improving inference efficiency. Simultaneously, state data is encoded to obtain state features, allowing for feature reuse. Furthermore, sine-cosine position encoding is used to finely map time-step parameters, fusing time features with noise-action features and enhancing them through MLP to obtain temporal action features. This allows the model to accurately perceive the current denoising stage, and finally, in conjunction with visual-language features, inputs them into the LLM of the visual-language multimodal large model to complete velocity field prediction, improving the accuracy of velocity field prediction and inference efficiency.

[0191] pass Figure 3 After the robot control model is trained using the method shown, it can be deployed to the controller of robot 10 for actual barcode scanning operations. Figure 4 The following is a flowchart illustrating a robot control method provided in an embodiment of this application. The process mainly includes the following steps: S401, acquire the robot's current multimodal data in the target work scenario; S402, input the current multimodal data into the trained robot control model, and output the target action sequence that matches the current multimodal data; S403 controls the robot to perform the target task based on the target action sequence.

[0192] The current multimodal data includes at least the current state data, the current observed image, and the current task instructions. The current state data includes at least the current joint state data of the robot and the current working state data of the actuators mounted on the robot.

[0193] In this embodiment, the target operation scenario is as follows: Figure 1 The scene shown depicts robot 10 performing a barcode scanning operation on a stack of cargo containers 20.

[0194] The current multimodal data is input into the trained robot control model, which starts with random noise and generates the target action sequence through progressive denoising.

[0195] For example, the model first generates random noise that follows a standard normal distribution as an initial noise action sequence. Then, based on the current multimodal data, it uses a flow matching algorithm to iteratively perform T-step denoising (T can be set according to real-time requirements, such as 10 steps, 50 steps, etc., which is not limited here) to gradually transform the noise action sequence into a smooth and coherent action sequence.

[0196] In each denoising step, the model infers the corresponding current predicted velocity field based on the current time step parameters, the current noisy action sequence, and the current multimodal data. It then updates the current noisy action sequence according to the current predicted velocity field and simultaneously updates the current time step parameters according to a preset step size. The updated noisy action sequence and time step parameters are then used as input for the next denoising process. After T iterations of denoising, the final noisy action sequence is the target action sequence output by the model.

[0197] The target action sequence includes a first subsequence and a second subsequence. The first subsequence is used to control the joint movement of the robot, and the second subsequence is used to control the actuators on the robot to perform corresponding operations at corresponding times. The dimension of the first subsequence is related to the number of robot joints, and the dimension of the second subsequence is related to the actuator trigger signal.

[0198] In this embodiment, the first subsequence is an 18-dimensional vector: the first 14 dimensions correspond to the angle increments of 14 bi-arm joints, arranged in the order of 7 joints in the left arm and 7 joints in the right arm; the 15th to 18th dimensions correspond to the angle increments of 4 lumbar joints; the second subsequence is the 19th dimension vector, which corresponds to the scanning trigger signal. When the value of this dimension is greater than the target threshold (e.g., 0.5), it indicates that the scanner is triggered to perform the scanning operation.

[0199] Optionally, after outputting the target action sequence, the target action sequence can be post-processed, such as smoothing the joint angle increment and performing threshold judgment on the scan trigger signal, to improve the smoothness and reliability of the control command.

[0200] Finally, the robot's control system parses the target action sequence into specific control commands.

[0201] For example, the incremental joint angles of the first 18 dimensions are converted into motor control signals for each joint, driving the arms and waist to perform actions according to the planned motion trajectory; when the value corresponding to the 19th dimension scanning trigger signal is greater than the target threshold, the scanner is controlled to perform the scanning operation at the corresponding moment. By executing a 50-step action sequence, the robot's end-effector scanner can traverse the target labels to be scanned on the current side, and complete the identification and recording of the target labels to be scanned.

[0202] During execution, the robot can provide real-time feedback on the current joint status and barcode scanner status, forming a closed-loop control to ensure the accuracy of motion execution and the reliability of the operation.

[0203] Using the above method, the current multimodal data is input into the trained robot control model, and the target action sequence is generated by gradually denoising from random noise, thus achieving end-to-end control from perception to action. Furthermore, through the model's capabilities in multimodal understanding and action generation, it can adapt to complex scene changes in cargo stacking and barcode scanning operations, generating smooth and accurate action sequences, thereby improving the robot's success rate and operational efficiency in logistics scenarios.

[0204] In some embodiments, to ensure the orderly execution of multi-label scanning operations on the current side, before starting the scanning operation on the current side, each target label to be scanned on the current side is first assigned a unique number (e.g., label 1, label 2... label N) and a corresponding scanning status (not scanned / scanned). Based on the number of identified labels, the system constructs the current task instruction: if there is only one target label to be scanned on the current side, the current task instruction includes "use a single arm to perform the scanning operation"; if there are two or more target labels to be scanned, the current task instruction includes "use dual arms to work collaboratively and scan each target label to be scanned sequentially".

[0205] When scanning each target label, the system determines the position coordinates (three-dimensional coordinates of the label in the robot's body coordinate system) of the current target label according to a preset scanning order (e.g., from high to low, from left to right), and incorporates these coordinates into the current task command. Subsequently, the system acquires the robot's current multimodal data (including the currently observed image, current state data, and the aforementioned current task command), inputs it into the trained robot control model, and the model outputs a target action sequence that matches the current position of the target label. Based on this target action sequence, the robot is controlled to perform the scanning operation.

[0206] Using the above method, a structured multi-label scanning workflow is constructed. Before the operation, label numbering, scanning status initialization, and dual-arm activation strategy configuration based on the number of labels are uniformly completed, and single-arm / dual-arm collaborative modes are integrated into the task instructions. During the execution phase, the current label's position coordinates are dynamically determined according to a preset order, and the task instructions are updated in real time, driving the model to output a target action sequence that accurately matches the label position. This not only ensures the logical consistency of dual-arm collaborative operations in multi-label scenarios but also enables the model to accurately perceive the operation intent through flexible construction of task instructions, effectively improving the robot's execution efficiency and success rate for complex scanning tasks.

[0207] In some embodiments, exemplarily described, after controlling the robot to perform a target task according to the target action sequence, the method further includes: If no execution result information is received from the actuator within the preset time, the execution is determined to have failed, and the current failure count is obtained. If the current failure count is less than the preset number (e.g., 3 times, the specific number depends on the situation and is not limited here), the robot control method described above is re-executed; if the current failure count is equal to the preset number, an exception message is generated.

[0208] For example, after each scanning operation, if the scanner receives a scanning result for the current target label, the scanning status of the current target label is updated to "scanned," and the robot control method described above continues to be executed for the next target label according to the preset scanning order. If no scanning result is received, the scanning is considered a failure, and the failure count for the current target label is incremented by 1 (initially 0). If the current failure count is less than the preset count, the robot control method is re-executed for the current target label. If the current failure count equals the preset count, the current workflow is paused, and a corresponding error message is generated to prompt the operator for manual intervention.

[0209] Optionally, when the number of scanned labels reaches the total number of labels, it indicates that the scanning task of all target labels to be scanned has been completed on the current side. Then, the robot is controlled to move to the next observation surface according to the preset observation position queue, and the above steps are repeated until the scanning task of all sides of the cargo container stack 20 is completed.

[0210] In actual operation, robots encounter various environmental changes and unforeseen circumstances. How to enable the robot control model to continuously learn and optimize from these real-world experiences, thereby constantly improving the success rate, is a further issue addressed in this application. Figure 5 A flowchart illustrating a robot control model optimization method is shown, which mainly includes the following steps: S501, acquire the robot's first observation data at the inference trigger moment, and the second observation data at the moment the target action sequence is completed; S502, the value of the first observation data and the second observation data are estimated by a pre-trained value estimation network to obtain the first value estimate and the second value estimate; S503, if the difference between the first value estimate and the second value estimate meets the preset condition, then the operation data generated by the robot before the completion time of the target action sequence is determined as the target operation data; S504, based on the target operation data, optimizes the robot control model to obtain an updated robot control model.

[0211] In the process of outputting the target action sequence based on the robot control model and controlling the robot to perform the barcode scanning operation according to the target action sequence, the first observation data of the robot at the inference trigger time (t0) and the second observation data at the time when the target action sequence is completed are obtained.

[0212] Where t0 is the time when the robot control model receives input data in the target operation scenario and infers the target action sequence of the robot.

[0213] The first observation data includes at least the first visual data and the first motion state data, which are used for subsequent extraction of visual features and motion state features, respectively.

[0214] The first visual data consists of a sequence of images acquired by the robot's vision sensors, used for subsequent visual feature extraction. The vision sensors include at least a head camera, a left wrist camera, and a right wrist camera; the specific number of viewpoints can be determined based on the robot's actual configuration and is not limited here. The number of frames in the first image sequence can be set according to actual needs (e.g., three frames: t0-2, t0-1, and t) to capture temporal information, and is not limited here.

[0215] The first motion state data is the robot's first state history, used for subsequent motion state feature extraction. The first state history can contain state data for multiple time steps (for example, 30 time steps from t0-29 to t0). The state data for each time step includes at least robot joint state data and barcode scanner working state data (whether scanning is triggered). The specific number of time steps can be set according to actual needs and is not limited here.

[0216] Similarly, the second observation data includes at least the second visual data and the second motion state data, and its content structure is the same as that of the first observation data, which will not be described again here.

[0217] The robot control model starts reasoning from t0 and outputs a 50-step target action sequence after the reasoning time. After the robot executes this 50-step action sequence, it reaches the execution completion time, and obtains the second observation data at this time, which also includes the second image sequence and the second state history.

[0218] Then, the first observation data is input into the pre-trained value estimation network, and the value estimation network outputs the normalized estimated remaining time V to the target job completion before , as the first value estimate, for example, 0.35, indicating that the estimated remaining target job completion time is 350 frames. Similarly, the second observation data is input into the value estimation network to obtain V after , as the first value estimate, for example, 0.28, indicating that the estimated remaining target job completion time is 280 frames.

[0219] Next, calculate the difference between V before and V after as the value change ΔV. For example, ΔV = V before - V after = 0.35 - 0.28 = 0.07, indicating that after executing the 50-step action sequence, the estimated remaining target job completion time by the value estimation network is reduced from 350 frames to 280 frames, that is, the estimated execution progress of the target job is advanced by 70 frames.

[0220] Judge whether ΔV is greater than or equal to the preset change threshold. The preset change threshold can be determined according to the difference between the theoretical execution progress and the actual execution progress, which is not limited here. For example, theoretically, executing 50 steps of actions should advance the execution progress of the target job by 50 frames; but considering the execution delay of the robotic arm, actually executing 50 steps corresponds to an advancement effect of about 80 frames. Then the preset change threshold threshold can be set as: threshold = (80 + 20) / 1000 = 0.1 (corresponding to 100 frames), that is, it is required that the advancement effect of this reasoning reaches more than 100 frames to be judged as better than the average level.

[0221] If ΔV < threshold, it is determined that the operation data generated by the robot before the execution completion time of this target action sequence is not the target operation data. If ΔV ≥ threshold, the operation data generated by the robot before the execution completion time of this target action sequence is determined as the target operation data and added to the incremental data set.

[0222] Optionally, the difference satisfying the preset condition can also be: the absolute value of ΔV is greater than or equal to threshold, and the specific condition can be set according to the actual application scenario, which is not limited here.

[0223] Optionally, before adding the target operation data to the incremental dataset, the target operation data can be manually sampled to ensure data quality. For example, a portion of the target operation data can be randomly selected for manual review to check the rationality of the action sequence, the accuracy of label recognition, etc., to ensure the reliability of the incremental data.

[0224] Once the incremental dataset accumulates to a preset amount, the robot control model is optimized based on this dataset to obtain an updated model. For example, when the incremental dataset accumulates 50 sets of target operation data, the original training data is mixed with the incremental dataset to form an optimized training set. This optimized training set is then used to incrementally optimize the robot control model, updating its parameters and resulting in an updated model. The incremental dataset is then cleared, and the next round of high-quality data collection and filtering begins. The preset data amount can be set according to actual needs and is not limited here. Through this incremental learning mechanism, the model can continuously learn optimal operation strategies during actual deployment, achieving autonomous evolution and performance improvement.

[0225] Using the method described above, based on a value estimation network, the remaining completion time is evaluated before and after the robot executes the action sequence. By comparing the results based on the threshold of ΔV, precise quantitative filtering of robot operation data is achieved. This method automatically identifies target operation data with better-than-average propulsion performance and accumulates them as incremental datasets for subsequent hybrid training and optimization of the robot control model. This not only effectively solves the problem of distinguishing between high-quality and low-quality operation data in traditional methods but also enables the model to continuously learn from high-quality operation data, improving the accuracy and efficiency of the control strategy. This enhances the robot's adaptability and success rate in complex barcode scanning scenarios.

[0226] In some embodiments, the method for obtaining the value estimation network includes, but is not limited to: First, acquire the robot's historical operation data in the target task scenario. The historical operation data includes at least the robot's historical observation data and target task completion indication information.

[0227] In response to the operation signal that controls the robot to perform the target operation in the target operation scenario, the corresponding historical observation data of the robot is collected. The historical observation data includes at least the sample image sequence and sample state history corresponding to each historical moment from the moment the operation signal is received to the moment the operation signal is completed.

[0228] For data points near the start of the sequence, if the sample image sequence has less than 3 frames or the sample state history has less than 30 time steps, zero padding (filling the missing parts with zeros) or edge copying (copying the state data of the first frame forward to fill it) can be used for processing. The specific processing method can be determined according to the actual application scenario, and is not limited here.

[0229] The target task completion indication information can be represented as the frame number of the target task completion. The annotation method can be: manually annotating the frame number of the target task completion in each set of data, that is, the moment when the target tag to be scanned was successfully scanned. For example, if there is only one target tag to be scanned on the current side, the completion time of the single-arm scanning operation is marked; if there are two or more target tags to be scanned on the current side, the completion time of the dual-arm collaborative operation is marked.

[0230] Optionally, the task completion indicator information can also be expressed as the percentage of the target task completed or other quantitative indicators that can reflect the progress of the target task completion, depending on the specific circumstances, and is not limited here.

[0231] Then, using historical observation data as input and target task completion information as supervision signal, the initial value estimation network is trained to obtain the value estimation network. The value estimation network is used to estimate the target task completion progress at the corresponding time based on the robot's observation data at different times.

[0232] For example, an initial value estimation network is constructed, including an observation encoder, a multimodal fusion module, and a value prediction head. For each set of historical observation data, each historical time point (t) is used to... i The sample image sequence and sample state history are used as input, with t i The corresponding target task completion indicator information (i.e., the target task completion frame number completion_frame) is used as a supervision signal, which is passed through the observation encoder, the multimodal fusion module, and the value prediction head respectively, and outputs t. i The corresponding predicted value estimate is value_pred i .

[0233] For t i The corresponding remaining number of frames is remaining_frames i The calculation method is as follows: remaining_frames i =completion_frame-t i Considering the maximum data length is 1400 frames, the sampling rate is 30Hz, and the target task duration is approximately 50 seconds, a normalization factor α (e.g., α=1000, the specific value of α can be adjusted according to the actual task duration, and is not limited here) can be used to normalize the remaining_frames. i After normalization, the target value estimate, value_target, is obtained. i =remaining_frames i / α. Ensure value_targeti Within a preset range, such as [0, 1.4], it helps to stabilize the training process and avoid gradient instability or convergence difficulties due to excessively large numerical ranges.

[0234] Based on value_pred i and value_target i The difference is used to calculate the value loss, which can be obtained through the mean squared error loss function. By minimizing the value loss, the parameters of the initial value estimation network are optimized to obtain a pre-trained value estimation network. The hyperparameter settings during the pre-training process of the value estimation network are as follows: Optimizer: Adaptive Moment Estimator, a gradient-based first-order optimization algorithm that adaptively adjusts the learning rate of each parameter; Learning rate: 1e-4 (empirical value), used to control the step size of parameter updates. A smaller learning rate helps stabilize training. Learning rate scheduling strategy: ReduceLROnPlateau (learning rate decay strategy), which is used to multiply the learning rate by a decay factor (such as 0.1) when the validation loss stops decreasing, so that the model can converge more finely; Batch size: 64, the number of samples used each time parameters are updated; Training epochs: 100 epochs, which is the number of times the training dataset is fully traversed.

[0235] The hyperparameter settings mentioned above are for illustrative purposes only. Specific values ​​can be adjusted according to the actual application scenario, and are not limited here.

[0236] During training, the parameters of the observation encoder are kept frozen, and only the parameters of the multimodal fusion module and the value prediction head are trained. This reduces the number of training parameters, prevents overfitting, accelerates convergence, and ensures the consistency of feature extraction.

[0237] Using the above method, a value estimation network is trained with historical observation data as input and manually labeled target task completion frame numbers as supervision signals. This network can accurately predict the normalized remaining task completion time based on image sequences and state history at any given time, providing a reliable value assessment benchmark for subsequent robot control models.

[0238] In some embodiments, it is exemplarily illustrated that a pre-trained value estimation network is used to estimate the value of the first observation data and the second observation data respectively, to obtain a first value estimate and a second value estimate, including but not limited to: First, visual features are extracted from the visual data in the target observation data to obtain visual features; the target observation data is either the first observation data or the second observation data.

[0239] Taking the first observation data as an example, RGB image sequences from the head camera at multiple historical moments are acquired, with each frame having standard RGB three channels. Each frame in the sequence is input into the image encoder in the observation encoder. The image encoder can employ a convolutional neural network architecture (such as the ResNet series) and is initialized using pre-trained weights from the ImageNet dataset to improve feature extraction capabilities. Each frame outputs a feature map after passing through the image encoder, which is then subjected to global average pooling to obtain a feature vector, and finally linearly projected to obtain fixed-dimensional image frame features. This process is repeated for each frame to obtain the image feature sequence from the head perspective.

[0240] Similarly, the image sequences from the left and right wrist cameras are processed in the same way to obtain left wrist image feature sequences and right wrist image feature sequences. Based on the image feature sequences from the three perspectives, visual features are generated.

[0241] Alternatively, the image encoder can also employ other convolutional neural network architectures, such as ResNet34, ResNet50, and other ResNet variants, or attention-based image coding structures like the Visual Transformer. The specific network architecture used can be selected based on the actual application scenario, computing resources, and performance requirements; no limitations are imposed here.

[0242] Simultaneously, the motion state data in the target observation data is extracted by using the joint encoder in the observation encoder to obtain motion state features.

[0243] For example, the state history of 30 time steps from t0-29 to t0 is obtained. Each time step contains 18-dimensional joint states and 1-dimensional scanner working states, totaling 19 dimensions, with an input shape of (30, 19). This state history is input into a joint encoder, which can use a sequence modeling network (such as a two-layer long short-term memory network, where the first and second layers of the long short-term memory network have a hidden layer dimension of 128) to model the temporal information. The hidden state of the last time step of the sequence is taken as the 128-dimensional feature representation of the entire sequence, and then linearly projected to obtain fixed-dimensional joint features (e.g., 128-dimensional), which serve as the motion state features.

[0244] Alternatively, sequence modeling networks can also use other temporal modeling architectures instead of long short-term memory networks, such as gated recurrent units (GRUs) and Transformer networks. The specific network architecture used can be selected based on factors such as the actual application scenario, the length of temporal dependencies, and computational resources, and is not limited here.

[0245] Then, the visual features and motion state features are fused across modalities to obtain the target fusion features.

[0246] For example, visual features and motion state features are input into a multimodal fusion module. The multimodal fusion module unifies the dimensions of the visual features and motion state features, and then performs cross-modal fusion on the unified visual features and motion state features to obtain the target fused features.

[0247] Finally, the value estimation network is used to perform value mapping on the fused features to obtain the estimated value corresponding to the target observation data.

[0248] For example, the fused features are input into the value prediction head, which can employ an MLP structure containing multiple linear layers and non-linear activation functions, and incorporate a Dropout mechanism to prevent overfitting. The final linear layer maps the fused features to a one-dimensional output, representing a normalized estimate of the remaining completion time, which serves as the value estimate. The processing flow is as follows: The value prediction head employs a three-layer MLP structure for value mapping. First, the fused features are input into the first linear layer, which maps the input features to 256-dimensional hidden features. Then, the ReLU (Revised Linear Unit) activation function is applied to these hidden features, setting negative values ​​to zero and keeping positive values ​​unchanged to introduce non-linear transformation capability. Next, Dropout processing is applied to the activated features, randomly discarding a portion of neurons at a preset ratio to prevent overfitting, resulting in the output of the first hidden layer. This preset ratio can be set according to the actual application scenario, for example, 0.3; the specific value is not limited here. Then, the output of the first hidden layer is input into the second linear layer, which maps the 256-dimensional features to 128-dimensional hidden features. Similarly, the ReLU activation function and Dropout processing are applied sequentially to these features, resulting in the output of the second hidden layer. Finally, the output of the second hidden layer is input into the third linear layer, which maps the 128-dimensional features to a 1-dimensional output, serving as the value estimate. This value estimate represents the remaining time until the target task is completed, in thousands of frames.

[0249] Using the above method, visual features are extracted from multi-view image sequences from the head camera, left wrist camera, and right wrist camera using an image encoder. At the same time, motion state features are obtained by temporal modeling of the state history using a joint encoder. Subsequently, the visual features and motion state features are dimensionally unified and fused across modes using a multimodal fusion module. Finally, the value prediction head maps the result to a normalized estimate of the remaining completion time. This method can accurately quantify the progress of the target task before and after the robot performs the action, providing a reliable value assessment basis for the automatic screening of subsequent target operation data and ensuring the objectivity and accuracy of data screening.

[0250] In some embodiments, it is exemplarily illustrated that visual features are extracted from visual data in target observation data to obtain visual features, including but not limited to: First, acquire at least one image sequence from the robot's perspective in the target observation data.

[0251] For example, in a barcode scanning scenario, image sequences from three perspectives are obtained from target observation data: the most recent 3 frames from the head camera (Sequence I). head The most recent 3 frames from the left wrist camera (I) left The most recent 3 frames from the right wrist camera (I) right The image sequence shape for each viewpoint is (sequence length, number of image channels, image height, image width), where the sequence length is 3, the image is RGB three-channel, and the resolution is 224×224, i.e., the shape is (3, 3, 224, 224).

[0252] Then, feature extraction is performed on the image frames in the image sequence from each viewpoint to obtain the image frame features from each viewpoint.

[0253] For example, for I head Its three frames I head0 I head1 I head2 Each image frame is input into a shared-weighted image encoder. The backbone network extracts a feature map of shape (512, 7, 7), which is then subjected to global average pooling to obtain a 512-dimensional feature vector. This vector is then linearly projected to obtain a 512-dimensional head image frame feature. This process is repeated for the three frames to obtain a head image frame feature sequence of shape (3, 512). Similarly, I... left and I right The image frame features of the left wrist and the right wrist were obtained, both with a shape of (3, 512).

[0254] Next, the image frame features from each viewpoint are processed temporally to obtain the temporal enhancement features for each viewpoint.

[0255] For example, for head image frame features, left wrist image frame features, and right wrist image frame features, a temporal self-attention mechanism is applied to obtain temporal enhanced features for the head, left wrist, and right wrist, respectively. The calculation process of the temporal self-attention mechanism is as follows: First, through three independent linear transformation layers, the image frame feature sequence of each viewpoint is mapped into a query vector, a key vector, and a value vector, respectively. Then, the dot product of the query vector and the key vector is calculated, and the result is scaled by dividing by the square root of the attention head dimension, and then normalized using the softmax function to obtain the attention weights. Finally, the attention weights are multiplied by the value vector to obtain the temporal enhanced features. The attention head dimension can be set according to actual needs and is not limited here; for example, it can be set to 64.

[0256] Finally, the temporal enhancement features from each perspective are fused across perspectives to obtain visual features.

[0257] For example, bidirectional cross-attention is applied to the temporal enhancement features from the three perspectives, fusing the temporal enhancement features from the left and right wrist perspectives into the head perspective. The fusion process is as follows: Using the temporal enhancement features of the head as the query and the temporal enhancement features of the left wrist as the key and value, cross attention is calculated to obtain the fused features of the head and the left side; using the temporal enhancement features of the head as the query and the temporal enhancement features of the right (left) wrist as the key and value, cross attention is calculated to obtain the fused features of the head and the right (left) wrist.

[0258] The fusion features of the head and left side, the fusion features of the head and right side, the temporal enhancement features of the head, the temporal enhancement features of the left wrist, and the temporal enhancement features of the right wrist are concatenated along the feature dimension to obtain a multi-view feature sequence.

[0259] Finally, attention pooling is used to aggregate multi-view feature sequences to obtain visual features. Specifically, attention pooling uses a learnable query vector to perform a weighted summation of the sequence dimensions, thereby aggregating multi-view feature sequences into visual features for a single frame.

[0260] Alternatively, cross-view fusion can also employ other fusion strategies, such as multi-view feature averaging, weighted summation, or global attention fusion based on Transformer. The specific choice can be made according to actual needs and is not limited here.

[0261] Using the above method, image sequences from three perspectives—head, left wrist, and right wrist—are acquired. A shared-weight image encoder is used to extract frame features from each perspective. Then, a temporal self-attention mechanism is applied to the feature sequences from each perspective to capture inter-frame temporal dependencies, resulting in temporally enhanced features. Next, bidirectional cross-attention is used to fuse the features from the left and right wrist perspectives into the head perspective, and these features are then concatenated with the original features from each perspective to form a rich multi-view feature sequence. Finally, these features are aggregated into visual features through attention pooling. This improves the representational power of visual features, providing more accurate and robust visual input for subsequent value estimation networks, and enhancing the accuracy and stability of value estimation.

[0262] In some embodiments, it is exemplarily illustrated that visual features and motion state features are fused across modally to obtain target fused features, including but not limited to: First, the visual features and motion state features are transformed in dimension to obtain visual projection features and motion state projection features with unified dimension.

[0263] For example, visual features and motion state features are input into a linear projection layer to obtain visual projection features and motion state projection features of a preset dimension. The preset dimension is preferably 512 dimensions, and the specific value can be set according to the actual application scenario. It is not limited here.

[0264] Then, cross-modal interaction is performed based on visual projection features and motion state projection features to obtain visual cross-modal features and motion state cross-modal features.

[0265] For example, using visual projection features as the query and motion state projection features as the key and value, cross-attention is calculated to obtain visual cross-modal features. Similarly, using motion state projection features as the query and visual projection features as the key and value, cross-attention is calculated to obtain motion state cross-modal features.

[0266] Simultaneously, the fusion weights are determined based on visual projection features and motion state projection features.

[0267] For example, visual projection features and motion projection features are concatenated and then input into a linear transformation layer (Linear_gate), followed by a sigmoid activation function to obtain gating coefficients as fusion weights, with values ​​ranging from (0, 1). These gating coefficients represent the weight proportion of the visual modality in the fusion process.

[0268] Finally, the visual cross-modal features and motion state cross-modal features are weighted and fused based on the fusion weights to obtain the target fusion features.

[0269] For example, the target fusion feature = fusion weight × visual cross-modal feature + (1 - fusion weight) × motion state cross-modal feature. This target fusion feature contains both visual and motion state information, and the fusion weight can be dynamically adjusted according to the input, achieving adaptive cross-modal fusion.

[0270] Alternatively, cross-modal interaction can also be achieved using other interaction mechanisms, such as a bidirectional long short-term memory network: visual projection features and motion state projection features are concatenated and input into the bidirectional long short-term memory network, and information interaction between the two modalities is achieved through bidirectional temporal modeling, with the hidden state at the last time step taken as the target fusion feature. Or a multimodal Transformer: visual projection features and motion state projection features are used as two independent input sequences, and multi-head self-attention and cross-attention calculations are performed through the encoder layer of the Transformer to achieve deep modal interaction. The specific interaction mechanism used can be selected according to the actual application scenario, and is not limited here.

[0271] Alternatively, the fusion weights can be determined based on other combinations of visual features and motion state features, such as dot product, bilinear pooling, etc. The specific weight determination method can be selected according to the actual application scenario, and is not limited here.

[0272] The above method projects visual features and motion state features onto a unified dimension, then uses a cross-attention mechanism for bidirectional interaction, integrating visual features with motion state information and vice versa, resulting in enhanced visual and motion state cross-modal features. Simultaneously, based on the concatenation of the two projected features and a gated linear layer, dynamic fusion weights are generated through sigmoid activation. These weights are adjusted in real-time according to the modal confidence of the input data, and finally, the target fused feature is obtained through weighted fusion. This dynamic gating mechanism achieves adaptive cross-modal fusion of visual and motion state features, enabling the model to dynamically balance the contributions of the two modalities based on the relative reliability of visual and motion state information in the current observation. This generates a more discriminative and robust fused representation, providing a more accurate input basis for subsequent value estimation.

[0273] In some embodiments, it is exemplarily illustrated that the robot control model is optimized based on target operational data to obtain an updated robot control model, including but not limited to: The original parameters of the robot control model are frozen, and incremental parameters are added to the specified network layer of the robot control model. Based on the target operation data, the incremental parameters are optimized to obtain the updated robot control model.

[0274] For example, once the incremental dataset has accumulated 50 complete sets of operation data, these target operation data are used to optimize the incremental parameters. During training, historical operation data (150 sets) are mixed with the incremental dataset to form an optimized training set. Based on this optimized training set, the flow matching algorithm is used to optimize the incremental parameters, resulting in an updated robot control model.

[0275] By using the above method, only a small number of incremental parameters that are highly relevant to the target operation scenario are updated, enabling the model to efficiently learn high-quality behavioral patterns in the target operation data. This allows for precise adaptation and rapid iteration of the control strategy, thereby continuously improving the robot's execution efficiency and success rate in barcode scanning scenarios while maintaining the generalization ability of the basic model.

[0276] In some embodiments, exemplarily illustrated, after obtaining the updated robot control model, the method further includes: Obtain the performance metrics of the updated robot control model on the test dataset.

[0277] For example, the performance of the updated robot control model can be evaluated on an independent test set containing multiple sets of operational data that were not used in training, such as 20 sets. The specific number can be set according to actual needs and is not limited here. Performance metrics include, but are not limited to: task completion rate (the percentage of successful barcode scanning), barcode scanning success rate (the percentage of successful barcode scanning), and average time (the average time required from start to finish).

[0278] If the performance metrics do not meet the preset retention conditions, the robot control model will be rolled back to the previous version, the incremental dataset will be cleared to prevent error accumulation, and the number of rollbacks will be accumulated.

[0279] The preset retention conditions can be: the updated robot control model is no less than the previous robot control model in all performance indicators, or each indicator is no less than 95% of the previous model. Specific retention conditions can be set according to actual application needs and are not limited here. If the preset retention conditions are not met, the robot control model will automatically roll back to the previous model, the incremental dataset will be cleared, and the rollback counter will be incremented by 1 (the initial value of the rollback counter is 0).

[0280] Stop optimizing the robot control model when the number of rollbacks equals the preset rollback threshold.

[0281] For example, the preset rollback threshold can be set to 3 times, and the specific number can be set according to the actual application scenario, which is not limited here. When the number of rollbacks is less than the preset rollback threshold, the system continues to run and collect target operation data for the next round of incremental optimization. If the performance of the updated robot control model does not meet the preset retention condition (i.e., 3 consecutive rollbacks) after 3 consecutive incremental optimizations, the stop condition is triggered, the incremental optimization process is paused, and the model performance is prevented from continuously degrading and invalid data is continuously accumulated.

[0282] At this point, the system can send an alarm to technical personnel for manual intervention to investigate issues such as data quality or environmental changes. If the data quality is subsequently restored or the environmental issues are resolved, the incremental optimization process can be manually restarted.

[0283] Using the above methods, the updated robot control model undergoes multi-dimensional performance evaluation. When the model performance fails to meet preset retention conditions, it automatically rolls back to the previous version, clears the incremental dataset, and accumulates the number of rollbacks. When the number of consecutive rollbacks reaches a preset rollback threshold, the incremental optimization process is automatically paused and an alarm is triggered, preventing the continuous accumulation of model degradation due to abnormal data quality or environmental changes. This effectively ensures the stability and reliability of the robot control model in actual deployment, ensuring that the model always maintains a controllable performance benchmark. It provides a safety net for incremental learning, preventing negative effects from being introduced during the optimization process. Simultaneously, the rollback counting and alarm mechanism provides clear anomaly indicators for maintenance personnel, facilitating timely manual intervention and problem investigation.

[0284] Optionally, if an abnormal situation occurs during the above operation (such as robot 10 losing its positioning, failure to collect observation data, barcode scanner malfunction, or cargo box falling off), robot 10 can pause the current process and report the abnormality, waiting for manual intervention or executing a preset abnormality handling strategy. This application embodiment does not limit the abnormality handling method.

[0285] The detection method for cargo box falling can be as follows: the visual sensor mounted on the robot 10 collects the point cloud data corresponding to the cargo box stack 20 in real time, and detects whether there is a local point cloud missing in the area of ​​the cargo box stack 20 based on the point cloud data.

[0286] For example, the real-time point cloud data collected in the current frame is compared with the pre-collected standard point cloud model of the cargo container stack 20. If a point cloud hole is detected in an area where there should be a cargo container (i.e., the point cloud density in that area is less than a preset density threshold or is completely missing), it is determined that a cargo container has fallen.

[0287] Alternatively, the comparison method can also be to calculate the point cloud density difference and / or height difference between real-time point cloud data and a standard point cloud model in the region. When the point cloud density difference is greater than a first threshold and / or the height difference is greater than a second threshold, it is determined to be a point cloud hole. The specific detection algorithm can be determined according to the actual application scenario, and is not limited here.

[0288] The above method enables the robot 10 to traverse all observation positions in an orderly manner according to the preset observation position queue after completing the scanning operation on the current side, thereby achieving comprehensive scanning of multiple sides of the cargo container stack 20. After the operation is completed, it automatically returns to the waiting position and has the ability to handle abnormalities, which improves the automation level and operational reliability of the system.

[0289] Based on the above methods, this application also provides a software system architecture to support pose planning, model reasoning, and motion control of the robot 10. This software system adopts a layered architecture, with each layer communicating through standard interfaces, including but not limited to: Perception layer: Running on the edge computing power board of the robot 10 itself, it is responsible for tasks such as camera data acquisition, image preprocessing, target detection and segmentation, and provides structured observation information for subsequent pose planning and model inference.

[0290] Decision layer: Running on a graphics processing unit (GPU), it includes a visual-language-action model inference module and a value estimation network evaluation module. The visual-language-action model inference module is used for execution... Figure 3 The trained robot control model generates a sequence of target actions based on the observation data output from the perception layer; the value estimation network evaluation module is used for execution. Figure 5 The value estimation network shown evaluates the job progress effect before and after the robot performs actions.

[0291] Control layer: responsible for path planning, motion control, and motor drive, converting the target action sequence output by the decision layer into specific motor control instructions to drive robot 10 to complete the barcode scanning operation.

[0292] Communication Layer: Utilizing Robot Operating System 2 (ROS2) or a custom communication protocol, data interaction between the perception, decision-making, and control layers is achieved. Message passing employs an asynchronous, non-blocking mode to ensure real-time communication between modules and meet the low-latency requirements of barcode scanning operations.

[0293] The layered architecture described above decouples the functions of perception, decision-making, and control, allowing each module to be developed, tested, and optimized independently. At the same time, standard interfaces ensure the scalability and maintainability of the system.

[0294] During the execution of the method, this application embodiment also integrates multiple layers of security mechanisms to ensure the safety of the operation process: (1) Collision detection: The motion space of the robotic arm is monitored in real time by a depth camera. The perception layer continuously outputs the environmental point cloud around the robotic arm. When the control layer detects that the distance between the robotic arm and the obstacle is less than the safety threshold, it immediately interrupts the execution of the current action sequence and sends a collision warning to the decision layer.

[0295] (2) Joint limits: Each joint is equipped with both software and hardware limits. The software limit is implemented in the control layer through kinematic constraints. When the joint angle in the target action sequence exceeds the preset range, the control layer refuses to execute the instruction. The hardware limit is implemented through physical limit switches at the joint ends, providing a last line of protection when the software limit fails.

[0296] (3) Force control protection: A six-dimensional force sensor is integrated at the end of the robot's robotic arm to monitor the end contact force in real time during the barcode scanning operation. When an abnormal external force is detected (such as an accidental collision or barcode scanner jamming), the control layer immediately triggers a protective stop to prevent damage to the equipment or goods.

[0297] (4) Emergency Stop: The robot body is equipped with a physical emergency stop button, which can cut off the power supply at any time. This button operates independently of the software system and can be directly triggered by the operator in any abnormal situation to ensure the highest priority safety response.

[0298] (5) Model output supervision: After the decision layer outputs the target action sequence, the control layer checks the rationality of the action sequence. The checks include: whether the joint angle exceeds the limit, whether the end speed exceeds the safety threshold, and whether the timing of the code scanning is reasonable. For abnormal instructions that fail the check, the control layer refuses to execute them and reports back to the decision layer, which then triggers a replanning or abnormal handling process.

[0299] It should be understood that, although Figure 2-5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0300] In some embodiments, a robot is provided, comprising: an observation module, a first determination module, a second determination module, and a planning module, wherein: The observation module is used to acquire observation data corresponding to the current side of the cargo stack; where the current side is the side of the robot facing the cargo stack. The first determining module is used to determine the target barcode label to be scanned on the current side and the position coordinates of the target barcode label in the robot body coordinate system based on the observation data. The second determining module is used to determine the planar orientation features corresponding to the current side based on the observation data; The planning module is used to plan the target pose of the robot when it is facing the current side for scanning operations, based on the position coordinates and planar orientation characteristics.

[0301] For specific limitations regarding the robot, please refer to the limitations on the pose planning method mentioned above, which will not be repeated here. Each module in the robot described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0302] In some embodiments, a computer device is provided, which may be a server. In one example, it includes a processor, memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores pose planning data. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a pose planning method. The display screen of the computer device may be a liquid crystal display screen or an e-ink display screen. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad provided on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0303] Those skilled in the art will understand that the structures shown in the above examples are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0304] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire the observation data corresponding to the current side of the cargo stack; where the current side is the side of the robot facing the cargo stack. Based on the observation data, determine the target label to be scanned on the current side, and the position coordinates of the target label to be scanned in the robot's body coordinate system; Based on the observation data, determine the plane orientation characteristics corresponding to the current side. Based on the position coordinates and planar orientation characteristics, the target pose of the robot is planned so that it is currently performing a barcode scanning operation on the side.

[0305] In some embodiments, when the processor executes a computer program, it further performs the following steps: Based on the observation data, the first pixel region corresponding to the target label to be scanned is extracted; Calculate the center pixel coordinates of the first pixel region and obtain the depth value corresponding to the center pixel coordinates from the observation data; Based on preset coordinate transformation parameters, the center point pixel coordinates and depth values ​​are transformed to the robot body coordinate system to obtain the position coordinates of the target label to be scanned in the robot body coordinate system.

[0306] In some embodiments, when the processor executes a computer program, it further performs the following steps: Based on the observation data, the second pixel region corresponding to the current side is extracted, and the depth dataset corresponding to the second pixel region is obtained from the observation data. Based on the second pixel region and depth dataset, generate a point cloud dataset of the current side in the robot body coordinate system; Perform plane fitting on the point cloud dataset, and determine the normal vector of the current side based on the fitting result, as the plane orientation feature.

[0307] In some embodiments, when the processor executes a computer program, it further performs the following steps: The position coordinates are projected onto a preset plane in the robot's body coordinate system to obtain the projection center point; The target position is determined based on the projection center point and the plane orientation characteristics. The distance between the target position and the projection center point is the preset working distance. The preset working distance is used to ensure that the robot's scanning device can perform scanning operations on the target label within the effective working distance. Based on the plane orientation characteristics, determine the target orientation so that the robot faces the current side.

[0308] In some embodiments, when the processor executes a computer program, it further performs the following steps: Acquire pre-collected image data corresponding to the cargo containers stacked on multiple sides; The area of ​​the label to be scanned on the current side is determined based on the pre-acquired image data; Based on the observation data and the area of ​​the label to be scanned, the target label to be scanned on the current side is determined.

[0309] In some embodiments, when the processor executes a computer program, it further performs the following steps: In response to the robot being in a preset waiting position, monitoring data is acquired in a specified direction in the robot's body coordinate system; Multi-target tracking is performed based on monitoring data. Multi-target tracking includes at least tracking forklifts and stacks of containers, with forklifts used to move stacks of containers. If the tracking results determine that the stack of containers meets the preset static condition and the forklift meets the preset departure condition, then the pose planning method is executed.

[0310] In some embodiments, when the processor executes a computer program, it further performs the following steps: In response to the completion signal of the current side scanning operation, obtain the index of the current observation position; The next observation position is determined based on the current observation position index and the preset observation position queue; the preset observation position queue includes multiple observation positions, each located at a different preset position facing the container stack; Control the robot to move to the next observation position and execute the pose planning method; Once the scanning operation has been completed at all observation positions, the robot is controlled to return to the preset waiting position.

[0311] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: Acquire the observation data corresponding to the current side of the cargo stack; where the current side is the side of the robot facing the cargo stack. Based on the observation data, determine the target label to be scanned on the current side, and the position coordinates of the target label to be scanned in the robot's body coordinate system; Based on the observation data, determine the plane orientation characteristics corresponding to the current side. Based on the position coordinates and planar orientation characteristics, the target pose of the robot is planned so that it is currently performing a barcode scanning operation on the side.

[0312] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: Based on the observation data, the first pixel region corresponding to the target label to be scanned is extracted; Calculate the center pixel coordinates of the first pixel region and obtain the depth value corresponding to the center pixel coordinates from the observation data; Based on preset coordinate transformation parameters, the center point pixel coordinates and depth values ​​are transformed to the robot body coordinate system to obtain the position coordinates of the target label to be scanned in the robot body coordinate system.

[0313] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: Based on the observation data, the second pixel region corresponding to the current side is extracted, and the depth dataset corresponding to the second pixel region is obtained from the observation data. Based on the second pixel region and depth dataset, generate a point cloud dataset of the current side in the robot body coordinate system; Perform plane fitting on the point cloud dataset, and determine the normal vector of the current side based on the fitting result, as the plane orientation feature.

[0314] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: The position coordinates are projected onto a preset plane in the robot's body coordinate system to obtain the projection center point; The target position is determined based on the projection center point and the plane orientation characteristics. The distance between the target position and the projection center point is the preset working distance. The preset working distance is used to ensure that the robot's scanning device can perform scanning operations on the target label within the effective working distance. Based on the plane orientation characteristics, determine the target orientation so that the robot faces the current side.

[0315] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: Acquire pre-collected image data corresponding to the cargo containers stacked on multiple sides; The area of ​​the label to be scanned on the current side is determined based on the pre-acquired image data; Based on the observation data and the area of ​​the label to be scanned, the target label to be scanned on the current side is determined.

[0316] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: In response to the robot being in a preset waiting position, monitoring data is acquired in a specified direction in the robot's body coordinate system; Multi-target tracking is performed based on monitoring data. Multi-target tracking includes at least tracking forklifts and stacks of containers, with forklifts used to move stacks of containers. If the tracking results determine that the stack of containers meets the preset static condition and the forklift meets the preset departure condition, then the pose planning method is executed.

[0317] In some embodiments, when a computer program is executed by a processor, it further performs the following steps: In response to the completion signal of the current side scanning operation, obtain the index of the current observation position; The next observation position is determined based on the current observation position index and the preset observation position queue; the preset observation position queue includes multiple observation positions, each located at a different preset position facing the container stack; Control the robot to move to the next observation position and execute the pose planning method; Once the scanning operation has been completed at all observation positions, the robot is controlled to return to the preset waiting position.

[0318] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0319] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0320] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A pose planning method applied to robots, characterized in that, The method includes: Acquire observation data corresponding to the current side of the cargo stack; wherein, the current side is the side of the robot facing the cargo stack; Based on the observation data, the target barcode label to be scanned on the current side is determined, as well as the position coordinates of the target barcode label in the robot body coordinate system; Based on the observation data, the planar orientation feature corresponding to the current side is determined; Based on the position coordinates and the plane orientation features, the target pose of the robot is planned so that it is performing a barcode scanning operation on the current side.

2. The method according to claim 1, characterized in that, The step of determining the target barcode label to be scanned on the current side based on the observation data, and the position coordinates of the target barcode label in the robot's body coordinate system, includes: Based on the observation data, the first pixel region corresponding to the target label to be scanned is extracted; Calculate the center pixel coordinates of the first pixel region, and obtain the depth value corresponding to the center pixel coordinates from the observation data; Based on preset coordinate transformation parameters, the center point pixel coordinates and the depth value are transformed to the robot body coordinate system to obtain the position coordinates of the target label to be scanned in the robot body coordinate system.

3. The method according to claim 1, characterized in that, The step of determining the planar orientation feature corresponding to the current side based on the observation data includes: Based on the observation data, the second pixel region corresponding to the current side is extracted, and the depth dataset corresponding to the second pixel region is obtained from the observation data; Based on the second pixel region and the depth dataset, a point cloud dataset of the current side surface in the robot body coordinate system is generated; The point cloud dataset is fitted with a plane, and the normal vector of the current side is determined based on the fitting result, which is used as the plane orientation feature.

4. The method according to any one of claims 1-3, characterized in that, The target pose includes at least the target position and the target orientation; the step of planning the target pose for the robot to perform barcode scanning operation on the current side based on the position coordinates and the planar orientation features includes: The position coordinates are projected onto a preset plane in the robot's body coordinate system to obtain the projection center point; The target position is determined based on the projection center point and the plane orientation feature, wherein the distance between the target position and the projection center point is a preset working distance, and the preset working distance is used to ensure that the robot's scanning device can perform scanning operations on the target label to be scanned within the effective working distance; Based on the planar orientation characteristics, the target orientation is determined so that the robot is facing the current side.

5. The method according to any one of claims 1-3, characterized in that, Before obtaining the observation data corresponding to the current side view of the cargo container stack, the method further includes: Acquire pre-collected image data corresponding to the stack of cargo containers from multiple sides; The area of ​​the label to be scanned on the current side is determined based on the pre-acquired image data; The step of determining the target tag to be scanned on the current side based on the observation data includes: Based on the observation data and the area of ​​the label to be scanned, the target label to be scanned on the current side is determined.

6. The method according to any one of claims 1-3, characterized in that, Before obtaining the observation data corresponding to the current side view of the cargo container stack, the method further includes: In response to the robot being positioned at a preset waiting position, monitoring data is acquired in a specified direction within the robot's body coordinate system; Multi-target tracking is performed based on the monitoring data, and the multi-target tracking includes at least tracking a forklift and a stack of containers, wherein the forklift is used to move the stack of containers. If, based on the tracking results, it is determined that the stack of containers meets the preset static condition and the forklift meets the preset departure condition, then the pose planning method is executed.

7. The method according to any one of claims 1-3, characterized in that, After planning the target pose for the robot to perform barcode scanning operation on the current side based on the position coordinates and the planar orientation features, the method further includes: In response to the scanning operation completion signal of the current side, obtain the current observation bit index; The next observation position is determined based on the current observation position index and the preset observation position queue; wherein, the preset observation position queue includes multiple observation positions, each of which is located at a different preset position facing the cargo container stack; Control the robot to move to the next observation position and execute the pose planning method; Once the scanning operation has been completed at all observation positions, the robot is controlled to return to the preset waiting position.

8. A robot, characterized in that, The robot includes: An observation module is used to acquire observation data corresponding to the current side of the cargo stack; wherein, the current side is the side of the robot facing the cargo stack; The first determining module is used to determine, based on the observation data, the target barcode label to be scanned on the current side and the position coordinates of the target barcode label to be scanned in the robot body coordinate system; The second determining module is used to determine the planar orientation feature corresponding to the current side based on the observation data; The planning module is used to plan the target pose of the robot performing the barcode scanning operation on the current side based on the position coordinates and the plane orientation features.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images