Robot scanning methods, devices, electronic equipment, storage media and software products

By matching the target object image with preset item images, and combining the pose information of the candidate item images for weighted fusion and coordinate transformation, the problems of invalid movement and invisible barcodes during robot scanning are solved, and efficient barcode scanning operation is achieved.

CN123103772APending Publication Date: 2026-11-17SHANGHAI QIONCHE INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611594791.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, robots have difficulty adapting to the differences in the shape, size, and barcode placement of different products when scanning, resulting in invalid movements that increase scanning time, and the inability to determine the effective scanning pose when the barcode is not visible.

Method used

By matching the target object image with preset item images, and combining the pose information of the candidate item images for weighted fusion and coordinate transformation, the three-dimensional pose of the target object and the end-effector grasping pose are determined. Combined with the calibration pose of the barcode scanning device and the robot's motion, the target barcode pose is calculated, reducing invalid actions and improving the scanning success rate.

Benefits of technology

It shortens the scanning time, improves the scanning success rate, and reduces sorting or outbound errors caused by misidentification of appearance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103772A_ABST
    Figure CN123103772A_ABST
Patent Text Reader

Abstract

This disclosure provides a robot barcode scanning method, apparatus, electronic device, storage medium, and program product. The method includes: acquiring an image of a target object and selecting matching candidate object images; weighted fusing and transforming the three-dimensional poses of the candidate object images to the robot's working coordinate system to obtain the three-dimensional pose of the target object; determining the end-effector grasping pose based on the geometric features of the target object and controlling the robot's end-effector to grasp the target object; determining the relative pose of the target object using the three-dimensional pose and the end-effector grasping pose, and transforming the first relative pose of the target barcode in the end-effector coordinate system to the robot's working coordinate system; determining the end-effector scanning pose by combining the calibration pose of the scanning device, and controlling the robot's end-effector to move to that pose for scanning. This application can determine the end-effector scanning pose when the target barcode is not currently visible, reducing invalid scanning actions and shortening scanning time. Therefore, this method can effectively and accurately scan target objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this application relate to the technical field of robot control, and more particularly to a robot barcode scanning method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In scenarios where robots grasp goods, it is often necessary to verify the product's identity by scanning barcodes. Currently, the scanning device can be fixedly installed in the robot's workspace, and the robot guides the product along a preset scanning trajectory, causing multiple surfaces of the product to sequentially pass through the scanning device's reading area.

[0003] However, different products vary in shape, size, and barcode placement, making it difficult to adapt a uniform, fixed scanning trajectory to various products and different gripping states. When using dense scanning actions covering multiple surfaces, invalid movements unrelated to the actual barcode position can easily occur, increasing scanning time. Some solutions detect the barcode and determine the scanning posture based on the current image, but when the barcode is located on the back, bottom, or obscured of the product, it is difficult to obtain the barcode's position and orientation, making it impossible to determine an effective scanning posture.

[0004] Furthermore, changes in the robot's gripping position and orientation alter the spatial relationship between the product, barcode, and the robot's end effector. Scanning based solely on the product's position or a preset presentation posture may result in barcodes being positioned, oriented, or positioned at distances that do not meet the reading requirements. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a robot barcode scanning method, apparatus, electronic device, storage medium, and program product.

[0006] In a first aspect, embodiments of this disclosure provide a robot barcode scanning method, the method comprising: Acquire the target object image and select at least one candidate item image that matches the target object image from multiple preset item images; The three-dimensional poses of the objects in each candidate object image are weighted and fused in the corresponding image space and then transformed to the preset robot working coordinate system to obtain the three-dimensional pose of the target object in the robot working coordinate system. When determining the robot end effector's grasping pose based on the geometric features of the target object, the robot end effector's grasping pose in the robot's working coordinate system is determined, and the robot end effector's grasping pose is controlled according to the grasping pose. By using the three-dimensional pose of the target object and the grasping pose of the end effector, the relative pose of the target object in the end effector coordinate system of the robot end effector after the robot grasps the target object is determined. The relative pose of the target barcode on the target object is converted from the first relative pose of the target barcode in the end coordinate system to the second relative pose of the target barcode in the robot working coordinate system by using the relative pose of the target object. Determine the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system. The end-effector scanning pose represents the pose of the robot end-effector in the robot's working coordinate system when it moves the target barcode from the relative pose of the second target barcode to the pose corresponding to the calibration pose. Control the robot's end effector to move to the end-scanning position to scan the target barcode.

[0007] In some alternative implementations, each item image includes a corresponding two-dimensional image and a corresponding depth map.

[0008] Accordingly, before acquiring the target image, the method further includes: Using two-dimensional images of each item from multiple angles and the corresponding depth map, the corresponding three-dimensional coordinate system of the item and the three-dimensional pose of the item in the three-dimensional coordinate system are determined. There is a preset transformation relationship between the three-dimensional coordinate system of the image space of each item and the corresponding three-dimensional coordinate system of the item. Determine the 3D position and normal vector of the barcode in the corresponding image space for each item image; The three-dimensional position and normal vector of each barcode are transformed into the corresponding three-dimensional coordinate system of the item and fused to obtain the relative pose of the barcode.

[0009] In some optional implementations, the three-dimensional position of each barcode and its corresponding normal vector are transformed to the corresponding three-dimensional coordinate system of the item and fused to obtain the corresponding barcode relative pose, including: By using the 3D pose of each item in its corresponding 3D coordinate system and the corresponding 3D coordinate system, the 3D position of the barcode in the corresponding image space and the corresponding barcode normal vector are inverted to obtain the relative pose of the barcode in the corresponding 3D coordinate system.

[0010] In some optional implementations, after inverting the barcode's 3D position and the corresponding barcode normal vector in the corresponding image space using the item's 3D pose in the corresponding item 3D coordinate system and the corresponding item 3D coordinate system, the method further includes: When multiple item images have the same barcode, the result of inverting the three-dimensional position of the barcode in the image space corresponding to each item image with the same barcode and the corresponding barcode normal vector is determined as the initial barcode relative pose corresponding to the same barcode. The relative poses of the initial barcodes corresponding to the same barcode are fused to obtain the corresponding relative pose of the barcode.

[0011] In some optional implementations, acquiring an image of the target object includes: Use a pre-set robot camera to capture images of the target object; Accordingly, the preset 3D poses of the items in each candidate item image are weighted and fused in their respective image spaces and then transformed to a preset robot working coordinate system to obtain the target object's 3D pose in the robot working coordinate system, including: The 3D poses of each candidate item in the corresponding image space are weighted to obtain the initial 3D pose of the target item in the corresponding image space. The initial 3D pose is transformed into the robot's working coordinate system using the preset camera extrinsic parameters of the robot camera, thus obtaining the 3D pose of the target object.

[0012] In some optional implementations, the relative pose of the target object in the end-effector coordinate system after the robot's end-effector grasps the target object is determined using the target object's three-dimensional pose and the end-effector grasping pose, including: By dividing the target object pose matrix, which represents the three-dimensional pose of the target object, by the grasping pose matrix, which represents the grasping pose of the end effector, the relative pose matrix of the target object is obtained. The relative pose matrix of the target object represents the relative pose of the target object in the end effector coordinate system when the robot's end effector grasps the target object. In response to ensuring that the relative pose of the target object is consistent between when the robot end-effector grasps the target object and after the robot end-effector grasps the target object, the relative pose of the target object when the robot end-effector grasps the target object is determined as the relative pose of the target object after the robot end-effector grasps the target object.

[0013] In some optional implementations, before converting the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose, the method further includes: Given that each candidate item image with a barcode corresponds to the same barcode relative pose, the same barcode corresponding to each candidate item image is determined as the target barcode, and the corresponding barcode relative pose is transformed to the end coordinate system to obtain the first target barcode relative pose.

[0014] In some optional implementations, the relative pose of a pre-set target barcode on the target object in the end-effector coordinate system is transformed to a second target barcode relative pose in the robot's working coordinate system using the target object's relative pose, including: Using the relative pose of the target object in the end-effector coordinate system, the inverse operation is performed on the first target barcode relative pose in the end-effector coordinate system to obtain the transformation matrix of the end-effector coordinate system relative to the robot working coordinate system. Using the transformation matrix, the relative pose of the first target barcode is transformed to the relative pose of the second target barcode in the robot's working coordinate system.

[0015] In some optional implementations, the calibration pose is a preset pose of the barcode scanning device in the robot's working coordinate system; Accordingly, determining the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system includes: Based on the pose of the scanning device in the robot's working coordinate system, and provided that the center of the target barcode is within the preset reading area of ​​the scanning device, and the angle between the barcode normal vector corresponding to the target barcode and the optical axis of the scanning device is within a preset angle range, the pose of the robot end effector grasping the target object in the robot's working coordinate system is determined as the end effector scanning pose.

[0016] Secondly, embodiments of this disclosure provide a robot barcode scanning device, the device comprising: The acquisition module is configured to acquire the target object image and select at least one candidate object image that matches the target object image from a plurality of preset object images corresponding to each object. The target object 3D pose determination module is configured to perform weighted fusion of the preset 3D poses of the objects in the corresponding image space of each candidate object image and then transform them to the preset robot working coordinate system to obtain the target object 3D pose in the robot working coordinate system. The grasping module is configured to determine the end-effector grasping pose in the robot's working coordinate system based on the geometric features of the target object, and to control the end-effector to grasp the target object according to the end-effector grasping pose. The target object relative pose determination module is configured to use the target object's three-dimensional pose and the end-effector grasping pose to determine the target object's relative pose in the end-effector coordinate system after the robot's end-effector grasps the target object. The target barcode relative pose determination module is configured to use the target object relative pose to convert the first target barcode relative pose in the end coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose. The end-effector scanning pose determination module is configured to determine the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system. The end-effector scanning pose represents the pose of the robot end-effector in the robot's working coordinate system when the robot end-effector moves the target barcode from the relative pose of the second target barcode to the pose corresponding to the calibration pose. The barcode scanning module is configured to control the robot's end effector to move to the end-scanning pose in order to scan the target barcode.

[0017] In some alternative implementations, each item image includes a corresponding two-dimensional image and a corresponding depth map.

[0018] Accordingly, the robot scanning device also includes a preprocessing module, which is configured to: Using two-dimensional images of each item from multiple angles and the corresponding depth map, the corresponding three-dimensional coordinate system of the item and the three-dimensional pose of the item in the three-dimensional coordinate system are determined. There is a preset transformation relationship between the three-dimensional coordinate system of the image space of each item and the corresponding three-dimensional coordinate system of the item. Determine the 3D position and normal vector of the barcode in the corresponding image space for each item image; The three-dimensional position and normal vector of each barcode are transformed into the corresponding three-dimensional coordinate system of the item and fused to obtain the relative pose of the barcode.

[0019] In some optional implementations, the three-dimensional position of each barcode and its corresponding normal vector are transformed to the corresponding three-dimensional coordinate system of the item and fused to obtain the corresponding barcode relative pose, including: By using the 3D pose of each item in its corresponding 3D coordinate system and the corresponding 3D coordinate system, the 3D position of the barcode in the corresponding image space and the corresponding barcode normal vector are inverted to obtain the relative pose of the barcode in the corresponding 3D coordinate system.

[0020] In some optional implementations, after inverting the barcode's 3D position and corresponding barcode normal vector in the corresponding image space using the item's 3D pose in the corresponding item's 3D coordinate system and the corresponding item's 3D coordinate system, the preprocessing module further performs: When multiple item images have the same barcode, the result of inverting the three-dimensional position of the barcode in the image space corresponding to each item image with the same barcode and the corresponding barcode normal vector is determined as the initial barcode relative pose corresponding to the same barcode. The relative poses of the initial barcodes corresponding to the same barcode are fused to obtain the corresponding relative pose of the barcode.

[0021] In some alternative implementations, the acquisition module is further configured as follows: Use a pre-set robot camera to capture images of the target object; Accordingly, the target object 3D pose determination module is also specifically configured as follows: The 3D poses of each candidate item in the corresponding image space are weighted to obtain the initial 3D pose of the target item in the corresponding image space. The initial 3D pose is transformed into the robot's working coordinate system using the preset camera extrinsic parameters of the robot camera, thus obtaining the 3D pose of the target object.

[0022] In some optional implementations, the target object relative pose determination module is further configured as follows: By dividing the target object pose matrix, which represents the three-dimensional pose of the target object, by the grasping pose matrix, which represents the grasping pose of the end effector, the relative pose matrix of the target object is obtained. The relative pose matrix of the target object represents the relative pose of the target object in the end effector coordinate system when the robot's end effector grasps the target object. In response to ensuring that the relative pose of the target object is consistent between when the robot end-effector grasps the target object and after the robot end-effector grasps the target object, the relative pose of the target object when the robot end-effector grasps the target object is determined as the relative pose of the target object after the robot end-effector grasps the target object.

[0023] In some optional implementations, the target barcode relative pose determination module is further specifically configured to perform the following steps before converting the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose: Given that each candidate item image with a barcode corresponds to the same barcode relative pose, the same barcode corresponding to each candidate item image is determined as the target barcode, and the corresponding barcode relative pose is transformed to the end coordinate system to obtain the first target barcode relative pose.

[0024] In some optional implementations, the target barcode relative pose determination module is further configured as follows: Using the relative pose of the target object in the end-effector coordinate system, the inverse operation is performed on the first target barcode relative pose in the end-effector coordinate system to obtain the transformation matrix of the end-effector coordinate system relative to the robot working coordinate system. Using the transformation matrix, the relative pose of the first target barcode is transformed to the relative pose of the second target barcode in the robot's working coordinate system.

[0025] In some optional implementations, the calibration pose is a preset pose of the barcode scanning device in the robot's working coordinate system; Accordingly, the end-effector scanning pose determination module is also specifically configured as follows: Based on the pose of the scanning device in the robot's working coordinate system, and provided that the center of the target barcode is within the preset reading area of ​​the scanning device, and the angle between the barcode normal vector corresponding to the target barcode and the optical axis of the scanning device is within a preset angle range, the pose of the robot end effector grasping the target object in the robot's working coordinate system is determined as the end effector scanning pose.

[0026] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0027] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation of the first aspect.

[0028] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the implementations of the first aspect.

[0029] To address the challenges of determining an effective scanning pose for a robot when the barcode is not currently visible, based on the target object's pose and grasping state, and the resulting issues of numerous invalid actions and lengthy scanning times along fixed scanning trajectories, the robot scanning method, apparatus, electronic device, storage medium, and program products disclosed in this disclosure reduce the impact of single matching biases by matching the target object image with preset item images, combining the pose information corresponding to candidate item images for weighted fusion and coordinate transformation, and unifying the poses of the target object and the robot's end effector to the robot's working coordinate system. The relative pose of the target object is determined based on its three-dimensional pose and the end effector's grasping pose, and the target barcode pose is calculated by combining preset barcode spatial relationships, enabling the end effector to determine the scanning pose even when the barcode is not visible in the current image. By combining the scanning device's calibration pose, reading conditions, and robot motion constraints, the target barcode moves with the robot's end effector to a readable position and orientation, reducing invalid actions in fixed, dense scanning trajectories and shortening scanning time. Meanwhile, by continuously scanning, fine-tuning the pose, and compensating for errors in grasping and execution, the success rate of scanning is improved; the identity of the target item is confirmed by the read barcode content, reducing sorting or outbound errors caused by misidentification due to appearance matching. Attached Figure Description

[0030] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure may be applied; Figure 2A A flowchart of one embodiment of the robot barcode scanning method according to the present disclosure; Figure 2B A breakdown flowchart of step 2020 of an embodiment of the robot scanning method according to the present disclosure; Figure 2C A breakdown flowchart of step 2040 of an embodiment of the robot scanning method according to the present disclosure; Figure 3 This is a breakdown flowchart of a process 300 for determining the relative pose of a barcode according to an embodiment of the present disclosure. Figure 4 This is a schematic diagram of the structure of one embodiment of the robot barcode scanning device according to the present disclosure; Figure 5 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 An exemplary system architecture 100 is shown, which can be applied to embodiments of the robot scanning method, apparatus, electronic device, storage medium, and program product of this application.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired communication links, wireless communication links, fiber optic links, etc.

[0035] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to upload item images, configure scanning parameters, initiate target item scanning tasks, or obtain scanning results. Various client applications can be installed on terminal devices 101, 102, and 103, such as target item scanning applications, robot control applications, item data management applications, web browsers, instant messaging applications, or client applications capable of calling target item scanning services.

[0036] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with information input and information output devices. Information input devices can include keyboards, mice, touchscreens, microphones, cameras, etc., while information output devices can include displays, speakers, etc. Terminal devices 101, 102, and 103 can include, but are not limited to, smartphones, tablets, laptops, desktop computers, and smart interactive terminals. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices and can be implemented as a single software module or multiple software modules; this application does not limit this.

[0037] In some embodiments, system architecture 100 can also communicate with a robot and a barcode scanning device. The robot includes a robot body, a robot camera mounted on the robot body, and a robot end effector. The robot camera is used to capture images of the target object and provide these images to terminal devices 101, 102, 103, and / or server 105. The robot end effector is used to grasp the target object. The robot can grasp the target object and move it according to control commands sent by terminal devices 101, 102, 103, and / or server 105. The barcode scanning device can be fixedly installed in the robot's workspace to read the target barcode on the target object and provide the scanning result to terminal devices 101, 102, 103, and / or server 105.

[0038] In some embodiments, the robot barcode scanning method provided in this application can be executed by server 105. Accordingly, the robot barcode scanning device can be disposed in server 105. Server 105 can be used to perform operations such as target object image acquisition, candidate item image selection, target object three-dimensional pose determination, end-effector grasping pose determination, target object relative pose calculation, target barcode relative pose conversion, and end-effector scanning pose determination, and send control commands to the robot to control the robot end to grasp the target object and move the target barcode on the target object to a position that the scanning device can read.

[0039] In some embodiments, the robot barcode scanning method provided in this application can be executed by terminal devices 101, 102, and 103. Accordingly, a robot barcode scanning device can be disposed in terminal devices 101, 102, and 103. For example, terminal devices 101, 102, and 103 can locally store preset object images and corresponding pose data, acquire target object images captured by robot cameras, determine end-effector grasping pose and end-effector scanning pose based on the target object images and preset data, and send corresponding control commands to the robot.

[0040] In some embodiments, the robot barcode scanning method provided in this application can also be executed collaboratively by terminal devices 101, 102, 103 and server 105. For example, terminal devices 101, 102, 103 can be used to acquire target object images captured by robot cameras and send the target object images to server 105; server 105 can be used to select candidate object images that match the target object images, determine the target object's three-dimensional pose, end-effector grasping pose, target object's relative pose, and end-effector scanning pose, and send the calculation results to terminal devices 101, 102, 103; terminal devices 101, 102, 103 can send control commands to the robot according to the calculation results and obtain the scanning results returned by the barcode scanning device. Accordingly, at least some modules in the robot barcode scanning device can be set in terminal devices 101, 102, 103, and at least some modules can be set in server 105.

[0041] In some embodiments, where the server directly communicates with the robot, robot camera, and barcode scanning device, the robot barcode scanning method provided in this application can be executed solely by the server 105. In this case, the system architecture 100 may not include terminal devices 101, 102, and 103. In other embodiments, in a local barcode scanning control scenario, the robot barcode scanning method provided in this application can be executed solely by terminal devices 101, 102, and 103. In this case, the system architecture 100 may not include the server 105.

[0042] Server 105 can be hardware or software. When server 105 is hardware, it can be implemented as a single server or as a distributed server cluster composed of multiple servers. When server 105 is software, it can be implemented as a single software module or as multiple software modules; this application does not limit this.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown in the diagram is for illustrative purposes only. Depending on the actual application requirements, system architecture 100 can include any number of terminal devices, networks, and servers.

[0044] In some alternative implementations, refer to Figure 2AThe diagram illustrates a flow 200 of an embodiment of the robot barcode scanning method of this disclosure, which includes the following steps 201 to 207: Step 201: Acquire the target object image and select at least one candidate object image that matches the target object image from a set of multiple preset object images.

[0045] In this embodiment, the executing entity of the robot scanning method can be the aforementioned terminal device or server. The executing entity acquires the target object image of the current object to be grasped, and matches the target object image with preset item images to select candidate item images that can be used for subsequent 3D pose calculation of the target object. For example, in a warehouse sorting scenario, the robot needs to grasp a target object from multiple items and confirm the product identity of the target object by scanning the code. The executing entity can acquire the target object image before grasping and perform the corresponding image matching operation.

[0046] Here, "item" refers to a commodity object with a corresponding shape, size, and product identity, such as beverages, boxed food, or daily necessities with specific packaging and capacity. Different specifications, models, or packaging of goods can correspond to different items. "Target object" refers to the item that needs to be picked up and scanned by the robot, which can be any one of a set of preset items.

[0047] An item image refers to a pre-saved image representing the appearance of a corresponding item. The same item can correspond to multiple images from different viewing angles. For example, for the same boxed product, images of the item viewed from the front, side, and oblique angles can be pre-saved. A target object image refers to an image captured during runtime that contains the current target object, reflecting its current appearance and placement.

[0048] Furthermore, the candidate item image can be an image selected from a pre-defined set of item images that matches the target item image. Both the item image and the target item image can contain color images that characterize the appearance of the item, and the executing entity can use these color images for matching.

[0049] Specifically, after the executing entity obtains the target object image, it can compare the appearance of the target object in the target object image with the corresponding item images of multiple items. Based on the correspondence between visible features such as shape, surface texture, and packaging pattern, the matching result between the target object image and each item image is determined.

[0050] The matching process can be performed between images of different items or between images of the same item from different viewing angles, thereby selecting candidate images of items that correspond to the appearance of the current target object.

[0051] The executing entity can also obtain the matching confidence of each candidate item image based on the matching results.

[0052] The matching confidence score is a numerical value that characterizes the reliability of the match between the candidate item image and the target item image. A higher matching confidence score indicates a more reliable correspondence between the candidate item image and the visible appearance of the current target item. The executing entity can record the ID and matching confidence score of each candidate item image to obtain the corresponding item data through the ID, and provide a basis for the subsequent weighted fusion of pose information.

[0053] For example, among the preset fifty item images, multiple candidate item images that are the same as or similar in appearance to the target item image can be selected, and the number of each candidate item image and its matching confidence level can be obtained.

[0054] Because the appearance of the same object varies from different viewing angles, and the object may be partially occluded during actual placement, a single image of the object cannot fully reflect all visible features. By selecting multiple candidate images, matching results from different viewing angles can be retained. This allows subsequent processing to combine the pose information from multiple candidate images to determine the object's 3D pose, reducing estimation bias caused by relying solely on a single matching result.

[0055] In this embodiment, the executing entity performs image matching on the target object without actively detecting barcodes in the target object image. Furthermore, the selection of candidate item images does not depend on the barcode detection results. For example, even if the front of the target object faces the image acquisition direction and the barcode is located on the back of the target object, the executing entity can still select candidate item images based on the packaging pattern, outline, and texture visible from the front. The same processing procedure can also be used when the barcode is located on the side, bottom, or obscured by other objects.

[0056] Therefore, whether a barcode is observable in the current image is not a necessary condition for selecting candidate item images. The executing entity establishes a correspondence between the visible appearance of the target object and the item image, allowing target objects whose barcodes are not currently visible to still enter the subsequent pose calculation and scanning process. Simultaneously, the candidate item images obtained through image matching can be used for product identification after the barcode content is obtained through subsequent scanning, reducing errors caused by judging product identity solely based on appearance matching results.

[0057] Step 202: After weighted fusion of the preset 3D poses of the items in the corresponding image space of each candidate item image, the object is transformed into the preset robot working coordinate system to obtain the 3D pose of the target object in the robot working coordinate system.

[0058] Based on the candidate item images selected in step 201, the executing entity can obtain the 3D pose of the items in each candidate item image, perform weighted fusion by combining the matching results of each candidate item image, and transform the fused pose into the robot's working coordinate system within the robot's workspace. Thus, the candidate item images obtained through image matching can be further transformed into the position and orientation of the target object in the robot's workspace, i.e., the 3D pose of the target object in the robot's working coordinate system, providing a spatial basis for subsequent grasping and barcode scanning pose calculations.

[0059] In this context, the 3D pose of an object refers to its position and orientation within the 3D coordinate system of the corresponding image space. This orientation can characterize features such as the object's orientation in the image space. The 3D coordinate system of the image space can be the camera coordinate system used when acquiring the image of the corresponding object, which differs from the 2D coordinate system that only represents the pixel positions of the image. The 3D pose of an object can be represented by translation and rotation data, or by a pose matrix that includes both translation and rotation data. For example, the same object can correspond to different 3D poses at different viewing angles, with each 3D pose reflecting the object's position and orientation relative to the corresponding image acquisition direction.

[0060] Furthermore, the robot working coordinate system refers to the coordinate system used to uniformly describe the position and orientation of each object in the robot's workspace, and the robot base coordinate system can be used as the robot working coordinate system.

[0061] Furthermore, the target object's 3D pose represents the target object's pose in the robot's working coordinate system, which is used to characterize the target object's current position and orientation within the robot's workspace. For example, the target object's 3D pose can characterize the position of a boxed product relative to the robot's base, as well as the orientation of the boxed product's top and sides in the workspace.

[0062] Specifically, the 3D pose of the object associated with each candidate object image can be retrieved based on its image number. Since different candidate object images correspond to different viewing angles, the retrieved 3D pose can be adjusted by combining the matching relationship between the target object image and each candidate object image, ensuring that the pose information used in the fusion represents the current target object's position and orientation in the current image space. For example, the 3D pose of the object corresponding to the candidate object image can be corrected based on the matched features and their spatial relationships, making it correspond to the current observation state of the target object. This avoids mixing pre-stored poses from different shooting states, which could lead to a failure to reflect the current placement of the target object.

[0063] During the weighted fusion process, the executing entity can determine the fusion weight of the corresponding pose information based on the matching confidence of each candidate item image obtained in step 201. Candidate item images with higher matching confidence can have their corresponding pose information assigned a larger fusion weight; candidate item images with lower matching confidence can have their corresponding pose information assigned a smaller fusion weight. When multiple matching confidences are obtained, the ratio between each matching confidence and the sum of the matching confidences can be determined as the corresponding fusion weight, ensuring that the sum of all fusion weights is one.

[0064] Furthermore, the position and attitude information can be fused separately.

[0065] Specifically, for position, the position components can be weighted according to the fusion weights to obtain the fused position. For pose, the rotation data can first be converted into a unified rotation representation, then fused according to the fusion weights, and the fusion result converted into the corresponding pose data. For example, quaternions can be used to represent the rotation data, and after unifying the quaternion representation of the direction, weighted calculation and normalization are performed to obtain the fused rotation data. By processing position and pose separately, the fusion result can simultaneously represent the position and orientation of the target object while maintaining the validity of the rotation representation. When only a single candidate object image is selected, the fusion weight of its corresponding pose information can be set to one.

[0066] After obtaining the fused pose, the result can be transformed to the robot's working coordinate system based on a pre-defined transformation relationship between the image space 3D coordinate system and the robot's working coordinate system. This transformation relationship simultaneously represents the position offset and orientation difference between the two coordinate systems. Therefore, the position and orientation in the fused result can be transformed accordingly to obtain the target's 3D pose in the robot's working coordinate system. It should be noted that the coordinate transformation changes the representation reference of the pose; the target's position and orientation in actual space do not change due to this transformation.

[0067] Since the matching results of different candidate item images may be affected by viewing angle, local occlusion, and appearance similarity, determining the fusion weights through matching confidence allows more reliable matching results to contribute more significantly to the target object's 3D pose, reducing the impact of single matching biases on pose estimation. Furthermore, transforming the fusion results to the robot's working coordinate system allows the target object's 3D pose and the subsequent robot end effector's pose to use a unified spatial representation benchmark, thus supporting the calculation of the relative pose between the target object and the robot end effector. Even when the target object image does not contain a barcode, the pose can still be calculated based on the matching results between the target object and candidate item images, without needing to use the barcode position in the current image as input for pose calculation.

[0068] Step 203: Determine the end-effector grasping pose in the robot working coordinate system when the robot end-effector grasps the target object based on the geometric features of the target object, and control the robot end-effector to grasp the target object according to the end-effector grasping pose.

[0069] Based on the obtained target image and the target's three-dimensional pose in step 202, the position and orientation suitable for the robot's end effector to grasp can be determined according to the target's geometric features, thus obtaining the end effector grasping pose, and controlling the robot's end effector to reach the pose to perform grasping.

[0070] Geometric features are used to describe the spatial morphology of a target object, and may include, for example, the object's shape, size, surface contour, and local surface orientation. For instance, boxed goods may have a flat top surface and oppositely positioned sides, while bottled goods may have curved sidewalls. Different geometric features can correspond to different gripping positions and gripping orientations.

[0071] The end effector grasping pose refers to the position and orientation of a robot's end effector in its working coordinate system when performing a grasping action. Position characterizes the spatial location the robot's end effector needs to reach, while orientation characterizes the direction the end effector faces when it comes into contact with the target object. For example, when using a suction cup to pick up boxed goods, the end effector grasping pose can align the suction cup with the top surface of the product and position the suction cup's contact area on a suitable surface for adhesion; when using a gripper, the end effector grasping pose can align the gripping direction of the gripper with a suitable surface on the target object for gripping.

[0072] Specifically, based on the current geometric characteristics of the target object, a suitable surface or area for gripping can be determined. Combined with the gripping method and structural dimensions of the robot's end effector, the corresponding contact position and approach direction can be determined. When using a suction method, an area with sufficient contact area and a surface orientation suitable for suction cup adhesion can be selected. When using a clamping method, a clamping area adapted to the opening and closing range of the grippers can be selected based on the width of the target object and the distance between relative surfaces. Thus, the gripping position can be determined according to the actual spatial shape of different target objects, reducing insufficient contact or unstable clamping caused by mismatch between the gripping position and the robot's end effector structure.

[0073] Based on the determined contact position, approach direction, and spatial relationship between the robot's end effector and the contact component, the corresponding position and orientation of the robot's end effector can be further determined. For example, there may be a fixed distance between the contact surface of the suction cup and the origin of the robot's end effector coordinate system. After determining the surface position that the suction cup needs to contact, the position that the origin of the robot's end effector coordinate system needs to reach can be determined by combining this fixed distance and the orientation of the suction cup. By uniformly representing the calculated position and orientation in the robot's working coordinate system, the end effector grasping pose can be obtained.

[0074] As can be seen, the end-effector grasping pose is calculated based on the current geometric features of the target object. Its determination process does not rely on a fixed grasping position in a pre-saved object image, nor does it directly use the target object's 3D pose obtained in step 202 as the end-effector grasping pose. The target object's 3D pose describes its position and orientation within the robot's workspace, while the end-effector grasping pose describes the position and orientation of the robot's end-effector during grasping; the two correspond to different objects. For example, even if the same target object has the same 3D pose, different end-effector grasping poses can be obtained when using top-face suction and side-face gripping methods.

[0075] After determining the end-effector grasping pose, the robot's end-effector can be controlled to move from its current position to the corresponding grasping position, and its orientation can be adjusted to ensure that the end-effector contacts the target object in a predetermined approach direction. Once the end-effector reaches the grasping pose, the gripper can be controlled to close or initiate suction, allowing the robot to grasp or pick up the target object. By simultaneously determining the grasping position and orientation, the contact relationship between the robot's end-effector and the target object can be adapted to the object's geometric characteristics, providing pose data in a unified coordinate system for subsequent calculations of the target object's relative pose based on its 3D pose and the end-effector grasping pose.

[0076] Step 204: Using the three-dimensional pose of the target object and the end-effector grasping pose, determine the relative pose of the target object in the end-effector coordinate system after the robot end-effector grasps the target object.

[0077] Based on the target object's 3D pose obtained in step 202 and the end-effector grasping pose determined in step 203, the spatial relationship between the target object and the robot's end effector during grasping can be determined, and this spatial relationship can be used to characterize the target object's position and orientation relative to the robot's end effector after grasping. Both the target object's 3D pose and the end effector grasping pose are expressed in the robot's working coordinate system, thus allowing for relative relationship calculations under a unified coordinate reference to obtain the target object's relative pose.

[0078] The end effector coordinate system can be a coordinate system that is attached to the robot's end effector and moves with it. Its origin and coordinate axis directions can be predetermined according to the structure and control requirements of the robot's end effector. For example, the origin of the end effector coordinate system can be located at the mounting center of the gripper or the tool reference point corresponding to the suction component.

[0079] Therefore, the relative pose of the target object can be its position and orientation in the end-effector coordinate system. This relative pose can also characterize the transformation relationship between the target object's three-dimensional coordinate system and the end-effector coordinate system. This pose reflects the spatial offset and orientation difference of the target object relative to the robot's end-effector after it has been grasped.

[0080] Specifically, the spatial relationship between the target object's 3D coordinate system and the robot's working coordinate system can be determined based on the target object's 3D pose. The spatial relationship between the end effector's coordinate system and the robot's working coordinate system can be determined based on the end effector's grasping pose. Then, the target object's position and orientation are transformed to the end effector's coordinate system. During the calculation, not only the positional difference between the target object and the robot's end effector needs to be considered, but also the robot's end effector's orientation needs to be taken into account, expressing this positional difference along each coordinate axis of the end effector's coordinate system. Simultaneously, based on the correspondence between the target object's orientation and the robot's end effector's orientation, the target object's posture in the end effector's coordinate system is determined.

[0081] For example, when a robot's end effector picks up a target object from above a boxed product, the center of the target object can be located below the origin of the end effector's coordinate system, with a certain distance between them. If the picking position deviates from the center of the target object, the center of the target object can also undergo a lateral shift relative to the end effector's coordinate system. The relative pose of the target object can simultaneously characterize this distance, the lateral shift, and the orientation difference between the target object and the robot's end effector. Therefore, even when grasping the same target object, different grasping positions and orientations may correspond to different relative poses of the target object.

[0082] When the target object is stably grasped and does not slip or rotate relative to the robot's end effector, the relative relationship between the target object and the robot's end effector remains unchanged during subsequent motion. At this time, the position and orientation of the robot's end effector in the robot's working coordinate system can change with motion, while the relative pose of the target object in the end effector coordinate system remains unchanged. Therefore, the changes in the target object's position and orientation during robot end effector movement can be deduced. This embodiment uses the assumption that the relative relationship between the target object and the robot's end effector remains unchanged after grasping and is consistent with the relative relationship calculated during grasping as the basis for subsequent calculations.

[0083] Through the above processing, the pose of the target object in the robot's workspace can be correlated with the motion of the robot's end effector. When subsequently calculating the position and orientation of the target barcode, the spatial relationship between the target barcode and the robot's end effector can be determined by combining the relative pose of the target object, thereby calculating the end-effector's required scanning pose. Actual grasping errors or target object slippage may cause deviations in this spatial relationship; their impact on scanning can be compensated for during subsequent adjustments to the end-effector's scanning pose.

[0084] Step 205: Using the relative pose of the target object, the first relative pose of the target barcode in the end coordinate system is converted to the second relative pose of the target barcode in the robot working coordinate system.

[0085] Based on the target object's relative pose determined in step 204, the pre-saved spatial data of the target barcode can be combined with the current grasping state to determine the target barcode's position and orientation relative to the robot's end effector, i.e., the target barcode's relative pose. Furthermore, based on this, the target barcode's position and orientation in the robot's working coordinate system can be determined, i.e., the second target barcode's relative pose. Therefore, even when the target barcode does not appear in the current target object image, the second target barcode's relative pose can be calculated using the pre-saved spatial data of the target barcode and the spatial relationships determined during grasping, without needing to re-detect the target barcode after grasping.

[0086] The target barcode can be a barcode on the target item that needs to be read by a scanning device, and it can be used to represent the product information corresponding to the target item.

[0087] In this step, the position and orientation of the target barcode in the end-effector coordinate system are determined as the first target barcode relative pose, while the second target barcode relative pose is the position and orientation of the same target barcode in the robot's working coordinate system. Both use different coordinate references to describe the spatial state of the same target barcode, and the coordinate transformation itself does not cause actual movement of the target barcode.

[0088] The position of the target barcode can be represented by the three-dimensional position of the barcode center, and the orientation can be represented by the normal vector of the area where the barcode is located, so that subsequent processing steps can take into account both the position and surface orientation of the target barcode.

[0089] Specifically, based on the candidate item images selected in step 201, the relative pose and content of the barcodes associated with each candidate item image can be obtained. When candidate item images with barcodes correspond to the same relative pose of a barcode, the same barcode corresponding to these candidate item images can be identified as the target barcode, and the relevant spatial data corresponding to the target barcode can be obtained. For example, candidate item images of the same packaged product from multiple viewing angles can be associated with the same barcode. Since the position and orientation of the barcode on the item remain consistent, the target barcode can be determined through the association between the candidate item images and preset data, without requiring the target barcode to appear simultaneously in the current target item image.

[0090] The pre-saved relative pose of the barcode is used to characterize the spatial relationship between the barcode and the item itself, that is, the position and orientation of the target barcode in the corresponding three-dimensional coordinate system of the item. Based on this, the relative pose of the barcode can be transformed into the end-effector coordinate system to obtain the first target barcode relative pose. For example, if the barcode is located on the side of a boxed product, when the robot end-effector picks up the target product from the top surface of the product, it can determine the position of the barcode center relative to the origin of the end-effector coordinate system, as well as the direction of the barcode normal vector in the end-effector coordinate system, based on the preset position and orientation of the barcode on the product, and the spatial offset and orientation difference of the product relative to the robot end-effector. Thus, the first target barcode relative pose can reflect the spatial relationship between the target barcode and the robot end-effector after this grasping.

[0091] Furthermore, based on the obtained relative pose of the first target barcode, the transformation relationship between the end-effector coordinate system and the robot working coordinate system can be determined by combining the correspondence between the relative pose of the target object and the three-dimensional pose of the target object. In this way, the position and orientation of the target barcode can be transformed to the robot working coordinate system to obtain the relative pose of the second target barcode.

[0092] The transformation of the barcode center position needs to consider both the positional offset and orientation difference between the two coordinate systems. The transformation of the barcode normal vector is used to change the representation reference of its direction, without adding the positional offset between the origins of the coordinate systems. For example, if the robot end effector has already grasped the target object but has not yet changed its grasping pose, the transformation can be completed by combining the end effector's grasping pose; if the robot end effector has already moved after grasping, the corresponding transformation relationship can be updated by combining the robot end effector's current pose in the robot's working coordinate system.

[0093] Because the pre-defined spatial relationship between the target barcode and the target object, and the relative relationship between the target object and the robot end effector after stable grasping, can remain unchanged, the movement of the robot end effector can correspondingly cause changes in the position and orientation of the target barcode. After converting the target barcode to the robot's working coordinate system, the relative pose of the second target barcode and the calibration pose of the scanning device can adopt a unified coordinate reference, thereby supporting subsequent calculations of the end-effector scanning pose that allows the target barcode to enter the effective reading area of ​​the scanning device, reducing the reliance on fixed scanning actions covering multiple surfaces of the target object.

[0094] Step 206: Determine the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system.

[0095] Among them, the end-effector scanning pose refers to the pose of the robot end-effector in the robot's working coordinate system when the robot end-effector moves the target barcode from the relative pose of the second target barcode to the pose corresponding to the calibrated pose.

[0096] Based on the relative pose of the second target barcode obtained in step 205, the position and orientation of the target barcode that can be read by the scanning device can be determined by combining the calibration pose of the scanning device. Based on the spatial relationship between the target barcode and the robot end effector, the end effector scanning pose that the robot end effector needs to reach can be determined.

[0097] The calibration pose can be the pre-calibrated position and orientation of the barcode scanning device in the robot's working coordinate system. When the barcode scanning device is fixedly installed in the robot's workspace, its calibration pose can serve as a spatial reference for determining the barcode reading position and orientation.

[0098] Furthermore, the end-effector scanning pose represents the position and orientation of the robot's end effector.

[0099] Furthermore, the target barcode pose corresponding to the calibration pose indicates the position and orientation that the target barcode needs to be in order to meet the reading conditions of the scanning device. For example, the target barcode can be located in the reading area in front of the scanning device, with its surface facing the scanning device and maintaining a suitable reading distance. When the robot's end effector reaches the end-effector scanning pose, it can present the target barcode at that position using the grasped object.

[0100] Specifically, the preset reading area and optical axis position of the scanning device in the robot's working coordinate system can be determined based on the calibration posture of the scanning device.

[0101] Furthermore, when the center of the target barcode is within the preset reading area of ​​the scanning device, and the angle between the barcode normal vector corresponding to the target barcode and the optical axis of the scanning device is within the preset angle range, the corresponding pose of the robot end effector in the robot working coordinate system can be determined as the end effector scanning pose.

[0102] The preset reading area refers to the spatial region where barcodes can be read, determined based on the scanning capability of the barcode scanner. The optical axis refers to the reference axis representing the scanning direction of the barcode scanner. The barcode normal vector is used to represent the orientation of the surface on which the barcode is located, and the angle between it and the optical axis can be calculated according to a unified directional convention. For example, even if the center of the barcode is located within the reading area, if the surface on which the barcode is located is tilted too much relative to the scanning direction, the projection of the barcode in the scanning field of view may be compressed, leading to reading difficulties. Therefore, simultaneously determining the position of the barcode center and the orientation of the barcode surface can reduce the problem of failing to read the barcode even if the position requirement is met.

[0103] Furthermore, the reading distance range can be determined based on the actual size of the target barcode and the scanning device's reading capability, and it can be judged whether the distance between the target barcode and the scanning device is within this range. The actual size of the target barcode can be obtained from preset data associated with the target object, and the scanning device's reading capability can be determined based on device parameters or pre-test results. For example, for small barcodes, a distance that is too far may prevent the barcode details from being effectively distinguished; a distance that is too close may prevent the barcode from being completely within the reading range. By combining the distance condition with the barcode center position and the angle condition corresponding to the barcode normal vector, the target barcode pose that simultaneously satisfies the requirements of position, orientation, and distance can be obtained.

[0104] Based on the first target barcode relative pose in the end-effector coordinate system, the target barcode pose that meets the above recognition conditions can be further calculated as the candidate pose of the robot end-effector.

[0105] Specifically, the target barcode pose that meets the recognition conditions can be represented as a target pose matrix of the barcode coordinate system relative to the robot's working coordinate system, and the relative pose of the first target barcode can be represented as a relative pose matrix of the barcode coordinate system relative to the end effector coordinate system. Since the relative relationship between the target barcode and the robot's end effector remains unchanged after grasping, the relative pose matrix can be inverted, and then the target pose matrix can be multiplied by the inverse matrix to obtain the pose matrix of the end effector coordinate system relative to the robot's working coordinate system. The corresponding position and orientation are then determined as candidate poses for the robot's end effector.

[0106] Since the spatial relationship between the target barcode and the robot end effector remains unchanged after stable grasping, the position and orientation of the target barcode can be determined based on this, without having to repeatedly rotate the target object to find the barcode one by one.

[0107] During the solution process, the robot's reachable area, singular pose constraints, and obstacle avoidance areas in the workspace can be combined to examine candidate poses and the motion processes leading to them. For example, candidate poses that the robot cannot reach, those that would cause the robot to approach singular poses, and motion paths that would cause the robot or the grasped object to collide with surrounding obstacles can be eliminated, thereby determining the end-effector scanning pose that can actually be executed.

[0108] When solving multiple feasible poses in parallel, the first feasible pose calculated can be determined as the end-effector scanning pose, reducing the waiting time for pose solving. When multiple feasible poses are obtained simultaneously, the pose with the shortest motion time can be selected based on the estimated time required for the robot's end-effector to move from the current pose to each feasible pose; alternatively, the pose with the smallest change in position and attitude relative to the current end-effector pose can be selected by comparing the position and attitude changes of each feasible pose relative to the current end-effector pose. Therefore, the end-effector scanning pose can both meet the barcode reading requirements and reduce unnecessary movement by considering the robot's current state.

[0109] Step 207: Control the robot end effector to move to the end-scanning pose to scan the target barcode.

[0110] Based on the end-effector scanning pose determined in step 206, the robot end-effector can be controlled to move the grasped target object, causing the position and orientation of the target barcode to change accordingly with the movement of the robot end-effector, and bringing the target barcode into the spatial area that the scanning device can read. During the movement, the robot end-effector maintains its grasp of the target object, keeping the relative relationship between the target object and the robot end-effector as stable as possible, so that the actual barcode pose reached corresponds to the aforementioned calculation result.

[0111] Specifically, the motion path can be determined based on the current pose of the robot's end effector and the end effector's scanning pose, and the robot's end effector can be controlled to move along this path. When the robot's end effector approaches the end effector's scanning pose, the scanning device can be activated, keeping it in scanning mode. Approaching the end effector's scanning pose can be determined by the positional deviation between the robot's end effector and the end effector's scanning pose, or by the remaining distance on the motion path. Since the target barcode may enter the valid reading area before reaching the final pose, activating the scanning device in advance can take advantage of the reading opportunity during the approach process, reducing the time consumed by waiting to activate the scanning device after reaching the final pose.

[0112] After the robot's end effector reaches the scanning pose or its vicinity, the scanning device can continue scanning and perform small-range pose fine-tuning around the end effector's scanning pose. For example, the robot's end effector can be slightly translated or slightly rotated, causing a limited change in position or orientation of the target barcode near the calculated reading position. Due to actual grasping errors, robot execution errors, slippage of the target object relative to the robot's end effector, and errors in the preset pose data, the actual barcode pose may deviate from the calculated result. Through the above fine-tuning, the target barcode can pass through adjacent readable positions, compensating for the impact of this deviation on scanning.

[0113] If a successful barcode scan is still not achieved after minor pose adjustments, the adjustment range for position or orientation can be expanded, and the target barcode can continue to be read during the adjustment process. If scanning still fails after expanding the adjustment range, the robot's end effector can be controlled to move the target object along a preset, fixed backup scanning trajectory.

[0114] For example, a fixed backup scanning trajectory can allow the target object to pass through the scanning device's reading area in multiple directions, increasing the chances of the target barcode being read. In this way, the normal scanning process can prioritize the end-point scanning pose calculated based on the current target object and grasping status, and then gradually increase the adjustment range if there are calculation deviations or actual reading conditions are not met.

[0115] Upon successful scanning, the barcode content read by the scanning device can be acquired and output, and the product identity corresponding to the target item can be confirmed based on the barcode content. For example, the read barcode content can be compared with the barcode content associated with and saved as candidate item images to verify whether the current target item matches the item obtained from the image matching. Since different items may have similar appearances, there is a possibility of misidentification in image matching results. Further confirmation through barcode content can reduce the problem of incorrectly sorting or shipping items with similar appearances.

[0116] By first calculating the end-scanning pose and then performing proximity scanning, local fine-tuning, and failure recovery, normal movement can be made to directly revolve around the expected position and orientation of the target barcode, reducing invalid movements in fixed dense scanning trajectories. At the same time, adjusting and fixing backup scanning trajectories step by step can compensate for actual pose deviations and improve the scanning success rate for different targets and under different gripping states.

[0117] Based on this, by matching the target object image with preset item images, and combining the pose information corresponding to the candidate item images for weighted fusion and coordinate transformation, the impact of single matching deviations is reduced, and the poses of the target object and the robot end effector are unified to the robot's working coordinate system. The relative pose of the target object is determined based on its 3D pose and the end effector's grasping pose. Then, the target barcode pose is calculated by combining the preset barcode spatial relationships, ensuring that the end effector's scanning pose can still be determined even when the barcode is not visible in the current image. By combining the scanning device's calibration pose, reading conditions, and robot motion constraints, the target barcode moves with the robot end effector to a readable position and orientation, reducing invalid actions in fixed, dense scanning trajectories and shortening scanning time. Simultaneously, continuous scanning, pose fine-tuning, and failure recovery compensation for grasping and execution deviations improve the scanning success rate; the identification of the target object is confirmed using the read barcode content, reducing sorting or outbound errors caused by misidentification due to appearance matching.

[0118] In some alternative implementations, refer to Figure 3The document illustrates a process 300 for determining the relative pose of a barcode according to an embodiment of the present disclosure. The process 300 for determining the relative pose of a barcode may include the following steps 301 to 304: Step 301: Using the two-dimensional images of each item from multiple angles and the corresponding depth map, determine the corresponding three-dimensional coordinate system of the item and the three-dimensional pose of the item in the three-dimensional coordinate system.

[0119] This embodiment can be executed before acquiring the target object image in step 201. Three-dimensional spatial data is established for each preset item, and the established three-dimensional spatial data is associated with and saved with the corresponding item image. Different items may have different shapes, sizes, and barcode placement positions. Therefore, image acquisition and three-dimensional reconstruction can be performed separately for each item, so that the subsequent calculation of the barcode's relative pose has a spatial reference corresponding to that item.

[0120] Each item image can include a two-dimensional image of the item and a corresponding depth map. The two-dimensional image can be a color image representing the item's appearance, and the depth map can be an image recording the surface depth of the item at the corresponding image location. The two-dimensional image and the depth map from the same viewing angle together constitute the corresponding item image. During acquisition, a color depth camera can be used to acquire multiple sets of item images from the front, side, back, and different tilt angles, or an image sequence can be acquired while continuously changing the viewing angle. Each set of two-dimensional images and depth maps can be associated according to the corresponding shooting time and viewing angle, so that the item surface area in the two-dimensional image can correspond to the corresponding depth data.

[0121] Specifically, based on the depth map and pre-calibrated camera parameters, the image positions corresponding to the object's surface can be converted into 3D spatial points, obtaining point clouds of the object's surface from various viewing angles. Then, based on the geometric correspondence of overlapping surfaces under different viewing angles, and combined with the appearance features in the 2D images, the point clouds are registered, converted to a unified reconstruction space, and fused to obtain a 3D model of the object. For example, the front and sides of a boxed product can be observed relatively completely in different images. By fusing the data through the correspondence of overlapping areas, a 3D model of the object containing multiple surfaces can be formed, reducing the spatial data loss caused by a single viewing angle not being able to cover the back or sides of the object.

[0122] Based on a 3D model of an object, a 3D coordinate system bound to the model can be established. The object's 3D coordinate system refers to the coordinate system used to express the position of each spatial point in the object model. It maintains a fixed spatial relationship with the object and is not redefined as the image acquisition direction changes. For example, the same packaged product may have different image representations under different viewing angles, but the same corner point in its model can always be represented using the same object 3D coordinate system.

[0123] The object's 3D coordinate system can be automatically generated during the 3D reconstruction process. For example, the reference origin and reference direction used in the reconstruction can be fixed to the object's 3D model as the origin and coordinate axis directions of the object's 3D coordinate system, without the need for manual specification. Alternatively, the origin can be adjusted to the object's center, centroid, or a preset corner point as needed for data reading or subsequent processing. When adjusting the coordinate reference, the spatial points and associated pose data in the model can be converted simultaneously, ensuring that different data still use a consistent object 3D coordinate system.

[0124] After obtaining the 3D model and coordinate system of the object, the transformation relationship between the image space 3D coordinate system and the object's 3D coordinate system can be determined based on the registration relationship corresponding to each viewing angle during the 3D reconstruction process. The 3D pose of each object image can be used to record this spatial correspondence, and can be represented by a pose matrix containing position and orientation data. Spatial points in the object model have uniform coordinates in the object's 3D coordinate system, but the position and orientation of the same object relative to different image spaces can be different. Therefore, it is necessary to save the pose data corresponding to each object image separately, rather than saving only a pose result that does not distinguish between viewing angles.

[0125] Through the above processing, multi-angle two-dimensional appearance data can be correlated with three-dimensional geometric data, and a unified spatial representation benchmark can be established for the same item. Subsequently, the three-dimensional position and normal vector of the barcode obtained from different item images can be expressed in the same item's three-dimensional coordinate system according to the transformation relationship corresponding to each image, thus providing a basis for comparison and fusion of barcode data obtained from different viewing angles.

[0126] Step 302: Determine the three-dimensional position and normal vector of the barcode in the corresponding image space for each item image.

[0127] Based on the object images obtained from multiple viewing angles in step 301, barcode detection and recognition can be performed in the two-dimensional images of each object image to determine the image region, two-dimensional position, and barcode content of the barcode. Combined with the corresponding depth map, the three-dimensional position and normal vector of the barcode in the corresponding image space can be determined. This processing can be performed when the object data is pre-established, which differs from processing that does not actively detect barcodes in the target object image during runtime.

[0128] The three-dimensional position of the barcode can be its location in the corresponding three-dimensional coordinate system of the image space, which can be represented by the three-dimensional coordinates of the barcode center. The barcode normal vector is the direction vector perpendicular to the local surface on which the barcode is located, used to represent the orientation of that surface. For example, when a barcode is attached to the side of a boxed product, the orientation of that local surface can be used to determine the barcode normal vector, and according to a unified directional convention, the barcode normal vector should point outward from the surface of the item.

[0129] Specifically, the boundaries and center position of the barcode region can be determined based on the barcode detection results in the 2D image. Then, the depth data of the barcode region can be read based on the correspondence between the 2D image and the depth map. Combined with camera parameters, the center of the barcode or multiple image positions within the barcode region can be converted into corresponding 3D spatial points in the image space. If the depth data at the barcode center is valid, the 3D position of the barcode center can be directly determined. If there is a lack of depth at the center, the valid spatial points within the barcode region can be used to fit a local surface, and then the 3D position of the barcode can be determined by combining this with the viewing direction corresponding to the barcode center.

[0130] For the barcode normal vector, a local plane fitting can be performed based on the effective spatial points within the barcode area and adjacent surfaces. The normal direction of the fitted plane is then determined as the barcode normal vector, and normalization is performed.

[0131] In specific examples, when the object has a known regular geometric shape, the geometric constraints of the object's 3D model can be used to optimize the 3D position and normal vector of the barcode. For instance, for a cuboid object, the corresponding plane can be determined based on the point cloud of the face where the barcode is located, and the 3D position of the barcode can be corrected using the constraint that the barcode is located on that plane, while the normal vector of the barcode can be corrected using the normal direction of that plane. Since single depth data may be affected by noise, combining multiple spatial points and the geometric relationships of the object's surface can reduce local errors in position and orientation estimation.

[0132] It should be noted that not all object images viewed from different angles need to contain barcodes. In object images where barcodes are detected, the barcode content, 3D position, and normal vector can be saved. If no barcode is detected or the corresponding depth data is insufficient to determine the barcode's spatial state, the object image can be excluded from the data source for that barcode pose calculation. Through this processing, the barcode region in a 2D image can be converted into spatial position and surface orientation data, providing input for subsequent coordinate transformations.

[0133] Step 303: Using the transformation matrix of the item's 3D pose in the corresponding item 3D coordinate system relative to the image space, the 3D position of the barcode in the corresponding image space and the corresponding barcode normal vector are transformed to the item's 3D coordinate system. When multiple item images have the same barcode, the result of inverting the 3D position of the barcode in the image space and the corresponding barcode normal vector of each item image with the same barcode is determined as the initial barcode relative pose corresponding to the same barcode.

[0134] Based on the barcode's 3D position and normal vector obtained in step 302, the barcode's 3D position and normal vector can be transformed from the corresponding image space to the item's 3D coordinate system by combining the item's 3D pose corresponding to each item image in step 301. Since different item images may be viewed from different positions and orientations, the position and normal vector of the same barcode may differ in different image spaces; after transformation to a unified coordinate system bound to the item, the differences caused by different observation coordinate references can be eliminated.

[0135] Specifically, the transformation matrix of the item's 3D coordinate system relative to the image space's 3D coordinate system can be obtained from the item's 3D pose in the corresponding item image. This transformation matrix is ​​then inverted to obtain the inverse transformation matrix, which transforms the image space to the item's 3D coordinate system. Using this inverse transformation matrix, the barcode's 3D position and normal vector are then transformed to the item's 3D coordinate system.

[0136] Furthermore, for the three-dimensional position of the barcode, the rotation relationship and position offset in the inverse transformation matrix can be used for transformation to obtain the position of the barcode center in the three-dimensional coordinate system of the item. For the barcode normal vector, its direction can be transformed using the rotation relationship in the inverse transformation matrix, without adding the position offset between the coordinate system origin, and the transformed vector is normalized. Thus, the position and orientation of the barcode can be expressed simultaneously in a unified three-dimensional coordinate system of the item.

[0137] For example, a barcode on the side of the same packaged product can appear in two images taken from the front and oblique sides, respectively. Its 3D position and normal vector differ in the corresponding space of the two images. After transformation using the inverse transformation matrix corresponding to each image, both sets of data represent the position and orientation of the barcode relative to the packaged product itself, and therefore should ideally be consistent.

[0138] If the same barcode content is detected in multiple images of the same item, and the detection area corresponds to the same barcode, these item images can be identified as item images with the same barcode. For each item image, the converted barcode position and normal vector can be used to determine an initial barcode relative pose for that barcode.

[0139] The initial barcode relative pose represents the position and orientation of the barcode relative to the three-dimensional coordinate system of the item, calculated from a single item image. Different item images can generate corresponding initial barcode relative poses.

[0140] By converting the data of the same barcode in different image spaces to a unified three-dimensional coordinate system of the item, the relative poses of each initial barcode can have a consistent representation benchmark, thereby supporting subsequent comparison, abnormal data identification and fusion processing.

[0141] Step 304: Fuse the relative poses of each initial barcode corresponding to the same barcode to obtain the corresponding relative pose of the barcode.

[0142] Based on the multiple initial barcode relative poses obtained in step 303, the position and orientation of the same barcode can be fused by combining the calculation results from different viewing angles to obtain the barcode relative pose associated with the corresponding item. Since each initial barcode relative pose is expressed in the same three-dimensional coordinate system of the item, they can be directly compared.

[0143] Specifically, the positional differences and normal vector direction differences between the relative poses of each initial barcode can be compared first. For example, the distance between the center positions of each barcode and the angle between the normal vectors of each barcode can be calculated to determine whether the relative pose of a certain initial barcode deviates significantly from other calculation results. If the difference between the relative pose of a certain initial barcode and other results exceeds a preset allowable range, the corresponding item image can be identified as noise data in the 3D reconstruction or barcode pose calculation, the initial barcode relative pose corresponding to that image can be removed, and the corresponding record can be deleted from the data participating in this fusion to avoid abnormal results affecting the final barcode relative pose.

[0144] Furthermore, for the retained initial relative pose of the barcodes, the position data and normal vector data can be fused separately. For example, the 3D coordinates of the centers of each barcode can be averaged to obtain the fused barcode position; for the normal vectors, their directions can be unified first, and then the normal vectors can be averaged and normalized to obtain the fused barcode normal vector. Unifying the direction of the normal vectors can prevent the normal vectors of the same surface from canceling each other out during fusion due to opposite estimated directions.

[0145] Weighted fusion can also be performed based on the quality of the observation data corresponding to the relative poses of each initial barcode. For example, weights can be determined based on the proportion of effective depth data within the barcode area or the local surface fitting error, so that the calculation result with more effective depth data and smaller surface fitting error has a larger fusion weight. By weighting the barcode position and the barcode normal vector after unified pointing separately, and then normalizing the fused normal vector, the corresponding relative pose of the barcode can be obtained.

[0146] After fusion is complete, the relative pose of the barcode can be associated and saved with the corresponding item number, the item's 3D model, the item's 3D coordinate system, the item image, and the corresponding 3D pose of the item in each image. The recognized barcode content is also saved. The actual size of the barcode can also be determined and saved based on the corresponding 3D spatial points and boundaries of the barcode area. For example, the actual width and height of the barcode area can be determined based on the spatial distance between barcode boundaries for subsequent use in determining the reading distance.

[0147] Since depth measurements and local surface estimations may have biases under different observation angles, fusing multiple valid results can reduce the impact of single observation errors on barcode position and orientation. Combined with outlier removal, it can also reduce the interference of 3D reconstruction bias or barcode region positioning errors on the fusion results. The obtained barcode relative pose is bound to the object itself. When the object's position and orientation change, it can be used as preset data to estimate the barcode's spatial state, so that subsequent scanning processes do not need to rely on the visibility of the barcode in the current target object image.

[0148] Based on this, steps 301 to 304 establish a 3D model of the item and a bound 3D coordinate system by fusing 2D images and depth maps of each item from multiple viewing angles, reducing surface data loss caused by a single viewing angle. Through barcode detection and depth data recovery, the 2D position of the barcode is transformed into a 3D position and normal vector, and measurement errors are reduced by incorporating geometric constraints on the item surface. Utilizing the coordinate transformation relationships corresponding to each item image, barcode data from different viewing angles are unified into the item's 3D coordinate system, eliminating coordinate reference differences and providing a basis for comparing and fusing the relative poses of multiple initial barcodes of the same barcode. Further, through outlier removal and multi-result fusion, single-observation bias is reduced, resulting in a stable barcode relative pose. After this pose is associated with and saved with the item data, the position and orientation of the barcode can be calculated at runtime based on the target object and the grasping state, allowing for the calculation of the end-scan pose even when the barcode is not visible in the current image.

[0149] In some alternative implementations, refer to Figure 2B It illustrates a breakdown flow 2020 of an embodiment of step 202 of the robot scanning method of this disclosure, which includes steps 2021 to 2022: Step 2021: Weight the 3D poses of each candidate item in the corresponding image space to obtain the initial 3D pose of the target item in the corresponding image space.

[0150] Based on the candidate object images selected in step 201 and the matching confidence of each candidate object image, the three-dimensional pose of the object in the corresponding image space can be obtained. The initial three-dimensional pose is obtained by weighted fusion of the matching relationship between the target object image and each candidate object image.

[0151] Among them, the candidate item is the item corresponding to the candidate item image; the initial 3D pose is the position and orientation of the target object in the current image space 3D coordinate system after fusion, which has not yet been transformed to the robot working coordinate system.

[0152] Specifically, based on the image number of each candidate item, the associated 3D pose of the item can be read. Then, based on the corresponding features between the target image and each candidate item image, the 3D pose of each candidate item image can be associated with the current observation state of the target object. Since each candidate item image may be acquired at different observation positions and angles, the 3D poses of the items participating in the fusion need to be unified to the image space corresponding to the current target object image to avoid directly fusing pose data under different observation coordinate references.

[0153] When the target object image contains both a 2D image and a depth map, the 3D spatial correspondence between the target object image and candidate object images can be determined by combining the depth data corresponding to the matching features. Then, the 3D pose of the object corresponding to the candidate object image can be adjusted based on this correspondence. When only a 2D image is acquired, the correspondence between 2D feature points in the current image and 3D feature points in the object model can be established by combining the preset actual object size, the corresponding 3D model, and camera parameters. The corresponding pose data can then be determined through perspective multi-point pose solving. Therefore, the pose information provided by each candidate object image can be used to estimate the position and orientation of the target object in the current image space.

[0154] After establishing a unified representation benchmark for pose data, fusion weights can be determined based on the matching confidence of each candidate object image. Higher matching confidence corresponds to a larger weight for the pose data, while lower matching confidence corresponds to a smaller weight. For example, if the sum of all matching confidences is greater than zero, the ratio of each matching confidence to this sum can be determined as the corresponding fusion weight, ensuring that the sum of all fusion weights is one. This weight allocation allows candidate results with a more reliable correspondence to the currently visible appearance of the target object to contribute more significantly to the fusion result.

[0155] For position data in pose estimation, the position components along the three coordinate axes can be weighted and summed separately to obtain the fused position. For attitude data, each rotation data point can be converted into a unified quaternion representation, and the signs of the quaternions representing the same rotation can be adjusted to have a consistent representation direction. Then, weighted calculations and normalization are performed to obtain effective rotation data. Combining the fused position and rotation data yields the initial 3D pose. When there is only one candidate object image, the fusion weight of its corresponding pose data can be set to one, and the pose unified to the current image space can be determined as the initial 3D pose.

[0156] For example, images of the front and oblique sides of the same packaged item may match the target image, but the front image has a higher matching confidence, allowing its corresponding pose data to be given greater weight. Meanwhile, the pose data from the oblique side image supplements the spatial estimation provided by another viewing angle. Therefore, the fusion result does not simply use the pre-stored pose of a single candidate item image, but rather combines multiple matching results to determine the current position and orientation of the target object.

[0157] The above processing reduces the impact of local occlusion, viewing angle differences, or feature matching deviations on pose estimation of individual candidate object images. The obtained initial 3D pose uses a unified coordinate reference in the current image space and can be used as input for coordinate transformation in step 2022, ensuring that the subsequent target object 3D pose has a clear data source.

[0158] Step 2022: Use the preset camera extrinsic parameters of the robot camera to convert the initial three-dimensional pose to the robot's working coordinate system to obtain the three-dimensional pose of the target object.

[0159] Based on the initial 3D pose obtained in step 2021, the preset camera extrinsic parameters of the robot camera can be obtained. Based on the camera extrinsic parameters, the spatial correspondence between the current image space and the robot working coordinate system can be established, and the position and orientation of the target object can be transformed to the robot working coordinate system to obtain the 3D pose of the target object.

[0160] Camera extrinsic parameters characterize the position and orientation of the robot's camera coordinate system relative to the robot's working coordinate system. These parameters can include rotation and translation parameters, or be represented as a transformation matrix that incorporates rotation and position offset. For example, translation parameters can characterize the position of the origin of the robot's camera coordinate system relative to the origin of the robot's working coordinate system, while rotation parameters can characterize the difference between the coordinate axis directions of the two coordinate systems. When the current 3D coordinate system of the image space is the robot's camera coordinate system, the transformation can be directly performed using the corresponding camera extrinsic parameters.

[0161] Camera extrinsic parameters can be obtained and saved through pre-calibration. For example, a calibrated object with a known spatial position can be acquired, and its corresponding position in the robot camera coordinate system and the robot working coordinate system can be combined to determine the rotational relationship and positional offset between the two coordinate systems. When the robot camera is fixedly mounted relative to the robot base, the corresponding camera extrinsic parameters can be used after installation and calibration. When the robot camera changes position with moving parts on the robot body, the camera extrinsic parameters corresponding to the current image can be determined based on the pre-calibrated installation relationship and the pose of the part when acquiring the target image, thereby avoiding the use of conversion data that is inconsistent with the camera position at the time of image acquisition.

[0162] Specifically, the position components of the initial 3D pose can be converted into the coordinate axis directions of the robot's working coordinate system by first utilizing the rotation relationship in the camera's extrinsic parameters. Then, the position offset between the origins of the two coordinate systems is superimposed to obtain the position of the target object in the robot's working coordinate system. For the attitude in the initial 3D pose, the rotation relationship of the robot camera relative to the robot's working coordinate system can be combined with the rotation relationship of the target object relative to the robot camera to obtain the attitude of the target object in the robot's working coordinate system. This attitude conversion does not superimpose the position offset.

[0163] When using a pose matrix representation, the camera extrinsic parameters can be represented as a transformation matrix from the robot's camera coordinate system to its working coordinate system. The initial 3D pose can be represented as the pose matrix of the target object relative to the robot's camera coordinate system. Multiplying the transformation matrix and the initial 3D pose matrix sequentially yields the pose matrix of the target object relative to the robot's working coordinate system. From the position and rotation data in this matrix, the 3D pose of the target object in the robot's working coordinate system can be determined. If the pre-saved camera extrinsic parameters use the opposite transformation direction, they can be inverted before performing the transformation to maintain a consistent coordinate transformation direction.

[0164] For example, the position of a target object in the robot's camera coordinate system is expressed based on the camera's origin and viewing direction; the spatial position used for robot control is based on the robot's working coordinate system. By combining the camera's position and orientation relative to the robot's working coordinate system, the target object's position observed by the camera can be converted into a spatial position usable by the robot. This process changes the reference for expressing the target object's pose; the target object's actual position and orientation in space remain unchanged due to the coordinate transformation.

[0165] Through the above processing, the initial 3D pose can be converted into a target object 3D pose adapted to robot control, reducing position and orientation deviations caused by differences between the camera coordinate reference and the robot's working coordinate reference. Simultaneously, both the target object's 3D pose and the subsequent end-effector grasping pose are expressed in the robot's working coordinate system, allowing for the calculation of the relative spatial relationship between the target object and the robot's end effector, providing a consistent spatial data foundation for subsequent barcode pose conversion and end-effector scanning pose determination.

[0166] Based on this, steps 2021 to 2022 unify the candidate pose data into the current image space by combining the matching relationship between the target object image and the candidate object images, and assign fusion weights according to the matching confidence, so that the more reliable matching results contribute more to the initial 3D pose, reducing the impact of single viewing angle, local occlusion, and feature matching bias on pose estimation. By fusing position and orientation data separately, the spatial position and orientation of the target object can be preserved simultaneously. Furthermore, by utilizing the camera extrinsic parameters of the robot camera, the initial 3D pose is transformed to the robot's working coordinate system, eliminating the coordinate reference difference between camera observation and robot control, allowing the 3D pose of the target object and the end-effector grasping pose to participate in calculations under a unified coordinate system. Thus, the spatial state of the target object can still be obtained when the target barcode is not currently visible, providing a continuous data foundation for relative pose calculation after grasping and end-effector scanning pose determination.

[0167] In some alternative implementations, refer to Figure 2C It illustrates a breakdown flow 2040 of an embodiment of step 204 of the robot scanning method of this disclosure, which includes steps 2041 to 2042: Step 2041: Divide the target object pose matrix, which represents the three-dimensional pose of the target object, by the grasping pose matrix, which represents the end-grasping pose, to obtain the target object relative pose matrix.

[0168] Based on the target object's 3D pose and the end effector's grasping pose obtained from the aforementioned steps, the target object pose matrix and the grasping pose matrix can be established respectively. Since both use the robot's working coordinate system as the representation reference, the common coordinate reference can be eliminated through matrix operations to determine the position and orientation of the target object relative to the robot's end effector during grasping.

[0169] The target pose matrix can also represent the spatial transformation relationship between the target's 3D coordinate system and the robot's working coordinate system, and the grasping pose matrix can also represent the spatial transformation relationship between the end effector coordinate system and the robot's working coordinate system. Both matrices can be represented using a 4x4 homogeneous transformation matrix containing rotation and translation data. The target relative pose matrix represents the spatial transformation relationship between the target's 3D coordinate system and the end effector coordinate system, converting spatial points in the target's 3D coordinate system into corresponding spatial points in the end effector coordinate system.

[0170] Specifically, the target object pose matrix can be established based on the three-dimensional pose of the target object determined before grasping, while the grasping pose matrix can be established based on the end-effector grasping pose adopted by the robot during grasping. When the target object does not move between acquiring its three-dimensional pose and performing the grasping, both matrices can jointly represent the spatial state of the target object and the robot's end effector during grasping, avoiding the need to calculate relative relationships using pose data from different object states.

[0171] Furthermore, the coordinate transformation can be represented by dividing the target pose matrix by the grasping pose matrix, thus using the inverse of the grasping pose matrix. When spatial points are represented by column vectors, and the pose matrix represents the transformation from the object's own coordinate system to the robot's working coordinate system, the grasping pose matrix can be inverted first to obtain the inverse transformation matrix from the robot's working coordinate system to the end effector coordinate system. This inverse transformation matrix is ​​then multiplied left by the target pose matrix to obtain the target's relative pose matrix. This operational sequence sequentially completes the transformation from the target's 3D coordinate system to the robot's working coordinate system, and then to the end effector coordinate system.

[0172] For relative position calculation, the position difference between the target object and the robot end effector in the robot's working coordinate system can be calculated first. Then, using the inverse rotation relationship in the grasping pose matrix, this position difference can be converted into position components in the end effector coordinate system. Furthermore, the inverse rotation relationship corresponding to the robot end effector's orientation can be combined with the rotation relationship corresponding to the target object's orientation to obtain the rotation data of the target object relative to the robot end effector. Therefore, relative pose calculation considers both position offset and orientation difference, rather than simply comparing the position coordinates of the two objects.

[0173] For example, when the robot's end effector picks up a target object from an off-center position on the top of a boxed product, the center of the target object may simultaneously have a downward and lateral positional offset relative to the origin of the end effector's coordinate system. Furthermore, when the robot's end effector is tilted towards the target object, there may be orientation differences. These offsets and differences can be uniformly recorded using the target object's relative pose matrix, enabling subsequent processing to calculate the spatial relationship between the target object and the robot's end effector based on the actual gripping position and orientation used in this case.

[0174] Through the above calculations, the poses of the target object and the robot end effector in the robot's working coordinate system can be converted into relative poses between them, preserving the influence of different grasping positions and orientations on the connection relationship. The resulting target object relative pose matrix can serve as the data basis for determining the target object's relative pose after grasping in the next step, and provide a basis for subsequently establishing the spatial correspondence between the robot end effector's motion and the target object's motion.

[0175] Step 2042: In response to determining that the relative pose of the target object is consistent when the robot end-effector grasps the target object and after the robot end-effector grasps the target object, the relative pose of the target object when the robot end-effector grasps the target object is determined as the relative pose of the target object after the robot end-effector grasps the target object.

[0176] Based on the target object's relative pose matrix obtained in step 2041, the target object's relative pose calculated during grasping can be used as the target object's position and orientation in the end-effector coordinate system after grasping, provided that the robot end-effector stably grasps the target object and the target object does not undergo displacement or rotation relative to the robot end-effector. In this case, the target object moves together with the robot end-effector, and the spatial relationship between them does not change due to changes in the position and orientation of the robot end-effector in the robot's workspace.

[0177] Specifically, after the robot's end effector establishes a stable connection with the target object through gripping or adsorption, the positional offset and orientation difference of the target object relative to the robot's end effector can be treated as fixed data based on this connection. For example, when the gripper remains in a gripping state and the target object does not slide within the gripping area, the position and orientation of the target object relative to the gripper remain unchanged; when the suction component maintains effective adsorption and the target object does not rotate relative to the suction component, the relative pose of the target object determined during gripping can also be used. In practical applications, existing operational data such as the gripper's closed state, gripping feedback, or adsorption pressure can be combined to determine whether the gripping connection remains effective. Based on stable gripping, the calculation assumption of maintaining a consistent relative pose is adopted, eliminating the need to reacquire the target object's pose after each robot end effector movement.

[0178] For example, after the robot's end effector picks up a boxed product and moves towards the barcode scanner, the position and orientation of the boxed product in the robot's working coordinate system change accordingly. However, the distance of its center relative to the origin of the end effector's coordinate system, its lateral offset, and its relative orientation can remain unchanged. Therefore, the target object's relative pose matrix obtained in step 2041 can be retained, and this matrix can be used to describe the fixed spatial relationship between the target object and the robot's end effector when the robot's end effector pose changes.

[0179] The above processing is based on the assumption that the relative relationship after grasping is consistent with the relative relationship calculated during grasping. In actual work, visual pose estimation errors before grasping, robot execution errors, and slight slippage of the target object after grasping may cause the actual spatial relationship to deviate from the calculated result. Such deviations can be compensated for in the subsequent fine-tuning process near the end-effector scanning pose.

[0180] By utilizing the relative pose of the target object under stable grasping conditions, the need for repeated observations and relative pose calculations after grasping can be reduced, and a clear spatial correspondence can be maintained between the subsequent motion of the robot's end effector and the motion of the target object. Combined with the pre-established spatial relationship of the barcode on the target object, the pose of the target barcode relative to the robot's end effector can be further determined, thereby supporting subsequent barcode pose conversion and end-effector scanning pose calculation.

[0181] Based on this, steps 2041 to 2042 combine the target object pose matrix and the grasping pose matrix, and use inverse matrix operations to transform the position and orientation of the target object in the robot's working coordinate system to the end effector coordinate system, obtaining a target object relative pose matrix that can simultaneously reflect positional offset and orientation differences, thus expressing the spatial relationship formed by different grasping positions and orientations. When the target object is stably grasped and does not slip or rotate relative to the robot's end effector, the relative pose of the target object determined during grasping can be used, reducing the need for repeated observation and calculation after the robot's end effector moves. Since this relative relationship remains unchanged with stable grasping, a clear correlation can be established between the movement of the robot's end effector and the movement of the target object and target barcode, providing a basis for barcode pose conversion and end-effector scanning pose inverse calculation, and adapting subsequent scanning actions to the current grasping state, reducing barcode presentation errors caused by ignoring grasping offsets.

[0182] In some optional embodiments of step 205, a transformation relationship between the end-effector coordinate system and the robot working coordinate system can be established by combining the target object's relative pose, the target object's three-dimensional pose, and the robot's end-effector pose. This transformation relationship is then used to convert the first target barcode relative pose in the end-effector coordinate system into the second target barcode relative pose in the robot working coordinate system. This process converts the spatial state of the target barcode relative to the robot end-effector into the spatial state of the target barcode within the robot's working space.

[0183] Specifically, the relative pose of the target object represents its position and orientation in the end-effector coordinate system, but it does not directly represent the relationship between the end-effector coordinate system and the robot's working coordinate system. Therefore, a correspondence between the two coordinate systems can be established first using the target object's 3D pose and its relative pose. When using column vectors to represent spatial points, the relative pose matrix can be inverted, and then the target pose matrix can be multiplied sequentially by this inverse matrix to obtain the transformation matrix between the end-effector coordinate system and the robot's working coordinate system. This transformation matrix corresponds to the robot's end-effector pose in the robot's working coordinate system at the same moment.

[0184] When the coordinate relationship is represented by a matrix transforming the robot's working coordinate system to its end effector coordinate system, the matrix can be inverted. This inversion operation determines the reverse relationship of the coordinate transformation, resulting in a transformation matrix from the end effector coordinate system to the robot's working coordinate system. This transformation matrix is ​​then used to transform the relative pose of the first target barcode. The transformation direction and multiplication order of the matrix can be determined according to a unified pose representation convention, avoiding transformation errors caused by different coordinate relationship directions.

[0185] For the barcode center position in the relative pose of the first target barcode, its coordinate orientation can be changed first according to the rotation relationship in the transformation matrix, and then the position of the origin of the end-effector coordinate system in the robot's working coordinate system can be superimposed to obtain the three-dimensional position of the barcode center in the robot's working coordinate system. For the barcode normal vector, its orientation can be transformed using only the rotation relationship, and the transformed vector can be normalized without superimposing the positional offset between the coordinate system origins. Combining the transformed barcode position and orientation yields the relative pose of the second target barcode.

[0186] For example, when the target barcode is located on the side of the boxed product being grasped, the first target barcode relative pose records the position of the barcode center relative to the robot end effector and the orientation of the barcode surface. Combining the position and orientation of the robot end effector in the robot's working coordinate system, the current position and orientation of the side containing the barcode in the workspace can be determined. If the robot end effector moves after grasping the product, the second target barcode relative pose can be updated using a transformation relationship corresponding to the current end effector pose, while the first target barcode relative pose remains unchanged under stable grasping conditions.

[0187] Through the above processing, the relative relationship between the target barcode and the robot end effector can be converted into the barcode pose in the robot's working coordinate system, so that the target barcode pose and the calibration pose of the scanning device have a consistent expression benchmark. Since the calculation uses the pre-established barcode spatial relationship and the relative relationship determined after grasping, the conversion can still be completed when the target barcode does not appear in the current target object image, providing barcode position and orientation data for subsequent end effector scanning pose calculation.

[0188] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a robot barcode scanning device, which is similar to... Figure 2A Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0189] like Figure 4As shown, the robot barcode scanning device 400 in this embodiment includes: a data acquisition module 401, a target object three-dimensional pose determination module 402, a gripping module 403, a target object relative pose determination module 404, a target barcode relative pose determination module 405, an end-effector scanning pose determination module 406, and a scanning module 407. The acquisition module 401 is configured to acquire the target object image and select at least one candidate object image that matches the target object image from a plurality of preset object images corresponding to each object. The target object 3D pose determination module 402 is configured to perform weighted fusion of the preset 3D poses of the objects in the corresponding image space of each candidate object image and then convert them to a preset robot working coordinate system to obtain the target object 3D pose in the robot working coordinate system. The grasping module 403 is configured to determine the end-effector grasping pose in the robot working coordinate system when the robot end-effector grasps the target object based on the geometric features of the target object, and to control the robot end-effector to grasp the target object according to the end-effector grasping pose. The target object relative pose determination module 404 is configured to determine the target object relative pose in the end coordinate system of the robot end after the robot end grasps the target object using the three-dimensional pose of the target object and the end-effector grasping pose. The target barcode relative pose determination module 405 is configured to use the target object relative pose to convert the first target barcode relative pose in the end coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose. The end-effector scanning pose determination module 406 is configured to determine the end-effector scanning pose corresponding to the preset calibration pose in the robot working coordinate system. The end-effector scanning pose represents the pose of the robot end in the robot working coordinate system when the robot end moves the target barcode from the relative pose of the second target barcode to the pose corresponding to the calibration pose. The barcode scanning module 407 is configured to control the robot end effector to move to the end-scanning pose in order to scan the target barcode.

[0190] In this embodiment, the specific processing of the acquisition module 401, the target object 3D pose determination module 402, the grasping module 403, the target object relative pose determination module 404, the target barcode relative pose determination module 405, the end-effector scanning pose determination module 406, and the scanning module 407 of the robot barcode scanning device 400, and the resulting technical effects can be found in the following references: Figure 2A The relevant descriptions of steps 201, 202, 203, 204, 205, 206 and 207 in the corresponding embodiments will not be repeated here.

[0191] In some alternative implementations, each item image includes a corresponding two-dimensional image and a corresponding depth map.

[0192] Accordingly, the robot scanning device also includes a preprocessing module 408, which is configured to: Using two-dimensional images of each item from multiple angles and the corresponding depth map, the corresponding three-dimensional coordinate system of the item and the three-dimensional pose of the item in the three-dimensional coordinate system are determined. There is a preset transformation relationship between the three-dimensional coordinate system of the image space of each item and the corresponding three-dimensional coordinate system of the item. Determine the 3D position and normal vector of the barcode in the corresponding image space for each item image; The three-dimensional position and normal vector of each barcode are transformed into the corresponding three-dimensional coordinate system of the item and fused to obtain the relative pose of the barcode.

[0193] In some optional implementations, the three-dimensional position of each barcode and its corresponding normal vector are transformed to the corresponding three-dimensional coordinate system of the item and fused to obtain the corresponding barcode relative pose, including: By using the 3D pose of each item in its corresponding 3D coordinate system and the corresponding 3D coordinate system, the 3D position of the barcode in the corresponding image space and the corresponding barcode normal vector are inverted to obtain the relative pose of the barcode in the corresponding 3D coordinate system.

[0194] In some optional implementations, after inverting the barcode's 3D position and the corresponding barcode normal vector in the corresponding image space using the item's 3D pose in the corresponding item 3D coordinate system and the corresponding item 3D coordinate system, the preprocessing module 408 further performs: When multiple item images have the same barcode, the result of inverting the three-dimensional position of the barcode in the image space corresponding to each item image with the same barcode and the corresponding barcode normal vector is determined as the initial barcode relative pose corresponding to the same barcode. The relative poses of the initial barcodes corresponding to the same barcode are fused to obtain the corresponding relative pose of the barcode.

[0195] In some alternative implementations, the acquisition module 401 is further configured as follows: Use a pre-set robot camera to capture images of the target object; Accordingly, the target object three-dimensional pose determination module 402 is also specifically configured as follows: The 3D poses of each candidate item in the corresponding image space are weighted to obtain the initial 3D pose of the target item in the corresponding image space. The initial 3D pose is transformed into the robot's working coordinate system using the preset camera extrinsic parameters of the robot camera, thus obtaining the 3D pose of the target object.

[0196] In some optional implementations, the target object relative pose determination module 404 is further configured as follows: By dividing the target object pose matrix, which represents the three-dimensional pose of the target object, by the grasping pose matrix, which represents the grasping pose of the end effector, the relative pose matrix of the target object is obtained. The relative pose matrix of the target object represents the relative pose of the target object in the end effector coordinate system when the robot's end effector grasps the target object. In response to ensuring that the relative pose of the target object is consistent between when the robot end-effector grasps the target object and after the robot end-effector grasps the target object, the relative pose of the target object when the robot end-effector grasps the target object is determined as the relative pose of the target object after the robot end-effector grasps the target object.

[0197] In some optional implementations, the target barcode relative pose determination module 405 is further specifically configured to perform the following steps before converting the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose: Given that each candidate item image with a barcode corresponds to the same barcode relative pose, the same barcode corresponding to each candidate item image is determined as the target barcode, and the corresponding barcode relative pose is transformed to the end coordinate system to obtain the first target barcode relative pose.

[0198] In some optional implementations, the target barcode relative pose determination module 405 is further configured as follows: Using the relative pose of the target object in the end-effector coordinate system, the inverse operation is performed on the first target barcode relative pose in the end-effector coordinate system to obtain the transformation matrix of the end-effector coordinate system relative to the robot working coordinate system. The relative pose of the target barcode and the second target barcode in the robot's working coordinate system is obtained by using a transformation matrix.

[0199] In some optional implementations, the calibration pose is a preset pose of the barcode scanning device in the robot's working coordinate system; Accordingly, the end-effector scanning pose determination module 406 is also specifically configured as follows: Based on the pose of the scanning device in the robot's working coordinate system, and provided that the center of the target barcode is within the preset reading area of ​​the scanning device, and the angle between the barcode normal vector corresponding to the target barcode and the optical axis of the scanning device is within a preset angle range, the pose of the robot end effector grasping the target object in the robot's working coordinate system is determined as the end effector scanning pose.

[0200] The following is for reference. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device 500 suitable for implementing embodiments of this application is shown. Figure 5 The electronic device 500 shown is merely an example and should not be construed as a limitation on the functionality and scope of use of the embodiments of this application.

[0201] like Figure 5 As shown, the electronic device 500 may include a processing unit 501 and a memory. The memory may include a read-only memory (ROM) 502, a random access memory (RAM) 503, and / or a storage device 508. The processing unit 501 may include at least one of a central processing unit, a graphics processing unit, a neural network processor, a tensor processor, a digital signal processor, or other processors with data processing capabilities. The processing unit 501 may perform various appropriate actions and processes according to a program stored in the ROM 502 or a program loaded from the storage device 508 into the RAM 503.

[0202] The RAM 503 can also store various programs and data required for the operation of the electronic device 500. The processing device 501, ROM 502, and RAM 503 can be connected to each other via the bus 504. The input / output interface 505 can also be connected to the bus 504.

[0203] In some embodiments, the following devices may be connected to the input / output interface 505: an input device 506, an output device 507, a storage device 508, and a communication device 509. The input device 506 may include at least one of a touchscreen, touchpad, keyboard, mouse, camera, microphone, etc.; the output device 507 may include at least one of a display screen, speaker, vibrator, etc.; the storage device 508 may include a hard disk, solid-state drive, flash memory, or other storage device; and the communication device 509 may be used to enable the electronic device 500 to exchange data with other devices via wired or wireless communication. It should be understood that... Figure 5 The devices shown are merely examples. In actual implementation, electronic device 500 may include more or fewer devices than those shown, and this application does not limit this.

[0204] In some embodiments, the memory may store a computer program. When the computer program is executed by the processing device 501, the processing device 501 may perform actions such as... Figure 2A The robot scanning method shown.

[0205] For example, when the electronic device 500 is used to perform the robot scanning method, the processing device 501 can perform the following processing: acquire the target object image, and select at least one candidate object image that matches the target object image from a plurality of preset object images corresponding to each object; perform weighted fusion of the preset three-dimensional poses of the objects in each candidate object image in the corresponding image space and transform them to a preset robot working coordinate system to obtain the target object's three-dimensional pose in the robot working coordinate system; determine the end-effector grasping pose in the robot working coordinate system when the robot end-effector grasps the target object based on the geometric features of the target object, and control the robot end-effector to grasp the target object according to the end-effector grasping pose. Using the target object's 3D pose and the end-effector's grasping pose, determine the target object's relative pose in the robot's end-effector coordinate system after the robot's end-effector grasps the target object; using the target object's relative pose, transform the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot's working coordinate system; determine the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system, where the end-effector scanning pose represents the robot's end-effector's pose in the robot's working coordinate system when it moves the target barcode from the second target barcode relative pose to the pose corresponding to the calibration pose; control the robot's end-effector to move to the end-effector scanning pose to scan the target barcode.

[0206] In some embodiments, the electronic device 500 may be a server, a terminal device, a server cluster, a cloud computing device, an edge computing device, or other devices with computing and storage capabilities. When the electronic device 500 is used to control a robot to scan barcodes, the electronic device 500 may be a server, a server cluster, or a cloud computing device; when the electronic device 500 is used to control the movement of a robot, the electronic device 500 may be a terminal device, a server, an edge computing device, or a device system composed of a terminal device and a server working together. This application does not limit this.

[0207] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program causes the processor to perform actions such as... Figure 2A The robot scanning method shown.

[0208] In some embodiments, a computer-readable storage medium may include, but is not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, or any combination thereof. For example, a computer-readable storage medium may include random access memory, read-only memory, flash memory, hard disk, solid-state drive, optical disk, mobile storage device, etc. A computer-readable storage medium may be a storage medium included in an electronic device, or it may be a standalone storage medium that can be read by an electronic device.

[0209] This application also provides a computer program product, including a computer program. When executed by a processor, the computer program causes the processor to perform actions such as... Figure 2A The robot scanning method shown.

[0210] In some embodiments, the computer program may be written using one or more programming languages ​​or a combination thereof. The programming languages ​​may include object-oriented programming languages ​​or procedural programming languages, and this application does not limit the specific programming languages ​​used. The computer program may be executed on a terminal device, on a server, or jointly executed by a terminal device and a server.

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0212] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself.

[0213] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A robot barcode scanning method, characterized in that, include: Acquire an image of the target object, and select at least one candidate item image that matches the target object image from multiple preset item images corresponding to each item; The three-dimensional poses of the objects in each candidate object image are weighted and fused in the corresponding image space and then transformed to the preset robot working coordinate system to obtain the three-dimensional pose of the target object in the robot working coordinate system. When the robot end effector grasps the target object based on the geometric features of the target object, the robot end effector grasping pose in the robot working coordinate system is determined, and the robot end effector grasps the target object according to the grasping pose. Using the three-dimensional pose of the target object and the grasping pose of the end effector, the relative pose of the target object in the end effector coordinate system of the robot end effector after the robot end effector grasps the target object is determined; Using the relative pose of the target object, the first relative pose of the preset target barcode on the target object in the end coordinate system is transformed to the second relative pose of the target barcode in the robot working coordinate system; Determine the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system. The end-effector scanning pose represents the pose of the robot end in the robot's working coordinate system when the robot end moves the target barcode from the second target barcode relative pose to the pose corresponding to the calibration pose. The robot end effector is controlled to move to the end-scanning pose in order to scan the target barcode.

2. The robot barcode scanning method according to claim 1, characterized in that, Each of the item images includes a corresponding two-dimensional image and a corresponding depth map; Before acquiring the target image, the method further includes: Using two-dimensional images of each item from multiple angles and corresponding depth maps, the corresponding three-dimensional coordinate system of the item and the three-dimensional pose of the item in the three-dimensional coordinate system are determined. There is a preset transformation relationship between the three-dimensional coordinate system of the image space corresponding to each item and the corresponding three-dimensional coordinate system of the item. Determine the 3D position and normal vector of the barcode in the corresponding image space for each item image; The three-dimensional position and normal vector of each barcode are transformed into the corresponding three-dimensional coordinate system of the item and fused to obtain the relative pose of the barcode.

3. The robot barcode scanning method according to claim 2, characterized in that, The process of transforming the three-dimensional position of each barcode and its corresponding normal vector to the corresponding three-dimensional coordinate system of the item and fusing them to obtain the corresponding barcode relative pose includes: By using the transformation matrix of the item's 3D pose in the corresponding item 3D coordinate system relative to the image space, the 3D position of the barcode in the corresponding image space and the corresponding barcode normal vector are transformed to the item 3D coordinate system, thereby determining the barcode's relative pose in the corresponding item 3D coordinate system.

4. The robot barcode scanning method according to claim 3, characterized in that, After transforming the barcode's 3D position and corresponding barcode normal vector from the image space to the object's 3D coordinate system using a transformation matrix of the object's 3D pose in the corresponding object's 3D coordinate system relative to the image space, the method further includes: When multiple item images have the same barcode, the result of inverting the three-dimensional position of the barcode in the image space corresponding to each item image with the same barcode and the corresponding barcode normal vector is determined as the initial barcode relative pose corresponding to the same barcode. The relative poses of the initial barcodes corresponding to the same barcode are fused to obtain the corresponding relative pose of the barcode.

5. The robot barcode scanning method according to claim 2, characterized in that, The target object image acquired includes: Images of the target object are captured using a pre-set robot camera; The step of weightedly fusing the preset 3D poses of the items in each candidate item image in the corresponding image space and then converting them to a preset robot working coordinate system to obtain the 3D pose of the target object in the robot working coordinate system includes: The three-dimensional poses of each candidate item in the corresponding image space are weighted to obtain the initial three-dimensional pose of the target object in the corresponding image space. The initial three-dimensional pose is transformed into the robot's working coordinate system using the preset camera extrinsic parameters of the robot camera to obtain the three-dimensional pose of the target object.

6. The robot barcode scanning method according to claim 1, characterized in that, The step of determining the relative pose of the target object in the end-effector coordinate system after the robot end-effector grasps the target object using the three-dimensional pose of the target object and the grasping pose of the end-effector includes: By dividing the target object pose matrix representing the three-dimensional pose of the target object by the grasping pose matrix representing the end-effector grasping pose, a target object relative pose matrix is ​​obtained. The target object relative pose matrix represents the target object relative pose in the end-effector coordinate system when the robot end-effector grasps the target object. In response to determining that the relative pose of the target object is consistent when the robot end-effector grasps the target object and after the robot end-effector grasps the target object, the relative pose of the target object when the robot end-effector grasps the target object is determined as the relative pose of the target object after the robot end-effector grasps the target object.

7. The robot barcode scanning method according to claim 5, characterized in that, Before converting the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot working coordinate system using the target object relative pose, the method further includes: When each candidate item image with a barcode corresponds to the same barcode relative pose, the same barcode corresponding to each candidate item image is determined as the target barcode, and the corresponding barcode relative pose is transformed to the end coordinate system to obtain the first target barcode relative pose.

8. The robot barcode scanning method according to claim 7, characterized in that, The step of converting the relative pose of a pre-set target barcode on the target object in the first target barcode relative pose in the end-effector coordinate system to the second target barcode relative pose in the robot working coordinate system using the relative pose of the target object includes: Using the relative pose of the target object in the end-effector coordinate system, the first relative pose of the target barcode in the end-effector coordinate system is inverted to obtain the transformation matrix of the end-effector coordinate system relative to the robot working coordinate system. Using the transformation matrix, the relative pose of the first target barcode is transformed to the relative pose of the second target barcode in the robot's working coordinate system.

9. The robot barcode scanning method according to claim 1, characterized in that, The calibration pose is a preset pose of the barcode scanning device in the robot's working coordinate system; Determining the end-effector scanning pose corresponding to the preset calibration pose in the robot's working coordinate system includes: Based on the pose of the scanning device in the robot's working coordinate system, and provided that the center of the target barcode is within the preset reading area of ​​the scanning device, and the angle between the barcode normal vector corresponding to the target barcode and the optical axis of the scanning device is within a preset angle range, the pose of the robot end effector grasping the target object in the robot's working coordinate system is determined as the end effector scanning pose.

10. A robot barcode scanning device, characterized in that, include: The acquisition module is configured to acquire the target object image and select at least one candidate object image that matches the target object image from a plurality of preset object images corresponding to each object. The target object 3D pose determination module is configured to perform weighted fusion of the preset 3D poses of the objects in the corresponding image space of each candidate object image and then convert them to a preset robot working coordinate system to obtain the target object 3D pose in the robot working coordinate system. The grasping module is configured to determine the end-effector grasping pose in the robot working coordinate system when the robot end-effector grasps the target object based on the geometric features of the target object, and to control the robot end-effector to grasp the target object according to the end-effector grasping pose; The target object relative pose determination module is configured to use the three-dimensional pose of the target object and the end-effector grasping pose to determine the target object relative pose in the end-effector coordinate system of the robot end-effector after the robot end-effector grasps the target object. The target barcode relative pose determination module is configured to use the target object relative pose to convert the preset target barcode relative pose on the target object into a first target barcode relative pose in the end coordinate system to a second target barcode relative pose in the robot working coordinate system; The end-effector scanning pose determination module is configured to determine the end-effector scanning pose corresponding to the preset calibration pose in the robot working coordinate system. The end-effector scanning pose represents the pose of the robot end in the robot working coordinate system when the robot end moves the target barcode from the second target barcode relative pose to the pose corresponding to the calibration pose. The barcode scanning module is configured to control the robot end effector to move to the end-scanning pose in order to scan the target barcode.

11. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes computer program instructions, wherein when the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 9.