Express sheet identification method and system, electronic equipment and computer readable storage medium

Abstract

The invention discloses an express sheet recognition method and system, electronic equipment and a computer readable storage medium, the express sheet recognition method is applied to a sorting robot comprising an end execution tool, and the express sheet recognition method comprises the steps that an initial image collected by a point location collection device matched with the sorting robot is acquired, a target object in the initial image is determined, and an express sheet area is detected; when the surface sheet area is not detected in the initial image, obtaining an object point cloud of the target object, and based on the object point cloud, controlling the end execution tool to move the target object and rotate the target object in the view of the point location acquisition device according to a preset track; when the surface sheet area is detected in the visual field in the rotating process of the target object, the preset track is interrupted, a reference surface single-point cloud of the surface sheet area is obtained, and a reference observation pose of the observation surface single area is obtained based on the reference surface single-point cloud; and adjusting the position of the target object by using the reference observation pose and the point location observation pose matched with the point location acquisition device until the express sheet information is identified. According to the scheme, the express sheet identification efficiency and accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of information recognition technology, and in particular to a waybill recognition method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of the logistics industry, accurately identifying waybill information has become a crucial issue. Existing technologies, besides traditional manual sorting, include methods that use multiple data acquisition devices at various locations to capture images from multiple angles for identification. However, these methods still struggle to guarantee the efficiency of waybill identification, and their accuracy cannot be guaranteed when the data acquisition devices at multiple locations cannot acquire a complete waybill. Therefore, improving the efficiency and accuracy of waybill identification has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a waybill recognition method, system, electronic device, and computer-readable storage medium that can improve the efficiency and accuracy of waybill recognition.

[0004] To address the aforementioned technical problems, this application provides a waybill recognition method applied to a sorting robot including an end effector. The waybill recognition method includes: acquiring an initial image captured by a point acquisition device matched to the sorting robot; determining a target object in the initial image and detecting a waybill region; in response to the absence of a waybill region detected in the initial image, acquiring an object point cloud of the target object; based on the object point cloud, controlling the end effector to move the target object and rotate the target object within the field of view of the point acquisition device according to a preset trajectory; in response to the detection of the waybill region within the field of view during the rotation of the target object, interrupting the preset trajectory and acquiring a reference waybill point cloud of the waybill region; based on the reference waybill point cloud, obtaining a reference observation pose for observing the waybill region; and adjusting the position of the target object using the reference observation pose and the point observation pose matched by the point acquisition device until the waybill information of the waybill region is recognized.

[0005] To address the aforementioned technical problems, a second aspect of this application provides a waybill recognition system applied to a sorting robot including an end effector. The waybill recognition system comprises: a detection module, an execution module, an observation pose calculation module, and a closed-loop feedback module. The detection module acquires an initial image from a point acquisition device matched to the sorting robot, identifies the target object in the initial image, and detects the waybill area. The execution module, in response to the absence of the waybill area in the initial image, acquires a point cloud of the target object and, based on the point cloud, controls the end effector to move the target object and rotate it within the field of view of the point acquisition device along a preset trajectory. The observation pose calculation module, in response to the detection of the waybill area within the field of view during the rotation of the target object, interrupts the preset trajectory and acquires a reference waybill point cloud of the waybill area, obtaining a reference observation pose for observing the waybill area based on the reference waybill point cloud. The closed-loop feedback module uses the reference observation pose and the point observation pose matched by the point acquisition device to adjust the position of the target object until the waybill information in the waybill area is recognized.

[0006] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0007] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0008] The beneficial effects of this application are as follows: Unlike existing technologies, this application acquires the initial image collected by the point acquisition device matched to the sorting robot, identifies the target object from the initial image, and detects whether the target object currently includes a shipping label area. When no shipping label area is detected in the initial image, the application acquires the object point cloud corresponding to the target object. Based on the object point cloud, the application controls the end effector to operate on the target object to move it, and controls the end effector to rotate the target object within the field of view of the point acquisition device according to a preset trajectory. This adjusts the position of the target object within the field of view of the point acquisition device, thereby increasing the probability of detecting the shipping label area within the field of view. When a shipping label area is detected within the field of view of the point acquisition device, the application interrupts the preset trajectory and acquires the reference shipping label point cloud corresponding to the shipping label area. Based on the reference shipping label point cloud, the application determines the reference observation pose that can observe the shipping label area, thus clarifying the pose required to observe the shipping label area. By utilizing the reference observation pose that can observe the target object and the point observation pose that matches the point acquisition device, the position of the target object is adjusted so that the dynamically changing reference observation pose moves closer to the fixed point observation pose. This allows for the acquisition of a more complete and high-precision label area within the field of view of the point acquisition device. Through continuous adjustment and detection, the label information of the label area is identified, improving the flexibility of label recognition, reducing the probability of label recognition failure, and increasing the efficiency and accuracy of label recognition. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one implementation method of the waybill recognition method of this application; Figure 2 This is a flowchart illustrating another embodiment of the waybill recognition method of this application; Figure 3 This is a schematic diagram of one embodiment of the waybill recognition system of this application; Figure 4 This is a schematic diagram of another embodiment of the waybill recognition system of this application; Figure 5 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0012] The label recognition method provided in this application is applied to a sorting robot, which includes an end effector and a corresponding execution subject that is a processing unit capable of data processing. The processing unit is integrated into the robot or exists independently of the robot and interacts with the robot for data.

[0013] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the waybill recognition method of this application. The method includes: S101: Obtain the initial image collected by the point acquisition device matched by the sorting robot, determine the target object in the initial image and detect the label area.

[0014] Specifically, the initial image is acquired by the point acquisition device matched with the sorting robot, the target object is identified from the initial image, and it is detected whether the target object currently includes a label area.

[0015] It should be noted that the location acquisition device matched with the sorting robot can be set in the head area of ​​the sorting robot, or at a preset location in the target scene where the sorting robot is located. This application does not impose any specific restrictions on this.

[0016] In one embodiment, an initial image is acquired by the point acquisition device matched by the sorting robot, the initial image is semantically segmented, the image region whose semantic segmentation result matches the target object is determined, the image region is detected by label detection, and the label region is determined to be detected in the initial image.

[0017] In one embodiment, an initial image is acquired by a point acquisition device matched with the sorting robot, target detection is performed on the initial image to determine the target object in the initial image, image segmentation is performed on the target object, and it is determined whether at least a portion of the shipping label area is obtained through segmentation, thereby determining whether the shipping label area is detected in the initial image.

[0018] In some implementation scenarios, initial images are acquired by a point acquisition device on the head area of ​​the sorting robot or at preset points within the target scene. A pre-trained region detection model is then used to detect target objects and their shipping label areas within the initial images. Specifically, the region detection model is used for semantic segmentation and shipping label detection of the initial images.

[0019] In some implementation scenarios, initial images are acquired by a point acquisition device on the head area of ​​the sorting robot and another at preset points within the target scene. A pre-trained target detection model is then used to detect target objects in the initial images and the label areas on the target objects. The detection model is used for target detection and image segmentation of the initial images.

[0020] S102: In response to the absence of a surface area detected in the initial image, acquire the object point cloud of the target object, and based on the object point cloud, control the end effector to move the target object and rotate the target object within the field of view of the point acquisition device according to a preset trajectory.

[0021] Specifically, when no shipping label area is detected in the initial image, the object point cloud corresponding to the target object is acquired. Based on the object point cloud, the end effector is controlled to operate on the target object to move it, and the end effector is controlled to rotate the target object within the field of view of the point acquisition device according to a preset trajectory.

[0022] It should be noted that, in the absence of a detected label area, the object point cloud corresponding to the target object is obtained based on the initial image.

[0023] In one embodiment, the initial image is matched with depth information. In response to the absence of a surface area detected in the initial image, the image region corresponding to the target object is extracted. Using the intrinsic parameters and depth information of the point acquisition device, at least some pixels in the image region are converted to three-dimensional space to obtain the object point cloud of the target object.

[0024] In one embodiment, the depth of pixel matching in the initial image is estimated using a trained depth prediction model to obtain the depth information matched in the initial image. Based on the initial image and the depth information, the point cloud information in the initial image is determined, and the object point cloud matching the target object is extracted from the point cloud information.

[0025] Furthermore, based on the object point cloud, the execution action of the end effector tool moving the target object and rotating the target object according to the preset trajectory is determined. Thus, the end effector tool actively moves the target object and drives the target object to rotate. By controlling the end effector tool, the position of the target object in the field of view of the point acquisition device is adjusted, so as to increase the probability of detecting the label area in the field of view.

[0026] In one embodiment, based on the object point cloud, the execution point where the end-effector holds the target object is determined, the end-effector is controlled to hold the execution point to move the target object to the center of the field of view of the point acquisition device, and the end-effector is controlled to drive the target object to rotate according to a preset trajectory.

[0027] In one embodiment, based on the object point cloud and the position of the end effector, the movement path of the end effector to move the target object is determined. The movement path is used to control the end effector to move the target object to a preset area within the field of view of the point acquisition device. The preset trajectory is used to control the end effector to drive the target object to rotate.

[0028] It should be noted that the end effector can be a gripper or a suction cup, thus adapting to targets of different sizes and shapes.

[0029] In some implementation scenarios, when the end effector selects a gripper, the largest surface is selected from the target object's surface, and the execution point is selected from the largest surface. The direction of the shortest axis of the target object is used as the gripper's gripping direction, thereby selecting the corresponding execution point on the opposite surface. After the gripper grasps the target object, it moves the target object to the center of the point acquisition device's field of view, and then controls the gripper to drive the target object to rotate according to a preset trajectory.

[0030] In some implementation scenarios, when the target execution tool is a suction cup, an execution surface with a curvature less than the curvature threshold and an area greater than the area threshold is selected from the surface of the target object. Multiple execution points are selected from the execution surface as the suction points of the suction cup. After the suction cup picks up the target object, it moves the target object to a preset area within the field of view of the point acquisition device, and then the suction cup is controlled to drive the target object to rotate according to a preset trajectory.

[0031] S103: In response to the detection of a shipping label area within the field of view during the rotation of the target object, the preset trajectory is interrupted and the reference shipping label point cloud of the shipping label area is obtained. Based on the reference shipping label point cloud, the reference observation pose of the observed shipping label area is obtained.

[0032] Specifically, the target object rotates continuously along a preset trajectory within the field of view of the point acquisition device. The point acquisition device can acquire images and perform label detection within the field of view. When a label area is detected within the field of view of the point acquisition device, the preset trajectory is interrupted and the reference label point cloud corresponding to the label area is obtained. Based on the reference label point cloud, the reference observation pose for observing the label area is determined, thereby clarifying the pose required to observe the label area.

[0033] It should be noted that when a shipping label area is detected within the field of view of the location acquisition device, the preset trajectory changes uniformly for all targets, which means that the change in the preset trajectory cannot guarantee that the shipping label area will continuously increase within the field of view. Therefore, the preset trajectory is interrupted to control the target object so that the shipping label area can be displayed more efficiently within the field of view of the location acquisition device.

[0034] It is understandable that a reference point cloud of the waybill area is determined based on the images acquired by the point acquisition device. The method for obtaining the reference point cloud of the waybill area is similar to the method for obtaining the object point cloud corresponding to the target object, and will not be elaborated upon here.

[0035] Furthermore, based on the reference surface point cloud, the pose of the observable surface region is fitted to obtain the reference observation pose of the observable surface region.

[0036] In one embodiment, the reference surface point cloud is segmented to obtain multiple point cloud regions. The normal vector of each point cloud region is determined. Using the normal vectors of all point cloud regions, the observation direction and observation reference point of the surface region are determined. Based on the observation direction, observation reference point and parameters of the point acquisition device, the reference observation pose is obtained.

[0037] In one embodiment, the complete surface area is fitted based on the reference surface point cloud to obtain the estimated surface point cloud, and the reference observation pose of the observed surface area is obtained based on the normal vector and center point of the estimated surface point cloud.

[0038] S104: Using the reference observation pose and the point observation pose matched by the point acquisition device, adjust the position of the target object until the waybill information in the waybill area is identified.

[0039] Specifically, by using the reference observation pose that can observe the target object and the point observation pose that matches the point acquisition device, the position of the target object is adjusted so that the dynamically changing reference observation pose moves closer to the fixed point observation pose, so as to collect a more complete and high-precision surface area within the field of view of the point acquisition device.

[0040] Understandably, by continuously adjusting and detecting until the waybill information in the waybill area is identified, the flexibility of waybill recognition is improved, thereby reducing the probability of waybill recognition failure and improving the efficiency and accuracy of waybill recognition.

[0041] In one embodiment, the point observation pose matched by the point acquisition device is obtained, wherein the point observation pose is used to indicate the current optimal shooting pose of the point acquisition device, the pose deviation between the reference observation pose and the point observation pose is obtained, and based on the pose deviation, the next execution action of the end effector is determined, and the position of the target object is adjusted by the next execution action.

[0042] In one embodiment, the point observation pose matched by the point acquisition device is obtained, wherein the point observation pose is used to indicate the current optimal shooting pose of the point acquisition device. Based on the reference observation pose and the point observation pose, the adjustment direction and adjustment distance of the end effector are determined, and the end effector is driven by the adjustment direction and adjustment distance to adjust the position of the target object.

[0043] Understandably, the target object continuously adjusts its position within the field of view of the point acquisition device and continuously identifies the information on the waybill in the waybill area. The end effector is controlled by the robotic arm of the sorting robot. During the movement of the robotic arm, the waybill can be identified in real time and the posture of the target object can be dynamically adjusted, increasing the probability that the waybill can be collected in the optimal state. Moreover, the closed-loop robotic arm control greatly reduces the risk of information omission and collection failure, improving the reliability and stability of waybill recognition.

[0044] The above scheme acquires the initial image from the location acquisition device matched to the sorting robot, identifies the target object from the initial image, and detects whether the target object currently includes a shipping label area. If no shipping label area is detected in the initial image, the object point cloud corresponding to the target object is acquired. Based on the object point cloud, the end effector is controlled to operate on the target object to move it, and the end effector is controlled to rotate the target object within the field of view of the location acquisition device according to a preset trajectory. This adjusts the position of the target object within the field of view of the location acquisition device, thereby increasing the probability of detecting the shipping label area within the field of view. If a shipping label area is detected within the field of view of the location acquisition device, the preset trajectory is interrupted, and a reference shipping label point cloud corresponding to the shipping label area is acquired. Based on the reference shipping label point cloud, a reference observation pose for observing the shipping label area is determined, thus clarifying the pose required to observe the shipping label area. By utilizing the reference observation pose that can observe the target object and the point observation pose that matches the point acquisition device, the position of the target object is adjusted so that the dynamically changing reference observation pose moves closer to the fixed point observation pose. This allows for the acquisition of a more complete and high-precision label area within the field of view of the point acquisition device. Through continuous adjustment and detection, the label information of the label area is identified, improving the flexibility of label recognition, reducing the probability of label recognition failure, and increasing the efficiency and accuracy of label recognition.

[0045] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the waybill recognition method of this application, the method including: S201: Obtain the initial image collected by the point acquisition device matched by the sorting robot, determine the target object in the initial image and detect the label area.

[0046] Specifically, the initial image is acquired by the point acquisition device matched with the sorting robot, the initial image is identified, the target object in the initial image is determined, and it is detected whether the target object currently includes the label area.

[0047] It is understandable that the point acquisition device takes a global picture of the target object to obtain an initial image. The target object is usually a package with a label attached. The package can be detected in the initial image, but the label on the package may not be in the field of view of the point acquisition device, so the label area cannot be detected.

[0048] S202: In response to the detection of a shipping label area in the initial image, acquire the target shipping label point cloud of the shipping label area, obtain the target observation pose of the observed shipping label area based on the target shipping label point cloud, adjust the end-of-line acquisition device on the end-of-line execution tool to the target observation pose, acquire the shipping label image of the shipping label area, and identify the shipping label information of the shipping label area from the shipping label image.

[0049] Specifically, when a shipping label region is detected in the initial image, the target shipping label point cloud corresponding to the shipping label region is obtained, and based on the target shipping label point cloud, the target observation pose that can observe the shipping label region at an ideal angle is determined.

[0050] Furthermore, the end-effector is equipped with an end-of-line acquisition device. The end-of-line acquisition device on the end-of-line actuator is adjusted to the target observation pose so that the end-of-line acquisition device can acquire the image of the waybill area at an ideal angle, thereby recognizing the waybill image to obtain the waybill information and improving the probability of recognizing accurate waybill information.

[0051] Understandably, when the label area is identified in the initial image, the accuracy of label recognition is improved by determining the target observation pose and actively adjusting the end-of-line acquisition device to the target observation pose.

[0052] In one embodiment, acquiring the target point cloud of the shipping label area, and obtaining the target observation pose of the observed shipping label area based on the target point cloud, includes: obtaining the target point cloud of the shipping label area based on the initial image and the parameters of the point acquisition device; obtaining the normal vector corresponding to each point in the target point cloud and its adjacent partial points; determining the observation direction and observation reference point of the shipping label area using the normal vectors corresponding to all points; obtaining the shipping label contour information corresponding to the shipping label area based on the target point cloud; and obtaining the target observation pose of the observed shipping label area based on the observation direction, observation reference point, and shipping label contour information.

[0053] Specifically, the depth information of the initial image is acquired. Using the depth information and the parameters of the point acquisition device, the surface label area in the initial image is mapped to three-dimensional space to generate a target surface label point cloud corresponding to the surface label area, enabling the target surface label point cloud to accurately match the surface label area in the actual scene. The parameters of the point acquisition device include its intrinsic parameters.

[0054] Furthermore, in the target surface point cloud, a point cloud sub-region composed of each point and its adjacent partial points is determined, and the normal vector of the point cloud sub-region is obtained as the normal vector corresponding to the point. Using the normal vectors corresponding to all points, the average normal direction of all normal vectors is obtained as the observation direction of the surface region. Using the normal vectors corresponding to all points, the average point position of all normal vectors is obtained, and the point closest to the average point position is used as the observation reference point.

[0055] Understandably, based on the target surface point cloud, the surface contour information corresponding to the surface region in three-dimensional space is obtained. The surface contour information includes the three-dimensional bounding box corresponding to the surface region. Let the three axes of the three-dimensional bounding box be x, y, and z. Ignore the axis with the smallest angle with the average normal direction (denoted as the z-axis), take the longer of the remaining two axes as the y-axis, and the remaining axis is the x-axis.

[0056] Furthermore, based on the observation direction, observation reference point, and surface plan outline information, the target observation pose of the observed surface plan area is obtained by integrating multiple specific information, ensuring that high-quality surface plan images can be acquired when the target observation pose is in place.

[0057] It should be noted that, based on the observation direction, observation reference point, and surface package contour information, the target observation pose of the observed surface package area is obtained, including: based on the surface package contour information, determining the major axis of the surface package area, obtaining the direction orthogonal to the observation direction and having the smallest angle with the major axis of the surface package as the major axis direction of the observed surface package area, and determining the minor axis direction that matches the observation direction and the major axis direction; obtaining the focusing distance of the point acquisition device, moving the focusing distance from the observation reference point along the observation direction to obtain the observation position, and generating a pose that matches the observation direction, the major axis direction, and the minor axis direction at the observation position as the target observation pose of the observed surface package area.

[0058] Specifically, the shipping label area corresponds to a rectangle. Based on the shipping label outline information, the major axis of the shipping label within the area can be determined. The direction orthogonal to the observation direction and with the smallest angle to the major axis of the shipping label is taken as the direction of the major axis of the observed shipping label area. By minimizing the angle between the major axis and the major axis of the shipping label, it can be ensured that the major axis of the shipping label is as parallel as possible to the long side of the image in the image acquired by the terminal acquisition device, thus ensuring that the shipping label area can be completely displayed in the image. Furthermore, after obtaining the observation direction and the major axis direction, the minor axis direction of the observed shipping label area can be fitted to obtain it.

[0059] Furthermore, the focusing distance of the end acquisition device is obtained, and the focusing distance is moved along the observation direction with the observation reference point as the starting point to obtain the observation position. At the observation position, the corresponding pose is generated according to the observation direction, the major axis direction and the minor axis direction, which serves as the target observation pose for a single area of ​​the observation surface, ensuring the clarity and completeness of the acquired image.

[0060] Optionally, the three axes of the target observation pose are denoted as X, Y, and Z. The X-axis corresponds to the minor axis, the Y-axis corresponds to the major axis, and the Z-axis corresponds to the observation direction. The Z-axis of the target observation pose is consistent with the average normal direction. The Y-axis of the observation pose is calculated by solving the two conditions: it is orthogonal to the average normal direction and has the smallest angle with the major axis of the plane. The X-axis is calculated using the right-hand rule. The position of the target observation pose starts from the observation reference point. Based on the optimal focusing distance of the end-effector, the target moves back a corresponding distance along the Z-axis to obtain the most suitable observation position, thereby generating the target observation pose at the observation position.

[0061] In one embodiment, adjusting the end-effector acquisition device on the end-effector to the target observation pose, acquiring a label image of the label area, and identifying label information of the label area from the label image includes: obtaining the relative pose of the end-effector and the end-effector acquisition device; obtaining the target end-effector pose of the end-effector based on the relative pose and the target observation pose; adjusting the end-effector to the target end-effector pose so that the end-effector acquisition device is adjusted to the target observation pose; acquiring a label image of the label area using the end-effector acquisition device; and identifying label information of the label area from the label image.

[0062] Specifically, the relative pose between the end effector and the end acquisition device is obtained, and based on the relative pose and the target observation pose, the target end pose required by the end effector when the end acquisition device is adjusted to the target end pose is determined.

[0063] Furthermore, the end effector is adjusted to the target end pose so that the end acquisition device is adjusted to the target observation pose, thereby ensuring that the end acquisition device can acquire high-precision shipping label images. The end acquisition device is used to acquire shipping label images of the shipping label area at an ideal angle, thereby recognizing the shipping label images to obtain shipping label information and increasing the probability of recognizing shipping label information.

[0064] S203: In response to the absence of a surface area detected in the initial image, acquire the object point cloud of the target object, and based on the object point cloud, control the end effector to move the target object and rotate the target object within the field of view of the point acquisition device according to a preset trajectory.

[0065] Specifically, when no shipping label area is detected in the initial image, the object point cloud corresponding to the target object is acquired. Based on the object point cloud, the end effector is controlled to operate on the target object to move it, and the end effector is controlled to rotate the target object within the field of view of the point acquisition device according to a preset trajectory.

[0066] In one embodiment, an object point cloud of the target object is obtained based on an initial image and parameters of the point acquisition device; the execution point where the end effector abuts the target object is determined from the object point cloud; the execution trajectory of the end effector moving the target object is obtained based on the execution point and parameters of the point acquisition device; the end effector abuts the execution point using the execution trajectory and moves the target object to a preset area within the field of view of the point acquisition device; the end effector rotates the target object using the preset trajectory.

[0067] Specifically, the depth information of the initial image is obtained, and the target object in the initial image is mapped to three-dimensional space using the depth information and the parameters of the point acquisition device to generate the object point cloud corresponding to the target object, so that the object point cloud can accurately match the target object in the actual scene.

[0068] Furthermore, the execution points when the end-effector holds the target object are selected from the object point cloud. Using the execution points that the end-effector needs to hold and the parameters of the point acquisition device, the execution trajectory of the end-effector moving the target object is generated. The execution trajectory is used to control the end-effector holding the execution points, thereby ensuring that the target object is not easily dropped after being held by the end-effector. The target object is moved to a preset area within the field of view of the point acquisition device. The preset trajectory is used to control the end-effector to drive the target object to rotate so that the label area can be detected within the field of view of the point acquisition device.

[0069] It should be noted that determining the execution point of the end effector against the target object from the object point cloud includes: in response to the detection of a surface area in the initial image, obtaining the target surface area point cloud of the surface area, filtering the target surface area point cloud from the object point cloud to obtain an updated object point cloud; and determining the execution point of the end effector against the target object from the object point cloud based on the object point cloud and the type of the end effector.

[0070] Specifically, when a shipping label region is detected in the initial image, the target shipping label point cloud of the shipping label region is obtained, and the target shipping label point cloud is filtered from the object point cloud in order to update the object point cloud. This avoids the end-effector selecting points on the target shipping label point cloud as execution points and reduces the probability that the shipping label region is occluded by the end-effector.

[0071] Furthermore, based on the object point cloud and the type of the end-effector, the execution points of the end-effector against the target are selected from the object point cloud so that the execution points can avoid the target point cloud and the execution points match the type of the end-effector.

[0072] It is understandable that end effector tools include gripper-type tools and suction cup-type tools, and different types of end effector tools can achieve corresponding execution points.

[0073] S204: In response to the detection of a shipping label area within the field of view during the rotation of the target object, the preset trajectory is interrupted and a reference shipping label point cloud of the shipping label area is obtained. Based on the reference shipping label point cloud, the reference observation pose of the observed shipping label area is obtained.

[0074] Specifically, the target object rotates continuously along a preset trajectory within the field of view of the point acquisition device. The point acquisition device can acquire images and perform label detection within the field of view. When a label area is detected within the field of view of the point acquisition device, the preset trajectory is interrupted and the reference label point cloud corresponding to the label area is obtained. Based on the reference label point cloud, the reference observation pose for observing the label area is determined, thereby clarifying the pose required to observe the label area.

[0075] Understandably, the point acquisition transposition continuously acquires images within the field of view. After obtaining a newly acquired image, it performs label detection. Once a label area matching the label's characteristics is detected, the preset trajectory is interrupted. Typically, the label area detected within the field of view is an incomplete portion.

[0076] Furthermore, a reference point cloud of the shipping label corresponding to the shipping label area is obtained, and the normal vector corresponding to each point in the reference point cloud is determined. Using the normal vector corresponding to each point, the average normal direction of all normal vectors is obtained as the observation direction of the shipping label area. Using the normal vector corresponding to all points, the average point position of all normal vectors is obtained, and the point closest to the average point position is used as the observation reference point.

[0077] It is understandable that, based on the reference surface cloud, the surface contour information of the surface region in three-dimensional space is obtained. Based on the observation direction, the observation reference point, and the surface contour information, the reference observation pose of the observed surface region is obtained by integrating multiple specific information.

[0078] S205: Using the reference observation pose and the point observation pose matched by the point acquisition device, adjust the position of the target object until the waybill information in the waybill area is identified.

[0079] Specifically, by using the reference observation pose that can observe the target object and the point observation pose that matches the point acquisition device, the position of the target object is adjusted so that the dynamically changing reference observation pose moves closer to the fixed point observation pose, so as to collect a more complete and high-precision surface area within the field of view of the point acquisition device.

[0080] In one embodiment, based on the parameters and field of view of the point acquisition device, the point observation pose matched by the point acquisition device is determined; the pose deviation between the reference observation pose and the point observation pose is obtained, and the position of the target object is adjusted based on the pose deviation; in response to the unidentified waybill information in the waybill area, the reference waybill point cloud of the waybill area and the reference observation pose of the observed waybill area are updated, and the process returns to the step of obtaining the pose deviation between the reference observation pose and the point observation pose, and adjusting the position of the target object based on the pose deviation.

[0081] Specifically, the orientation and focal length of the image acquired by the point acquisition device, as well as the orientation within the field of view, are obtained. Based on the image orientation, focal length, and orientation within the field of view, the matching point observation pose of the point acquisition device is determined. Here, the point observation pose of the point acquisition device at this time is denoted as P0. The Z-axis of P0 is consistent with the field of view direction, and the Y-axis is consistent with the width (horizontal direction) of the image. The position of P0 is at the center of the field of view, and the distance is determined by the optimal focusing distance of the point acquisition device.

[0082] Furthermore, the pose deviation between the reference observation pose and the point observation pose is obtained. Based on the pose deviation, the next action of the end effector is determined. By interrupting the preset trajectory and actively generating the next action of the end effector, the position of the target object is adjusted with higher efficiency, and a more comprehensive surface area is obtained within the field of view.

[0083] Understandably, if the waybill information in the waybill area is still not identified after adjustment, the reference waybill point cloud and the reference observation pose of the observed waybill area are updated, and the process returns to obtaining the pose deviation between the reference observation pose and the point observation pose. Based on the pose deviation, the position of the target object is adjusted, thereby improving the efficiency of waybill recognition through continuous adjustment and closed-loop feedback.

[0084] It should be noted that after acquiring the object point cloud of the target object, and based on the object point cloud, controlling the end effector to move the target object and rotate the target object within the field of view of the point acquisition device according to the preset trajectory, the process also includes: in response to the completion of the preset trajectory or the failure to identify the waybill information in the waybill area after the pose deviation is less than the deviation threshold, generating the corresponding abnormal prompt information for the target object.

[0085] Specifically, if the preset trajectory is executed but the waybill information in the waybill area is not identified, or if the pose deviation between the reference observation pose and the point observation pose is less than the deviation threshold but the waybill information in the waybill area is still not identified, then an abnormal prompt message corresponding to the target object is generated to indicate that the waybill identification is abnormal.

[0086] Optionally, the error message includes the reason for the waybill recognition error, which is related to waybill damage, waybill obstruction, and waybill not being affixed.

[0087] In this embodiment, when a shipping label area is identified in the initial image, the accuracy of shipping label recognition is improved by determining the target observation pose and actively adjusting the end-effector to the target observation pose. When no shipping label area is identified in the initial image, the end-effector actively moves the target object and rotates it. By controlling the end-effector to adjust the position of the target object within the field of view of the point acquisition device, the probability of detecting the shipping label area within the field of view is increased. When a shipping label area is identified within the field of view, the preset trajectory is interrupted and the next action of the end-effector is actively generated, thereby adjusting the position of the target object with higher efficiency, acquiring a more comprehensive shipping label area within the field of view, and improving the efficiency and accuracy of shipping label recognition.

[0088] Please see Figure 3 , Figure 3This is a schematic diagram of one embodiment of the label recognition system of this application. The system is applied to a sorting robot including an end effector. The label recognition system 30 includes: a detection module 301, an execution tool module 302, an observation pose calculation module 303, and a closed-loop feedback module 304. The detection module 301 is used to acquire an initial image collected by a point acquisition device matched with the sorting robot, determine the target object in the initial image, and detect the label area. The execution tool module 302 is used to acquire the object point cloud of the target object in response to the absence of a label area detected in the initial image, and based on the object point cloud, control the end effector to move the target object and rotate the target object within the field of view of the point acquisition device according to a preset trajectory. The observation pose calculation module 303 is used to interrupt the preset trajectory and acquire a reference label point cloud of the label area in response to the detection of a label area within the field of view during the rotation of the target object, and obtain a reference observation pose of the observed label area based on the reference label point cloud. The closed-loop feedback module 304 is used to adjust the position of the target object using the reference observation pose and the point observation pose matched with the point acquisition device until the label information of the label area is recognized.

[0089] In the above scheme, the detection module 301 acquires the initial image collected by the point acquisition device matched to the sorting robot, identifies the target object from the initial image, and detects whether the target object currently includes a shipping label area. If no shipping label area is detected in the initial image, the execution tool module 302 acquires the object point cloud corresponding to the target object. Based on the object point cloud, it controls the end effector to operate on the target object to move it, and controls the end effector to rotate the target object within the field of view of the point acquisition device according to a preset trajectory. This adjusts the position of the target object within the field of view of the point acquisition device, thereby increasing the probability of detecting the shipping label area within the field of view. If a shipping label area is detected within the field of view of the point acquisition device, the observation pose calculation module 303 interrupts the preset trajectory and acquires the reference shipping label point cloud corresponding to the shipping label area. Based on the reference shipping label point cloud, it determines the reference observation pose that can observe the shipping label area, thus clarifying the pose required to observe the shipping label area. The closed-loop feedback module 304 utilizes the reference observation pose that can observe the target object and the point observation pose that matches the point acquisition device to adjust the position of the target object, so that the dynamically changing reference observation pose moves closer to the fixed point observation pose. This allows for the acquisition of a more complete and high-precision label area within the field of view of the point acquisition device. Through continuous adjustment and detection, the label information in the label area is recognized, improving the flexibility of label recognition, reducing the probability of label recognition failure, and improving the efficiency and accuracy of label recognition.

[0090] Please see Figure 4 , Figure 4This is a schematic diagram of another embodiment of the waybill recognition system of this application. The system is applied to a sorting robot including an end effector. The waybill recognition system 30 includes: a detection module 301, an execution tool module 302, an observation pose calculation module 303, a closed-loop feedback module 304, an operation pose calculation module 305, and an information recognition and data management module 306. The operation pose calculation module 305 is used to determine the execution point where the end effector abuts the target object from the object point cloud, and obtains the execution trajectory of the end effector moving the target object based on the execution point and the parameters of the point acquisition device. The information recognition and data management module 306 is used to identify and store the waybill information in the waybill area.

[0091] It is understood that the waybill recognition system 30 can implement the waybill recognition method in any of the above embodiments.

[0092] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 40 includes a memory 401 and a processor 402 coupled to each other. The memory 401 stores program data (not shown in the figure). The processor 402 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0093] Please see Figure 6 , Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, it implements the method in any of the above embodiments. For related descriptions, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0094] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A face single identification method, characterized by, The face sheet identification method is applied to a sorting robot comprising an end execution tool, and comprises the following steps: An initial image collected by a point position collection device matched with the sorting robot is acquired, a target object in the initial image is determined, and a face sheet region is detected; In response to the face sheet region not being detected in the initial image, an object point cloud of the target object is acquired, the end execution tool is controlled to move the target object and rotate the target object according to a preset trajectory in a field of view of the point position collection device based on the object point cloud; In response to the face sheet region being detected in the field of view during the rotation of the target object, the preset trajectory is interrupted, a reference face sheet point cloud of the face sheet region is acquired, and a reference observation pose for observing the face sheet region is obtained based on the reference face sheet point cloud; The position of the target object is adjusted by using the reference observation pose and a point position observation pose matched with the point position collection device until face sheet information of the face sheet region is identified.

2. The face identification method according to claim 1, wherein, After the initial image collected by the point position collection device matched with the sorting robot is acquired, the target object in the initial image is determined, and the face sheet region is detected, the method further comprises the following steps: In response to the face sheet region being detected in the initial image, a target face sheet point cloud of the face sheet region is acquired, a target observation pose for observing the face sheet region is obtained based on the target face sheet point cloud, an end collection device on the end execution tool is adjusted to the target observation pose, a face sheet image of the face sheet region is collected, and face sheet information of the face sheet region is identified from the face sheet image.

3. The face identification method according to claim 2, wherein, The target face sheet point cloud of the face sheet region is acquired, and the target observation pose for observing the face sheet region is obtained based on the target face sheet point cloud, which comprises the following steps: The target face sheet point cloud of the face sheet region is obtained based on the initial image and parameters of the point position collection device; The normal vector corresponding to each point is obtained based on each point and its adjacent partial points in the target face sheet point cloud, the observation direction and the observation reference point of the face sheet region are determined by using the normal vectors corresponding to all points, the face sheet contour information corresponding to the face sheet region is obtained based on the target face sheet point cloud; The target observation pose for observing the face sheet region is obtained based on the observation direction, the observation reference point and the face sheet contour information.

4. The face identification method according to claim 3, wherein, The target observation pose for observing the face sheet region is obtained based on the observation direction, the observation reference point and the face sheet contour information, which comprises the following steps: The face sheet long axis of the face sheet region is determined based on the face sheet contour information, a direction orthogonal to the observation direction and having the smallest angle with the face sheet long axis is obtained as a long axis direction for observing the face sheet region, and a short axis direction matched with the observation direction and the long axis direction is determined; The focusing distance of the end collection device is acquired, the observation position is obtained by moving the focusing distance along the observation direction from the observation reference point, the pose matched with the observation direction, the long axis direction and the short axis direction is generated at the observation position, and the target observation pose for observing the face sheet region is obtained.

5. The face identification method according to claim 2, wherein, The adjusting the end collecting device on the end executing tool to the target observation pose, collecting a face single image of the face single region, and identifying face single information of the face single region from the face single image, comprises: Obtaining a relative pose of the end executing tool and the end collecting device, and obtaining a target end pose of the end executing tool based on the relative pose and the target observation pose; Adjusting the end executing tool to the target end pose, so that the end collecting device is adjusted to the target observation pose, collecting a face single image of the face single region by using the end collecting device, and identifying face single information of the face single region from the face single image.

6. The face identification method according to claim 1, wherein, The obtaining of the object point cloud of the target object and the control of the end executing tool to move the target object and rotate the target object according to a preset trajectory within a field of view of a point collecting device based on the object point cloud, comprises: Obtaining an object point cloud of the target object based on the initial image and parameters of the point collecting device; Determining an execution point at which the end executing tool abuts against the target object from the object point cloud, and obtaining an execution trajectory of the end executing tool to move the target object based on the execution point and the parameters of the point collecting device; Controlling the end executing tool to abut against the execution point by using the execution trajectory, and moving the target object to a preset region within the field of view of the point collecting device, and controlling the end executing tool to rotate the target object by using the preset trajectory.

7. The face identification method according to claim 6, wherein, The determining of the execution point at which the end executing tool abuts against the target object from the object point cloud, comprises: In response to detecting the face single region in the initial image, obtaining a target face single point cloud of the face single region, and filtering the target face single point cloud from the object point cloud to obtain an updated object point cloud; Determining the execution point at which the end executing tool abuts against the target object from the object point cloud based on the object point cloud and the type of the end executing tool.

8. The face identification method according to claim 1, characterized in that, The adjusting of the position of the target object by using the point observation pose matched by the reference observation pose and the point collecting device until the face single information of the face single region is identified, comprises: Determining the point observation pose matched by the point collecting device based on the parameters of the point collecting device and the field of view; Obtaining a pose deviation between the reference observation pose and the point observation pose, and adjusting the position of the target object based on the pose deviation; In response to not identifying the face single information of the face single region, updating a reference face single point cloud of the face single region and a reference observation pose of observing the face single region, and returning to the step of obtaining the pose deviation between the reference observation pose and the point observation pose and adjusting the position of the target object based on the pose deviation.

9. The face identification method according to claim 8, wherein, After the obtaining of the object point cloud of the target object and the control of the end executing tool to move the target object and rotate the target object according to a preset trajectory within a field of view of a point collecting device based on the object point cloud, further comprising: In response to the preset track being executed or the pose deviation being less than a deviation threshold, and no face sheet information of the face sheet region being identified, generating abnormal prompt information corresponding to the target object.

10. A face single identification system characterized by, The face sheet recognition system is applied to a sorting robot including an end execution tool, and the face sheet recognition system comprises: a detection module configured to acquire an initial image collected by a point position acquisition device matched with the sorting robot, determine a target object in the initial image, and detect a face sheet region; an execution tool execution module configured to, in response to the face sheet region not being detected in the initial image, acquire an object point cloud of the target object, control the end execution tool to move the target object and rotate the target object according to a preset track within a field of view of the point position acquisition device based on the object point cloud; an observation pose calculation module configured to, in response to the face sheet region being detected within the field of view, interrupt the preset track, acquire a reference face sheet point cloud of the face sheet region, and obtain a reference observation pose for observing the face sheet region based on the reference face sheet point cloud; a closed-loop feedback module configured to adjust a position of the target object by using the reference observation pose and a point position observation pose matched with the point position acquisition device until face sheet information of the face sheet region is identified.

11. An electronic device, comprising: comprise: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method according to any one of claims 1-9.

12. A computer readable storage medium having stored thereon program data, wherein, The program data, when executed by the processor, implements the method according to any one of claims 1-9.

Images