Trunk 3D reconstruction method and device based on binocular camera, medium and equipment

Abstract

The invention provides a luggage case 3D reconstruction method and device based on a binocular camera, a medium and equipment, and relates to the technical field of point cloud reconstruction, and the method comprises the steps: controlling the binocular camera to collect a target image corresponding to a target luggage case in response to the detection that the target luggage case reaches a target position, so as to obtain corresponding target images Xa and Xb; respectively carrying out foreground identification on Xa and Xb according to a preset identification method so as to obtain corresponding target ROI regions Qa and Qb; performing three-dimensional reconstruction on the target luggage case according to Qa and Qb to obtain a point cloud bounding box corresponding to the target luggage case; and according to the point cloud bounding box corresponding to the target luggage case, obtaining position information and contour information of the target luggage case. The position information and the contour information, obtained through three-dimensional reconstruction, of the target luggage case are more accurate. And an accurate data basis is provided for other subsequent identification and transfer work of the luggage case.

Description

Technical Field

[0001] This application relates to the field of point cloud reconstruction technology, and in particular to a method, apparatus, medium and equipment for 3D reconstruction of a suitcase based on a binocular camera. Background Technology

[0002] In busy airport baggage handling systems, the efficient and accurate processing and transfer of thousands of suitcases is crucial. With the rapid development of automation technology, robotic arms, as a key component of modern airport logistics automation, are widely used in baggage handling, loading, and other operational processes. These robotic arms not only significantly improve work efficiency but also reduce human error, enhancing the overall safety and reliability of operations.

[0003] However, for a robotic arm to accurately grasp and transfer suitcases, the primary prerequisite is the ability to precisely determine the suitcase's position and size. Due to the complex and ever-changing airport environment, the variety of suitcases—different sizes and shapes—and their frequent close stacking, pose a significant challenge to the robotic arm's positioning and size recognition.

[0004] In related technologies, stereo cameras are used to acquire images of suitcases. A typical stereo camera setup involves connecting the optical axes of the two imaging lenses on the same horizontal plane, with the line perpendicular to the direction of the conveyor belt's movement. In this setup, the background of the acquired image includes not only the suitcase and conveyor belt but also other background elements, making it complex and difficult to extract the foreground. The accuracy of the extracted foreground is also relatively low. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method, apparatus, medium, and device for 3D reconstruction of a suitcase based on a binocular camera, which at least partially solves the problems existing in the prior art.

[0006] In a first aspect of this application, a method for 3D reconstruction of a suitcase based on a binocular camera is provided, the method comprising the following steps:

[0007] S100, in response to detecting that the target suitcase has arrived at the target position, the binocular camera is controlled to acquire the target image corresponding to the target suitcase to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are converged and installed vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera.

[0008] S200, perform foreground recognition on Xa and Xb respectively according to the preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein, the target ROI region is a region that only contains the image of the target suitcase.

[0009] S300: Based on Qa and Qb, perform 3D reconstruction of the target suitcase to obtain the point cloud bounding box corresponding to the target suitcase.

[0010] S400 obtains the location and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

[0011] In a second aspect of this application, a binocular camera-based 3D reconstruction device for a suitcase is provided, the device comprising:

[0012] The acquisition unit, in response to detecting that the target suitcase has arrived at the target position, controls the binocular camera to acquire the target image corresponding to the target suitcase, so as to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are converged and installed vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera.

[0013] The recognition unit is used to perform foreground recognition on Xa and Xb respectively according to a preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein the target ROI region is a region that contains only the image of the target suitcase.

[0014] The construction unit is used to perform 3D reconstruction of the target suitcase based on Qa and Qb to obtain the point cloud bounding box corresponding to the target suitcase.

[0015] The size acquisition unit is used to obtain the position and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

[0016] In a third aspect of this application, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the aforementioned binocular camera-based 3D reconstruction method for a suitcase.

[0017] In a fourth aspect of this application, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0018] This application has at least the following beneficial effects:

[0019] This application provides a binocular camera-based 3D reconstruction method for suitcases. In response to detecting that a target suitcase has arrived at a target location, an image of the target suitcase is acquired using an RGBD binocular camera. The binocular camera includes two imaging lenses. In this application, the two imaging lenses are converged and mounted vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt. The width of the viewing angle corresponding to the Euler angle of any imaging lens is equal to the width of the conveyor belt where the target suitcase is located. Compared to conventional binocular cameras where the line connecting the optical axes of the two imaging lenses on the same horizontal plane is perpendicular to the movement direction of the conveyor belt, the binocular camera provided in this application is positioned along the movement direction of the conveyor belt where the target suitcase is located, and the width of the viewing angle corresponding to the Euler angle of each imaging lens is limited to the width of the conveyor belt where the target suitcase is located. In other words, the background of the target suitcase image acquired by the binocular camera system configured in this application primarily consists of the conveyor belt, with little or no other background, reducing the background complexity of the image and making foreground extraction easier. This results in more accurate point cloud construction. Finally, the location and contour information of the target suitcase obtained through 3D reconstruction are also more precise, providing a precise data foundation for subsequent suitcase identification and transfer operations. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 Flowchart of a binocular camera-based 3D reconstruction method for a suitcase provided in this application embodiment;

[0022] Figure 2 A structural block diagram of a binocular camera-based 3D reconstruction device for a suitcase provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0025] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0026] Please refer to Figure 1 As shown, embodiments of this application provide a method for 3D reconstruction of a suitcase based on a binocular camera, the method comprising the following steps:

[0027] S100, in response to detecting that the target suitcase has arrived at the target position, the binocular camera is controlled to acquire the target image corresponding to the target suitcase to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are converged and installed vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera.

[0028] Specifically, an RGBD binocular camera is a camera system that combines RGB (red, green, and blue primary colors) color image capture and depth measurement. It typically includes two cameras, acquiring depth information through binocular parallax, structured light, or TOF (Time-of-Flight) technologies, and combining this information with the RGB color image. In this application, the first target image includes a first target color image and a first target depth image; the second target image includes a second target color image and a second target depth image. The two imaging lenses of the binocular camera are convergently mounted vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the direction of movement of the conveyor belt where the target suitcase is located; the width of the viewing angle corresponding to the Euler angle of any imaging lens is equal to the width of the conveyor belt where the target suitcase is located. Compared to conventional binocular cameras where the optical axes of the two imaging lenses are on the same horizontal plane and the line connecting them is perpendicular to the direction of movement of the conveyor belt, the binocular camera provided in this embodiment is set along the direction of movement of the conveyor belt where the target suitcase is located. Furthermore, the width of the viewing angle corresponding to the Euler angle of each imaging lens is limited to the width of the conveyor belt where the target suitcase is located. That is, the background (excluding the suitcase) of the image of the target suitcase acquired by the binocular camera system set in this embodiment basically only includes the conveyor belt, excluding or including only a small portion of other background, thus reducing the background complexity of the image corresponding to the target suitcase.

[0029] S200, perform foreground recognition on Xa and Xb respectively according to the preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein, the target ROI region is a region that only contains the image of the target suitcase.

[0030] Specifically, the preset recognition methods include:

[0031] S210, Obtain the target background image corresponding to the target color map; wherein, the target background image is the image in the corresponding target color map that does not include the target suitcase part; the target color map is the target color map corresponding to Xa or Xb;

[0032] Here, the target background image is the image in the corresponding target color map that does not include the target suitcase. That is, before acquiring the image corresponding to the suitcase, while the corresponding position on the conveyor belt is empty, an image of the corresponding position is acquired using an RGBD binocular camera from the same viewing angle as the target background image. Furthermore, since the image corresponding to each suitcase in this application is taken after the target suitcase arrives at the corresponding target position, and the binocular camera is pre-calibrated (i.e., the viewing angle is fixed), the pre-obtained target background image can be applied to foreground recognition of each suitcase.

[0033] S220, the target color map and the corresponding target background image are input into the pre-trained foreground separation model to obtain the mask processing area corresponding to the target color map; wherein, the mask processing area corresponding to the target color map is the area corresponding to the target suitcase.

[0034] Here, the target color image and the corresponding target background image are input into a pre-trained foreground separation model. The input to this model is the target color image and the corresponding target background image, and the output is the mask processing region corresponding to the target color image. The mask processing region is the region corresponding to the target suitcase. In other words, the suitcase portion of the color image is extracted using the pre-trained foreground separation model and used as the mask processing region. The mask processing includes processing a two-dimensional matrix array, specifically, a zero-one matrix.

[0035] S230, based on the masking processing area, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area; the target depth map is the target depth map corresponding to Xa or Xb.

[0036] Here, based on the masking area obtained using the target color map, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the depth map. The ROI region of the target depth map is the masking area, i.e., the area corresponding to the target suitcase.

[0037] In one embodiment, the ROI region corresponding to the target depth map is obtained through the following steps:

[0038] The zero-one matrix is ​​multiplied element-wise with the depth map to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the mask processing region.

[0039] Here, since the target color map and target depth map are registered, there is a one-to-one correspondence between pixels. The mask region obtained from the target color map (i.e., a zero-one matrix, where the foreground value is 1 and the non-foreground value is 0) is multiplied element-wise with the depth map to obtain the corresponding ROI region in the target depth map and the corresponding non-ROI region in the depth map.

[0040] In this embodiment, the background in the target color image is subtracted using a pre-trained foreground separation model to obtain the mask processing area corresponding to the target color image. Specifically, the suitcase portion in the color image is extracted using the pre-trained foreground separation model as the mask processing area. Furthermore, since the corresponding target color image and target depth image are registered, there is a one-to-one correspondence between pixels. Then, based on the mask processing area obtained using the target color image, the target depth image corresponding to the target color image is masked to obtain the ROI region corresponding to the depth image. This completes the foreground recognition of the depth image, removing the background portion from the depth image acquired by the RGBD stereo camera, and obtaining the depth image portion corresponding to the target suitcase (i.e., the foreground). Compared to determining the ROI region of the target depth image using a pre-set bounding box (not based on the target color image corresponding to the target suitcase), this embodiment determines the ROI region in the corresponding target depth image based on the mask region extracted from the target color image corresponding to the target suitcase, resulting in higher adaptability and accuracy. In addition, when obtaining the mask region, this embodiment does not require manual input of prompt words compared to the automatic segmentation model, and can directly obtain the corresponding mask region based on the pre-trained foreground separation model.

[0041] S300: Based on Qa and Qb, perform 3D reconstruction of the target suitcase to obtain the point cloud bounding box corresponding to the target suitcase.

[0042] Specifically, 3D reconstruction is performed based on the oped 3D algorithm. Step S300 also includes:

[0043] S310: Obtain the corresponding point cloud based on Qa and Qb.

[0044] S320: Stitch together the point clouds corresponding to Qa and Qb to obtain the initial point cloud cluster corresponding to the target suitcase.

[0045] S330 denoises the initial point cloud clusters corresponding to the target suitcase according to a preset denoising method to obtain the point cloud bounding box corresponding to the target suitcase.

[0046] Here, because errors occur when the camera acquires depth data, the initial point cloud corresponding to the target suitcase contains some linear noise due to the ROI region of the target depth map obtained through the mask region mentioned above. Therefore, denoising is required.

[0047] Specifically, the preset noise reduction methods include:

[0048] S331 marks each point cloud within the initial point cloud cluster as an unvisited point cloud.

[0049] S332, Traverse the initial point cloud cluster. If the number of unvisited point clouds within a preset distance range corresponding to any unvisited point cloud is greater than a preset number, then the unvisited point cloud is determined as the initial core object point; otherwise, it is determined as noise.

[0050] S333, add the initial core object point and the unvisited point cloud contained within the preset distance range corresponding to the initial core object point to the target cluster.

[0051] S334 marks each point cloud within the target cluster as a visited point cloud.

[0052] S335, if any visited point cloud in the target cluster contains an unvisited point cloud within a preset distance range, then add each unvisited point cloud to the candidate list.

[0053] S336, Traverse the candidate list. If the number of point clouds contained within the preset distance range corresponding to any unvisited point cloud is greater than the preset number, then the unvisited point cloud is identified as a key core object point; otherwise, it is identified as noise and removed from the candidate list.

[0054] S337, add the key core object point and each unvisited point cloud within the preset distance range corresponding to the key core object point to the target cluster, and delete it from the candidate list; then jump to step S334 until the candidate list is empty.

[0055] S338, when the candidate list is empty, the corresponding target cluster is determined as the target point cloud corresponding to the target suitcase.

[0056] Steps S331-S338 above constitute the DBSCAN point cloud clustering method. Since the initial point cloud clusters of the target suitcase obtained in this application consist of a main suitcase point cloud plus some linear noise point clouds, and the DBSCAN clustering algorithm is a density-based spatial clustering algorithm, it divides regions with sufficiently high density into clusters and can discover clusters of arbitrary shapes in noisy datasets. In this application, because the point cloud of the linear noise part is not dense enough, by setting reasonable parameters, it can be marked as noise points and filtered out during the clustering process. For the main suitcase point cloud, its corresponding point cloud is sufficiently dense. The core object point is determined by the number of points within its neighborhood (within a preset distance range), and the clustering expands outward from any core object point until all point clouds are traversed, thus obtaining the main suitcase and excluding all linear noise parts with insufficient point cloud density from the cluster corresponding to the main suitcase point cloud. This achieves denoising of the suitcase point cloud data with linear noise.

[0057] In one exemplary embodiment of this application, after step S334, the method further includes:

[0058] S339, if no unvisited point cloud is included within the preset distance range corresponding to each visited point cloud in the target cluster, then the clustering ends, and the corresponding target cluster is determined as the target point cloud cluster corresponding to the target suitcase.

[0059] Specifically, if no unvisited point cloud is included within the preset distance range corresponding to each visited point cloud in the target cluster, it means that the corresponding dense points are already in the target cluster. At this time, the clustering ends, and the corresponding target cluster is determined as the target point cloud cluster corresponding to the target suitcase.

[0060] S400 obtains the location and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

[0061] Specifically, the obtained point cloud bounding box is located in a spatial Cartesian coordinate system constructed by the binocular camera, and it marks the position and outline information of the target suitcase. It should be noted that the conveyor belt is located on the plane formed by the x and y axes of the spatial Cartesian coordinate system; the outline of the point cloud bounding box is a hexahedron with eight vertices; and the point cloud bounding box has corresponding dimensions.

[0062] In one exemplary embodiment of this application, after step S400, the method further includes:

[0063] S500 selects one vertex from the eight vertices of the point cloud bounding box and identifies it as the critical vertex.

[0064] S600, vectors are drawn from the key vertices to the three vertices directly connected to the edges of the hexahedron corresponding to the bounding box contour of the point cloud to obtain the first vector YX, the second vector EX, and the third vector SX; wherein any two vectors among YX, EX, and SX are perpendicular to each other.

[0065] Specifically, the first vector YX, the second vector EX, and the third vector SX must contain the height of the target suitcase in the current pose (the edge perpendicular to the plane where the conveyor belt is located).

[0066] S700, obtain the normal vector GF of the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system at the location of the key vertex.

[0067] S800 determines the height WG corresponding to the current pose of the target suitcase based on YX, EX, SX and GF.

[0068] Specifically, step S800 includes:

[0069] S810, if any of the angles YJ, EJ, SJ, or SJ between YX and GF are less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase.

[0070] Here, the normal vector GF corresponding to the location of the key vertex in the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system is perpendicular to the plane formed by the x-axis and y-axis. If the angle between any vector YX, EX, SX and the line containing GF is less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase. It should be noted that the angle between any vector YX, EX, SX and the line containing GF is less than or equal to 90 degrees. Since the bounding box is obtained based on the actual shape of the suitcase, the bounding box may not be a cuboid, but it is an approximately cuboid hexahedron. Therefore, the height corresponding to the bounding box should be approximately perpendicular to the plane formed by the x-axis and y-axis, that is, approximately parallel to the line containing GF. Therefore, the vector with an angle less than the preset angle value is determined as the height vector. As an example, the preset angle value is 8°.

[0071] S820 determines the height WG corresponding to the current pose of the target suitcase based on the height vector.

[0072] Here, after determining the height vector, the magnitude of the height vector is obtained, which gives the height WG corresponding to the current pose of the target suitcase.

[0073] S900, if WG / DB is greater than the preset ratio threshold, then the current pose of the target suitcase is determined to be unsafe; where DB is the length of the shortest side other than WG in the size corresponding to the point cloud bounding box.

[0074] Specifically, if WG / DB is greater than the preset ratio threshold, it means that the surface of the target suitcase in contact with the conveyor belt in its current pose is not the largest surface of the suitcase, the contact area is small, and the height is high. The suitcase may tip over during the conveyor belt's transport. Therefore, in this case, the current pose of the target suitcase is determined to be unsafe.

[0075] In this embodiment, a vector is drawn from any key vertex of the bounding box corresponding to the suitcase to the three vertices directly connected to it. One of the three vectors is guaranteed to be the height of the target suitcase in its current pose (the edge perpendicular to the plane of the conveyor belt). The normal vector GF corresponding to the position of the key vertex in the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system is perpendicular to the plane formed by the x-axis and y-axis. If the angle between any of the vectors YX, EX, and SX and the line containing GF is less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase. After determining the height vector, the magnitude of the height vector is obtained, thus obtaining the height WG corresponding to the current pose of the target suitcase. If WG / DB is greater than a preset ratio threshold, it indicates that the surface of the target suitcase in contact with the conveyor belt in its current pose is not the largest surface of the suitcase, the contact area is small, and the height is high. The suitcase may tip over during the conveyor belt transportation process. Therefore, in this case, the current pose of the target suitcase is deemed unsafe.

[0076] This embodiment determines the height of the target suitcase in the current Cartesian coordinate system by using the normal vector corresponding to the plane formed by the x and y axes, without introducing other data or requiring additional equipment for measurement. This allows for rapid location of the suitcase's height in the current Cartesian coordinate system, and uses this information to determine whether the suitcase's current pose is safe. This embodiment can promptly identify and handle suitcases with unsafe poses, preventing property damage caused by suitcases slipping or falling, thus improving the user experience.

[0077] In one exemplary embodiment of this application, after step S600, the method further includes:

[0078] S1000, obtain the perpendicularity values ​​YD, ED, and SD of the planes formed by YX, EX, and SX with the x-axis and y-axis of the spatial rectangular coordinate system, respectively; where YD = |90°-YA|; ED = |90°-EA|; SD = |90°-SA|; YA, EA, and SA are the angles between YX, EX, and SX and the planes formed by the x-axis and y-axis of the spatial rectangular coordinate system, respectively.

[0079] S1100, obtain the vector with the largest verticality value among YD, ED and SD and determine it as the height vector corresponding to the current pose of the target suitcase; and jump to step S820.

[0080] Specifically, the higher the perpendicularity value among YD, ED, and SD, the closer it is to being perpendicular to the plane formed by the x and y axes of the Cartesian coordinate system. This value is then determined as the height corresponding to the current pose of the target suitcase. It should be noted that since the bounding box is derived from the actual shape of the suitcase, it may not be a cuboid, but it is a hexahedron approximating a cuboid. Therefore, the height corresponding to the bounding box should be approximately perpendicular to the plane formed by the x and y axes.

[0081] In one exemplary embodiment of this application, after step S800, the method further includes:

[0082] S1200, if WG / DB is less than or equal to the preset proportional threshold, then the current luggage position is determined to be safe.

[0083] Specifically, if WG / DB is less than or equal to a preset proportional threshold, it means that the surface of the target suitcase in contact with the conveyor belt in the current pose should be the largest surface of the suitcase. The target suitcase in this pose is relatively safe and not likely to tip over. Therefore, the current suitcase pose is determined to be safe.

[0084] In one exemplary embodiment of this application, the target suitcase is in a stationary state when the point cloud bounding box corresponding to the current pose of the target suitcase is obtained.

[0085] Specifically, when the target suitcase is stationary, the image obtained by the binocular camera is clearer, which facilitates subsequent processing.

[0086] In one exemplary embodiment of this application, after step S400, the method further includes:

[0087] S1300, the point cloud bounding box is determined as the initial point cloud cluster; wherein, each point cloud data in the initial point cloud cluster has corresponding location information.

[0088] Specifically, each point cloud data has corresponding location information, which can be the three-dimensional coordinates of the point cloud data in the aforementioned spatial rectangular coordinate system.

[0089] S1400, based on the initial point cloud cluster, obtain the corresponding initial point cloud centroid point DA.

[0090] Specifically, the initial point cloud cluster includes not only the point cloud information of the target suitcase surface, but also the point cloud information corresponding to a portion of the depth data, and obtains its corresponding initial point cloud centroid DA.

[0091] S1500, traverse each point cloud according to the nearest neighbor algorithm to obtain a list of sub-point cloud clusters D = (D1, D2, ..., D...). i D n ); i = 1, 2, ..., n; where n is the number of sub-point cloud clusters obtained by traversing the bounding box of the point cloud according to the nearest neighbor algorithm; D i The i-th sub-point cloud cluster is obtained by traversing the bounding box of the point cloud using the nearest neighbor algorithm; where each sub-point cloud cluster has a corresponding sub-centroid, and the distance from each point cloud in each sub-point cloud cluster to the corresponding sub-centroid is less than a preset distance threshold.

[0092] Specifically, the nearest neighbor algorithm is used to traverse each point cloud, dividing all point clouds into several sub-point cloud clusters. In this embodiment, since there may be baggage tags attached to the target suitcase, these tags are mostly long and narrow, protruding from the corresponding position on the target suitcase itself. This creates point cloud noise. For the several sub-point cloud clusters obtained by traversing each point cloud using the nearest neighbor algorithm, the distance from each point cloud in each sub-point cloud cluster to its corresponding sub-centroid is less than a preset distance threshold. That is, the point clouds in each sub-point cloud cluster are neighboring, and the point clouds in the cluster are relatively dense. In other words, the nearest neighbor algorithm divides the rich point clouds near the cluster center into a sub-point cloud cluster based on the initially set cluster center and set parameters (cluster radius, minimum number of point clouds, etc.). And each sub-point cloud cluster has a corresponding sub-centroid.

[0093] S1600, based on D and DA, the center of gravity distance fluctuation value ZB is obtained.

[0094] Specifically, step S1600 includes:

[0095] S1610, based on D and DA, obtain the centroid distance list DZ = (DZ1, DZ2, ..., DZ...). i ..., DZ n ); among them, DZ i D i The distance between the corresponding sub-centroid and the DA centroid.

[0096] S1620, based on DZ, the centroid distance fluctuation value ZB is obtained; where ZB satisfies the following condition: ZB=∑ n i＝1 (DZ i -avg(DZ)) 2 ) / n; where avg() is the preset average value determination function.

[0097] Here, the distance between each sub-centroid and the initial point cloud centroid is obtained, resulting in DZ, and then the corresponding centroid distance fluctuation value ZB is obtained.

[0098] S1700, if ZB is greater than the preset fluctuation value threshold, then the non-target point cloud is removed from the initial point cloud corresponding to the current pose of the target suitcase according to the preset removal method, so as to obtain the updated target point cloud corresponding to the current pose of the target suitcase.

[0099] Specifically, if ZB is greater than the preset fluctuation threshold, it indicates that D contains outlier data, meaning that the centroid of one or more sub-point cloud clusters in D may be too far from the centroid of the initial point cloud cluster. In airport scenarios, baggage tags are mostly long and narrow. When acquiring the initial point cloud cluster of a suitcase, if it includes point cloud noise from the protruding tag portion, the centroid of the initial point cloud cluster of the target suitcase obtained in this case will be farther away from the protruding tag portion compared to the centroid of the initial point cloud cluster of the target suitcase without the protruding tag portion noise. That is, the centroid of the point cloud cluster corresponding to the protruding tag portion is farther from the overall centroid of the target suitcase plus the protruding tag. Therefore, when ZB is greater than the preset fluctuation threshold, it is very likely due to the large fluctuation value caused by outliers corresponding to the protruding tag portion. Therefore, in this embodiment, when ZB is greater than the preset fluctuation value threshold, the non-target point cloud is removed from the initial point cloud corresponding to the current pose of the target suitcase according to the preset removal method, so as to obtain the updated target point cloud corresponding to the current pose of the target suitcase.

[0100] In this embodiment, the nearest neighbor algorithm is used to identify point clouds that are relatively close to each other as a sub-point cloud cluster. This divides all point clouds corresponding to the target suitcase into several sub-point cloud clusters. Then, the distance between the centroid of each sub-point cloud cluster and the centroid of the initial point cloud cluster corresponding to the target suitcase is obtained. Further, fluctuation values ​​are obtained based on these distances, and the magnitude of these fluctuation values ​​is used to determine if outliers exist. Since the protruding label portion is located away from the centroid of the initial point cloud cluster containing the protruding label portion of the target suitcase, it is determined whether the initial point cloud cluster of the target suitcase contains the protruding label portion. Finally, this portion is removed, resulting in the point cloud cluster corresponding to the target suitcase. Due to the large amount of point cloud data, this embodiment divides all point clouds into several sub-point cloud clusters, saving data processing resources. Furthermore, by utilizing the relationship between the protruding label and the centroid of the initial point cloud cluster, it is possible to determine whether there is a protruding label portion on the target suitcase and remove its corresponding point cloud data. This allows the robotic arm's positioning algorithm to accurately calculate the true position and orientation of the suitcase, improving the safety during suitcase transport.

[0101] In one exemplary embodiment of this application, step S1700 includes:

[0102] S1710, if ZB is greater than the preset fluctuation threshold, then arrange the centroid distances in DZ in ascending order to obtain PZ.

[0103] S1720, PZ is grouped according to the binary search method until the non-target point cloud segmentation distance is determined; wherein, the fluctuation value corresponding to the centroid distance of all point cloud clusters that is less than the non-target point cloud segmentation distance is less than or equal to a preset fluctuation value threshold.

[0104] Specifically, as an example: In the first binary search, PZ is divided into two parts from the middle, and the fluctuation value of each part is obtained. The part with a fluctuation value greater than a preset fluctuation value threshold is further divided into two parts from the middle for binary search, until the non-target point cloud segmentation distance is determined. The fluctuation value corresponding to the centroid distance of all point cloud clusters that is less than the non-target point cloud segmentation distance is less than or equal to the preset fluctuation value threshold.

[0105] S1730, delete the sub-point cloud clusters corresponding to several centroid distances that are greater than the non-target point cloud segmentation distance from the initial point cloud clusters corresponding to the current pose of the target suitcase, and obtain the updated target point cloud clusters corresponding to the current pose of the target suitcase.

[0106] Specifically, sub-point cloud clusters corresponding to centroid distances greater than the non-target point cloud segmentation distance are likely to be the sub-point cloud clusters corresponding to the protruding label portion. Therefore, the sub-point cloud clusters corresponding to centroid distances greater than the non-target point cloud segmentation distance are deleted from the initial point cloud cluster corresponding to the current pose of the target suitcase, resulting in the updated target point cloud cluster corresponding to the current pose of the target suitcase. Each sub-point cloud cluster contains 100 to 200 point cloud data points.

[0107] Please refer to Figure 2 As shown, an embodiment of this application provides a 3D reconstruction device 100 for a suitcase based on a binocular camera, the device comprising:

[0108] The acquisition unit 110 is configured to, in response to detecting that a target suitcase has arrived at the target position, control the binocular camera to acquire the target image corresponding to the target suitcase, so as to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are mounted in a converging manner vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera.

[0109] The recognition unit 120 is used to perform foreground recognition on Xa and Xb respectively according to a preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein the target ROI region is a region that contains only the image of the target suitcase.

[0110] Construction unit 130 performs 3D reconstruction of the target suitcase based on Qa and Qb to obtain the point cloud bounding box corresponding to the target suitcase.

[0111] The size acquisition unit 140 is used to obtain the position and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

[0112] Embodiments of this application also provide a computer program product including program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above according to various exemplary embodiments of this application.

[0113] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0114] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0115] In an exemplary embodiment of this application, an electronic device capable of implementing the above-described method is also provided.

[0116] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0117] An electronic device according to this embodiment of the present application. The electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present application.

[0118] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0119] The memory stores program code that can be executed by a processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this application.

[0120] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0121] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0122] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0123] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0124] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of this application.

[0125] In exemplary embodiments of this application, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this application may also be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0126] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0129] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0130] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0131] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0132] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for 3D reconstruction of a suitcase based on a stereo camera, characterized in that, The method includes: S100, in response to detecting that the target suitcase has arrived at the target position, the binocular camera is controlled to acquire the target image corresponding to the target suitcase to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are converged and installed vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera; S200, perform foreground recognition on Xa and Xb respectively according to the preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein, the target ROI region is a region that only contains the target suitcase image; S300, perform 3D reconstruction of the target suitcase based on Qa and Qb to obtain the point cloud bounding box corresponding to the target suitcase; S400 obtains the location and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

2. The method for 3D reconstruction of a suitcase based on a stereo camera according to claim 1, characterized in that... When the binocular camera acquires the target image corresponding to the target suitcase, the target suitcase is in a stationary state.

3. The method for 3D reconstruction of a suitcase based on a binocular camera according to claim 1, characterized in that, 3D reconstruction is performed using the oped3D algorithm.

4. The method for 3D reconstruction of a suitcase based on a binocular camera according to claim 1, characterized in that, Step S300 includes: S310, obtain the corresponding point cloud based on Qa and Qb; S320, stitch together the point clouds corresponding to Qa and Qb to obtain the initial point cloud cluster corresponding to the target suitcase; S330 denoises the initial point cloud clusters corresponding to the target suitcase according to a preset denoising method to obtain the point cloud bounding box corresponding to the target suitcase.

5. The method for 3D reconstruction of a suitcase based on a binocular camera according to claim 1, characterized in that, Binocular cameras need to be calibrated before use.

6. A 3D reconstruction device for a suitcase based on a binocular camera, characterized in that, The device includes: The acquisition unit, in response to detecting that a target suitcase has arrived at the target position, controls the binocular camera to acquire the target image corresponding to the target suitcase, to obtain corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are convergently mounted vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera; The recognition unit is used to perform foreground recognition on Xa and Xb respectively according to a preset recognition method to obtain the corresponding target ROI regions Qa and Qb; wherein, the target ROI region is a region that contains only the image of the target suitcase; The construction unit performs 3D reconstruction of the target suitcase based on Qa and Qb to obtain the point cloud bounding box corresponding to the target suitcase; The size acquisition unit is used to obtain the position and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

7. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 7.

Images