Method and device for determining ROI (Region of Interest) in depth map, medium and electronic equipment

Abstract

The invention provides a method and device for determining an ROI area in a depth map, a medium and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining a target image of a target luggage case at a target position of a conveyor belt according to an RGBD binocular camera; obtaining a target background image corresponding to the target color map; inputting the target color image and the corresponding target background image into a pre-trained foreground separation model to obtain a mask processing area corresponding to the target color image; and according to the mask processing region, performing mask processing on a target depth map corresponding to the target color map to obtain an ROI region corresponding to the target depth map. According to the application, the ROI region in the corresponding target depth map is determined according to the mask region extracted from the target color map corresponding to the target luggage case, and the adaptation degree and the accuracy are higher.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods, apparatus, media and electronic devices for determining ROI regions in depth maps. Background Technology

[0002] In modern airport baggage handling systems, the efficient and accurate transfer of large numbers of suitcases is crucial for ensuring on-time flight departures and improving passenger satisfaction. With the rapid development of automation technology, robotic arms are increasingly used in airport baggage handling scenarios. They can automatically complete tasks such as grasping, carrying, and loading suitcases, greatly improving processing efficiency and reducing labor costs. However, the precise grasping and transfer of suitcases by robotic arms depends on accurate judgment of the suitcase's location. In the complex airport environment, suitcases vary in size, shape, color, and surface material, posing a significant challenge to the robotic arm's positioning. To overcome this challenge, it is necessary to acquire three-dimensional information of the suitcases so that the robotic arm can accurately identify and grasp the target suitcase.

[0003] To acquire 3D information about a suitcase, an RGBD stereo camera is an ideal choice. An RGBD camera can simultaneously capture color images (RGB images) and depth images (depth maps) of a scene. The depth map provides distance information from the object to the camera, a crucial foundation for 3D reconstruction. By combining the RGB and depth maps, the complete 3D shape of the suitcase can be constructed, enabling accurate location determination. However, to accurately acquire the complete 3D shape of the suitcase, it is necessary to determine the Region of Interest (ROI) in the depth map. Current technologies mostly directly divide the depth map into regions, identifying the suitcase's boundaries by analyzing abrupt changes in depth values ​​and texture variations, and thus determining the ROI. Due to the characteristics of depth maps, the ROI determined in this way is not accurate. Therefore, a method that can accurately locate the ROI in the depth map is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method, apparatus, medium, and electronic device for determining the Region of Interest (ROI) in a depth map, which at least partially solves the problems existing in the prior art.

[0005] In a first aspect of this application, a method for determining the Region of Interest (ROI) in a depth map is provided, the method comprising the following steps:

[0006] S100: Acquire a target image of the target suitcase located at the target position on the conveyor belt using an RGBD binocular camera; wherein the target image includes a target color map and a target depth map.

[0007] S200, Obtain the target background image corresponding to the target color map; wherein, the target background image is the image in the corresponding target color map that does not include the target suitcase portion.

[0008] S300: Input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing area corresponding to the target color map; wherein, the mask processing area corresponding to the target color map is the area corresponding to the target suitcase.

[0009] S400, based on the masking processing area, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

[0010] In a second aspect of this application, an apparatus for determining a Region of Interest (ROI) in a depth map is provided, the apparatus comprising:

[0011] The target image acquisition unit is used to acquire a target image of a target suitcase located at a target position on the conveyor belt using an RGBD binocular camera; wherein the target image includes a target color map and a target depth map.

[0012] The background image acquisition unit is used to acquire the target background image corresponding to the target color map; wherein, the target background image is the image in the corresponding target color map that does not include the target suitcase part.

[0013] The processing region determination unit is used to input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing region corresponding to the target color map; wherein, the mask processing region corresponding to the target color map is the region corresponding to the target suitcase.

[0014] The ROI region determination unit is used to perform masking processing on the target depth map corresponding to the target color map according to the masking processing area to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

[0015] In a third aspect of this application, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the aforementioned method for determining the ROI region in the depth map.

[0016] In a fourth aspect of this application, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0017] This application has at least the following beneficial effects:

[0018] The method for determining the Region of Interest (ROI) in a depth map provided in this application uses a pre-trained foreground separation model to subtract the background from the target color image, obtaining the mask processing region corresponding to the target color image. Specifically, the pre-trained foreground separation model extracts the suitcase portion from the color image as the mask processing region. Furthermore, since the corresponding target color image and target depth image are registered, there is a one-to-one correspondence between pixels. Then, based on the mask processing region obtained using the target color image, the target depth image corresponding to the target color image is masked to obtain the ROI region corresponding to the depth image. This completes the foreground recognition of the depth image, removing the background portion from the depth image acquired by the RGBD stereo camera, and obtaining the depth image portion corresponding to the target suitcase (i.e., the foreground). Compared to determining the ROI region of the target depth image using a pre-defined bounding box (not based on the target color image corresponding to the target suitcase), this application determines the ROI region in the corresponding target depth image based on the mask region extracted from the target color image corresponding to the target suitcase, resulting in higher adaptability and accuracy. In addition, when obtaining the mask region, this application does not require manual input of prompt words compared to automatic segmentation models, and can directly obtain the corresponding mask region based on the pre-trained foreground separation model. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the method for determining the ROI region in a depth map provided in this application embodiment;

[0021] Figure 2 This is a structural block diagram of the device for determining the ROI region in the depth map provided in the embodiments of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0024] It should be noted that the following description covers various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0025] Please refer to Figure 1 As shown, embodiments of this application provide a method for determining the Region of Interest (ROI) in a depth map, the method comprising the following steps:

[0026] S100: Acquire a target image of the target suitcase located at the target position on the conveyor belt using an RGBD binocular camera; wherein the target image includes a target color map and a target depth map.

[0027] Specifically, methods for acquiring the target image include:

[0028] In response to detecting that the target suitcase has arrived at the target location, the binocular camera is controlled to acquire the target image corresponding to the target suitcase, so as to obtain the corresponding target images Xa and Xb; wherein, Xa is the first target image obtained by the first imaging lens in the binocular camera; Xb is the second target image obtained by the second imaging lens in the binocular camera; the two imaging lenses of the binocular camera are converged and mounted vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the movement direction of the conveyor belt where the target suitcase is located; the width of the viewing angle range corresponding to the Euler angle of any imaging lens is the width of the conveyor belt where the target suitcase is located; the binocular camera is an RGBD binocular camera.

[0029] Specifically, an RGBD binocular camera is a camera system that combines RGB (red, green, and blue primary colors) color image capture and depth measurement. It typically includes two cameras, acquiring depth information through binocular parallax, structured light, or TOF (Time-of-Flight) technologies, and combining this information with the RGB color image. In this application, the first target image includes a first target color image and a first target depth image; the second target image includes a second target color image and a second target depth image. The two imaging lenses of the binocular camera are convergently mounted vertically above the central axis of the conveyor belt where the target suitcase is located, and the line connecting the optical axes of the two imaging lenses on the same horizontal plane is parallel to the direction of movement of the conveyor belt where the target suitcase is located; the width of the viewing angle corresponding to the Euler angle of any imaging lens is equal to the width of the conveyor belt where the target suitcase is located. Compared to conventional binocular cameras where the optical axes of the two imaging lenses are on the same horizontal plane and the line connecting them is perpendicular to the direction of movement of the conveyor belt, the binocular camera provided in this embodiment is set along the direction of movement of the conveyor belt where the target suitcase is located. Furthermore, the width of the viewing angle corresponding to the Euler angle of each imaging lens is limited to the width of the conveyor belt where the target suitcase is located. That is, the background (excluding the suitcase) of the image of the target suitcase acquired by the binocular camera system set in this embodiment basically only includes the conveyor belt, excluding or including only a small portion of other background, thus reducing the background complexity of the image corresponding to the target suitcase.

[0030] It should be noted that when the binocular camera acquires the target image of the target suitcase, the suitcase must be stationary. Furthermore, the binocular camera requires calibration before use.

[0031] S200, Obtain the target background image corresponding to the target color map; wherein, the target background image is the image in the corresponding target color map that does not include the target suitcase portion.

[0032] Specifically, the target color map is the target color map corresponding to Xa or Xb. The target background image is the image in the corresponding target color map that does not include the target suitcase portion. That is, before acquiring the image corresponding to the suitcase, when the corresponding position on the conveyor belt is empty, the image at the corresponding position is acquired using an RGBD binocular camera from the same viewing angle as the target background image. Furthermore, since the image corresponding to each suitcase in this application is taken after the target suitcase arrives at the corresponding target position, and the binocular camera is pre-calibrated (i.e., the viewing angle is fixed), the pre-obtained target background image can be applied to foreground recognition of each suitcase.

[0033] S300: Input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing area corresponding to the target color map; wherein, the mask processing area corresponding to the target color map is the area corresponding to the target suitcase.

[0034] Specifically, the target color image and the corresponding target background image are input into a pre-trained foreground separation model. The input to this model is the target color image and the corresponding target background image, and the output is the mask region corresponding to the target color image. The mask region is the area corresponding to the target suitcase. In other words, the pre-trained foreground separation model extracts the suitcase portion from the color image as the mask region. The mask processing includes processing a two-dimensional matrix array, specifically a zero-one matrix.

[0035] S400, based on the masking processing area, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

[0036] Specifically, based on the masked processing area obtained using the target color map, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the depth map. Here, the ROI region of the target depth map is the masked processing area, that is, the area corresponding to the target suitcase.

[0037] In one embodiment, the ROI region corresponding to the target depth map is obtained through the following steps:

[0038] The zero-one matrix is ​​multiplied element-wise with the depth map to obtain the ROI region corresponding to the target depth map; wherein the ROI region of the target depth map is the mask processing region.

[0039] Here, since the target color map and target depth map are registered, there is a one-to-one correspondence between pixels. The mask region obtained from the target color map (i.e., a zero-one matrix, where the foreground value is 1 and the non-foreground value is 0) is multiplied element-wise with the depth map to obtain the corresponding ROI region in the target depth map and the corresponding non-ROI region in the depth map.

[0040] In this embodiment, the background in the target color image is subtracted using a pre-trained foreground separation model to obtain the mask processing area corresponding to the target color image. Specifically, the suitcase portion in the color image is extracted using the pre-trained foreground separation model as the mask processing area. Furthermore, since the corresponding target color image and target depth image are registered, there is a one-to-one correspondence between pixels. Then, based on the mask processing area obtained using the target color image, the target depth image corresponding to the target color image is masked to obtain the ROI region corresponding to the depth image. This completes the foreground recognition of the depth image, removing the background portion from the depth image acquired by the RGBD stereo camera, and obtaining the depth image portion corresponding to the target suitcase (i.e., the foreground). Compared to determining the ROI region of the target depth image using a pre-set bounding box (not based on the target color image corresponding to the target suitcase), this embodiment determines the ROI region in the corresponding target depth image based on the mask region extracted from the target color image corresponding to the target suitcase, resulting in higher adaptability and accuracy. In addition, when obtaining the mask region, this embodiment does not require manual input of prompt words compared to the automatic segmentation model, and can directly obtain the corresponding mask region based on the pre-trained foreground separation model.

[0041] In one exemplary embodiment of this application, after step S100, the method further includes:

[0042] S500 performs 3D reconstruction of the target suitcase based on Qa and Qb to obtain the point cloud bounding box corresponding to the target suitcase.

[0043] Specifically, 3D reconstruction is performed based on the oped 3D algorithm. Step S500 further includes:

[0044] S510 obtains the corresponding point cloud based on Qa and Qb.

[0045] S520: stitch together the point clouds corresponding to Qa and Qb to obtain the initial point cloud cluster corresponding to the target suitcase.

[0046] S530 denoises the initial point cloud clusters corresponding to the target suitcase according to a preset denoising method to obtain the point cloud bounding box corresponding to the target suitcase.

[0047] Here, because errors occur when the camera acquires depth data, the initial point cloud corresponding to the target suitcase contains some linear noise due to the ROI region of the target depth map obtained through the mask region mentioned above. Therefore, denoising is required.

[0048] Specifically, the preset noise reduction methods include:

[0049] S531 marks each point cloud within the initial point cloud cluster as an unvisited point cloud.

[0050] S532, Traverse the initial point cloud cluster. If the number of unvisited point clouds within a preset distance range corresponding to any unvisited point cloud is greater than a preset number, then the unvisited point cloud is determined as the initial core object point; otherwise, it is determined as noise.

[0051] S533, add the initial core object point and the unvisited point cloud contained within the preset distance range corresponding to the initial core object point to the target cluster.

[0052] S534 marks each point cloud within the target cluster as a visited point cloud.

[0053] S535, if any visited point cloud in the target cluster contains an unvisited point cloud within a preset distance range, then each unvisited point cloud is added to the candidate list.

[0054] S536, Traverse the candidate list. If the number of point clouds contained within a preset distance range corresponding to any unvisited point cloud is greater than a preset number, then the unvisited point cloud is identified as a key core object point; otherwise, it is identified as noise and removed from the candidate list.

[0055] S537, add the key core object point and each unvisited point cloud within the preset distance range corresponding to the key core object point to the target cluster, and delete it from the candidate list; then jump to step S534 until the candidate list is empty.

[0056] S538, when the candidate list is empty, the corresponding target cluster is determined as the target point cloud corresponding to the target suitcase.

[0057] Steps S531-S538 above constitute the DBSCAN point cloud clustering method. Since the initial point cloud clusters of the target suitcase obtained in this application consist of a main suitcase point cloud plus some linear noise point clouds, and the DBSCAN clustering algorithm is a density-based spatial clustering algorithm, it divides regions with sufficiently high density into clusters and can discover clusters of arbitrary shapes in noisy datasets. In this application, because the point cloud of the linear noise part is not dense enough, by setting reasonable parameters, it can be marked as noise points and filtered out during the clustering process. For the main suitcase point cloud, its corresponding point cloud is sufficiently dense. The core object point is determined by the number of points within its neighborhood (within a preset distance range), and the clustering expands outward from any core object point until all point clouds are traversed, thus obtaining the main suitcase and excluding all linear noise parts with insufficient point cloud density from the cluster corresponding to the main suitcase point cloud. This achieves denoising of the suitcase point cloud data with linear noise.

[0058] In one exemplary embodiment of the application, after step S538, the method further includes:

[0059] S539, if no unvisited point cloud is included within the preset distance range corresponding to each visited point cloud in the target cluster, then the clustering ends and the corresponding target cluster is determined as the target point cloud cluster corresponding to the target suitcase.

[0060] Specifically, if no unvisited point cloud is included within the preset distance range corresponding to each visited point cloud in the target cluster, it means that the corresponding dense points are already in the target cluster. At this time, the clustering ends, and the corresponding target cluster is determined as the target point cloud cluster corresponding to the target suitcase.

[0061] S600 obtains the location and outline information of the target suitcase based on the point cloud bounding box corresponding to the target suitcase.

[0062] Specifically, the obtained point cloud bounding box is located in a spatial Cartesian coordinate system constructed by the binocular camera, and it marks the position and outline information of the target suitcase. It should be noted that the conveyor belt is located on the plane formed by the x and y axes of the spatial Cartesian coordinate system; the outline of the point cloud bounding box is a hexahedron with eight vertices; and the point cloud bounding box has corresponding dimensions.

[0063] In one exemplary embodiment of this application, after step S600, the method further includes:

[0064] S700 selects one vertex from the eight vertices of the point cloud bounding box and identifies it as the critical vertex.

[0065] S800, from the key vertex to the three vertices directly connected to the edge of the hexahedron corresponding to the bounding box contour of the point cloud, vectors are drawn to obtain the first vector YX, the second vector EX, and the third vector SX; wherein any two vectors among YX, EX, and SX are perpendicular to each other.

[0066] Specifically, the first vector YX, the second vector EX, and the third vector SX must contain the height of the target suitcase in the current pose (the edge perpendicular to the plane where the conveyor belt is located).

[0067] S900, obtain the normal vector GF corresponding to the position of the key vertex in the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system.

[0068] S1000 determines the height WG corresponding to the current pose of the target suitcase based on YX, EX, SX and GF.

[0069] Specifically, step S1000 includes:

[0070] S1010, if any of the angles YJ, EJ, SJ, or SJ between YX and GF are less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase.

[0071] Here, the normal vector GF corresponding to the location of the key vertex in the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system is perpendicular to the plane formed by the x-axis and y-axis. If the angle between any vector YX, EX, SX and the line containing GF is less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase. It should be noted that the angle between any vector YX, EX, SX and the line containing GF is less than or equal to 90 degrees. Since the bounding box is obtained based on the actual shape of the suitcase, the bounding box may not be a cuboid, but it is an approximately cuboid hexahedron. Therefore, the height corresponding to the bounding box should be approximately perpendicular to the plane formed by the x-axis and y-axis, that is, approximately parallel to the line containing GF. Therefore, the vector with an angle less than the preset angle value is determined as the height vector. As an example, the preset angle value is 8°.

[0072] S1020, Determine the height WG corresponding to the current pose of the target suitcase based on the height vector.

[0073] Here, after determining the height vector, the magnitude of the height vector is obtained, which gives the height WG corresponding to the current pose of the target suitcase.

[0074] S1100, if WG / DB is greater than the preset ratio threshold, then the current pose of the target suitcase is determined to be unsafe; where DB is the length of the shortest side other than WG in the dimensions corresponding to the point cloud bounding box.

[0075] Specifically, if WG / DB is greater than the preset ratio threshold, it means that the surface of the target suitcase in contact with the conveyor belt in its current pose is not the largest surface of the suitcase, the contact area is small, and the height is high. The suitcase may tip over during the conveyor belt's transport. Therefore, in this case, the current pose of the target suitcase is determined to be unsafe.

[0076] In this embodiment, a vector is drawn from any key vertex of the bounding box corresponding to the suitcase to the three vertices directly connected to it. One of the three vectors is guaranteed to be the height of the target suitcase in its current pose (the edge perpendicular to the plane of the conveyor belt). The normal vector GF corresponding to the position of the key vertex in the plane formed by the x-axis and y-axis of the spatial rectangular coordinate system is perpendicular to the plane formed by the x-axis and y-axis. If the angle between any of the vectors YX, EX, and SX and the line containing GF is less than a preset angle value, then the vector corresponding to the angle less than the preset angle value is determined as the height vector corresponding to the current pose of the target suitcase. After determining the height vector, the magnitude of the height vector is obtained, thus obtaining the height WG corresponding to the current pose of the target suitcase. If WG / DB is greater than a preset ratio threshold, it indicates that the surface of the target suitcase in contact with the conveyor belt in its current pose is not the largest surface of the suitcase, the contact area is small, and the height is high. The suitcase may tip over during the conveyor belt transportation process. Therefore, in this case, the current pose of the target suitcase is deemed unsafe.

[0077] This embodiment determines the height of the target suitcase in the current Cartesian coordinate system by using the normal vector corresponding to the plane formed by the x and y axes, without introducing other data or requiring additional equipment for measurement. This allows for rapid location of the suitcase's height in the current Cartesian coordinate system, and uses this information to determine whether the suitcase's current pose is safe. This embodiment can promptly identify and handle suitcases with unsafe poses, preventing property damage caused by suitcases slipping or falling, thus improving the user experience.

[0078] In one exemplary embodiment of this application, after step S800, the method further includes:

[0079] S1200, obtain the perpendicularity values ​​YD, ED, and SD of the planes formed by YX, EX, and SX with the x-axis and y-axis of the spatial rectangular coordinate system, respectively; where YD = |90°-YA|; ED = |90°-EA|; SD = |90°-SA|; YA, EA, and SA are the angles between YX, EX, and SX and the planes formed by the x-axis and y-axis of the spatial rectangular coordinate system, respectively.

[0080] S1300, obtain the vector with the largest verticality value among YD, ED and SD and determine it as the height vector corresponding to the current pose of the target suitcase; and jump to step S820.

[0081] Specifically, the higher the perpendicularity value among YD, ED, and SD, the closer it is to being perpendicular to the plane formed by the x and y axes of the Cartesian coordinate system. This value is then determined as the height corresponding to the current pose of the target suitcase. It should be noted that since the bounding box is derived from the actual shape of the suitcase, it may not be a cuboid, but it is a hexahedron approximating a cuboid. Therefore, the height corresponding to the bounding box should be approximately perpendicular to the plane formed by the x and y axes.

[0082] In one exemplary embodiment of this application, after step S1000, the method further includes:

[0083] S1400, if WG / DB is less than or equal to the preset proportional threshold, then the current luggage position is determined to be safe.

[0084] Specifically, if WG / DB is less than or equal to a preset proportional threshold, it means that the surface of the target suitcase in contact with the conveyor belt in the current pose should be the largest surface of the suitcase. The target suitcase in this pose is relatively safe and not likely to tip over. Therefore, the current suitcase pose is determined to be safe.

[0085] In one exemplary embodiment of this application, the target suitcase is in a stationary state when the point cloud bounding box corresponding to the current pose of the target suitcase is obtained.

[0086] Specifically, when the target suitcase is stationary, the image obtained by the binocular camera is clearer, which facilitates subsequent processing.

[0087] In one exemplary embodiment of this application, after step S600, the method further includes:

[0088] S1500, the point cloud bounding box is determined as the initial point cloud cluster; wherein, each point cloud data in the initial point cloud cluster has corresponding location information.

[0089] Specifically, each point cloud data has corresponding location information, which can be the three-dimensional coordinates of the point cloud data in the aforementioned spatial rectangular coordinate system.

[0090] S1600, based on the initial point cloud cluster, obtain the corresponding initial point cloud centroid point DA.

[0091] Specifically, the initial point cloud cluster includes not only the point cloud information of the target suitcase surface, but also the point cloud information corresponding to a portion of the depth data, and obtains its corresponding initial point cloud centroid DA.

[0092] S1700, traverse each point cloud according to the nearest neighbor algorithm to obtain a list of sub-point cloud clusters D = (D1, D2, ..., D...). i D n ); i = 1, 2, ..., n; where n is the number of sub-point cloud clusters obtained by traversing the bounding box of the point cloud according to the nearest neighbor algorithm; D i The i-th sub-point cloud cluster is obtained by traversing the bounding box of the point cloud using the nearest neighbor algorithm; where each sub-point cloud cluster has a corresponding sub-centroid, and the distance from each point cloud in each sub-point cloud cluster to the corresponding sub-centroid is less than a preset distance threshold.

[0093] Specifically, the nearest neighbor algorithm is used to traverse each point cloud, dividing all point clouds into several sub-point cloud clusters. In this embodiment, since there may be baggage tags attached to the target suitcase, these tags are mostly long and narrow, protruding from the corresponding position on the target suitcase itself. This creates point cloud noise. For the several sub-point cloud clusters obtained by traversing each point cloud using the nearest neighbor algorithm, the distance from each point cloud in each sub-point cloud cluster to its corresponding sub-centroid is less than a preset distance threshold. That is, the point clouds in each sub-point cloud cluster are neighboring, and the point clouds in the cluster are relatively dense. In other words, the nearest neighbor algorithm divides the rich point clouds near the cluster center into a sub-point cloud cluster based on the initially set cluster center and set parameters (cluster radius, minimum number of point clouds, etc.). And each sub-point cloud cluster has a corresponding sub-centroid.

[0094] S1800, based on D and DA, the center of gravity distance fluctuation value ZB is obtained.

[0095] Specifically, step S1600 includes:

[0096] S1810, based on D and DA, obtain the centroid distance list DZ = (DZ1, DZ2, ..., DZ...). i ..., DZ n ); among them, DZ i D i The distance between the corresponding sub-centroid and the DA centroid.

[0097] S1820, based on DZ, the centroid distance fluctuation value ZB is obtained; where ZB satisfies the following condition: ZB=∑ n i＝1 (DZ i -avg(DZ)) 2 ) / n; where avg() is the preset average value determination function.

[0098] Here, the distance between each sub-centroid and the initial point cloud centroid is obtained, resulting in DZ, and then the corresponding centroid distance fluctuation value ZB is obtained.

[0099] S1900, if ZB is greater than the preset fluctuation threshold, then the non-target point cloud is removed from the initial point cloud corresponding to the current pose of the target suitcase according to the preset removal method, so as to obtain the updated target point cloud corresponding to the current pose of the target suitcase.

[0100] Specifically, if ZB is greater than the preset fluctuation threshold, it indicates that D contains outlier data, meaning that the centroid of one or more sub-point cloud clusters in D may be too far from the centroid of the initial point cloud cluster. In airport scenarios, baggage tags are mostly long and narrow. When acquiring the initial point cloud cluster of a suitcase, if it includes point cloud noise from the protruding tag portion, the centroid of the initial point cloud cluster of the target suitcase obtained in this case will be farther away from the protruding tag portion compared to the centroid of the initial point cloud cluster of the target suitcase without the protruding tag portion noise. That is, the centroid of the point cloud cluster corresponding to the protruding tag portion is farther from the overall centroid of the target suitcase plus the protruding tag. Therefore, when ZB is greater than the preset fluctuation threshold, it is very likely due to the large fluctuation value caused by outliers corresponding to the protruding tag portion. Therefore, in this embodiment, when ZB is greater than the preset fluctuation value threshold, the non-target point cloud is removed from the initial point cloud corresponding to the current pose of the target suitcase according to the preset removal method, so as to obtain the updated target point cloud corresponding to the current pose of the target suitcase.

[0101] In this embodiment, the nearest neighbor algorithm is used to identify point clouds that are relatively close to each other as a sub-point cloud cluster. This divides all point clouds corresponding to the target suitcase into several sub-point cloud clusters. Then, the distance between the centroid of each sub-point cloud cluster and the centroid of the initial point cloud cluster corresponding to the target suitcase is obtained. Further, fluctuation values ​​are obtained based on these distances, and the magnitude of these fluctuation values ​​is used to determine if outliers exist. Since the protruding label portion is located away from the centroid of the initial point cloud cluster containing the protruding label portion of the target suitcase, it is determined whether the initial point cloud cluster of the target suitcase contains the protruding label portion. Finally, this portion is removed, resulting in the point cloud cluster corresponding to the target suitcase. Due to the large amount of point cloud data, this embodiment divides all point clouds into several sub-point cloud clusters, saving data processing resources. Furthermore, by utilizing the relationship between the protruding label and the centroid of the initial point cloud cluster, it is possible to determine whether there is a protruding label portion on the target suitcase and remove its corresponding point cloud data. This allows the robotic arm's positioning algorithm to accurately calculate the true position and orientation of the suitcase, improving the safety during suitcase transport.

[0102] In one exemplary embodiment of this application, step S1900 includes:

[0103] S1910, if ZB is greater than the preset fluctuation threshold, then arrange the centroid distances in DZ in ascending order.

[0104] S1920, PZ is grouped according to the binary search method until the non-target point cloud segmentation distance is determined; wherein, the fluctuation value corresponding to the centroid distance of all point cloud clusters that is less than the non-target point cloud segmentation distance is less than or equal to a preset fluctuation value threshold.

[0105] Specifically, as an example: In the first binary search, PZ is divided into two parts from the middle, and the fluctuation value of each part is obtained. The part with a fluctuation value greater than a preset fluctuation value threshold is further divided into two parts from the middle for binary search, until the non-target point cloud segmentation distance is determined. The fluctuation value corresponding to the centroid distance of all point cloud clusters that is less than the non-target point cloud segmentation distance is less than or equal to the preset fluctuation value threshold.

[0106] S1930, delete the sub-point cloud clusters corresponding to several centroid distances that are greater than the non-target point cloud segmentation distance from the initial point cloud clusters corresponding to the current pose of the target suitcase, and obtain the updated target point cloud clusters corresponding to the current pose of the target suitcase.

[0107] Specifically, sub-point cloud clusters corresponding to centroid distances greater than the non-target point cloud segmentation distance are likely to be the sub-point cloud clusters corresponding to the protruding label portion. Therefore, the sub-point cloud clusters corresponding to centroid distances greater than the non-target point cloud segmentation distance are deleted from the initial point cloud cluster corresponding to the current pose of the target suitcase, resulting in the updated target point cloud cluster corresponding to the current pose of the target suitcase. Each sub-point cloud cluster contains 100 to 200 point cloud data points.

[0108] Please refer to Figure 2 As shown, an embodiment of this application provides a device 100 for determining the Region of Interest (ROI) in a depth map, the device comprising:

[0109] The target image acquisition unit 110 is used to acquire a target image of a target suitcase located at the target position on the conveyor belt using an RGBD binocular camera; wherein the target image includes a target color map and a target depth map.

[0110] Background image acquisition unit 120 is used to acquire a target background image corresponding to a target color map; wherein, the target background image is an image in the corresponding target color map that does not include the target suitcase portion.

[0111] The processing region determination unit 130 is used to input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing region corresponding to the target color map; wherein, the mask processing region corresponding to the target color map is the region corresponding to the target suitcase.

[0112] The ROI region determination unit 140 is used to perform masking processing on the target depth map corresponding to the target color map according to the masking processing area to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

[0113] Embodiments of this application also provide a computer program product including program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above according to various exemplary embodiments of this application.

[0114] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0115] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0116] In an exemplary embodiment of this application, an electronic device capable of implementing the above-described method is also provided.

[0117] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0118] An electronic device according to this embodiment of the present application. The electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0119] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0120] The memory stores program code that can be executed by a processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this application.

[0121] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0122] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0123] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0124] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of this application.

[0126] In exemplary embodiments of this application, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this application may also be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0127] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0128] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0129] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0130] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0131] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0132] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0133] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining the Region of Interest (ROI) in a depth map, characterized in that, The method includes: S100: Acquire a target image of the target suitcase located at the target position on the conveyor belt using an RGBD binocular camera; wherein, the target image includes a target color map and a target depth map; S200, Obtain the target background image corresponding to the target color map; wherein, the target background image is the image in the corresponding target color map that does not include the target suitcase portion; S300: Input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing area corresponding to the target color map; wherein, the mask processing area corresponding to the target color map is the area corresponding to the target suitcase; S400, based on the masking processing area, the target depth map corresponding to the target color map is masked to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

2. The method for determining the ROI region in a depth map according to claim 1, characterized in that, The target image includes a first target image and a second target image; the first target image is obtained using the first imaging lens of an RGBD binocular camera; the second target image is obtained using the second imaging lens of an RGBD binocular camera.

3. The method for determining the ROI region in a depth map according to claim 1, characterized in that, After step S400, the method further includes: S500, based on the mask processing area corresponding to the target color map and the ROI area corresponding to the target depth map, perform three-dimensional reconstruction of the target suitcase.

4. The method for determining the ROI region in a depth map according to claim 1, characterized in that, The mask is a zero-one matrix.

5. The method for determining the ROI region in a depth map according to claim 4, characterized in that, Step S500 includes: The zero-one matrix is ​​multiplied element-wise with the depth map to obtain the ROI region corresponding to the target depth map; wherein the ROI region of the target depth map is the mask processing region.

6. A device for determining the Region of Interest (ROI) in a depth map, characterized in that, The device includes: The target image acquisition unit is used to acquire a target image of a target suitcase located at a target position on the conveyor belt using an RGBD binocular camera; wherein the target image includes a target color map and a target depth map; The background image acquisition unit is used to acquire a target background image corresponding to the target color map; wherein, the target background image is an image in the corresponding target color map that does not include the target suitcase portion; The processing region determination unit is used to input the target color map and the corresponding target background image into the pre-trained foreground separation model to obtain the mask processing region corresponding to the target color map; wherein, the mask processing region corresponding to the target color map is the region corresponding to the target suitcase; The ROI region determination unit is used to perform masking processing on the target depth map corresponding to the target color map according to the masking processing area to obtain the ROI region corresponding to the target depth map; wherein, the ROI region of the target depth map is the masking processing area.

7. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 6.

Images