Parcel positioning method and apparatus, electronic device, and storage medium

Abstract

The application discloses a parcel positioning method and device, electronic equipment and storage medium, which are used for segmenting adhered parcels and positioning the parcels. In the embodiment of the application, the depth value of each pixel point in the projection depth map containing the parcel is determined based on the projection depth map, and a candidate segmentation point is determined from the pixel point according to the depth value; for each candidate segmentation point, a segmentation point is determined by using a local threshold segmentation method; then the segmentation point is removed from the projection depth map, and finally the position of the parcel is determined based on the remaining points in the projection depth map. In the application, the segmentation point is determined based on the projection depth map containing the parcel, which excludes the interference of the parcel surface texture, then the parcel is segmented based on the segmentation point, which can accurately segment the adhered parcel, the parcel is positioned based on the remaining points, and the accuracy of the three-dimensional position information of the parcel is improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional processing technology, and in particular to a package positioning method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of 3D vision technology, it has been applied to various aspects of the industrial field. Compared to 2D vision, 3D vision can acquire accurate spatial position information of objects, giving it a significant advantage in the industrial sector.

[0003] In the parcel sorting process, there exists a single-item separation parcel sorting device. This device sorts a large number of incoming parcels into orderly parcels with consistent spacing, ensuring that subsequent equipment can obtain information about each individual parcel. One of the core technologies of this single-item separation device is to acquire the positional information of each parcel using a 3D camera, and then control the movement of different motors to achieve parcel separation. However, while related technologies use edge detection methods for parcel separation, this method cannot separate parcels that are stuck together, leading to inaccurate parcel positioning. Summary of the Invention

[0004] The purpose of this application is to provide a package positioning method, apparatus, electronic device, and storage medium for separating and positioning adhered packages.

[0005] In a first aspect, embodiments of this application provide a package positioning method, including:

[0006] Based on the projection depth map containing the package, the depth value of each pixel in the projection depth map is determined, and candidate segmentation points are determined from the pixels according to the depth values;

[0007] For each candidate segmentation point, the candidate segmentation point is used as a target candidate segmentation point. Starting from the target candidate segmentation point, a specified number of two-dimensional coordinates of pixels are continuously acquired along the gradient descent direction in units of pixels. Based on the acquired two-dimensional coordinates, the slope of the straight line fitted to the corresponding pixel is determined. If the slope is greater than a specified threshold, the target candidate segmentation point corresponding to the slope is determined as the segmentation point. The horizontal coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value.

[0008] The segmentation point is removed from the projection depth map, and the location of the package is determined based on the remaining points in the projection depth map.

[0009] In the present application, the segmentation points are determined based on the projection depth map containing the package, the interference of the package surface texture is excluded, then the package is segmented based on the segmentation points, the package can be accurately segmented, the package is positioned based on the remaining points, and the accuracy of the package positioning is improved.

[0010] In some possible embodiments, the projection depth map containing the package is obtained according to the following method:

[0011] Obtain a color space (RGB) image and a depth image containing the package;

[0012] Perform deep learning instance segmentation on the RGB image to obtain the mask image;

[0013] Align the pixel points of the depth image with the pixel points of the RGB image based on the mapping relationship between the depth image and the RGB image to obtain the depth value of the pixel points;

[0014] Obtain point cloud data based on the depth value of the pixel points and the mask image;

[0015] Project the pixel points in the point cloud data to a reference plane to obtain a projection depth map containing the package.

[0016] In the present application, the RGB image containing the package is subjected to deep learning instance segmentation, which can roughly segment the adhered package, that is, the preliminary screening is realized, and the calculation amount for subsequent determination of segmentation points is reduced.

[0017] In some possible embodiments, the projection of the pixel points in the point cloud data to the reference plane to obtain the projection depth map containing the package comprises:

[0018] According to the platform plane equation pre-calibrated, project the pixel points in the point cloud data to the reference plane to obtain a projection map;

[0019] Determine the distance between each pixel point in the point cloud data and the projection map;

[0020] Generate the projection depth map based on the distance and the projection map.

[0021] In the present application, the projection depth map is used to determine the position of the package, which is convenient for calculating the length, width and other size information of the package, and further makes the positioning of the package more accurate.

[0022] In some possible embodiments, the obtaining of the point cloud data based on the depth value of the pixel points and the mask image comprises:

[0023] Input the depth value of the pixel points and the mask image into a depth camera with pre-calibrated intrinsic parameters;

[0024] Obtain the point cloud data generated by the depth camera.

[0025] In this application, point cloud data containing packages is acquired using a depth camera, which reduces the difficulty of subsequent calculations and saves computing power.

[0026] In some possible embodiments, determining candidate segmentation points from pixels based on the depth value includes:

[0027] For each pixel in the projection depth map, the pixel is taken as the target pixel.

[0028] Using the target pixel as the center, determine the pixel with the maximum depth value and the pixel with the minimum depth value within the specified area;

[0029] If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than a first threshold, and the depth value of the target pixel is less than a second threshold, then the target pixel is selected as a candidate segmentation point.

[0030] In this application, candidate segmentation points are determined from the projected depth map by using gradients, which reduces the computational load for subsequent segmentation point determination and thus improves efficiency.

[0031] In some possible embodiments, the second threshold is the depth value of the pixel with the largest depth value and the average depth value of the pixel with the smallest depth value within the specified area.

[0032] In some possible embodiments, determining the location of the package based on the remaining points in the projected depth map includes:

[0033] Perform connected component analysis on the remaining points to obtain the location of the package.

[0034] In this application, in order to improve the accuracy of package location, after determining the package's segmentation point, a connected component analysis is performed on the remaining points.

[0035] Secondly, this application also provides a package positioning device, the device comprising:

[0036] The candidate segmentation point determination module is used to determine the depth value of each pixel in the projection depth map based on the projection depth map containing the package, and to determine candidate segmentation points from the pixels according to the depth values.

[0037] The segmentation point determination module is used to, for each candidate segmentation point, take the candidate segmentation point as a target candidate segmentation point, and continuously acquire the two-dimensional coordinates of a specified number of pixels along the gradient descent direction, starting from the target candidate segmentation point; and determine the slope of the straight line fitted to the corresponding pixel based on the acquired two-dimensional coordinates. If the slope is greater than a specified threshold, the target candidate segmentation point corresponding to the slope is determined as the segmentation point; wherein the horizontal coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value;

[0038] The package location determination module is used to remove the segmentation point from the projection depth map and determine the location of the package based on the remaining points in the projection depth map.

[0039] In some possible embodiments, the projected depth map containing the package is obtained according to the following method:

[0040] Obtain the RGB image and depth image containing the package;

[0041] The RGB image is segmented using deep learning to obtain the mask image;

[0042] Based on the mapping relationship between the depth image and the RGB image, the pixels of the depth image are aligned with the pixels of the RGB image to obtain the depth value of the pixel;

[0043] Point cloud data is obtained based on the depth values ​​of the pixels and the mask image;

[0044] The pixels in the point cloud data are projected onto a reference plane to obtain a projection depth map containing the enclosed data.

[0045] In some possible embodiments, when the candidate segmentation point determination module performs the operation of projecting pixels in the point cloud data onto a reference plane to obtain a projection depth map containing the enclosed data, it is configured to:

[0046] According to the pre-calibrated platform plane equation, the pixels in the point cloud data are projected onto the reference plane to obtain a projection map;

[0047] Determine the distance between each pixel in the point cloud data and the projection map;

[0048] The projection depth map is generated based on the distance and the projection map.

[0049] In some possible embodiments, when the candidate segmentation point determination module performs point cloud data based on the depth values ​​of the pixels and the mask image, it is configured to:

[0050] The depth values ​​of the pixels and the mask image are input into a depth camera with pre-calibrated intrinsic parameters;

[0051] Obtain the point cloud data generated by the depth camera.

[0052] In some possible embodiments, when the segmentation point determination module performs the task of determining candidate segmentation points from pixels based on the depth value, it is configured to:

[0053] For each pixel in the projection depth map, the pixel is taken as the target pixel.

[0054] Using the target pixel as the center, determine the pixel with the maximum depth value and the pixel with the minimum depth value within the specified area;

[0055] If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than a first threshold, and the depth value of the target pixel is less than a second threshold, then the target pixel is selected as a candidate segmentation point.

[0056] In some possible embodiments, the second threshold is the depth value of the pixel with the largest depth value and the average depth value of the pixel with the smallest depth value within the specified area.

[0057] In some possible embodiments, when the package location determination module performs the task of determining the location of the package based on the remaining points in the projected depth map, it is configured to:

[0058] Perform connected component analysis on the remaining points to obtain the location of the package.

[0059] Thirdly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods provided in the first aspect embodiment of this application.

[0060] Fourthly, another embodiment of this application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for causing a computer to perform any of the methods provided in the first aspect of this application. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is an application scenario diagram of a package positioning method provided in an embodiment of this application;

[0063] Figure 2 A flowchart illustrating a package positioning method provided in an embodiment of this application;

[0064] Figure 3 A schematic flowchart illustrating the determination of a projection depth map for a package positioning method provided in this application embodiment;

[0065] Figure 4 A schematic diagram of a package mask for a package positioning method provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram illustrating the process of acquiring point cloud data for a package positioning method provided in an embodiment of this application.

[0067] Figure 6 A schematic flowchart illustrating the process of obtaining a projection depth map for a package positioning method provided in this application embodiment;

[0068] Figure 7 A schematic projection of a package positioning method provided in an embodiment of this application;

[0069] Figure 8 A flowchart illustrating the process of determining candidate segmentation points in a package positioning method provided in this application embodiment;

[0070] Figure 9 A schematic diagram illustrating the determination of candidate segmentation points in a package positioning method provided in an embodiment of this application;

[0071] Figure 10 A flowchart illustrating the determination of segmentation points in a package positioning method provided in this application embodiment;

[0072] Figure 11 A schematic diagram illustrating the determination of segmentation points in a package positioning method provided in this application embodiment;

[0073] Figure 12 A schematic diagram of an apparatus for a package positioning method provided in an embodiment of this application;

[0074] Figure 13 This is a schematic diagram of an electronic device for a package positioning method provided in an embodiment of this application. Detailed Implementation

[0075] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0076] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0077] The inventors discovered that with the development of 3D vision technology, it has been applied to various aspects of the industrial field. Compared to 2D vision, 3D vision can acquire accurate spatial position information of objects, giving it a significant advantage in industrial applications.

[0078] In the parcel sorting process, there exists a single-item separation parcel sorting device. This device sorts a large number of incoming parcels into orderly parcels with consistent spacing, ensuring that subsequent equipment can obtain information about each individual parcel. One of the core technologies of this single-item separation device is to acquire the position information of each parcel using a 3D camera, and then control the movement of different motors to achieve parcel separation. However, while related technologies use edge detection methods for parcel separation, this method cannot separate parcels that are stuck together.

[0079] In view of this, this application proposes a package positioning method, apparatus, electronic device, and storage medium to solve the above-mentioned problems. The inventive concept of this application can be summarized as follows: based on a projection depth map containing the package, the depth value of each pixel in the projection depth map is determined, and candidate segmentation points are determined from the pixels according to the depth values; for each candidate segmentation point, a local threshold segmentation method is used to determine the segmentation point; then the segmentation points are removed from the projection depth map, and finally the position of the package is determined based on the remaining points in the projection depth map.

[0080] For ease of understanding, the following detailed description of a package positioning method provided by this application, in conjunction with the accompanying drawings, will illustrate the following:

[0081] like Figure 1 The diagram shown illustrates an application scenario of the package location method according to an embodiment of this application. The diagram includes: a server 10, a memory 20, and a camera 30.

[0082] Server 10 retrieves the image containing the package captured by camera 30 from memory 20, processes the image to obtain a projection depth map containing the package, and then determines the depth value of each pixel in the projection depth map based on the depth value. Candidate segmentation points are then determined from the pixels based on the depth value. For each candidate segmentation point, it is used as a target candidate segmentation point. Starting from the target candidate segmentation point, a specified number of two-dimensional coordinates of pixels are continuously acquired along the gradient descent direction, pixel by pixel. The slope of the straight line fitted to the corresponding pixel is determined based on the acquired two-dimensional coordinates. If the slope is greater than a specified threshold, the target candidate segmentation point corresponding to the slope is determined as the segmentation point. The segmentation points are removed from the projection depth map, and the location of the package is determined based on the remaining points in the projection depth map.

[0083] The description in this application focuses on a single server or camera; however, those skilled in the art should understand that the illustrated camera 30, server 10, and memory 20 are intended to illustrate the operation of the camera, server, and memory involved in the technical solutions of this application. The detailed description of a single server and memory is at least for ease of explanation and does not imply any limitation on the number, type, or location of cameras and servers. It should be noted that adding additional modules to or removing individual modules from the illustrated environment does not change the underlying concept of the exemplary embodiments of this application.

[0084] It should be noted that the memory in the embodiments of this application can be, for example, a cache system, hard disk storage, memory storage, etc. Furthermore, the package positioning method proposed in this application is not only applicable to… Figure 1 The application scenarios shown are also applicable to any device that requires package location tracking.

[0085] The package location method proposed in this application will be described in detail below, such as... Figure 2 The diagram shown is an overall flowchart of a package positioning method provided in an embodiment of this application, wherein:

[0086] In step 201: Based on the projection depth map containing the package, determine the depth value of each pixel in the projection depth map, and determine candidate segmentation points from the pixels according to the depth values;

[0087] In step 202: For each candidate segmentation point, the candidate segmentation point is used as the target candidate segmentation point. Starting from the target candidate segmentation point, a specified number of two-dimensional coordinates of pixels are continuously acquired along the gradient descent direction in units of pixels. Based on the acquired two-dimensional coordinates, the slope of the straight line fitted to the corresponding pixel is determined. If the slope is greater than a specified threshold, the target candidate segmentation point corresponding to the slope is determined as the segmentation point. The horizontal coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value.

[0088] In step 203: the segmentation points are removed from the projection depth map, and the location of the package is determined based on the remaining points in the projection depth map.

[0089] For ease of understanding, the following is... Figure 2 The steps involved are explained in detail below:

[0090] In this embodiment of the application, in order to determine the three-dimensional information of the package's length, width, and height, the package's position is determined based on its projection depth map. The projection depth map can be obtained using methods such as... Figure 3 The steps shown are determined, wherein:

[0091] In step 301: Obtain the RGB image and depth image containing the package;

[0092] In step 302: Deep learning instance segmentation is performed on the RGB image to obtain a mask image;

[0093] like Figure 4 As shown, the left side is the RGB image containing the element A, and the right side is the mask image containing the element A.

[0094] In step 303: Based on the mapping relationship between the depth image and the RGB image, the pixels of the depth image are aligned with the pixels of the RGB image to obtain the depth value of the pixel;

[0095] In this application, after obtaining the coordinates of the pixels in the RGB image containing the package, the pixels in the RGB image are mapped to the depth image according to the pre-defined mapping relationship between the depth image and the RGB image, thereby obtaining the depth value of each pixel.

[0096] In step 304: Point cloud data is obtained based on the depth values ​​of pixels and the mask image;

[0097] In this embodiment of the application, the method for acquiring point cloud data can be specifically implemented as follows: Figure 5 The steps shown are as follows:

[0098] In step 501: The depth values ​​of the pixels and the mask image are input into a depth camera with pre-calibrated intrinsic parameters;

[0099] In step 502: Obtain point cloud data generated by the depth camera.

[0100] In this application, in order to make the generated point cloud data more accurate, the intrinsic parameters of the depth camera are calibrated in advance by those skilled in the art according to actual needs. In specific implementation, the depth values ​​of the pixels and the mask image can be directly input into the depth camera to obtain the point cloud data.

[0101] In step 305: the pixels in the point cloud data are projected onto the reference plane to obtain a projection depth map containing the wrapper.

[0102] In this application, step 305 can be implemented as follows: Figure 6 The steps shown are as follows:

[0103] In step 601: According to the pre-calibrated platform plane equation, the pixels in the point cloud data are projected onto the reference plane to obtain the projection map;

[0104] Among them, the projection diagram is as follows Figure 7 As shown.

[0105] In step 602: Determine the distance between each pixel in the point cloud data and the projection map;

[0106] In a more specific implementation, the distance between each pixel and the projection image can be determined by determining the Euclidean distance between the pixel and the projection image, or other methods can be used to determine the distance between the pixel and the projection image. This application does not limit this method.

[0107] In step 603: Generate a projection depth map based on the distance and projection map.

[0108] The following section provides a detailed explanation of the process of selecting candidate segmentation points from pixels, such as... Figure 8 As shown, where:

[0109] In step 801: For each pixel in the projection depth map, the pixel is taken as the target pixel;

[0110] In step 802: Using the target pixel as the center, determine the pixel with the maximum and minimum depth values ​​within the specified area;

[0111] like Figure 9 As shown, with pixel A as the target pixel, the pixel with the largest depth value within the specified area is pixel B, and the pixel with the smallest depth value is pixel C.

[0112] In step 803: If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than the first threshold, and the depth value of the target pixel is less than the second threshold, then the target pixel is selected as a candidate segmentation point.

[0113] In some embodiments, the first threshold is an empirical value determined by those skilled in the art based on experience, and the second preset value is the average depth value of the pixel with the largest depth value and the pixel with the smallest depth value within the specified area. Figure 9 For example, the second threshold is the average depth value of pixel B and pixel C.

[0114] In summary, the above is adopted Figure 8The steps described above iterate through all pixels to obtain all candidate segmentation points. The following is a detailed explanation of the steps for obtaining segmentation points, such as... Figure 10 As shown:

[0115] In step 1001: For each candidate segmentation point, the candidate segmentation point is used as the target candidate segmentation point;

[0116] In step 1002: Starting from the target candidate segmentation point, continuously acquire the two-dimensional coordinates of a specified number of pixels along the gradient descent direction in units of pixels; where the horizontal coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value;

[0117] In step 1003: the slope of the straight line fitted to the corresponding pixel point is determined based on the obtained two-dimensional coordinates;

[0118] In step 1004: If the slope is greater than a specified threshold, then the target candidate segmentation point corresponding to the slope is determined as the segmentation point.

[0119] For example: Figure 11 As shown, with pixel A as the target pixel and a specified number of 3, 3 pixels are selected starting from A along the gradient descent direction, and pixel A and the selected 3 pixels are fitted into a straight line.

[0120] In summary, candidate segmentation points are first determined based on the projection depth map containing the package, and then the segmentation point is selected from the candidate segmentation points. In this application, after the segmentation point is determined, connected component analysis is performed on the remaining points to obtain the location of the package.

[0121] In this application, the segmentation points are determined based on the projection depth map containing the package, eliminating the interference of the package surface texture. Then, the package is segmented based on the segmentation points, which can accurately separate the sticky packages. The package is located based on the remaining points, which improves the accuracy of the package's three-dimensional position information.

[0122] like Figure 12 As shown, based on the same inventive concept, a package positioning device 1200 is proposed, comprising:

[0123] The candidate segmentation point determination module 12001 is used to determine the depth value of each pixel in the projection depth map based on the projection depth map containing the package, and to determine candidate segmentation points from the pixels according to the depth values.

[0124] The segmentation point determination module 12002 is used to, for each candidate segmentation point, take the candidate segmentation point as a target candidate segmentation point, take the target candidate segmentation point as the starting point, continuously acquire the two-dimensional coordinates of a specified number of pixels in units of pixels along the gradient descent direction; and determine the slope of the straight line fitted to the corresponding pixel based on the acquired two-dimensional coordinates. If the slope is greater than a specified threshold, the target candidate segmentation point corresponding to the slope is determined as the segmentation point; wherein the horizontal coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value;

[0125] Package location determination module 12003 is used to remove the segmentation point from the projection depth map and determine the location of the package based on the remaining points in the projection depth map.

[0126] In some possible embodiments, the projected depth map containing the package is obtained according to the following method:

[0127] Obtain the RGB image and depth image containing the package;

[0128] The RGB image is segmented using deep learning to obtain the mask image;

[0129] Based on the mapping relationship between the depth image and the RGB image, the pixels of the depth image are aligned with the pixels of the RGB image to obtain the depth value of the pixel;

[0130] Point cloud data is obtained based on the depth values ​​of the pixels and the mask image;

[0131] The pixels in the point cloud data are projected onto a reference plane to obtain a projection depth map containing the enclosed data.

[0132] In some possible embodiments, when the candidate segmentation point determination module performs the operation of projecting pixels in the point cloud data onto a reference plane to obtain a projection depth map containing the enclosed data, it is configured to:

[0133] According to the pre-calibrated platform plane equation, the pixels in the point cloud data are projected onto the reference plane to obtain a projection map;

[0134] Determine the distance between each pixel in the point cloud data and the projection map;

[0135] The projection depth map is generated based on the distance and the projection map.

[0136] In some possible embodiments, when the candidate segmentation point determination module performs point cloud data based on the depth values ​​of the pixels and the mask image, it is configured to:

[0137] The depth values ​​of the pixels and the mask image are input into a depth camera with pre-calibrated intrinsic parameters;

[0138] Obtain the point cloud data generated by the depth camera.

[0139] In some possible embodiments, when the segmentation point determination module performs the task of determining candidate segmentation points from pixels based on the depth value, it is configured to:

[0140] For each pixel in the projection depth map, the pixel is taken as the target pixel.

[0141] Using the target pixel as the center, determine the pixel with the maximum depth value and the pixel with the minimum depth value within the specified area;

[0142] If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than a first threshold, and the depth value of the target pixel is less than a second threshold, then the target pixel is selected as a candidate segmentation point.

[0143] In some possible embodiments, the second threshold is the depth value of the pixel with the largest depth value and the average depth value of the pixel with the smallest depth value within the specified area.

[0144] In some possible embodiments, when the package location determination module performs the task of determining the location of the package based on the remaining points in the projected depth map, it is configured to:

[0145] Perform connected component analysis on the remaining points to obtain the location of the package.

[0146] Having introduced the package positioning method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0147] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0148] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of the package positioning method according to the various exemplary embodiments of this application described above.

[0149] The following reference Figure 13To describe an electronic device 130 according to this embodiment of the present application. Figure 13 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 13 As shown, the electronic device 130 is presented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0151] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0152] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0153] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0154] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0155] In some possible implementations, various aspects of the package positioning method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the package positioning method according to the various exemplary embodiments of this application described above.

[0156] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] The program product for package positioning according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0158] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0160] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0161] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0162] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A package positioning method, characterized in that, The method includes: Based on the projection depth map containing the package, the depth value of each pixel in the projection depth map is determined, and candidate segmentation points are determined from the pixels according to the depth values; Each candidate segmentation point is sequentially used as the target candidate segmentation point; Starting from the target candidate segmentation point, continuously acquire the two-dimensional coordinates of a specified number of pixels along the gradient descent direction, in units of pixels. The slope of the straight line fitted to the pixel is determined based on the two-dimensional coordinates of the pixel. The target candidate segmentation point whose slope of the fitted straight line is greater than a specified threshold is determined as the segmentation point; wherein the x-coordinate of the pixel represents the distance between it and the target candidate segmentation point, and the y-coordinate represents the depth value; The segmentation points are removed from the projection depth map, and the location of the package is determined based on the remaining points in the projection depth map; Determining the location of the package based on the remaining points in the projection depth map includes: performing connected component analysis on the remaining points to obtain the location of the package; The step of determining candidate segmentation points from pixels based on the depth value includes: For each pixel in the projection depth map, the pixel is taken as the target pixel. Using the target pixel as the center, determine the pixel with the maximum depth value and the pixel with the minimum depth value within the specified area; If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than a first threshold, and the depth value of the target pixel is less than a second threshold, then the target pixel is selected as a candidate segmentation point.

2. The method according to claim 1, characterized in that, The projection depth map containing the package was obtained using the following method: Obtain the RGB image and depth image containing the package; The RGB image is segmented using deep learning to obtain a mask image; Based on the mapping relationship between the depth image and the RGB image, the pixels of the depth image are aligned with the pixels of the RGB image to obtain the depth value of the pixel; Point cloud data is obtained based on the depth values ​​of the pixels and the mask image; The pixels in the point cloud data are projected onto a reference plane to obtain a projection depth map containing the enclosed data.

3. The method according to claim 2, characterized in that, The step of projecting the pixels in the point cloud data onto a reference plane to obtain a projection depth map containing the enclosed data includes: According to the pre-calibrated platform plane equation, the pixels in the point cloud data are projected onto the reference plane to obtain a projection map; Determine the distance between each pixel in the point cloud data and the projection map; The projection depth map is generated based on the distance and the projection map.

4. The method according to claim 2, characterized in that, The point cloud data obtained based on the depth values ​​of the pixels and the mask image includes: The depth values ​​of the pixels and the mask image are input into a depth camera with pre-calibrated intrinsic parameters; Obtain the point cloud data generated by the depth camera.

5. The method according to claim 1, characterized in that, The second threshold is the depth value of the pixel with the largest depth value and the average depth value of the pixel with the smallest depth value within the specified area.

6. A package positioning device, characterized in that, The device includes: The candidate segmentation point determination module is used to determine the depth value of each pixel in the projection depth map based on the projection depth map containing the package, and to determine candidate segmentation points from the pixels according to the depth values. The segmentation point determination module is used to sequentially use each candidate segmentation point as a target candidate segmentation point; starting from the target candidate segmentation point, continuously acquire the two-dimensional coordinates of a specified number of pixels along the gradient descent direction in units of pixels; determine the slope of the straight line fitted by the pixel points based on the two-dimensional coordinates of the pixels; and determine the target candidate segmentation point whose slope of the fitted straight line is greater than a specified threshold as the segmentation point; wherein the horizontal coordinate of the pixel point represents the distance between it and the target candidate segmentation point, and the vertical coordinate represents the depth value; The package location determination module is used to remove the segmentation point from the projection depth map and determine the location of the package based on the remaining points in the projection depth map; When the package location determination module determines the location of the package based on the remaining points in the projected depth map, it is configured as follows: Perform connected component analysis on the remaining points to obtain the location of the package; When the segmentation point determination module performs the function of determining candidate segmentation points from pixels based on the depth value, it is configured as follows: For each pixel in the projection depth map, the pixel is taken as the target pixel. Using the target pixel as the center, determine the pixel with the maximum depth value and the pixel with the minimum depth value within the specified area; If the depth difference between the pixel with the largest depth value and the pixel with the smallest depth value is greater than a first threshold, and the depth value of the target pixel is less than a second threshold, then the target pixel is selected as a candidate segmentation point.

7. The apparatus according to claim 6, characterized in that, The projection depth map containing the package was obtained using the following method: Obtain the RGB image and depth image containing the package; The RGB image is segmented using deep learning to obtain a mask image; Based on the mapping relationship between the depth image and the RGB image, the pixels of the depth image are aligned with the pixels of the RGB image to obtain the depth value of the pixel; Point cloud data is obtained based on the depth values ​​of the pixels and the mask image; The pixels in the point cloud data are projected onto a reference plane to obtain a projection depth map containing the enclosed data.

8. The apparatus according to claim 7, characterized in that, When the candidate segmentation point determination module performs the operation of projecting pixels from the point cloud data onto a reference plane to obtain a projection depth map containing the enclosed data, it is configured as follows: According to the pre-calibrated platform plane equation, the pixels in the point cloud data are projected onto the reference plane to obtain a projection map; Determine the distance between each pixel in the point cloud data and the projection map; The projection depth map is generated based on the distance and the projection map.

9. The apparatus according to claim 7, characterized in that, When the candidate segmentation point determination module executes the point cloud data based on the depth value of the pixel and the mask image, it is configured to: The depth values ​​of the pixels and the mask image are input into a depth camera with pre-calibrated intrinsic parameters; Obtain the point cloud data generated by the depth camera.

10. The apparatus according to claim 6, characterized in that, The second threshold is the depth value of the pixel with the largest depth value and the average depth value of the pixel with the smallest depth value within the specified area.

11. An electronic device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A computer storage medium, characterized in that, The computer storage medium stores a computer program that enables the computer to perform the method described in any one of claims 1-5.

Images