A shelf position detection method, device, equipment and storage medium

Abstract

The application discloses a shelf position detection method, device and equipment and a storage medium, acquires a three-dimensional point cloud image captured by a depth camera, detects a region where a shelf is located from the three-dimensional point cloud image based on a target detection algorithm of deep learning, segments a shelf leg from the region where the shelf is located based on a target segmentation algorithm of deep learning, and calculates a center position of the shelf and a posture of the shelf based on point cloud coordinates in the shelf leg. The application detects a shelf based on a deep learning algorithm on a three-dimensional point cloud image captured by a depth camera, detects a region where the shelf is located, segments a shelf leg from the region where the shelf is located based on the deep learning algorithm, and can process a more complex scene by using a machine learning algorithm, thereby improving the accuracy of shelf position detection.

Description

Technical Field

[0001] This invention relates to point cloud data processing technology, and more particularly to a method, apparatus, equipment, and storage medium for detecting shelf positions. Background Technology

[0002] Unmanned transport equipment refers to transport equipment equipped with electromagnetic or optical automatic guidance devices that can travel along a predetermined path.

[0003] During the docking process between the automated guided vehicle (AGV) and the shelving, it is necessary to sense the shelving's position. The traditional shelving position detection method is as follows: a depth camera is used to acquire depth images of the shelving, the depth images are converted to grayscale to obtain grayscale images, the grayscale images are processed, and an edge detection algorithm based on the enhanced Sobel operator is used to detect the left and right column regions. The point cloud data corresponding to the left and right column regions of the shelving are obtained and edge information is extracted. From the obtained edge information, the spatial position and orientation information of the shelving center are calculated through spatial geometric relationships.

[0004] As the complexity of the scenario increases, the accuracy of traditional shelf location detection methods will decrease significantly. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for detecting shelf positions, which can handle more complex scenarios and improve the accuracy of shelf position detection.

[0006] In a first aspect, the present invention provides a shelf position detection method, comprising:

[0007] Acquire 3D point cloud images captured by a depth camera;

[0008] A deep learning-based object detection algorithm detects the area where the shelf is located from the 3D point cloud image;

[0009] A deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located.

[0010] Based on the point cloud coordinates within the shelf legs, the center position and orientation of the shelf are calculated.

[0011] Optionally, a deep learning-based object detection algorithm detects the area where the shelf is located from the 3D point cloud image, including:

[0012] Divide all points in a 3D point cloud image into multiple 3D voxels of the same size;

[0013] Feature extraction is performed on points within each of the three-dimensional voxels to obtain a feature matrix composed of the feature vectors of each of the three-dimensional voxels;

[0014] Based on the feature matrix, classification detection and location regression are performed to obtain the three-dimensional bounding box of the area where the shelf is located.

[0015] Optionally, feature extraction is performed on points within each of the three-dimensional voxels to obtain a feature matrix composed of the feature vectors of each of the three-dimensional voxels, including:

[0016] For each of the three-dimensional voxels, the mean coordinates of all points within the three-dimensional voxel are calculated and used as the centroid coordinates of the three-dimensional voxel.

[0017] Calculate the difference between the coordinates of each point within the three-dimensional voxel and the coordinates of the centroid of the three-dimensional voxel to obtain the relative coordinates of the point and the centroid.

[0018] By concatenating the vector representation of the point coordinates with the vector representation of the relative coordinates, the position vector of each point within the three-dimensional voxel is obtained.

[0019] The position vectors of the points within the three-dimensional voxels are input into a fully connected network, which maps the position vectors of the points within the three-dimensional voxels to obtain the mapping features.

[0020] The mapping features are input into a residual convolutional network, which performs feature mining on the mapping features to obtain the feature vector of the three-dimensional voxel.

[0021] The feature vectors of each of the three-dimensional voxels are combined into a feature matrix.

[0022] Optionally, classification detection and location regression are performed based on the feature matrix to obtain the three-dimensional bounding box of the area where the shelf is located, including:

[0023] The feature matrix is ​​input into a region generation network, which processes the feature matrix to classify, detect, and regress the targets in the 3D point cloud image, thereby obtaining the 3D bounding box of the area where the shelf is located.

[0024] Optionally, a deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located, including:

[0025] Local features are extracted from all points within the area where the shelf is located to obtain the local features of the area where the shelf is located.

[0026] Pooling operations are performed on the local features in each dimension to obtain the global features of the area where the shelf is located;

[0027] The local features and the global features are concatenated along a dimension to obtain the fused features;

[0028] Based on the fusion feature, the shelf legs are segmented from the area where the shelf is located.

[0029] Optionally, before the deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located, the method further includes:

[0030] Perform point cloud completion on the targets within the area where the shelf is located.

[0031] Optionally, point cloud completion is performed on the targets within the area where the shelf is located, including:

[0032] The irregular point cloud data within the area where the shelf is located is converted into standardized three-dimensional mesh data;

[0033] The three-dimensional mesh data is input into a three-dimensional convolutional network for processing, resulting in the three-dimensional mesh data after convolution processing.

[0034] The 3D mesh data after convolution processing is inversely converted into point cloud data;

[0035] Sampling is performed on the inverse-converted point cloud data to obtain coarse point cloud data;

[0036] The points in the coarse point cloud data are offset to generate new points to complete the point cloud for the target in the area where the shelf is located.

[0037] Secondly, the present invention also provides a shelf position detection device, comprising:

[0038] The point cloud image acquisition module is used to acquire 3D point cloud images captured by the depth camera;

[0039] The shelf area detection module is used to detect the area where the shelf is located from the three-dimensional point cloud image based on a deep learning-based target detection algorithm.

[0040] The shelf leg segmentation module is used to segment the shelf legs from the area where the shelf is located based on a deep learning-based target segmentation algorithm.

[0041] The pose calculation module is used to calculate the center position and posture of the shelf based on the point cloud coordinates within the shelf legs.

[0042] Thirdly, the present invention also provides a computer device, comprising:

[0043] One or more processors;

[0044] Storage device for storing one or more programs;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the shelf location detection method as provided in the first aspect of the present invention.

[0046] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the shelf position detection method as provided in the first aspect of the present invention.

[0047] The shelf location detection method provided by this invention includes: acquiring a three-dimensional point cloud image captured by a depth camera; detecting the area where the shelf is located from the three-dimensional point cloud image using a deep learning-based object detection algorithm; segmenting the shelf legs from the area where the shelf is located using a deep learning-based object segmentation algorithm; and calculating the center position and orientation of the shelf based on the point cloud coordinates within the shelf legs. This application uses a deep learning algorithm to perform shelf detection on the three-dimensional point cloud image captured by a depth camera, detects the area where the shelf is located, and segments the shelf legs from the area where the shelf is located using a deep learning algorithm. Compared with traditional algorithms, this invention uses a machine learning algorithm that can handle more complex scenarios and improve the accuracy of shelf location detection.

[0048] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of a shelf position detection method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of a shelf position detection device provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0053] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] Figure 1 This is a flowchart illustrating a shelf location detection method provided in an embodiment of the present invention. This embodiment is applicable to situations where shelf location is detected based on a deep learning algorithm. The method can be executed by the shelf location detection device provided in this embodiment. This device can be implemented in software and / or hardware, and is typically configured in an electronic device, such as... Figure 1 As shown, the shelf location detection method includes the following steps:

[0057] S101. Acquire 3D point cloud images captured by a depth camera.

[0058] In this embodiment of the invention, when the unmanned device docks with the shelf, a depth camera is used to acquire a three-dimensional point cloud image within a preset range along the direction of travel of the unmanned device, and then uploads it to an electronic device. The depth camera can be a TOF (Time Of Flight) camera, which, as the name suggests, utilizes the time of flight of light. Its ranging principle involves continuously sending light pulses to the target, then using a sensor to receive the light returning from the object. By detecting the round-trip time of the light pulses, the distance to the target is obtained, thus achieving object ranging and ultimately obtaining a three-dimensional point cloud image. Each point in the three-dimensional point cloud image is represented by Pi(x,y,z), where (x,y) represents the pixel position corresponding to the horizontal and vertical coordinates, and z represents the distance of the pixel from the camera.

[0059] S102. A deep learning-based object detection algorithm detects the area where the shelf is located from a 3D point cloud image.

[0060] In this embodiment of the invention, a deep learning-based target detection algorithm detects the area where the shelf is located from a 3D point cloud image, that is, detects the shelf from the 3D point cloud image and uses a detection box to indicate the area where the shelf is located.

[0061] For example, in some embodiments of the present invention, the Voxelnet algorithm is used to detect the area where the shelf is located from a 3D point cloud image. For example, the shelf detection process based on the Voxelnet algorithm is as follows:

[0062] 1. Divide all points in a 3D point cloud image into multiple 3D voxels of the same size.

[0063] For a point cloud space composed of all points in a 3D point cloud image, the point cloud space is divided into multiple voxels of the same size.

[0064] 2. Extract features from points within each three-dimensional voxel to obtain a feature matrix composed of the feature vectors of each three-dimensional voxel.

[0065] For example, for each three-dimensional voxel, a convolutional network is used to extract features from each point within the three-dimensional voxel, and the features of each point are concatenated in the dimension to obtain a feature vector representing the mutual positional features of all points within the three-dimensional voxel. The feature vectors of all three-dimensional voxels are combined to obtain the feature matrix of the entire three-dimensional point cloud image.

[0066] For example, in this embodiment of the invention, the feature matrix extraction process is as shown in step 2-1:

[0067] 2-1. For each three-dimensional voxel, calculate the average coordinates of all points within the three-dimensional voxel, and use them as the centroid coordinates of the three-dimensional voxel.

[0068] In this embodiment of the invention, for each three-dimensional voxel, the mean coordinates of all points within the voxel are calculated, that is, the mean x-coordinate, mean y-coordinate, and mean z-coordinate of all points are calculated respectively, and the mean coordinates of all points are used as the centroid coordinates of the three-dimensional voxel. Of course, in other embodiments of the invention, other algorithms can also be used to calculate the centroid coordinates of the three-dimensional voxel, and this embodiment of the invention does not limit this.

[0069] 2-2. Calculate the difference between the coordinates of each point within the three-dimensional voxel and the coordinates of the centroid of the three-dimensional voxel to obtain the relative coordinates of the point and the centroid.

[0070] For each point within a three-dimensional voxel, calculate the difference between the coordinates (xi, yi, zi) of that point and the coordinates (vx, vy, vz) of the centroid of that three-dimensional voxel, and obtain the relative coordinates (xi-vx, yi-vy, zi-vz) of that point and the centroid.

[0071] 2-3. Concatenate the vector representation of the point coordinates with the vector representation of the relative coordinates to obtain the position vector of each point within the three-dimensional voxel.

[0072] For example, the vector representation of the coordinates (xi, yi, zi) of each point within a 3D voxel is concatenated with the vector representation of the relative coordinates (xi-vx, yi-vy, zi-vz) of that point with respect to the centroid, to obtain the position vector (xi, yi, zi, xi-vx, yi-vy, zi-vz) of each point within the 3D voxel. That is, the position vector of a point within a 3D voxel reflects both the position of the point in the 3D point cloud image and the position of the point within the 3D voxel.

[0073] 2-4. Input the position vectors of points within the 3D voxels into the fully connected network. The fully connected network maps the position vectors of points within the 3D voxels to obtain the mapped features.

[0074] For example, in this embodiment of the invention, a VFE (Voxel Feature Encoding) network is used to extract features from all points in a 3D voxel. For each point, a fully connected neural network maps it to a feature space. The feature space represents the integration of point features to describe the surface properties expressed by the voxel. The fully connected network consists of a fully connected layer, a normalization layer, and a ReLU activation function layer. The fully connected layer maps the point features to the feature space to obtain linear features. The normalization layer normalizes the linear features to obtain normalized features. The ReLU activation function layer normalizes the normalized features. All points within a voxel are abstracted into a point-wise feature. Then, a pooling layer (Element-wise Maxpooling) performs global max pooling on the point-wise feature to obtain a locally aggregated feature. Next, the locally aggregated feature is fused with the previous point-wise feature (Point-wise Concatenate) to obtain the mapped features of each voxel.

[0075] 2-5. Input the mapped features into the residual convolutional network. The residual convolutional network performs feature mining on the mapped features to obtain the feature vectors of the three-dimensional voxels.

[0076] In some embodiments of the present invention, the mapped features are input into a three-dimensional residual convolutional network (3D-ResNet). The residual convolutional network performs feature mining on the mapped features to obtain feature vectors of three-dimensional voxels. The residual convolutional network further abstracts the features, increases the receptive field, and learns geometric spatial representations, while avoiding the gradient descent and overfitting problems of deep networks.

[0077] 2-6. Combine the feature vectors of each three-dimensional voxel into a feature matrix.

[0078] In this embodiment of the invention, the feature vectors of each three-dimensional voxel are combined into a feature matrix.

[0079] 3. Based on the feature matrix, perform classification detection and location regression to obtain the three-dimensional bounding box of the area where the shelf is located.

[0080] In this embodiment of the invention, classification detection and location regression are performed based on the feature matrix to obtain the three-dimensional bounding box of the area where the shelf is located. For example, in some embodiments of the invention, a Region Proposal Network (RPN) is used to process the feature matrix to perform classification detection and location regression on objects in the three-dimensional point cloud image, thereby detecting the shelf and displaying it using a three-dimensional bounding box.

[0081] In 3D point cloud images, point cloud gaps are common, which can lead to reduced recognition accuracy during subsequent semantic segmentation, especially for small targets (such as shelf legs). To address this issue, in this embodiment of the invention, before the deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located, point cloud completion is performed on the targets within that area.

[0082] For example, in this embodiment of the invention, the GRNet point cloud completion algorithm is used to complete the point cloud of the target within the area where the shelf is located. For example, the steps of point cloud completion are as follows:

[0083] 1. Convert the irregular point cloud data in the area where the shelf is located into standardized 3D mesh data.

[0084] In this embodiment of the invention, irregular point cloud data within the area where the shelf is located is input into a Gridding layer. The Gridding layer uses an interpolation function to weight the eight vertices of the three-dimensional grid cell where the point is located. The interpolation function can accurately measure the geometric relationship of the point cloud, transforming the disordered point cloud into standardized three-dimensional grid data while preserving the spatial structure of the point cloud.

[0085] 2. Input the 3D mesh data into a 3D convolutional network for processing. The resulting 3D mesh data is obtained after convolution processing.

[0086] In this embodiment of the invention, three-dimensional mesh data is input into a three-dimensional convolutional network (3D-CNN) for processing. The resulting three-dimensional mesh data is then processed using convolution. The 3D CNN learns content-aware and spatial-aware features, thereby allowing the network to complete missing parts of incomplete point clouds.

[0087] 3. Convert the convolutional 3D mesh data into point cloud data.

[0088] The 3D mesh data after convolution is input into the Gridding Reverse layer. By replacing each 3D mesh cell with a new point whose coordinates are the weighted sum of the eight vertices of the mesh cell, the 3D mesh data after convolution is inversely converted into point cloud data.

[0089] 4. Sample the inverse-converted point cloud data to obtain coarse point cloud data.

[0090] To reduce feature redundancy, the inversely transformed point cloud data is input into the sampling layer for random sampling to obtain coarse point cloud data.

[0091] 5. Offset the points in the coarse point cloud data to generate new points and complete the point cloud for the target in the area where the shelf is located.

[0092] The coarse point cloud is fed into a multilayer perceptron. For each sampled point, the multilayer perceptron estimates multiple offsets for each sampled point, and then offsets the sampled point according to the offsets to obtain multiple new points, thereby completing the point cloud of the target in the area where the shelf is located.

[0093] S103. A deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located.

[0094] In this embodiment of the invention, a deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located. For example, the target segmentation algorithm can be the PointNet algorithm or the U-Net algorithm; this embodiment of the invention does not limit the specific algorithm used.

[0095] For example, the PointNet algorithm is used as an example to illustrate the target segmentation algorithm.

[0096] 1. Extract local features from all points within the area where the shelf is located to obtain the local features of the area where the shelf is located.

[0097] For example, in this embodiment of the invention, a T-Net is used to first normalize the original point cloud in the area where the shelf is located with different rotations and translations. Then, the normalized point cloud is input into a multilayer perceptron with n shared weights. The output features of the multilayer perceptron are then normalized by a T-Net to obtain the local features of the area where the shelf is located.

[0098] 2. Perform pooling operations on local features across all dimensions to obtain the global features of the area where the shelf is located.

[0099] For example, in an embodiment of the present invention, local features are first processed by a multilayer perceptron, and the processed features are sent to a pooling layer to perform max pooling operations in each dimension to obtain global features of the area where the shelf is located.

[0100] 3. Combine the local and global features in terms of dimensions to obtain the fused features.

[0101] In this embodiment of the invention, local features and global features are concatenated in a dimensional manner to obtain fused features.

[0102] 4. Based on the fusion features, the shelf legs are segmented from the area where the shelf is located.

[0103] For example, in an embodiment of the present invention, semantic segmentation is performed on multiple multilayer perceptrons that are sequentially connected by fused feature inputs, and the shelf legs are segmented from the area where the shelf is located.

[0104] S104. Calculate the center position and orientation of the shelf based on the point cloud coordinates within the shelf legs.

[0105] In this embodiment of the invention, after segmenting the shelf legs from the area where the shelf is located, the centers of the four shelf legs are calculated based on the point cloud coordinates and geometric relationships within the shelf legs. The average coordinate of the four shelf leg centers is taken as the center position of the shelf. The shelf's attitude, i.e., its azimuth relative to the unmanned equipment, can be deduced based on the centers of the four shelf legs and their geometric relationships. After determining the shelf's center position and attitude, the unmanned equipment docks with the shelf to perform shelf lifting or hanging operations.

[0106] The shelf location detection method provided in this invention includes: acquiring a three-dimensional point cloud image captured by a depth camera; detecting the area where the shelf is located from the three-dimensional point cloud image using a deep learning-based object detection algorithm; segmenting the shelf legs from the area where the shelf is located using a deep learning-based object segmentation algorithm; and calculating the center position and orientation of the shelf based on the point cloud coordinates within the shelf legs. This application uses a deep learning algorithm to perform shelf detection on the three-dimensional point cloud image captured by a depth camera, detects the area where the shelf is located, and segments the shelf legs from the area where the shelf is located using a deep learning algorithm. Compared with traditional algorithms, this invention uses a machine learning algorithm that can handle more complex scenarios and improve the accuracy of shelf location detection.

[0107] Figure 2 This is a schematic diagram of a shelf position detection device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the shelf position detection device includes:

[0108] The point cloud image acquisition module 201 is used to acquire three-dimensional point cloud images captured by a depth camera;

[0109] The shelf area detection module 202 is used to detect the area where the shelf is located from the three-dimensional point cloud image based on a deep learning-based target detection algorithm;

[0110] The shelf leg segmentation module 203 is used to segment the shelf legs from the area where the shelf is located based on a deep learning-based target segmentation algorithm.

[0111] The pose calculation module 204 is used to calculate the center position and pose of the shelf based on the point cloud coordinates within the shelf legs.

[0112] In some embodiments of the present invention, the shelf area detection module 202 includes:

[0113] The voxel segmentation submodule is used to segment all points in a 3D point cloud image into multiple 3D voxels of the same size.

[0114] The feature matrix extraction submodule is used to extract features from points within each of the three-dimensional voxels to obtain a feature matrix composed of the feature vectors of each of the three-dimensional voxels.

[0115] The classification and regression submodule is used to perform classification detection and location regression based on the feature matrix to obtain the three-dimensional bounding box of the area where the shelf is located.

[0116] In some embodiments of the present invention, the feature matrix extraction submodule includes:

[0117] The centroid coordinate calculation unit is used to calculate the average coordinates of all points within each three-dimensional voxel, which is then used as the centroid coordinates of the three-dimensional voxel.

[0118] The relative coordinate calculation unit is used to calculate the difference between the coordinates of each point within the three-dimensional voxel and the coordinates of the centroid of the three-dimensional voxel, so as to obtain the relative coordinates of the point and the centroid.

[0119] A position vector determination unit is used to concatenate the vector representation of the point coordinates with the vector representation of the relative coordinates to obtain the position vector of each point within the three-dimensional voxel.

[0120] The feature mapping unit is used to input the position vectors of points within the three-dimensional voxels into a fully connected network, and the fully connected network maps the position vectors of points within the three-dimensional voxels to obtain mapped features.

[0121] The feature vector mining unit is used to input the mapping features into the residual convolutional network, and the residual convolutional network performs feature mining on the mapping features to obtain the feature vector of the three-dimensional voxel;

[0122] The vector combination unit is used to combine the feature vectors of each of the three-dimensional voxels into a feature matrix.

[0123] In some embodiments of the present invention, the classification regression submodule includes:

[0124] The classification and regression unit is used to input the feature matrix into the region generation network. The region generation network processes the feature matrix and performs classification detection and position regression on the targets in the three-dimensional point cloud image to obtain the three-dimensional bounding box of the area where the shelf is located.

[0125] In some embodiments of the present invention, the shelf leg segmentation module 203 includes:

[0126] The local feature extraction submodule is used to extract local features from all points within the area where the shelf is located, so as to obtain the local features of the area where the shelf is located.

[0127] The global feature extraction submodule is used to perform pooling operations on the local features in each dimension to obtain the global features of the area where the shelf is located;

[0128] The feature fusion submodule is used to concatenate the local features and the global features in a dimensional direction to obtain fused features;

[0129] The shelf leg segmentation submodule is used to segment the shelf legs from the area where the shelf is located based on the fusion features.

[0130] In some embodiments of the present invention, the shelf position detection device further includes:

[0131] The point cloud completion module is used to complete the point cloud of the target in the area where the shelf is located before the deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located.

[0132] In some embodiments of the present invention, the point cloud completion module includes:

[0133] The point cloud normalization submodule is used to convert irregular point cloud data in the area where the shelf is located into normalized three-dimensional mesh data.

[0134] The 3D convolution submodule is used to input the 3D mesh data into a 3D convolutional network for processing, resulting in 3D mesh data after convolution processing.

[0135] The point cloud inversion submodule is used to inversely convert 3D mesh data after convolution into point cloud data;

[0136] The point cloud sampling submodule is used to sample the inversely transformed point cloud data to obtain coarse point cloud data.

[0137] The point cloud completion submodule is used to offset the points in the coarse point cloud data and generate new points to complete the point cloud for the target in the area where the shelf is located.

[0138] The above-mentioned shelf position detection device can execute the shelf position detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the shelf position detection method.

[0139] Figure 3 This is a schematic diagram of an electronic device provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0140] like Figure 3As shown, the electronic device includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0141] Multiple components in the electronic device are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as shelf location detection methods.

[0143] In some embodiments, the shelf location detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the shelf location detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the shelf location detection method by any other suitable means (e.g., by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0148] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0149] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0150] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the shelf location detection method provided in any embodiment of this application.

[0151] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting shelf position, characterized in that, include: Acquire 3D point cloud images captured by a depth camera; A deep learning-based object detection algorithm detects the area where the shelf is located from the 3D point cloud image; A deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located. Based on the point cloud coordinates within the shelf legs, calculate the center position and orientation of the shelf; Before the deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located, the following steps are also included: Perform point cloud completion on the targets within the area where the shelf is located; The step of performing point cloud completion on the targets within the area where the shelf is located includes: The irregular point cloud data within the area where the shelf is located is converted into standardized three-dimensional mesh data; The three-dimensional mesh data is input into a three-dimensional convolutional network for processing to obtain convolutional three-dimensional mesh data. The 3D mesh data after convolution processing is inversely converted into point cloud data; Sampling is performed on the inverse-converted point cloud data to obtain coarse point cloud data; The points in the coarse point cloud data are offset to generate new points to complete the point cloud for the target in the area where the shelf is located.

2. The shelf position detection method according to claim 1, characterized in that, A deep learning-based object detection algorithm detects the area where the shelf is located from the 3D point cloud image, including: Divide all points in a 3D point cloud image into multiple 3D voxels of the same size; Feature extraction is performed on points within each of the three-dimensional voxels to obtain a feature matrix composed of the feature vectors of each of the three-dimensional voxels; Based on the feature matrix, classification detection and location regression are performed to obtain the three-dimensional bounding box of the area where the shelf is located.

3. The shelf position detection method according to claim 2, characterized in that, Feature extraction is performed on points within each of the three-dimensional voxels to obtain a feature matrix composed of the feature vectors of each of the three-dimensional voxels, including: For each of the three-dimensional voxels, the mean coordinates of all points within the three-dimensional voxel are calculated and used as the centroid coordinates of the three-dimensional voxel. Calculate the difference between the coordinates of each point within the three-dimensional voxel and the coordinates of the centroid of the three-dimensional voxel to obtain the relative coordinates of each point within the three-dimensional voxel and the centroid. By concatenating the vector representation of the point coordinates with the vector representation of the relative coordinates, the position vector of each point within the three-dimensional voxel is obtained. The position vectors of the points within the three-dimensional voxels are input into a fully connected network, which maps the position vectors of the points within the three-dimensional voxels to obtain the mapping features. The mapping features are input into a residual convolutional network, which performs feature mining on the mapping features to obtain the feature vector of the three-dimensional voxel. The feature vectors of each of the three-dimensional voxels are combined into a feature matrix.

4. The shelf position detection method according to claim 2, characterized in that, Based on the feature matrix, classification detection and location regression are performed to obtain the three-dimensional bounding box of the area where the shelf is located, including: The feature matrix is ​​input into a region generation network, which processes the feature matrix to classify, detect, and regress the targets in the 3D point cloud image, thereby obtaining the 3D bounding box of the area where the shelf is located.

5. The shelf position detection method according to any one of claims 1-4, characterized in that, A deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located, including: Local features are extracted from all points within the area where the shelf is located to obtain the local features of the area where the shelf is located. Pooling operations are performed on the local features in each dimension to obtain the global features of the area where the shelf is located; The local features and the global features are concatenated along a dimension to obtain the fused features; Based on the fusion feature, the shelf legs are segmented from the area where the shelf is located.

6. A shelf position detection device, characterized in that, include: The point cloud image acquisition module is used to acquire 3D point cloud images captured by the depth camera; The shelf area detection module is used to detect the area where the shelf is located from the three-dimensional point cloud image based on a deep learning-based target detection algorithm. The shelf leg segmentation module is used to segment the shelf legs from the area where the shelf is located based on a deep learning-based target segmentation algorithm. The pose calculation module is used to calculate the center position and pose of the shelf based on the point cloud coordinates within the shelf legs. The point cloud completion module is used to complete the point cloud of the target in the area where the shelf is located before the deep learning-based target segmentation algorithm segments the shelf legs from the area where the shelf is located. The point cloud completion module includes: The point cloud normalization submodule is used to convert irregular point cloud data in the area where the shelf is located into normalized three-dimensional mesh data. The 3D convolution submodule is used to input the 3D mesh data into a 3D convolutional network for processing to obtain the 3D mesh data after convolution processing. The point cloud inversion submodule is used to inversely convert 3D mesh data after convolution into point cloud data; The point cloud sampling submodule is used to sample the inversely transformed point cloud data to obtain coarse point cloud data. The point cloud completion submodule is used to offset the points in the coarse point cloud data and generate new points to complete the point cloud for the target in the area where the shelf is located.

7. A computer device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the shelf location detection method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the shelf location detection method as described in any one of claims 1-5.

Images