A method and system for detecting the opening position of mobile shelving units based on convolutional neural networks

Abstract

This invention relates to a method and system for detecting the opening position of a mobile shelving unit based on a convolutional neural network. The method includes the following steps: acquiring top views of the mobile shelving unit under different opening states; for each top view, extracting multiple sample images and obtaining the edge points of the opening of the mobile shelving unit marked in each sample image; constructing a training dataset using the sample images and the corresponding edge points; constructing a lightweight convolutional neural network model and training the model based on the training dataset to obtain a trained mobile shelving unit opening position detection model; acquiring the top view of the mobile shelving unit in real time; extracting multiple images to be detected from the top view; inputting each image to be detected into the mobile shelving unit opening position detection model to obtain the edge points of the opening of the mobile shelving unit in each image; and calculating the opening position of the mobile shelving unit based on the edge points of the opening of the mobile shelving unit in each image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dense cabinet opening detection, and particularly relates to a dense cabinet opening position detection method and system based on a convolutional neural network. BACKGROUND

[0002] The dense cabinet system has the advantages of compact structure, small space occupation, large storage capacity of materials, etc. As an important material storage device, the dense cabinet system has been widely applied in warehouses of various industries. However, in the operation process of the dense cabinet system, it is difficult to realize real-time video monitoring of the opening position of the dense cabinet due to the problems of large cabinet area, non-fixed opening position, and shielding caused by the high cabinet, and it is difficult to realize full-process monitoring of the behavior of personnel in the warehouse.

[0003] In view of this problem, in recent years, a scheme of using a movable guide rail driving mechanism to drive a small field-of-view monitoring camera to the position directly above the opening has appeared. The core technology that can be realized by this scheme is real-time detection of the opening position of the dense cabinet. How to obtain a high-precision opening position while controlling the hardware cost is of great significance to engineering implementation.

[0004] The existing opening position detection method of the dense cabinet is mainly realized based on a magnetic induction sensor, an ultrasonic sensor, an infrared sensor, or a mechanical sensor. Although this method can realize high-precision detection of the opening position, it needs to install sensors and wiring on each cabinet of the dense cabinet, which has high cost and is difficult to maintain. At the same time, since the cabinet of the dense cabinet can move along the guide rail, the position is not fixed, which increases the difficulty of wiring construction and seriously affects the service life of the wire. There are the following problems.

[0005] 1. Sensors need to be installed on each cabinet of the dense cabinet, which not only has high cost but also increases the probability of failure; 2. Each sensor needs to be wired. Since the cabinet of the dense cabinet can move along the guide rail, not only is the wiring construction difficult, but also the frequent stretching and bending seriously affect the service life of the wire. SUMMARY

[0006] In view of the above analysis, the embodiments of the present application aim to provide a dense cabinet opening position detection method and system based on a convolutional neural network, to solve the problems of high hardware cost and poor maintainability of the existing dense cabinet opening detection method.

[0007] In one aspect, the embodiments of the present application provide a dense cabinet opening position detection method based on a convolutional neural network, comprising the following steps:

[0008] Obtaining overhead views of dense cabinets in different opening states, for each overhead view of the dense cabinet, extracting a plurality of sample images, obtaining edge points marked at the opening of the dense cabinet in each sample image, and constructing a training data set composed of the sample images and the corresponding edge points;

[0009] Constructing a lightweight convolutional neural network model, training the model based on the training data set, and obtaining a trained dense cabinet opening position detection model;

[0010] Real-time obtaining of overhead views of dense cabinets, extracting a plurality of to-be-detected images from the overhead views, inputting each to-be-detected image into the dense cabinet opening position detection model to obtain edge points at the opening of the dense cabinet in each to-be-detected image;

[0011] Based on the edge points at the opening of the dense cabinet in each to-be-detected image, the opening position of the dense cabinet is calculated.

[0012] Based on the above technical solution, the lightweight convolutional neural network model is an improved ShuffleNet model, and the improved ShuffleNet model comprises:

[0013] A first convolutional layer for extracting shallow features of input data and outputting shallow feature maps, the first convolutional layer being a one-dimensional convolutional layer;

[0014] A max-pooling layer for reducing the dimension of the shallow feature maps;

[0015] A plurality of ShuffleNet modules for extracting deep semantic features from the reduced shallow feature maps by using point-wise group convolution and channel shuffling;

[0016] A second convolutional layer for reducing the channel dimension of the deep semantic features and outputting a detection result.

[0017] Further, each ShuffleNet module comprises a one-dimensional ShuffleNet unit and a channel attention module connected in sequence; the convolutional layer of the one-dimensional ShuffleNet unit is a one-dimensional convolution.

[0018] Further, the loss Loss of the lightweight convolutional neural network model is calculated by the following formula:

[0019]

[0020]

[0021] Loss=Point_Loss+Confidence_Loss

[0022] Wherein, S represents the length of the feature output by the model; k iThis represents the position of the edge point within the i-th interval predicted by the model. c represents the actual edge point position within the i-th interval; i This represents the confidence level that there are edge points in the i-th interval; This indicates whether there is actually a target point in the i-th interval; the value is 1 if there is, and 0 otherwise. λ obj , λ noobj and λ point This represents the weighting coefficient.

[0023] Furthermore, the opening position of the mobile shelving unit is calculated based on the edge points at the opening of the mobile shelving unit in each image to be detected, including:

[0024] Based on the coordinate projection transformation, calculate the coordinates of the bottom of the cabinet corresponding to the edge point of the opening of the mobile shelving in each image to be detected;

[0025] Based on the standard calibration board, the camera is calibrated on the ground plane to obtain the camera's internal parameters, external parameters, and distortion coefficients; based on the camera's internal parameters, external parameters, and distortion coefficients, the coordinates of the bottom of the mobile shelving unit at the opening are converted into actual position data.

[0026] Furthermore, the coordinates (x, y) of the bottom of the cabinet at the opening of the mobile shelving unit are calculated using the following formula. * ,y * ):

[0027]

[0028] Where (x,y) are the coordinates of the edge point of the opening of the mobile shelving unit to be inspected, (x * ,y * (x) represents the coordinates of the bottom of the mobile shelving unit at the opening. c ,y c ) represents the coordinates of the image center point, H represents the camera installation height, and h represents the height of the mobile shelving unit.

[0029] Furthermore, the coordinates of the bottom of the mobile shelving unit at the opening are converted into actual position data using the following formula based on the camera's internal parameters, external parameters, and distortion coefficients:

[0030] According to the formula: Convert the pixel coordinates of the edge points to image coordinates (x, y, z); where (u, v) represents the pixel coordinates of the edge points, dx and dy represent the actual physical size of the pixels; and (u0, v0) represents the pixel coordinates of the image center point.

[0031] According to the formula Calculate the camera coordinates (X) of the edge points c ,Y c Z c); where f represents the camera focal length;

[0032] According to the formula +T calculates the world coordinates (X) of the edge point. w ,Y w Z w ), where R represents the rotation matrix and T represents the translation matrix.

[0033] Furthermore, after converting the pixel coordinates of the edge points to image coordinates, and before calculating the camera coordinates of the edge points, distortion correction is performed using the following formula:

[0034] Radial distortion correction is performed on the position data:

[0035]

[0036]

[0037] Tangential distortion correction is performed on the position data:

[0038]

[0039]

[0040] Where k1, k2, k3, p1, and p2 are distortion coefficients, and (x, y) represents the uncorrected coordinates in the image coordinate system. The coordinates are the corrected coordinates in the image coordinate system, and r represents the distance of the coordinate point (x, y) from the center point of the image.

[0041] Compared with existing technologies, this invention uses a low-cost area array industrial camera to acquire a top view of the mobile shelving unit and achieves high-precision detection of the opening location through a convolutional neural network model. While achieving detection results comparable to traditional methods using magnetic induction sensors, ultrasonic sensors, infrared sensors, or mechanical sensors, it features low cost and simple maintenance. This invention can accurately identify the opening location of the mobile shelving unit and, using a lightweight convolutional neural network, can perform real-time detection without special requirements for computer configuration or additional hardware costs, making it of great application value.

[0042] On the other hand, embodiments of the present invention provide a mobile shelving opening position detection system based on a convolutional neural network, comprising:

[0043] The dataset construction module is used to obtain top views of mobile shelving units under different opening states. For each top view of a mobile shelving unit, multiple sample images are extracted, and the edge points of the mobile shelving unit opening marked in each sample image are obtained. The sample images and the corresponding edge points are used to form a training dataset.

[0044] The model training module is used to construct a lightweight convolutional neural network model, and to train the model based on the training dataset to obtain a trained mobile shelving opening location detection model.

[0045] The edge point detection module is used to acquire a top view of the mobile shelving in real time, extract multiple images to be detected from the top view, and input each image to be detected into the mobile shelving opening position detection model to obtain the edge points at the opening of the mobile shelving in each image to be detected.

[0046] The opening position calculation module is used to calculate the opening position of the mobile shelving unit based on the edge points of the opening in each image to be detected.

[0047] Furthermore, the model training module calculates the loss using the following formula:

[0048]

[0049]

[0050] Loss=Point_Loss+Confidence_Loss

[0051] Where S represents the length of the features output by the model; k i This represents the position of the edge point within the i-th interval predicted by the model. c represents the actual edge point position within the i-th interval; i This represents the confidence level that there are edge points in the i-th interval; This indicates whether there is actually a target point in the i-th interval; the value is 1 if there is, and 0 otherwise. λ obj , λ noobj and λ point This represents the weighting coefficient.

[0052] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0053] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0054] Figure 1 This is a flowchart of the method for detecting the opening position of a mobile shelving unit based on a convolutional neural network, according to an embodiment of the present invention.

[0055] Figure 2 This is a block diagram of the mobile shelving opening position detection system based on a convolutional neural network according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of an image acquisition scene according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of sample image extraction according to an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the lightweight convolutional neural network model structure according to an embodiment of the present invention. Detailed Implementation

[0059] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0060] A specific embodiment of the present invention discloses a method for detecting the opening position of a mobile shelving unit based on a convolutional neural network, such as... Figure 1 As shown, it includes the following steps:

[0061] S1. Obtain top views of the mobile shelving unit under different opening states. For each top view of the mobile shelving unit, extract multiple sample images and obtain the edge points of the mobile shelving unit opening marked in each sample image. Combine the sample images and the corresponding edge points to form a training dataset.

[0062] S2. Construct a lightweight convolutional neural network model, train the model based on the training dataset, and obtain a trained mobile shelving opening location detection model.

[0063] S3. Real-time acquisition of the top view of the mobile shelving unit, extraction of multiple sets of images to be detected from the top view, inputting each image to be detected into the mobile shelving unit opening detection model to obtain the edge points of the mobile shelving unit opening in each image to be detected.

[0064] S4. Calculate the opening position of the mobile shelving unit based on the edge points at the opening of the mobile shelving unit in each image to be detected.

[0065] During implementation, an industrial area scan camera is installed above the mobile shelving unit to obtain a top-down view of the unit.

[0066] This application uses a low-cost area-scan industrial camera to acquire a top view of a mobile shelving unit, and achieves high-precision detection of the opening location of the shelving unit through a convolutional neural network model. While achieving detection results comparable to traditional methods using magnetic induction sensors, ultrasonic sensors, infrared sensors, or mechanical sensors, it features low cost and simple maintenance. This invention can accurately identify the opening location of the mobile shelving unit, and using a lightweight convolutional neural network enables real-time detection. It has no special requirements for computer configuration and does not increase computer hardware costs, making it of great application value.

[0067] When implementing, such as Figure 3 As shown, if the shooting range of one industrial area scan camera is insufficient to cover a mobile shelving area, multiple industrial area scan cameras can be used to capture the top view of the mobile shelving, and the multiple top views can be stitched together into a complete top view of the mobile shelving using an image stitching method.

[0068] Specifically, the following steps are used to stitch together the images captured by each image acquisition device to obtain a complete top view of the mobile shelving unit:

[0069] S01. Use the SIFT algorithm to extract feature points from each image;

[0070] The SIFT algorithm describes the detected feature points in an image using a 128-dimensional feature vector. Therefore, after an image is processed by the SIFT algorithm, each feature point is represented as a 128-dimensional feature description vector.

[0071] S02. Use the fast nearest neighbor algorithm to match feature points between two adjacent images to obtain the initial correspondence between feature points in adjacent images;

[0072] After identifying feature points, matching feature points not only requires the closest distance between their feature description vectors but also the ability to be distinguished from other feature points. Therefore, the ratio of the distance between the nearest neighbor and the second nearest neighbor is used to constrain feature point matching. Two images are stitched together as an example, referred to as the first image and the second image, respectively.

[0073] First, a KD-tree data structure is used to store the feature description vectors in an ordered manner. Using the feature points in the first image as a reference, the Fast Approximate K-Nearest Neighbor (FLANN) algorithm is then used to find the nearest and second nearest neighbors in the second image. If the ratio of the distance between the nearest and second nearest neighbors is less than a threshold, the matching relationship between the feature point and its nearest neighbor is preserved.

[0074] S03. Use the random sampling consistency algorithm to delete erroneous correspondences and calculate the homography matrix between adjacent images;

[0075] Specifically, it includes:

[0076] S031. Randomly select n sets of matching points from the matching point set S to form a subset s, where s is the assumed interior point and n is at least 4.

[0077] S032. Calculate the homography matrix from the feature points in the first image to the corresponding feature points in the second image based on the feature points in the subset s.

[0078] S033. Traverse all feature point pairs in dataset S except for subset s. For each feature point pair, if the error between the mapping point calculated by the feature point of the first image according to the transformation matrix and the feature point of the second image is within the given error e, then mark it as an interior point; otherwise, mark it as an exterior point.

[0079] S034. All interior points form a uniform set. If the number of points in the uniform set satisfies a given threshold TT, then the homography matrix is ​​recalculated using all interior points in the uniform set, and then the algorithm ends.

[0080] S035. If the number of interior points in the consistent set is less than the threshold TT, then select a new subset s and return to step 031.

[0081] If the number of points in the K uniform sets obtained after K iterations does not meet the given threshold, then select a uniform set with the largest number of interior points, recalculate the homography matrix using all interior points in the uniform set, and then terminate the algorithm.

[0082] By employing a random sampling consistency algorithm, outliers are eliminated, and the homography matrix is ​​calculated based solely on inliers, thereby removing erroneous correspondences and making feature point matching more accurate.

[0083] Among them, homography matrix according to Calculate the homography matrix H. Here, (x,y) represents the feature points in the first image of the feature point pair, and (x',y') represents the feature points in the second image of the feature point pair.

[0084] S04. Based on the homography matrix between adjacent images, different images are mapped onto the same image to obtain the stitched image.

[0085] The homography matrix between images is estimated using a sample consistency algorithm, and all images are then warped onto a common image plane. Typically, this common plane is the central image plane. One approach is to create a very large image, such as one filled entirely with zeros, parallel to the central image, and then warp all the images onto it to obtain the stitched image.

[0086] Specifically, in step S1, multiple sample images are extracted from the top view of the mobile shelving unit using the following method:

[0087] In the top view, multiple sets of sample images with a width of W and a length of L are extracted at equal intervals in a direction perpendicular to the direction of movement of the mobile shelving unit. The long side of each sample image is parallel to the direction of movement of the mobile shelving unit.

[0088] In practice, the image width can be 1 pixel, and the sample image is an L×1 image, such as... Figure 4 As shown in the figure, each line represents an L×1 dimensional sample image, and the interval between the lines is fixed.

[0089] To facilitate calculations, when installing an industrial area scan camera, one of the camera's coordinate axes can be aligned parallel to the direction of movement of the mobile shelving unit, thus making it easier to extract sample images.

[0090] After extracting the sample images, the opening positions of the mobile shelving units in the images can be manually labeled. Since the top view is obtained, the top edge points of adjacent cabinets at the opening of the mobile shelving units are labeled, and the coordinate positions are used as the labels corresponding to the sample images. The sample images and the corresponding labels constitute the training dataset.

[0091] To facilitate real-time detection and improve detection efficiency, a lightweight convolutional neural network model was constructed as a detection model for the opening position of mobile shelving units.

[0092] Specifically, the lightweight convolutional neural network model is an improved ShuffleNet model, such as... Figure 5 As shown in (b), the improved ShuffleNet model includes:

[0093] The first convolutional layer is used to extract shallow features from the input data and output a shallow feature map. The first convolutional layer is a one-dimensional convolutional layer.

[0094] Max pooling layers are used to reduce the dimensionality of shallow feature maps;

[0095] Multiple ShuffleNet modules are used to extract deep semantic features from the dimensionality-reduced shallow feature maps by employing pointwise group convolution and channel shuffling.

[0096] The second convolutional layer is used to reduce the channel dimension of deep semantic features and output the detection results.

[0097] Specifically, the first convolutional layer is a one-dimensional convolutional layer with a stride of 2 and 24 kernels. It is used to extract shallow features from the input data, reduce the data dimensionality, and increase the number of feature channels.

[0098] For example, the input sample image is a 4000×1×3 image, where the third dimension 3 represents the number of channels. The feature map obtained after the first convolutional layer is 2000×1×24, and the output feature map after the pooling layer is 1000×1×24.

[0099] The dimensionality-reduced feature maps are processed through multiple ShuffleNet modules to extract deep semantic features. Each module further reduces the feature map dimensionality while increasing the number of feature channels, gradually extracting deep semantic features from edge points. ShuffleNet employs pointwise group convolution and channel shuffling, effectively reducing the computational load of the network and significantly lowering computational costs while ensuring minimal loss of accuracy.

[0100] Specifically, in order to enhance the feature extraction capability of the network, each ShuffleNet module includes a one-dimensional ShuffleNet unit and a channel attention module connected in sequence; the convolutional layer of the one-dimensional ShuffleNet unit is a one-dimensional convolution.

[0101] That is, all convolutional layers in a ShuffleNet unit are replaced with one-dimensional convolutions, and a channel attention module is added after each ShuffleNet unit structure to further extract more accurate features. For example... Figure 5 As shown in (a), the channel attention module consists of global average pooling, two fully connected layers, one ReLU layer, and one sigmoid activation layer. It generates corresponding weights for each channel, enabling the network to autonomously select based on the importance of each feature channel. The features output by the ShuffleNet unit are multiplied by the weights calculated by the channel attention module to obtain the features with added attention weights.

[0102] The improved ShuffleNet model in this application removes the global average pooling and fully connected layers at the end of the original ShuffleNet model and replaces them with a single convolutional layer, namely the second convolutional layer. This convolutional layer has a kernel size of 1×1, two kernels, and a stride of 1. For example, if the input data is (125×1×1536), the output data is (125×1×2).

[0103] It's important to note that the lightweight convolutional neural network here outputs a feature dimension of (125×1×2), producing two feature maps, each 125×1 in size. This means the one-dimensional input vector is divided into 125 equal intervals, each containing two values: one representing the predicted edge location and the other representing the confidence score of the presence of an edge within that interval. In other words, the first element in each of the two feature maps represents the confidence score of the presence of an edge within the first interval of the input data and the coordinates of the predicted edge. Furthermore, since the input data is a one-dimensional vector, only one coordinate value is needed to represent the edge location.

[0104] After building a lightweight convolutional neural network model, the model is trained based on training sample data. During the network training process, the gradient descent method is used to backpropagate and update the parameters to optimize the model parameters, thereby obtaining a trained mobile shelving opening location detection model.

[0105] Specifically, the loss of the lightweight convolutional neural network model is calculated using the following formula, and the model parameters are updated accordingly:

[0106]

[0107]

[0108] Loss=Point_Loss+Confidence_Loss

[0109] Where S represents the length of the features output by the model; k i This represents the position of the edge point within the i-th interval predicted by the model. c represents the actual edge point position within the i-th interval; i This represents the confidence level that there are edge points in the i-th interval; This indicates whether there is actually a target point in the i-th interval; the value is 1 if there is, and 0 otherwise. λ obj , λ noobj and λ point This represents the weighting coefficient.

[0110] Specifically, It is determined based on the coordinates of the edge points in the label. For example, if the edge point coordinates of the sample are at the 50th pixel position, and the first interval is from the 1st to the 30th pixel position, then there is no target point in the first interval.

[0111] Once the model reaches the required accuracy or the predetermined number of iterations, training stops, and a well-trained mobile shelving opening location detection model is obtained.

[0112] After obtaining the trained model, in step S3, the top view of the mobile shelving is acquired in real time, and multiple images to be detected are extracted from the top view of the mobile shelving to be detected using the same method as in step S1; each image to be detected is input into the trained mobile shelving opening position detection model to obtain the edge points of the mobile shelving in each image to be detected.

[0113] Since the edge points of the mobile shelving obtained from the top view are the top edge points of the adjacent cabinets at the opening of the mobile shelving, they need to be converted into coordinates of the bottom of the cabinets to facilitate accurate calculation of the specific data in the world coordinate system of the opening position of the mobile shelving.

[0114] Specifically, in step S4, the opening position of the mobile shelving is calculated based on the edge points at the opening of the mobile shelving in each image to be detected, including:

[0115] S41. Calculate the coordinates of the bottom of the cabinet corresponding to each edge point in each image to be detected based on the coordinate projection transformation;

[0116] Specifically, the coordinates (x, y) of the bottom of the cabinet at the opening of the mobile shelving unit are calculated using the following formula. * ,y * ):

[0117]

[0118] Where (x,y) are the coordinates of the edge point at the opening of the mobile shelving unit in the image to be detected, (x * ,y * (x) represents the coordinates of the bottom of the mobile shelving unit at the opening. c ,y c ) represents the coordinates of the image center point, H represents the camera installation height, and h represents the height of the mobile shelving unit.

[0119] It should be noted that the coordinates here are the coordinates of the edge point of the opening of the mobile shelving in the image to be detected in the top view of the mobile shelving, and the coordinate system is the pixel coordinate system.

[0120] S42. Based on the standard calibration plate, calibrate the camera on the ground plane to obtain the camera's internal parameters, external parameters, and distortion coefficients; convert the coordinates of the bottom of the cabinet at the opening of the mobile shelving into actual position data according to the camera's internal parameters, external parameters, and distortion coefficients.

[0121] In practice, the Zhang Zhengyou calibration method can be used to calibrate the camera on the ground plane based on a standard calibration plate to obtain the camera's internal parameters, external parameters, and distortion coefficients.

[0122] The following formula is used to convert the coordinates of the bottom of the mobile shelving unit at the opening into actual position data based on the camera's internal parameters, external parameters, and distortion coefficients:

[0123] According to the formula: Convert the pixel coordinates of the edge points to image coordinates (x, y, z); where (u, v) represents the pixel coordinates of the edge points, dx and dy represent the actual physical size of the pixels; and (u0, v0) represents the pixel coordinates of the image center point.

[0124] According to the formula Calculate the camera coordinates (X) of the edge points c ,Y c Z c ); where f represents the camera focal length;

[0125] According to the formula Calculate the world coordinates (X) of the edge point w ,Y w Z w ), where R represents the rotation matrix and T represents the translation matrix.

[0126] Where dx, dy, (u0,v0) and f are the camera's internal parameters; R and T are the camera's external parameters.

[0127] Because distortions caused by the optical lens, including radial and tangential distortion, exist during the camera's imaging process, distortion correction is necessary. This distortion correction is performed in the image coordinate system.

[0128] Radial distortion correction is performed on the position data:

[0129]

[0130]

[0131] Tangential distortion correction is performed on the position data:

[0132]

[0133]

[0134] Where k1, k2, k3, p1, and p2 are distortion coefficients, and (x, y) represents the uncorrected coordinates in the image coordinate system. The coordinates are the corrected coordinates in the image coordinate system, and r represents the distance of the coordinate point (x, y) from the center point of the image. The farther away from the center point, the greater the distortion.

[0135] After obtaining the coordinates of all edge points in the world coordinate system, connect the edge points on the same side of the opening to obtain the edge lines on both sides. Based on the two edge lines, the actual opening position of the mobile shelving can be accurately obtained.

[0136] One specific embodiment of the present invention discloses a mobile shelving opening position detection system based on a convolutional neural network, such as... Figure 2 As shown, the system includes:

[0137] The dataset construction module is used to obtain top views of mobile shelving units under different opening states. For each top view of a mobile shelving unit, multiple sample images are extracted, and the edge points of the mobile shelving unit opening marked in each sample image are obtained. The sample images and the corresponding edge points are used to form a training dataset.

[0138] The model training module is used to construct a lightweight convolutional neural network model, and to train the model based on the training dataset to obtain a trained mobile shelving opening location detection model.

[0139] The edge point detection module is used to acquire a top view of the mobile shelving in real time, extract multiple images to be detected from the top view, and input each image to be detected into the mobile shelving opening position detection model to obtain the edge points at the opening of the mobile shelving in each image to be detected.

[0140] The opening position calculation module is used to calculate the opening position of the mobile shelving unit based on the edge points of the opening in each image to be detected.

[0141] Preferably, the model training module calculates the loss using the following formula:

[0142]

[0143]

[0144] Loss=Point_Loss+Confidence_Loss

[0145] Where S represents the length of the features output by the model; k i This represents the position of the edge point within the i-th interval predicted by the model. c represents the actual edge point position within the i-th interval; i This represents the confidence level that there are edge points in the i-th interval; This indicates whether there is actually a target point in the i-th interval; the value is 1 if there is, and 0 otherwise. λ obj , λ noobj and λ point This represents the weighting coefficient.

[0146] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation processes, please refer to the foregoing embodiments, which will not be repeated here.

[0147] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0148] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A dense cabinet opening position detection method based on a convolutional neural network, characterized by, The method comprises the following steps: obtaining overhead views of dense cabinets in different opening states, for each overhead view of the dense cabinets, extracting a plurality of sample images, obtaining edge points of the opening of the dense cabinets marked in each sample image, and constructing a training data set by combining the sample images and the corresponding edge points; constructing a lightweight convolutional neural network model, training the model based on the training data set, and obtaining a trained dense cabinet opening position detection model; obtaining an overhead view of the dense cabinet in real time, extracting a plurality of to-be-detected images from the overhead view, inputting each to-be-detected image into the dense cabinet opening position detection model to obtain edge points of the opening of the dense cabinet in each to-be-detected image; calculating the opening position of the dense cabinet based on the edge points of the opening of the dense cabinet in each to-be-detected image; the lightweight convolutional neural network model is an improved ShuffleNet model, and the improved ShuffleNet model comprises: a first convolutional layer for extracting shallow features of input data and outputting a shallow feature map, wherein the first convolutional layer is a one-dimensional convolutional layer; a max-pooling layer for reducing the dimension of the shallow feature map; a plurality of ShuffleNet modules for extracting deep semantic features from the reduced shallow feature map by using point-by-point group convolution and channel shuffling; a second convolutional layer for reducing the channel dimension of the deep semantic features and outputting a detection result. 2.The dense cabinet opening position detection method based on a convolutional neural network according to claim 1, characterized in that, Each ShuffleNet module comprises a one-dimensional ShuffleNet unit and a channel attention module connected in sequence, and the convolutional layer of the one-dimensional ShuffleNet unit is a one-dimensional convolution. 3.The dense cabinet opening position detection method based on a convolutional neural network according to claim 1, characterized in that, The loss Loss of the lightweight convolutional neural network model is calculated by the following formula: ； ； ； wherein, denotes the length of the feature represented by the model output; denotes the position of the edge point predicted by the model in the th interval, denotes the position of the actual edge point in the th interval; denotes the confidence that there is an edge point in the th interval; denotes whether there is a target point in the th interval or not, with value 1 if there is and 0 otherwise, , and denote the weight coefficients. 4.The dense cabinet opening position detection method based on a convolutional neural network according to claim 1, wherein, calculating the opening position of the dense cabinet based on the edge points of the opening of the dense cabinet in each to-be-detected image comprises: calculating the coordinates of the bottom of the cabinet corresponding to the edge points of the opening of the dense cabinet in each to-be-detected image according to coordinate projection transformation; calibrating the camera on the ground surface based on a standard calibration plate to obtain internal parameters, external parameters and distortion coefficients of the camera, and converting the coordinates of the bottom of the cabinet at the opening of the dense cabinet into actual position data according to the internal parameters, the external parameters and the distortion coefficients of the camera. 5.The dense cabinet opening position detection method based on a convolutional neural network according to claim 4, characterized in that, The coordinates of the bottom of the cabinet at the opening of the dense cabinet are calculated according to the following formula: ) ； wherein, is the coordinate of the edge point of the opening of the dense cabinet to be detected, is the coordinate of the bottom of the cabinet at the opening of the dense cabinet, is the coordinate of the center point of the image, is the installation height of the camera, is the height of the cabinet of the dense cabinet. 6.The dense cabinet opening position detection method based on a convolutional neural network according to claim 4, characterized in that, The coordinates of the bottom of the cabinet at the opening of the dense cabinet are converted into actual position data according to the internal parameters, the external parameters and the distortion coefficients of the camera by using the following formula: According to the formula: converts the pixel coordinates of the edge point to image coordinates ; wherein denotes the pixel coordinates of the edge point, and denotes the actual physical size of the pixel; denotes the pixel coordinates of the image center point; According to the formula The coordinates of the edge point in the camera coordinate system are calculated ; wherein f represents the focal length of the camera; According to the formula Calculate the world coordinates of edge points ,in, Represents the rotation matrix. This represents the translation matrix.

7. The dense cabinet opening position detection method based on a convolutional neural network according to claim 6, characterized in that, After converting the pixel coordinates of the edge points into image coordinates, before calculating the camera coordinates of the edge points, distortion correction is further performed by using the following formula: performing radial distortion correction on the position data: ； ； performing tangential distortion correction on the position data: ； ； wherein, , , , , is a distortion coefficient, denotes uncorrected coordinates in the image coordinate system, is the corrected coordinate in the image coordinate system, and r denotes the distance of the coordinate point from the image center point.

8. A dense cabinet opening position detection system based on a convolutional neural network, characterized by, comprising: a data set construction module for obtaining overhead views of dense cabinets in different opening states, for each overhead view of the dense cabinets, extracting a plurality of sample images, obtaining edge points of the opening of the dense cabinets marked in each sample image, and constructing a training data set by combining the sample images and the corresponding edge points; a model training module for constructing a lightweight convolutional neural network model, training the model based on the training data set, and obtaining a trained dense cabinet opening position detection model; An edge point detection module is configured to acquire a dense cabinet top view in real time, extract a plurality of to-be-detected images from the top view, and input each to-be-detected image into the dense cabinet opening position detection model to obtain edge points of a cabinet opening in each to-be-detected image; An opening position calculation module is configured to calculate the opening position of the dense cabinet based on the edge points of the cabinet opening in each to-be-detected image. The lightweight convolutional neural network model is an improved ShuffleNet model, and the improved ShuffleNet model comprises: A first convolutional layer is configured to extract shallow layer features of input data and output a shallow layer feature map, and the first convolutional layer is a one-dimensional convolutional layer; A max pooling layer is configured to reduce the dimension of the shallow layer feature map; A plurality of ShuffleNet modules are configured to extract deep layer semantic features from the reduced shallow layer feature map in a point-by-point group convolution and channel shuffle manner; A second convolutional layer is configured to reduce the channel dimension of the deep layer semantic features and output a detection result. 9.The dense cabinet opening position detection system based on a convolutional neural network according to claim 8, wherein, The model training module calculates a loss Loss by the following formula: ； ； ； wherein, denotes the length of the feature represented by the model output; denotes the position of the edge point predicted by the model in the th interval, denotes the position of the actual edge point in the th interval; denotes the confidence that there is an edge point in the th interval; denotes whether there is actually a target point in the th interval, with value 1 if there is and 0 otherwise, , and denote the weight coefficients.

Images