Real-time shelf stockout detection method and device based on inspection robot and storage medium

Abstract

The invention discloses a real-time shelf stockout detection method and device based on an inspection robot, and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: correcting a historical shelf image based on a shelf linear element of a shelf, obtaining a target neural network model through a training data set training model determined based on each corrected image, collecting a current shelf image of a preset point by using a camera on the inspection robot; processing the current shelf image by using the target neural network model to obtain a distorted flow field prediction result, generating a corrected image, feature points and descriptors based on distortion estimation parameters of the distorted flow field prediction result, and matching feature points of an image adjacent to the corrected image based on the feature points and the descriptors to obtain target matching point pairs; and determining a transformed image by using the homography matrix determined based on the target matching point pair and the corrected image so as to determine a vertical frame view, and detecting the vertical frame view to obtain an out-of-stock detection result. Therefore, the efficiency of detecting the goods in the goods shelf can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a real-time shelf out-of-stock detection method and device based on a patrol robot and a storage medium. BACKGROUND

[0002] At present, the shelf out-of-stock detection of retail stores mainly relies on manual patrol or fixed camera systems. Manual patrol is low in efficiency, high in cost, and easy to be affected by subjective factors; among them, although the fixed camera system can provide continuous monitoring, it has limited coverage, poor flexibility, and high delay due to dependence on cloud processing, which cannot meet the real-time demand. In the prior art, computer vision-based methods (such as target detection or segmentation) have been used for out-of-stock detection, but these methods usually run on the server side, increasing network dependency and processing delay. In addition, the complex lighting, occlusion and lens distortion problems of the shelf scene further reduce the detection accuracy.

[0003] The deficiencies of the above-mentioned technologies are as follows:

[0004] 1. Lack of real-time performance: traditional cloud processing methods have high delay (image uploading and result downloading are required), which cannot support mobile end-side real-time decision-making of the patrol robot.

[0005] 2. Image quality defects: images captured by multi-camera are easily affected by lens distortion (such as bending of straight line elements on the shelf), which leads to errors in subsequent segmentation and detection.

[0006] 3. Multi-image integration problem: there are gaps and overlaps between multiple camera images, which will reduce the consistency and detection efficiency of the overall view if not seamlessly spliced.

[0007] 4. Low accuracy of out-of-stock detection: semantic segmentation algorithms have poor robustness in complex retail scenes (such as dense goods and personnel interference), leading to false positives or false negatives.

[0008] 5. End-side resource limitations: the end-side devices (such as embedded GPUs) of the patrol robot have limited computing power, and existing deep learning models are difficult to deploy efficiently.

[0009] From the above, how to improve the efficiency of out-of-stock detection of goods in the shelf in the process of real-time shelf out-of-stock detection based on a patrol robot is a problem to be solved at present. SUMMARY

[0010] Therefore, the purpose of the present application is to provide a real-time shelf out-of-stock detection method and device based on a patrol robot and a storage medium, which can improve the efficiency of out-of-stock detection of goods in the shelf in the process of real-time shelf out-of-stock detection based on a patrol robot, and further improve the efficiency of the production process. The specific scheme is as follows:

[0011] In a first aspect, the present application provides a real-time shelf out-of-stock detection method based on an inspection robot, comprising:

[0012] correcting historical shelf images based on shelf straight line elements corresponding to shelves at preset positions, and determining a training data set based on the obtained correction images, to train an initial neural network model using the training data set to obtain a target neural network model, and then using a plurality of cameras on the inspection robot to collect current shelf images of the preset positions;

[0013] processing the current shelf images using the target neural network model to obtain distortion flow field prediction results, then generating correction images using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distortion flow field prediction results, and determining feature points and descriptors of the correction images, then matching feature points of images adjacent to the correction images based on the feature points and the descriptors to obtain target matching point pairs;

[0014] determining a homography matrix using a random sample consensus algorithm and based on the target matching point pairs, to align each of the correction images to the same coordinate system using the homography matrix to obtain transformed images, and processing the transformed images using a multi-band fusion algorithm to obtain a vertical frame view, and then performing out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result.

[0015] Optionally, the correction of historical shelf images based on shelf straight line elements corresponding to shelves at preset positions, and the determination of a training data set based on the obtained correction images, comprise:

[0016] determining shelf straight line elements corresponding to historical shelf images, setting the shelf straight line elements as geometric constraints, and then using a preset correction tool to perform distortion correction on the historical shelf images based on the geometric constraints to obtain correction images; wherein the size of the overlapping region between the collection regions of each camera is not less than a preset overlapping region threshold;

[0017] using a position sensor in the inspection robot to control the inspection robot to move to a preset position in a target store, and then using a plurality of cameras vertically arranged on a column of the inspection robot to collect current shelf images corresponding to the preset position based on a preset resolution and a preset frame rate.

[0018] Optionally, the determination of a training data set based on the obtained correction images, to train an initial neural network model using the training data set to obtain a target neural network model, comprises:

[0019] determine distances from a center of a plane in the corrected image to an optical axis center, and determine a plurality of first radial distortion coefficients for determining a distortion pattern of a central region of the corrected image and a plurality of second radial distortion coefficients for determining a fitting accuracy of an edge region of the corrected image;

[0020] perform image synthesis based on the first radial distortion coefficients, the second radial distortion coefficients, and the corrected image using a preset distortion formula to obtain a distortion image, and construct a training data set based on the corrected images and the distortion images, wherein the distortion images correspond to different distortion degrees;

[0021] construct an initial neural network model using an encoder-decoder architecture and including a residual connection, and train the initial neural network model using the training data set to obtain a target neural network model.

[0022] Optionally, the target neural network model is used to process the current shelf image to obtain a distortion flow field prediction result, a corrected image is generated based on distortion estimation parameters corresponding to the distortion flow field prediction result using a preset interpolation algorithm, feature points and descriptors of the corrected image are determined, feature points of an image adjacent to the corrected image are matched based on the feature points and the descriptors to obtain target matching point pairs, and the method includes the following steps:

[0023] The target neural network model is used to process the current shelf image based on a preset loss function to obtain a distortion flow field prediction result, and a corrected image is generated based on distortion estimation parameters determined based on the distortion flow field prediction result using a preset parameter estimation algorithm, and a preset bilinear interpolation algorithm is used to generate the corrected image based on the distortion flow field prediction result and the distortion estimation parameters, wherein a time length for determining the corrected image is not greater than a first preset processing time threshold.

[0024] First SIFT feature points and first ORB feature points are extracted from the corrected image, and descriptors are generated based on the first SIFT feature points and the first ORB feature points, adjacent images adjacent to the corrected image are determined, and second SIFT feature points and second ORB feature points corresponding to the adjacent images are determined.

[0025] The first SIFT feature points and the second SIFT feature points, and the first ORB feature points and the second ORB feature points are matched using a preset high-dimensional space nearest neighbor search technology and the descriptors to obtain initial matching point pairs, and the initial matching point pairs are filtered based on geometric consistency of a shelf scene to obtain target matching point pairs.

[0026] Optionally, the RANSAC algorithm is used to determine the homography matrix based on the target matching point pair, to align each of the corrected images to the same coordinate system using the homography matrix, to obtain a transformed image, and to process the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, including:

[0027] The RANSAC algorithm is used to estimate the target matching point pair to obtain a homography matrix, and to determine the coordinate points corresponding to each of the corrected images, to align the corrected images based on the coordinate points using the homography matrix, to obtain a transformed image;

[0028] The transformed image is decomposed into different spatial frequency bands, and a preset smoothing weighted average algorithm is used to perform brightness transition processing on a low-frequency band portion that meets a preset low-frequency condition to obtain a first processing result;

[0029] A high-frequency band portion that meets a preset high-frequency condition is directly superimposed or partially retained to obtain a second processing result, and a multi-band fusion algorithm is used to reconstruct and fuse the first processing result and the second processing result to obtain a vertical frame view; wherein the resolution corresponding to the vertical frame view is determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined to be not greater than a second preset processing time threshold.

[0030] Optionally, the vertical frame view is subjected to out-of-stock detection to obtain an out-of-stock detection result, including:

[0031] A preset segmentation network is used to perform semantic segmentation on the vertical frame view to obtain a target semantic segmentation result; the target semantic segmentation result includes a product category, a shelf category, a background category, and a personnel interference category corresponding to the vertical frame view;

[0032] Based on the target semantic segmentation result, the product categories corresponding to each product in the product region are extracted, and a product existence index is set based on the number of pixels in the product region, and then it is determined whether the product existence index is greater than a preset product area threshold;

[0033] If the product existence index is greater than the preset product area threshold, the state corresponding to the product is set to an in-stock state, and if the product existence index is not greater than the preset product area threshold, the state corresponding to the product is set to an out-of-stock state, and the out-of-stock location, product type, and confidence corresponding to the product are sent to the current user interface.

[0034] Optionally, after the vertical frame view is subjected to out-of-stock detection to obtain an out-of-stock detection result, it further includes:

[0035] The inspection time interval is determined based on production demand, and the inspection robot is called to re-trigger the step of using several cameras on the inspection robot to collect the current shelf image of the preset point at each inspection time interval; wherein, it is determined that the processing delay time corresponding to the out-of-stock detection result is not greater than a preset delay threshold.

[0036] In a second aspect, the present application provides a real-time shelf out-of-stock detection device based on a patrol robot, comprising:

[0037] A current shelf image acquisition module is used to correct historical shelf images based on shelf linear elements corresponding to shelves at preset locations, and to determine a training data set based on each corrected image obtained, so as to train an initial neural network model using the training data set to obtain a target neural network model, and then to use multiple cameras on the inspection robot to capture current shelf images at the preset locations;

[0038] a matching point pair determination module, configured to process the current shelf image using the target neural network model to obtain a distorted flow field prediction result, then generate a corrected image using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distorted flow field prediction result, determine feature points and descriptors of the corrected image, and then match feature points of an image adjacent to the corrected image based on the feature points and descriptors to obtain target matching point pairs;

[0039] The out-of-stock detection result determination module is used to use a random sampling consistency algorithm and determine a homography matrix based on the target matching point pairs, so as to use the homography matrix to align each of the corrected images into the same coordinate system to obtain a transformed image, and use a multi-band fusion algorithm to process the transformed image to obtain a vertical frame view, and then perform out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result.

[0040] Optionally, the out-of-stock detection result determination module includes:

[0041] An image coordinate system alignment unit is used to estimate the target matching point pairs using a RANSAC algorithm to obtain a homography matrix, and determine the coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment on the corrected images using the homography matrix and based on the coordinate points to obtain a transformed image;

[0042] a frequency band brightness transition unit, configured to decompose the transformed image into frequency bands of different spatial frequencies, and perform brightness transition processing on a low-frequency band portion that meets a preset low-frequency condition using a preset smoothing weighted average algorithm to obtain a first processing result;

[0043] The processing result reconstruction fusion unit is configured to perform direct superposition or partial reservation processing on the high-frequency band part satisfying the preset high-frequency condition to obtain a second processing result, and perform reconstruction fusion on the first processing result and the second processing result by using a multi-band fusion algorithm to obtain a vertical frame view. The vertical frame view corresponds to a resolution determined based on the number of cameras, and a processing time corresponding to the vertical frame view is not greater than a second preset processing time threshold.

[0044] In a third aspect, a computer readable storage medium is provided for storing a computer program, wherein the computer program is executed by a processor to implement the real-time shelf out-of-stock detection method based on the inspection robot.

[0045] As can be seen from the above, before performing the real-time shelf out-of-stock detection based on the inspection robot, the historical shelf image is corrected based on the shelf straight line element corresponding to the shelf at the preset point, and the training data set is determined based on the obtained each corrected image, so as to train the initial neural network model by using the training data set to obtain the target neural network model, and then the current shelf image of the preset point is collected by using the plurality of cameras on the inspection robot; the current shelf image is processed by using the target neural network model to obtain the distortion flow field prediction result, and then the corrected image is generated by using the preset interpolation algorithm and based on the distortion estimation parameter corresponding to the distortion flow field prediction result, and the feature points and the descriptor of the corrected image are determined, and then the feature points of the image adjacent to the corrected image are matched based on the feature points and the descriptor to obtain the target matching point pair; the homography matrix is determined by using the random sample consensus algorithm and based on the target matching point pair, so as to align each corrected image to the same coordinate system by using the homography matrix to obtain the transformed image, and the vertical frame view is obtained by processing the transformed image by using the multi-band fusion algorithm, and then the out-of-stock detection result is obtained by performing the out-of-stock detection on the vertical frame view.

[0046] It can be seen that, first, before real-time shelf out-of-stock detection based on the inspection robot, the historical shelf image needs to be corrected based on the shelf straight line element corresponding to the shelf at the preset point, and the training data set is determined based on the obtained each corrected image, so as to train the initial neural network model by using the training data set to obtain the target neural network model, and then the current shelf image of the preset point is collected by using a plurality of cameras on the inspection robot; secondly, the target neural network model is used to process the current shelf image to obtain the distortion flow field prediction result, and then the corrected image is generated by using the preset interpolation algorithm and based on the distortion estimation parameter corresponding to the distortion flow field prediction result, and the feature points and descriptors of the corrected image are determined, and then the target matching point pair is obtained by matching the feature points of the corrected image with the feature points of the image adjacent to the corrected image; finally, the homography matrix is determined by using the random sample consensus algorithm and based on the target matching point pair, so as to align each corrected image to the same coordinate system by using the homography matrix to obtain the transformed image, and the vertical frame view is obtained by processing the transformed image by using the multi-band fusion algorithm, and then the out-of-stock detection result is obtained by detecting the vertical frame view. In this way, the efficiency of the out-of-stock detection of the goods in the shelf is improved in the process of real-time shelf out-of-stock detection based on the inspection robot, and the efficiency of the production process is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0048] Figure 1 A flow chart of a real-time shelf out-of-stock detection method based on an inspection robot is disclosed in the present application.

[0049] Figure 2 A schematic diagram of the positional relationship between a specific store commodity shelf and an inspection robot is disclosed.

[0050] Figure 3 A flow chart of a specific image distortion correction process is disclosed in the present application.

[0051] Figure 4 A flow chart of a specific image stitching process is disclosed in the present application.

[0052] Figure 5 A structure schematic diagram of a real-time shelf out-of-stock detection device based on an inspection robot is disclosed in the present application. DETAILED DESCRIPTION

[0053] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0054] At present, the shelf out-of-stock detection of retail stores mainly relies on manual inspection or fixed camera systems. Manual inspection is low in efficiency, high in cost, and easy to be affected by subjective factors; among them, although the fixed camera system can provide continuous monitoring, it has limited coverage, poor flexibility, and high delay due to dependence on cloud processing, which cannot meet the real-time demand. Therefore, the present application provides a real-time shelf out-of-stock detection method based on an inspection robot, which can improve the efficiency of out-of-stock detection of goods in the shelf in the real-time shelf out-of-stock detection process based on the inspection robot, and further improve the efficiency of the production process.

[0055] Referring to Figure 1 The embodiments of the present application disclose a real-time shelf out-of-stock detection method based on an inspection robot, which comprises:

[0056] Step S11, correcting the historical shelf image based on the shelf straight line element corresponding to the shelf at the preset point, and determining a training data set based on the obtained correction image, to train an initial neural network model to obtain a target neural network model, and then collecting the current shelf image of the preset point by using a plurality of cameras on the inspection robot.

[0057] In this embodiment, when the inspection robot moves in the store, the multiple cameras (standard configuration is 3, which can be adjusted according to the height of the shelf) vertically arranged on the column of the inspection robot can synchronously collect images. Among them, the field of view angle of each camera is designed to cover a specific vertical area of the shelf (for example, each camera covers 1-2 meters in height), and the overlapping area of the fields of view of adjacent cameras is not less than 20%, so as to ensure no blind area. Furthermore, the parameters of each camera include a resolution of not less than 1920x1080, a frame rate of 30fps (Frames Per Second, i.e. frames per second), to adapt to the dynamic environment, in addition, the present application embodiment configures the collection trigger mechanism in the robot position sensor, such as LiDAR (Light Detection and Ranging, i.e. laser detection and ranging), so as to automatically start the image collection operation when the robot reaches the preset point.

[0058] In a specific embodiment, the schematic diagram of the positional relationship between the store commodity shelf and the inspection robot is as shown in Figure 2As shown, multiple cameras are arranged vertically on the robot column. The field of view of each camera can cover a certain area of ​​the shelf in the vertical direction. The fields of view of adjacent cameras overlap. The combined field of view range of multiple cameras can ensure that the field of view of each camera covers the entire shelf in the vertical direction.

[0059] Specifically, correcting historical shelf images based on shelf straight line elements corresponding to shelves at preset points, and determining a training data set based on the obtained corrected images can include: determining shelf straight line elements corresponding to historical shelf images to set the shelf straight line elements as geometric constraints, and then using a preset correction tool and based on the geometric constraints to perform distortion correction on the historical shelf images to obtain corrected images; wherein the size of the overlapping area between the acquisition areas of each camera is not less than a preset overlapping area threshold; using the position sensor in the inspection robot to control the inspection robot to move to the preset point in the target store, and then using a number of cameras arranged vertically on the column of the inspection robot to collect the current shelf image corresponding to the preset point based on a preset resolution and a preset frame rate.

[0060] It is worth mentioning that the optical imaging of the camera is not an ideal pinhole imaging, which leads to a certain deviation between the actual imaging point of the spatial point on the imaging plane and the ideal imaging point, and there is a complex nonlinear relationship between the two, namely lens distortion. Lens distortion is mainly divided into radial distortion and tangential distortion, with radial distortion being the main one. The shelf scene has many straight line elements, and this prior information can play a good role in the calibration process. The image distortion correction process is as follows: Figure 3 As shown;

[0061] In this embodiment, when capturing a large number of shelf images, covering different lighting and angles, this embodiment utilizes shelf linear elements (such as shelf edges) as geometric constraints and generates distortion-corrected reference images using manual calibration tools, such as checkerboard calibration in the OpenCV (OpenSource Computer Vision Library). Simultaneously, the Brown-Conrady radial distortion formula (Brown-Conrady Distortion Model) is used to synthesize images of varying degrees of distortion, constructing a training dataset (including distorted images and corresponding corrected images). The expression is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] in, is the horizontal coordinate of the image plane center, is the vertical coordinate of the image plane center, is the distance of the image plane center to the optical axis center, , and are low order radial distortion coefficients, dominating the distortion pattern (e.g. barrel / pincushion) in the central region, , and are high order radial distortion coefficients, enhancing the fitting accuracy in the edge region, especially important for ultra-wide lenses, is the horizontal coordinate of the synthesized image, is the vertical coordinate of the synthesized image.

[0066] Subsequently, an embodiment of the present application needs to design a simple and lightweight convolutional neural network (CNN, Convolutional Neural Network) to utilize the convolutional neural network and predict the distortion flow field based on a single distortion image, wherein the network structure adopts a U-Net encoder-decoder architecture (U-Net Encoder-Decoder Architecture): the encoder is 4 layers of convolution, each layer contains Conv2D (2D Convolution), BatchNorm (Batch Normalization) and ReLU (Rectified Linear Unit), and outputs a feature map; the decoder is 4 layers of deconvolution to restore the resolution; skip connections are added to avoid gradient vanishing. Specifically, based on the obtained correction images, a training data set is determined to train an initial neural network model to obtain a target neural network model, which can include: determining the distance of the plane center in the correction image to the optical axis center, and determining a plurality of first radial distortion coefficients for determining the distortion pattern of the central region of the correction image and a plurality of second radial distortion coefficients for determining the fitting accuracy of the edge region of the correction image; using a preset distortion formula and based on each first radial distortion coefficient, each second radial distortion coefficient and the correction image to perform image synthesis to obtain a distortion image, and based on each correction image and each distortion image to construct a training data set; each distortion image corresponds to a different distortion degree; an initial neural network model adopting an encoder-decoder architecture and including a residual connection is constructed, and the training data set is used to train the initial neural network model to obtain a target neural network model.

[0067] In step S12, the target neural network model is used to process the current shelf image to obtain a distortion flow field prediction result, a correction image is generated based on a preset interpolation algorithm and distortion estimation parameters corresponding to the distortion flow field prediction result, feature points and descriptors of the correction image are determined, feature points of an image adjacent to the correction image are matched based on the feature points and the descriptors, and target matching point pairs are obtained.

[0068] In this embodiment, the input of the target neural network model is a single distortion image (with a size of 256x256), the output is a distortion flow field (with the same size as the input), and the loss function adopts L2 norm (i.e., L2 Norm) (mean square error) to minimize the difference between the predicted flow field and the real flow field. It is worth mentioning that after the distortion flow field is predicted by the model, the model fitting method needs to be used to further estimate the distortion parameters in this embodiment, so as to improve the correction accuracy. In a specific implementation, the RANSAC algorithm (RANdom SAmple Consensus, i.e., RANdom SAmple Consensus) can be used to estimate the distortion parameters in this embodiment to improve the robustness. Finally, the correction image is generated based on the parameters and the flow field by using the bilinear interpolation, and the processing time of the model is controlled within 50 ms to meet the real-time requirements of the terminal side.

[0069] It is worth mentioning that since multiple cameras are deployed in the inspection robot, the images collected by these cameras need to be spliced to form a complete shelf view in the vertical direction to eliminate the gaps and overlaps between the images, so as to improve the efficiency and accuracy of subsequent processing. The process of splicing the images is as shown in Figure 4 Firstly, feature extraction and matching operations need to be performed on the images: firstly, SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) feature point extraction operations are performed on the multi-view images after distortion correction, and descriptors are calculated. Then, the feature points of adjacent images are matched based on FLANN (Fast Library for Approximate Nearest Neighbors), and the geometric consistency (such as the vertical shelf line) of the shelf scene is used to filter the mismatch.

[0070] Specifically, the target neural network model is used to process the current shelf image to obtain a distortion flow field prediction result, then a preset interpolation algorithm is used to generate a corrected image based on distortion estimation parameters corresponding to the distortion flow field prediction result, and feature points and descriptors of the corrected image are determined, and then the feature points and the descriptors are matched with feature points of an image adjacent to the corrected image to obtain target matching point pairs, which can include: using the target neural network model to process the current shelf image based on a preset loss function to obtain the distortion flow field prediction result, then using a preset parameter estimation algorithm based on the distortion flow field prediction result to determine the distortion estimation parameters, and using a preset bilinear interpolation algorithm based on the distortion flow field prediction result and the distortion estimation parameters to generate the corrected image; wherein the time length of the corrected image is not greater than a first preset processing time threshold; first SIFT feature points and first ORB feature points are extracted from the corrected image, and descriptors are generated based on the first SIFT feature points and the first ORB feature points, then adjacent images adjacent to the corrected image are determined, and second SIFT feature points and second ORB feature points corresponding to the adjacent images are determined; using a preset high-dimensional space nearest neighbor search technology and the descriptors, the first SIFT feature points and the second SIFT feature points, and the first ORB feature points and the second ORB feature points are matched to obtain initial matching point pairs, and the initial matching point pairs are filtered based on the geometric consistency of the shelf scene to obtain the target matching point pairs.

[0071] In step S13, a homography matrix is determined based on the target matching point pairs using a random sample consensus algorithm, and each of the corrected images is aligned to the same coordinate system using the homography matrix to obtain transformed images, and a multi-band fusion algorithm is used to process the transformed images to obtain a vertical frame view, and then the vertical frame view is subjected to out-of-stock detection to obtain an out-of-stock detection result.

[0072] In this embodiment, after obtaining the target matching point pairs, the image needs to be transformed and fused. In one specific embodiment, a homography matrix is applied to transform the image to align multiple images to a unified coordinate system, and the expression is as follows:

[0073] ;

[0074] wherein, is the original image coordinate, is the transformed coordinate, is the homography matrix, which is determined based on the target matching point pairs using the RANSAC algorithm.

[0075] Further, when fusing the images, the embodiments of the present application can use a multi-band blending algorithm to process the images by frequency (smooth transition for low-frequency part, and retain details for high-frequency part), and perform weighted average processing on the overlapping area to eliminate gaps and brightness differences. Finally, a complete vertical shelf view is generated, in which the resolution can be adaptively adjusted according to the number of cameras, for example, if the number of cameras is 3, the resolution of the spliced image is 4000x3000.

[0076] Specifically, the random sample consensus algorithm is used to determine the homography matrix based on the target matching point pair, to align the corrected images to the same coordinate system using the homography matrix to obtain the transformed image, and to process the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, which can include: using the RANSAC algorithm to estimate the target matching point pair to obtain the homography matrix, and determining the coordinate points corresponding to each corrected image, to align the coordinate system of the corrected image based on the coordinate points using the homography matrix, to obtain the transformed image; decompose the transformed image into different spatial frequency bands, and use a pre-set smooth weighted average algorithm to perform brightness transition processing on the low-frequency band part that meets the pre-set low-frequency condition to obtain a first processing result; perform direct superposition or partial retention processing on the high-frequency band part that meets the pre-set high-frequency condition to obtain a second processing result, and use a multi-band fusion algorithm to reconstruct and fuse the first processing result and the second processing result to obtain a vertical frame view; wherein the resolution of the vertical frame view is a resolution determined based on the number of cameras, and the processing time of the vertical frame view is determined to be not greater than a second pre-set processing time threshold.

[0077] It is worth mentioning that when deployed on an end-side device such as NVIDIA Jetson (i.e. an embedded AI computing platform), the embodiments of the present application can use GPU acceleration to speed up feature matching and fusion processes to ensure that the stitching delay is less than 100ms, and output the stitching result as a seamless see-through shelf view to facilitate subsequent semantic segmentation.

[0078] In this embodiment, after obtaining the spliced image, the embodiments of the present application need to use the algorithm deployed on the robot side to perform semantic segmentation on the spliced image, i.e. to identify and classify different elements (such as goods, shelves, personnel, etc.) in the image, so as to facilitate subsequent out-of-stock detection and other analysis. Subsequently, based on the results of image semantic segmentation, the robot will perform out-of-stock detection on the goods on the shelf. If the quantity of a certain good is found to be below a pre-set threshold or completely missing, the good will be marked as out-of-stock, and an out-of-stock detection result will be generated.

[0079] It is worth mentioning that the above semantic segmentation model is a lightweight segmentation network (DeepLabv3plus-MobileNetV2), and the input of the model is the spliced shelf image, and the output is the semantic category of each pixel (including: goods, shelves, background, personnel interference, etc.). Moreover, the training data set for training the semantic segmentation model is a pre-labeled shelf image, and data augmentation includes rotation, scaling and illumination change to improve generalization. In addition, in a specific embodiment, when the model performs inference on the edge side, the calculation amount can be reduced through quantization (INT8 precision) and pruning to control the segmentation delay within 200ms.

[0080] Furthermore, the embodiments of the present application can utilize the out-of-stock detection algorithm and extract each commodity in the commodity area based on the segmentation result (such as through connected component analysis). For each commodity category (such as beverages, snacks), the number of pixels in the area is counted, and the obtained statistical result is set as a commodity existence index. In addition, the out-of-stock detection logic is to set a commodity quantity threshold (for example, when the commodity area is less than 30% of the historical average, it is determined that the commodity is out of stock), and to determine the size relationship between the current commodity quantity and the commodity quantity threshold to determine whether the commodity is out of stock, or to trigger an alarm when the commodity is completely missing, wherein the output result of the algorithm includes the out-of-stock location, the commodity type and the confidence of the commodity.

[0081] Specifically, the out-of-stock detection on the vertical frame view to obtain the out-of-stock detection result can include: performing semantic segmentation on the vertical frame view by using a preset segmentation network to obtain a target semantic segmentation result; the target semantic segmentation result includes a commodity category, a shelf category, a background category and a personnel interference category corresponding to the vertical frame view; extracting the commodity category corresponding to each commodity in the commodity area based on the target semantic segmentation result, and setting a commodity existence index based on the number of pixels in the commodity area, and then determining whether the commodity existence index is greater than a preset commodity area threshold; if the commodity existence index is greater than the preset commodity area threshold, the state corresponding to the commodity is set to an in-stock state, and if the commodity existence index is not greater than the preset commodity area threshold, the state corresponding to the commodity is set to an out-of-stock state, and the out-of-stock location, the commodity type and the confidence corresponding to the commodity are sent to the current user interface.

[0082] It is worth mentioning that the application embodiment will display the detection result in real time on the robot interface and upload it to the store management system, and the above entire process is controlled by a loop structure, and the robot will repeat the inspection operation based on a preset interval (for example, every 30 minutes) after completing an inspection each time to ensure that the goods in the store always remain in a sufficient inventory state. Specifically, after obtaining the out-of-stock detection result by detecting the vertical frame view, the method can further include: determining an inspection time interval based on production needs, and calling the inspection robot to re-trigger the step of collecting the current shelf image of the preset point by the plurality of cameras on the inspection robot every inspection time interval; wherein the processing delay time corresponding to the out-of-stock detection result is not greater than the preset delay threshold.

[0083] As can be seen from the above, the application embodiment first needs to correct the historical shelf image based on the shelf straight line element corresponding to the shelf at the preset point before real-time shelf out-of-stock detection based on the inspection robot, and determine a training data set based on the obtained each corrected image to train an initial neural network model to obtain a target neural network model, and then collect the current shelf image of the preset point by a plurality of cameras on the inspection robot; secondly, the target neural network model is used to process the current shelf image to obtain a distortion flow field prediction result, and then a corrected image is generated based on the distortion estimation parameter corresponding to the distortion flow field prediction result by using a preset interpolation algorithm, and the feature points and descriptors of the corrected image are determined, and then the feature points of the image adjacent to the corrected image are matched based on the feature points and descriptors to obtain a target matching point pair; finally, the homography matrix is determined based on the target matching point pair by using the random sample consensus algorithm, so as to align each corrected image to the same coordinate system by using the homography matrix to obtain a transformed image, and the vertical frame view is obtained by processing the transformed image by using a multi-band fusion algorithm, and then the vertical frame view is detected for out-of-stock to obtain an out-of-stock detection result. In this way, the efficiency of detecting the out-of-stock of the goods in the shelf is improved in the process of real-time shelf out-of-stock detection based on the inspection robot, and the efficiency of the production process is further improved.

[0084] Correspondingly, referring to Figure 5 The application also provides a real-time shelf out-of-stock detection device based on an inspection robot, which comprises:

[0085] The current shelf image acquisition module 11 is configured to correct the historical shelf image based on the shelf straight line element corresponding to the shelf at the preset point, and determine a training data set based on the obtained each corrected image, so as to train an initial neural network model by using the training data set to obtain a target neural network model, and then collect the current shelf image of the preset point by a plurality of cameras on the inspection robot;

[0086] The matching point pair determination module 12 is configured to process the current shelf image by using the target neural network model to obtain a distortion flow field prediction result, generate a corrected image by using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distortion flow field prediction result, determine feature points and descriptors of the corrected image, match feature points of an image adjacent to the corrected image based on the feature points and the descriptors, and obtain target matching point pairs.

[0087] The out-of-stock detection result determination module 13 is configured to determine a homography matrix by using a random sample consensus algorithm and based on the target matching point pairs, align each of the corrected images to the same coordinate system by using the homography matrix to obtain transformed images, process the transformed images by using a multi-band fusion algorithm to obtain a vertical frame view, and perform out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result.

[0088] In some embodiments, the current shelf image acquisition module 11 can specifically include:

[0089] The corrected image determination unit is configured to determine a shelf straight line element corresponding to a historical shelf image, set the shelf straight line element as a geometric constraint condition, and perform distortion correction on the historical shelf image by using a preset correction tool and based on the geometric constraint condition to obtain a corrected image; a size of an overlapping region between the collection regions of each of the cameras is not less than a preset overlapping region threshold.

[0090] The shelf image collection unit is configured to control the inspection robot to move to a preset point by using a position sensor in the inspection robot in a target store, and collect a current shelf image corresponding to the preset point by using a plurality of cameras vertically arranged on a stand of the inspection robot and based on a preset resolution and a preset frame rate.

[0091] In some embodiments, the current shelf image acquisition module 11 can specifically include:

[0092] The radial distortion coefficient determination unit is configured to determine a distance from a center point in a plane in the corrected image to a center of an optical axis, determine a plurality of first radial distortion coefficients for determining a distortion form of a central region of the corrected image, and determine a plurality of second radial distortion coefficients for determining fitting accuracy of an edge region of the corrected image.

[0093] The training data set determination unit is configured to perform image synthesis by using a preset distortion formula and based on each of the first radial distortion coefficients, each of the second radial distortion coefficients, and the corrected image to obtain a distortion image, and construct a training data set based on each of the corrected images and each of the distortion images; the distortion images correspond to different distortion degrees.

[0094] The model training unit is configured to construct an initial neural network model adopting an architecture of an encoder and a decoder and including a residual connection, and train the initial neural network model by using the training data set to obtain a target neural network model.

[0095] In some embodiments, the matching point pair determination module 12 can specifically include:

[0096] The distortion estimation parameter determination unit is configured to process the current shelf image by using the target neural network model and based on a preset loss function to obtain a distortion flow field prediction result, and then determine distortion estimation parameters by using a preset parameter estimation algorithm and based on the distortion flow field prediction result, so as to generate a corrected image by using a preset bilinear interpolation algorithm and based on the distortion flow field prediction result and the distortion estimation parameters; wherein a time length for determining the corrected image is not greater than a first preset processing time threshold.

[0097] The feature point generation unit is configured to extract first SIFT feature points and first ORB feature points from the corrected image, and generate a descriptor based on the first SIFT feature points and the first ORB feature points, and then determine a neighboring image adjacent to the corrected image, and determine second SIFT feature points and second ORB feature points corresponding to the neighboring image.

[0098] The feature point matching unit is configured to match the first SIFT feature points and the second SIFT feature points, and the first ORB feature points and the second ORB feature points by using a preset high-dimensional space nearest neighbor search technology and the descriptor to obtain initial matching point pairs, and filter the initial matching point pairs based on geometric consistency of a shelf scene to obtain target matching point pairs.

[0099] In some embodiments, the out-of-stock detection result determination module 13 can specifically include:

[0100] The image coordinate system alignment unit is configured to estimate the target matching point pairs by using a RANSAC algorithm to obtain a homography matrix, and determine coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment on the corrected images based on the homography matrix and the coordinate points to obtain transformed images.

[0101] The frequency band brightness transition unit is configured to decompose the transformed images into frequency bands of different spatial frequencies, and perform brightness transition processing on a low-frequency frequency band part satisfying a preset low-frequency condition by using a preset smooth weighted average algorithm to obtain a first processing result.

[0102] The processing result reconstruction fusion unit is configured to perform direct superposition or partial reservation processing on the high-frequency band part satisfying the preset high-frequency condition to obtain a second processing result, and perform reconstruction fusion on the first processing result and the second processing result by using a multi-frequency band fusion algorithm to obtain a vertical frame view. The vertical frame view corresponds to a resolution determined based on the number of cameras, and a processing time corresponding to the vertical frame view is determined to be not greater than a second preset processing time threshold.

[0103] In some embodiments, the out-of-stock detection result determination module 13 can specifically include:

[0104] The semantic segmentation result determination unit is configured to perform semantic segmentation on the vertical frame view by using a preset segmentation network to obtain a target semantic segmentation result. The target semantic segmentation result includes a commodity category, a shelf category, a background category, and a personnel interference category corresponding to the vertical frame view.

[0105] The commodity category determination unit is configured to extract a commodity category corresponding to each commodity in a commodity region based on the target semantic segmentation result, and set a pixel quantity in the commodity region as a commodity existence index, and then determine whether the commodity existence index is greater than a preset commodity area threshold.

[0106] The index determination unit is configured to set a state corresponding to the commodity as an in-stock state if the commodity existence index is greater than the preset commodity area threshold, and set the state corresponding to the commodity as an out-of-stock state if the commodity existence index is not greater than the preset commodity area threshold, and send an out-of-stock position, a commodity type, and a confidence corresponding to the commodity to a current user interface.

[0107] In some embodiments, the real-time shelf out-of-stock detection device based on the inspection robot can further include:

[0108] The inspection time interval determination unit is configured to determine an inspection time interval based on production demand, and call the step of triggering the current shelf image of the preset point position collected by the plurality of cameras on the inspection robot every inspection time interval. The processing delay time corresponding to the out-of-stock detection result is determined to be not greater than a preset delay threshold.

[0109] Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to realize the foregoing disclosed real-time shelf out-of-stock detection method based on an inspection robot. For specific steps of the method, reference can be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0110] The various embodiments described in the specification are progressive in nature, and each embodiment highlights the differences from other embodiments. The same or similar parts among the various embodiments can be mutually referred to. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method.

[0111] Those skilled in the art will further appreciate that the individual steps of the examples described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or combinations of both. The various examples have been described in relation to the described embodiments, as a means of generalizing the interchangeability of hardware and software under the principles mentioned above. The particular implementation of an individual example in either hardware or software can be determined by the particular application and design constraints imposed on the overall system. Skilled artisans will appreciate that the principles described herein can be practiced in a variety of system environments, and that the described implementations are merely examples and not intended to limit the scope of the application.

[0112] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0113] Finally, it needs to be pointed out that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0114] The above describes the technical solutions provided by the present application in detail, and the principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only for helping to understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description of the specification should not be understood as limiting the present application.

Claims

1. A real-time shelf out-of-stock detection method based on a patrol robot, characterized in that: include: Correcting historical shelf images based on shelf linear elements corresponding to shelves at preset locations, and determining a training data set based on the obtained corrected images, so as to train an initial neural network model using the training data set to obtain a target neural network model, and then using several cameras on the inspection robot to capture current shelf images at the preset locations; Processing the current shelf image using the target neural network model to obtain a distorted flow field prediction result, then generating a corrected image using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distorted flow field prediction result, determining feature points and descriptors of the corrected image, and then matching feature points of an image adjacent to the corrected image based on the feature points and the descriptors to obtain target matching point pairs; A random sampling consistency algorithm is used to determine a homography matrix based on the target matching point pairs, so as to align each of the corrected images into the same coordinate system using the homography matrix to obtain a transformed image, and the transformed image is processed using a multi-band fusion algorithm to obtain a vertical frame view, and then out-of-stock detection is performed on the vertical frame view to obtain an out-of-stock detection result.

2. The real-time out-of-stock detection method for shelves based on patrol robots according to claim 1 is characterized in that: The method of correcting the historical shelf images based on the shelf straight line elements corresponding to the shelves at the preset points, and determining the training data set based on the obtained corrected images, includes: Determining shelf straight line elements corresponding to the historical shelf images, setting the shelf straight line elements as geometric constraints, and then using a preset correction tool and based on the geometric constraints to perform distortion correction on the historical shelf images to obtain a corrected image; wherein the size of the overlapping area between the acquisition areas of the cameras is not less than a preset overlapping area threshold; In the target store, the position sensor in the inspection robot is used to control the inspection robot to move to a preset point, and then several cameras arranged vertically on the column of the inspection robot are used to capture the current shelf image corresponding to the preset point based on a preset resolution and a preset frame rate.

3. The real-time shelf out-of-stock detection method based on the inspection robot according to claim 1 is characterized in that: The step of determining a training data set based on the obtained corrected images, and using the training data set to train an initial neural network model to obtain a target neural network model, includes: Determining a distance from a plane midpoint in the corrected image to the center of the optical axis, and determining a plurality of first radial distortion coefficients for determining a distortion form of a central region of the corrected image and a plurality of second radial distortion coefficients for determining a fitting accuracy of an edge region of the corrected image; Performing image synthesis based on the first radial distortion coefficients, the second radial distortion coefficients, and the corrected image using a preset distortion formula to obtain a distorted image, and constructing a training dataset based on the corrected image and the distorted image; each distorted image corresponds to a different degree of distortion; An initial neural network model is constructed that adopts an encoder and decoder architecture and includes a residual connection, and the initial neural network model is trained using the training data set to obtain a target neural network model.

4. The real-time shelf out-of-stock detection method based on patrol robots according to claim 1 is characterized in that: The method further comprises: processing the current shelf image using the target neural network model to obtain a distorted flow field prediction result, generating a corrected image using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distorted flow field prediction result, determining feature points and descriptors of the corrected image, and then matching feature points of an image adjacent to the corrected image based on the feature points and the descriptors to obtain target matching point pairs, including: Processing the current shelf image using the target neural network model and based on a preset loss function to obtain a distorted flow field prediction result, then determining distortion estimation parameters based on the distorted flow field prediction result using a preset parameter estimation algorithm, and generating a corrected image using a preset bilinear interpolation algorithm based on the distorted flow field prediction result and the distortion estimation parameters; wherein determining that the duration of the corrected image is no greater than a first preset processing time threshold; Extracting first SIFT feature points and first ORB feature points from the corrected image, generating a descriptor based on the first SIFT feature points and the first ORB feature points, then determining an adjacent image adjacent to the corrected image, and determining second SIFT feature points and second ORB feature points corresponding to the adjacent image; Using the preset high-dimensional space nearest neighbor search technology and the descriptor, the first SIFT feature point and the second SIFT feature point, as well as the first ORB feature point and the second ORB feature point are matched to obtain initial matching point pairs, and the initial matching point pairs are filtered based on the geometric consistency of the shelf scene to obtain target matching point pairs.

5. The real-time shelf out-of-stock detection method based on patrol robots according to claim 1 is characterized in that: The method comprises: using a random sampling consistency algorithm and determining a homography matrix based on the target matching point pairs, aligning the corrected images into the same coordinate system using the homography matrix to obtain a transformed image, and processing the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, including: The target matching point pairs are estimated using a RANSAC algorithm to obtain a homography matrix, and coordinate points corresponding to each of the corrected images are determined, so as to align the coordinate systems of the corrected images using the homography matrix and based on the coordinate points to obtain a transformed image; Decomposing the transformed image into frequency bands of different spatial frequencies, and performing brightness transition processing on a low-frequency band portion that meets a preset low-frequency condition using a preset smoothing weighted average algorithm to obtain a first processing result; Directly superimpose or partially retain the high-frequency band portion that meets the preset high-frequency condition to obtain a second processing result, and use a multi-band fusion algorithm to reconstruct and fuse the first processing result and the second processing result to obtain a vertical frame view; wherein, the resolution corresponding to the vertical frame view is a resolution determined based on the number of cameras, and it is determined that the processing time corresponding to the vertical frame view is not greater than a second preset processing time threshold.

6. The real-time out-of-stock detection method for shelves based on patrol robots according to claim 1, characterized in that: The performing out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result includes: Performing semantic segmentation on the vertical frame view using a preset segmentation network to obtain a target semantic segmentation result; the target semantic segmentation result includes a product type, a shelf type, a background type, and a human interference type corresponding to the vertical frame view; Extracting the product category corresponding to each product in the product area based on the target semantic segmentation result, setting the product existence index based on the number of pixels in the product area, and then determining whether the product existence index is greater than a preset product area threshold; If the commodity existence index is greater than the preset commodity area threshold, the status corresponding to the commodity is set to in-stock status; if the commodity existence index is not greater than the preset commodity area threshold, the status corresponding to the commodity is set to out-of-stock status, and the out-of-stock location, commodity type and confidence corresponding to the commodity are sent to the current user interface.

7. The real-time shelf out-of-stock detection method based on patrol robots according to claim 1 is characterized in that: After performing out-of-stock detection on the vertical frame view and obtaining the out-of-stock detection result, the method further includes: The inspection time interval is determined based on production demand, and the inspection robot is called to re-trigger the step of using several cameras on the inspection robot to collect the current shelf image of the preset point at each inspection time interval; wherein, it is determined that the processing delay time corresponding to the out-of-stock detection result is not greater than a preset delay threshold.

8. A real-time shelf out-of-stock detection device based on a patrol robot, characterized in that: include: A current shelf image acquisition module is used to correct historical shelf images based on shelf linear elements corresponding to shelves at preset locations, and to determine a training data set based on each corrected image obtained, so as to train an initial neural network model using the training data set to obtain a target neural network model, and then to use multiple cameras on the inspection robot to capture current shelf images at the preset locations; a matching point pair determination module, configured to process the current shelf image using the target neural network model to obtain a distorted flow field prediction result, then generate a corrected image using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distorted flow field prediction result, determine feature points and descriptors of the corrected image, and then match feature points of an image adjacent to the corrected image based on the feature points and descriptors to obtain target matching point pairs; The out-of-stock detection result determination module is used to use a random sampling consistency algorithm and determine a homography matrix based on the target matching point pairs, so as to use the homography matrix to align each of the corrected images into the same coordinate system to obtain a transformed image, and use a multi-band fusion algorithm to process the transformed image to obtain a vertical frame view, and then perform out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result.

9. The real-time shelf out-of-stock detection device based on a patrol robot according to claim 8, characterized in that: The out-of-stock detection result determination module includes: An image coordinate system alignment unit is used to estimate the target matching point pairs using a RANSAC algorithm to obtain a homography matrix, and determine the coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment on the corrected images using the homography matrix and based on the coordinate points to obtain a transformed image; a frequency band brightness transition unit, configured to decompose the transformed image into frequency bands of different spatial frequencies, and perform brightness transition processing on a low-frequency band portion that meets a preset low-frequency condition using a preset smoothing weighted average algorithm to obtain a first processing result; The processing result reconstruction and fusion unit is used to directly superimpose or partially retain the high-frequency frequency band portion that meets the preset high-frequency conditions to obtain a second processing result, and use a multi-band fusion algorithm to reconstruct and fuse the first processing result and the second processing result to obtain a vertical frame view; wherein the resolution corresponding to the vertical frame view is a resolution determined based on the number of cameras, and it is determined that the processing time corresponding to the vertical frame view is not greater than a second preset processing time threshold.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the real-time shelf out-of-stock detection method based on a patrol robot according to any one of claims 1 to 7 is implemented.

Images