Real-time shelf out-of-stock detection method and device based on inspection robot and storage medium

Abstract

The application discloses a real-time shelf out-of-stock detection method and device based on an inspection robot and a storage medium, relates to the technical field of computer vision, and comprises the following steps: correcting a historical shelf image based on a shelf linear element, training a model by using a training data set determined based on each corrected image to obtain a target neural network model, and collecting a current shelf image of a preset point by using a camera on the inspection robot; processing the current shelf image by using the target neural network model to obtain a distortion flow field prediction result, generating a corrected image, feature points and a descriptor based on distortion estimation parameters of the distortion flow field prediction result, matching the feature points and the descriptor with feature points of an image adjacent to the corrected image to obtain a target matching point pair; and determining a transformed image by using a homography matrix determined based on the target matching point pair and the corrected image, so as to determine a vertical frame view and detect the vertical frame view to obtain an out-of-stock detection result. In this way, the efficiency of detecting goods in a shelf can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a real-time shelf out-of-stock detection method and device based on a patrol robot and a storage medium. BACKGROUND

[0002] At present, the shelf out-of-stock detection of retail stores mainly relies on manual patrol or fixed camera systems. Manual patrol is low in efficiency, high in cost, and easy to be affected by subjective factors; among them, although the fixed camera system can provide continuous monitoring, it has limited coverage, poor flexibility, and high delay due to dependence on cloud processing, which cannot meet the real-time demand. In the prior art, computer vision-based methods (such as target detection or segmentation) have been used for out-of-stock detection, but these methods usually run on the server side, increasing network dependency and processing delay. In addition, the complex lighting, occlusion and lens distortion problems of the shelf scene further reduce the detection accuracy.

[0003] The deficiencies of the above-mentioned technologies are as follows:

[0004] 1. Lack of real-time performance: traditional cloud processing methods have high delay (image uploading and result downloading are required), which cannot support mobile end-side real-time decision-making of the patrol robot.

[0005] 2. Image quality defects: images captured by multi-camera are easily affected by lens distortion (such as bending of straight line elements on the shelf), which leads to errors in subsequent segmentation and detection.

[0006] 3. Multi-image integration problem: there are gaps and overlaps between multiple camera images, which will reduce the consistency and detection efficiency of the overall view if not seamlessly spliced.

[0007] 4. Low accuracy of out-of-stock detection: semantic segmentation algorithms have poor robustness in complex retail scenes (such as dense goods and personnel interference), leading to false positives or false negatives.

[0008] 5. End-side resource limitations: the end-side devices (such as embedded GPUs) of the patrol robot have limited computing power, and existing deep learning models are difficult to deploy efficiently.

[0009] From the above, how to improve the efficiency of out-of-stock detection of goods in the shelf in the process of real-time shelf out-of-stock detection based on a patrol robot is a problem to be solved at present. SUMMARY

[0010] Therefore, the purpose of the present application is to provide a real-time shelf out-of-stock detection method and device based on a patrol robot and a storage medium, which can improve the efficiency of out-of-stock detection of goods in the shelf in the process of real-time shelf out-of-stock detection based on a patrol robot, and further improve the efficiency of the production process. The specific scheme is as follows:

[0011] In a first aspect, the present application provides a real-time shelf out-of-stock detection method based on an inspection robot, comprising:

[0012] correcting historical shelf images based on shelf straight line elements corresponding to shelves at preset positions, and determining a training data set based on the obtained correction images, to train an initial neural network model using the training data set to obtain a target neural network model, and then using a plurality of cameras on the inspection robot to collect current shelf images of the preset positions;

[0013] processing the current shelf images using the target neural network model to obtain distortion flow field prediction results, then generating correction images using a preset interpolation algorithm and based on distortion estimation parameters corresponding to the distortion flow field prediction results, and determining feature points and descriptors of the correction images, then matching feature points of images adjacent to the correction images based on the feature points and the descriptors to obtain target matching point pairs;

[0014] determining a homography matrix using a random sample consensus algorithm and based on the target matching point pairs, to align each of the correction images to the same coordinate system using the homography matrix to obtain transformed images, and processing the transformed images using a multi-band fusion algorithm to obtain a vertical frame view, and then performing out-of-stock detection on the vertical frame view to obtain an out-of-stock detection result.

[0015] Optionally, the correction of historical shelf images based on shelf straight line elements corresponding to shelves at preset positions, and the determination of a training data set based on the obtained correction images, comprises:

[0016] determining shelf straight line elements corresponding to historical shelf images, setting the shelf straight line elements as geometric constraints, and then using a preset correction tool to perform distortion correction on the historical shelf images based on the geometric constraints to obtain correction images; wherein the size of the overlapping region between the collection regions of each camera is not less than a preset overlapping region threshold;

[0017] using a position sensor in the inspection robot to control the inspection robot to move to a preset position in a target store, and then using a plurality of cameras vertically arranged on a column of the inspection robot to collect current shelf images corresponding to the preset position based on a preset resolution and a preset frame rate.

[0018] Optionally, the determination of a training data set based on the obtained correction images, to train an initial neural network model using the training data set to obtain a target neural network model, comprises:

[0019] The distance from the midpoint of the plane in the corrected image to the center of the optical axis is determined, and several first radial distortion coefficients for determining the distortion morphology of the central region of the corrected image and several second radial distortion coefficients for determining the fitting accuracy of the edge region of the corrected image are determined.

[0020] Image synthesis is performed using a preset distortion formula and based on each of the first radial distortion coefficients, each of the second radial distortion coefficients, and the corrected image to obtain a distorted image. A training dataset is then constructed based on each of the corrected images and each of the distorted images; each of the distorted images corresponds to a different degree of distortion.

[0021] An initial neural network model with an encoder and decoder architecture and residual connections is constructed, and the initial neural network model is trained using the training dataset to obtain the target neural network model.

[0022] Optionally, the step of processing the current shelf image using the target neural network model to obtain a distorted flow field prediction result, then generating a corrected image using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distorted flow field prediction result, and determining the feature points and descriptors of the corrected image, and then matching feature points of images adjacent to the corrected image based on the feature points and descriptors to obtain target matching point pairs, includes:

[0023] The target neural network model is used to process the current shelf image based on a preset loss function to obtain a distorted flow field prediction result. Then, a preset parameter estimation algorithm is used to determine distortion estimation parameters based on the distorted flow field prediction result. A preset bilinear interpolation algorithm is used to generate a corrected image based on the distorted flow field prediction result and the distortion estimation parameters. The duration of the corrected image is determined to be no greater than a first preset processing time threshold.

[0024] First SIFT feature points and first ORB feature points are extracted from the corrected image, and a descriptor is generated based on the first SIFT feature points and first ORB feature points. Then, neighboring images adjacent to the corrected image are determined, and second SIFT feature points and second ORB feature points corresponding to the neighboring images are determined.

[0025] Using a preset high-dimensional spatial nearest neighbor search technique and the descriptor, the first SIFT feature point and the second SIFT feature point, and the first ORB feature point and the second ORB feature point are matched to obtain initial matching point pairs. The initial matching point pairs are then filtered based on the geometric consistency of the shelf scene to obtain target matching point pairs.

[0026] Optionally, the step of using a random sampling consensus algorithm and determining a homography matrix based on the target matching point pairs, aligning each of the corrected images to the same coordinate system using the homography matrix to obtain a transformed image, and processing the transformed image using a multi-band fusion algorithm to obtain a vertical frame view includes:

[0027] The RANSAC algorithm is used to estimate the target matching point pairs to obtain the homography matrix, and the coordinate points corresponding to each of the corrected images are determined. The coordinate system of the corrected images is then aligned using the homography matrix and based on the coordinate points to obtain the transformed image.

[0028] The transformed image is decomposed into frequency bands of different spatial frequencies, and a preset smoothing weighted average algorithm is used to perform brightness transition processing on the low-frequency band portion that meets the preset low-frequency conditions to obtain the first processing result.

[0029] The high-frequency band portion that meets the preset high-frequency conditions is directly superimposed or partially retained to obtain a second processing result. The first processing result and the second processing result are then reconstructed and fused using a multi-band fusion algorithm to obtain a vertical frame view. The resolution corresponding to the vertical frame view is determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined to be no greater than a second preset processing time threshold.

[0030] Optionally, the step of performing out-of-stock detection on the vertical frame view to obtain out-of-stock detection results includes:

[0031] The vertical frame view is semantically segmented using a preset segmentation network to obtain a target semantic segmentation result; the target semantic segmentation result includes the product type, shelf type, background type, and personnel interference type corresponding to the vertical frame view;

[0032] Based on the target semantic segmentation result, extract the product category corresponding to each product in the product area, and set the number of pixels in the product area as the product existence index. Then determine whether the product existence index is greater than the preset product area threshold.

[0033] If the product availability index is greater than the preset product area threshold, the product's status is set to "in stock". If the product availability index is not greater than the preset product area threshold, the product's status is set to "out of stock". The out-of-stock location, product type, and confidence level corresponding to the product are then sent to the current user interface.

[0034] Optionally, after performing out-of-stock detection on the vertical frame view and obtaining the out-of-stock detection result, the method further includes:

[0035] The inspection interval is determined based on production needs, and the inspection robot is invoked to re-trigger the step of using several cameras on the inspection robot to collect the current shelf image of the preset point every inspection interval; wherein, the processing delay time corresponding to the out-of-stock detection result is determined to be no greater than a preset delay threshold.

[0036] Secondly, this application provides a real-time shelf shortage detection device based on an inspection robot, comprising:

[0037] The current shelf image acquisition module is used to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset points, and to determine a training dataset based on the obtained corrected images, so as to train an initial neural network model using the training dataset to obtain a target neural network model, and then use several cameras on the inspection robot to collect current shelf images at the preset points.

[0038] The matching point pair determination module is used to process the current shelf image using the target neural network model to obtain the distortion flow field prediction result, then use a preset interpolation algorithm and generate a corrected image based on the distortion estimation parameters corresponding to the distortion flow field prediction result, and determine the feature points and descriptors of the corrected image, and then match the feature points of the image adjacent to the corrected image based on the feature points and the descriptors to obtain the target matching point pair;

[0039] The out-of-stock detection result determination module is used to determine the homography matrix based on the target matching point pairs using a random sampling consistency algorithm, so as to align each of the corrected images to the same coordinate system using the homography matrix to obtain the transformed image, and to process the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, and then to perform out-of-stock detection on the vertical frame view to obtain the out-of-stock detection result.

[0040] Optionally, the out-of-stock detection result determination module includes:

[0041] The image coordinate system alignment unit is used to estimate the target matching point pairs using the RANSAC algorithm to obtain the homography matrix and determine the coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment of the corrected images using the homography matrix and based on the coordinate points to obtain the transformed image.

[0042] The frequency band brightness transition unit is used to decompose the transformed image into frequency bands of different spatial frequencies, and to perform brightness transition processing on the low-frequency frequency band part that meets the preset low-frequency conditions using a preset smooth weighted average algorithm to obtain the first processing result.

[0043] The processing result reconstruction and fusion unit is used to directly superimpose or partially retain the high-frequency band portion that meets the preset high-frequency conditions to obtain a second processing result, and to reconstruct and fuse the first processing result and the second processing result using a multi-band fusion algorithm to obtain a vertical frame view; wherein, the resolution corresponding to the vertical frame view is a resolution determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined not to be greater than a second preset processing time threshold.

[0044] Thirdly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned real-time shelf shortage detection method based on an inspection robot.

[0045] As can be seen from the above, before performing real-time shelf out-of-stock detection based on inspection robots, this application needs to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset points, and determine a training dataset based on the obtained corrected images to train an initial neural network model to obtain a target neural network model. Then, several cameras on the inspection robot are used to collect current shelf images at preset points. The target neural network model is used to process the current shelf images to obtain distortion flow field prediction results. Then, a corrected image is generated using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distortion flow field prediction results. The feature points and descriptors of the corrected image are determined. Then, feature points of images adjacent to the corrected image are matched based on the feature points and descriptors to obtain target matching point pairs. A random sampling consensus algorithm is used to determine the homography matrix based on the target matching point pairs. The homography matrix is ​​used to align each corrected image to the same coordinate system to obtain a transformed image. A multi-band fusion algorithm is used to process the transformed image to obtain a vertical frame view. Then, out-of-stock detection is performed on the vertical frame view to obtain the out-of-stock detection result.

[0046] Therefore, before conducting real-time shelf shortage detection based on inspection robots, this application first needs to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset points, and determine a training dataset based on the obtained corrected images to train an initial neural network model to obtain a target neural network model. Then, several cameras on the inspection robot collect current shelf images at preset points. Second, the target neural network model processes the current shelf images to obtain distortion flow field prediction results. Then, a corrected image is generated using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distortion flow field prediction results, and the feature points and descriptors of the corrected image are determined. Then, feature points of images adjacent to the corrected image are matched based on the feature points and descriptors to obtain target matching point pairs. Finally, a random sampling consensus algorithm is used to determine the homography matrix based on the target matching point pairs, so that the homography matrix is ​​used to align each corrected image to the same coordinate system to obtain a transformed image. A multi-band fusion algorithm is used to process the transformed image to obtain a vertical frame view, and then the vertical frame view is used for shortage detection to obtain the shortage detection result. This improves the efficiency of detecting out-of-stock items on shelves during real-time shelf out-of-stock detection based on inspection robots, thereby enhancing the efficiency of the production process. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This application discloses a flowchart of a real-time shelf shortage detection method based on an inspection robot.

[0049] Figure 2 This is a schematic diagram illustrating the positional relationship between a specific store merchandise shelf and an inspection robot.

[0050] Figure 3 This is a schematic diagram of a specific process for correcting image distortion disclosed in this application;

[0051] Figure 4 This is a schematic diagram illustrating a specific process for stitching images as disclosed in this application;

[0052] Figure 5 This is a schematic diagram of a real-time shelf shortage detection device based on an inspection robot disclosed in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Currently, shelf-out-of-stock detection in retail stores mainly relies on manual inspections or fixed camera systems. Manual inspections are inefficient, costly, and susceptible to subjective factors; while fixed camera systems provide continuous monitoring, their coverage is limited, their flexibility is poor, and their reliance on cloud processing leads to high latency, failing to meet real-time requirements. Therefore, this application provides a real-time shelf-out-of-stock detection method based on an inspection robot, which can improve the efficiency of detecting out-of-stock items on shelves, thereby improving the efficiency of the production process.

[0055] See Figure 1 As shown, this embodiment of the invention discloses a real-time shelf out-of-stock detection method based on an inspection robot, including:

[0056] Step S11: Correct the historical shelf images based on the shelf line elements corresponding to the shelves at the preset points, and determine the training dataset based on the obtained corrected images, so as to train the initial neural network model using the training dataset to obtain the target neural network model, and then use several cameras on the inspection robot to collect the current shelf images at the preset points.

[0057] In this embodiment, as the inspection robot moves within the store, it can simultaneously acquire images through multiple cameras (standard configuration is 3, adjustable according to shelf height) vertically arranged on the robot's column. Each camera's field of view is designed to cover a specific vertical area of ​​the shelf (e.g., each camera covers a height of 1-2 meters), and the overlap between adjacent cameras' fields of view is no less than 20% to ensure no blind spots. Furthermore, each camera's parameters include a resolution of no less than 1920×1080 and a frame rate of 30fps (Frames Per Second) to adapt to dynamic environments. In addition, this embodiment incorporates a trigger mechanism in the robot's position sensor, such as LiDAR (Light Detection and Ranging), so that image acquisition is automatically initiated when the robot reaches a preset location.

[0058] In one specific implementation, a schematic diagram illustrating the positional relationship between the store's merchandise shelves and the inspection robot is shown below. Figure 2As shown, multiple cameras are vertically arranged on the robot column. The field of view of each camera can cover a certain area of ​​the shelf in the vertical direction. The fields of view of adjacent cameras overlap to some extent. The combined field of view of multiple cameras can ensure that the field of view of each camera covers the entire shelf in the vertical direction.

[0059] Specifically, the process of correcting historical shelf images based on the shelf line elements corresponding to the shelves at preset locations, and determining the training dataset based on the obtained corrected images, may include: determining the shelf line elements corresponding to the historical shelf images, setting the shelf line elements as geometric constraints, and then using preset correction tools and based on the geometric constraints to perform distortion correction on the historical shelf images to obtain corrected images; wherein the overlap area between the acquisition areas of each camera is not less than a preset overlap area threshold; in the target store, the position sensor in the inspection robot is used to control the inspection robot to move to the preset location, and then several cameras vertically arranged on the column of the inspection robot are used to acquire the current shelf image corresponding to the preset location based on a preset resolution and a preset frame rate.

[0060] It's worth noting that the camera's optical imaging is not ideal pinhole imaging, causing a certain deviation between the actual and ideal image points on the imaging plane. Furthermore, a complex nonlinear relationship exists between the two, known as lens distortion. Lens distortion is mainly divided into radial and tangential distortion, with radial distortion being predominant. Given the abundance of straight-line elements in the shelf scene, this prior information plays a significant role in the calibration process. The image distortion correction process is as follows: Figure 3 As shown;

[0061] In this embodiment, when capturing a large number of shelf images, it is necessary to cover different lighting and angles. Then, this embodiment utilizes the linear elements of the shelves (such as shelf edges) as geometric constraints, and generates distortion-corrected reference images using manual correction tools, such as the checkerboard calibration of OpenCV (OpenSource Computer Vision Library). Simultaneously, images with different degrees of distortion are synthesized using the Brown-Conrady Distortion Model to construct a training dataset (including distorted images and corresponding corrected images), and the expression is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] in, Let x be the x-coordinate of a point in the normalized image plane. The ordinate of a point in the normalized image plane. This is the distance from the midpoint of the normalized image plane to the center of the optical axis. , and All are low-order radial distortion coefficients, which dominate the distortion morphology of the central region (such as barrel / pincushion shape). , and These are all high-order radial distortion coefficients, which enhance the fitting accuracy in edge regions, and are especially crucial for ultra-wide-angle lenses. The x-coordinate of the synthesized image. This represents the ordinate of the synthesized image.

[0066] Subsequently, this application embodiment requires the design of a simple and lightweight convolutional neural network (CNN) to predict distorted flow fields based on a single distorted image using the convolutional neural network. The network structure adopts the U-Net encoder-decoder architecture: the encoder consists of 4 convolutional layers, each containing Conv2D (2D Convolution), BatchNorm (Batch Normalization), and ReLU (Rectified Linear Unit), outputting a feature map; the decoder consists of 4 deconvolutional layers to restore resolution; and residual connections (SkipConnections) are added to avoid gradient vanishing. Specifically, a training dataset is determined based on the obtained corrected images, and an initial neural network model is trained using the training dataset to obtain the target neural network model. This may include: determining the distance from the midpoint of the plane in the corrected image to the center of the optical axis, and determining several first radial distortion coefficients used to determine the distortion morphology of the central region of the corrected image and several second radial distortion coefficients used to determine the fitting accuracy of the edge region of the corrected image; using a preset distortion formula and based on each first radial distortion coefficient, each second radial distortion coefficient, and the corrected image, image synthesis is performed to obtain a distorted image, and a training dataset is constructed based on each corrected image and each distorted image; the degree of distortion corresponding to each distorted image is different; an initial neural network model with an encoder and decoder architecture and including residual connections is constructed, and the initial neural network model is trained using the training dataset to obtain the target neural network model.

[0067] Step S12: Process the current shelf image using the target neural network model to obtain the distortion flow field prediction result. Then, use a preset interpolation algorithm and generate a corrected image based on the distortion estimation parameters corresponding to the distortion flow field prediction result. Determine the feature points and descriptors of the corrected image. Then, match the feature points of images adjacent to the corrected image based on the feature points and descriptors to obtain target matching point pairs.

[0068] In this embodiment, since the input to the target neural network model is a single distorted image (sized to 256×256), and the output is a distorted flow field (sized the same as the input), the loss function uses the L2 norm (mean squared error) to minimize the difference between the predicted flow field and the actual flow field. It is worth mentioning that after predicting and obtaining the distorted flow field using the model, this embodiment requires further estimation of distortion parameters using a model fitting method to improve correction accuracy. In one specific implementation, this embodiment can use the RANSAC algorithm (RANdom Sampling Consensus) to estimate distortion parameters to improve robustness; finally, bilinear interpolation is applied to generate a corrected image based on the parameters and flow field, and the model processing time is controlled within 50ms to meet the real-time requirements of the end-side.

[0069] It is worth mentioning that, since the inspection robot is equipped with multiple cameras, this embodiment of the application needs to stitch the images captured by these cameras together to form a complete vertical view of the shelf, thereby eliminating gaps and overlaps between images and improving the efficiency and accuracy of subsequent processing. The image stitching process is as follows: Figure 4 As shown. First, feature extraction and matching operations need to be performed on the image: First, SIFT (Scale-Invariant Feature Transform) and ORB (Oriented Fast and Rotated BRIEF) feature points are extracted from the distortion-corrected multi-view image, and descriptors are calculated. Then, feature points of neighboring images are matched based on FLANN (Fast Library for Approximate Nearest Neighbors), and the geometric consistency of the shelf scene (such as vertical shelf lines) is used to filter out mismatches.

[0070] Specifically, the current shelf image is processed using a target neural network model to obtain a distorted flow field prediction result. Then, a corrected image is generated using a preset interpolation algorithm based on the distortion estimation parameters corresponding to the distorted flow field prediction result. Feature points and descriptors of the corrected image are determined. Then, feature points of adjacent images are matched based on the feature points and descriptors to obtain target matching point pairs. This can include: processing the current shelf image using a target neural network model based on a preset loss function to obtain a distorted flow field prediction result; determining distortion estimation parameters using a preset parameter estimation algorithm based on the distorted flow field prediction result; and generating a corrected image using a preset bilinear interpolation algorithm based on the distorted flow field prediction result and the distortion estimation parameters. The image is processed as follows: the duration of the image correction is determined to be no greater than a first preset processing time threshold; first SIFT feature points and first ORB feature points are extracted from the image correction, and a descriptor is generated based on the first SIFT feature points and first ORB feature points; then, neighboring images adjacent to the image correction are determined, and second SIFT feature points and second ORB feature points corresponding to the neighboring images are determined; using a preset high-dimensional space nearest neighbor search technique and descriptor, the first SIFT feature points and second SIFT feature points, and the first ORB feature points and second ORB feature points are matched to obtain initial matching point pairs, and the initial matching point pairs are filtered based on the geometric consistency of the shelf scene to obtain target matching point pairs.

[0071] Step S13: Using the random sampling consistency algorithm and based on the target matching point pair, determine the homography matrix, and use the homography matrix to align each of the corrected images to the same coordinate system to obtain the transformed image. Then, use the multi-band fusion algorithm to process the transformed image to obtain a vertical frame view. Finally, perform out-of-stock detection on the vertical frame view to obtain the out-of-stock detection result.

[0072] In this embodiment, after obtaining the target matching point pairs, the images need to be transformed and fused. In one specific implementation, the homography matrix is ​​applied to transform the images to align multiple images to a unified coordinate system, and the expression is as follows:

[0073] ;

[0074] in, These are the coordinates of the original image. These are the transformed coordinates. This is the homography matrix, which is determined using the RANSAC algorithm based on the target matching point pairs.

[0075] Furthermore, when fusing images, this embodiment of the application can employ a multi-band blending algorithm to perform frequency division processing on the images (smoothing the low-frequency parts and preserving the details of the high-frequency parts), and perform weighted averaging processing on the overlapping areas to eliminate gaps and brightness differences. Finally, a complete vertical shelf view is generated, where the resolution can be adaptively adjusted according to the number of cameras. For example, if the number of cameras is 3, the resolution of the stitched image is 4000×3000.

[0076] Specifically, a homography matrix is ​​determined using a random sampling consensus algorithm based on target matching point pairs. This homography matrix is ​​then used to align each corrected image to the same coordinate system, resulting in a transformed image. A multi-band fusion algorithm is then used to process the transformed image to obtain a vertical frame view. This process includes: estimating the target matching point pairs using the RANSAC algorithm to obtain a homography matrix and determining the coordinate points corresponding to each corrected image; aligning the corrected images to the same coordinate system using the homography matrix and these coordinate points to obtain a transformed image; decomposing the transformed image into frequency bands of different spatial frequencies; and using a preset smoothing weighted average algorithm to perform brightness transition processing on the low-frequency band portion that meets preset low-frequency conditions, resulting in a first processing result; and directly superimposing or partially retaining the high-frequency band portion that meets preset high-frequency conditions, resulting in a second processing result. Finally, a multi-band fusion algorithm is used to reconstruct and fuse the first and second processing results to obtain a vertical frame view. The resolution corresponding to the vertical frame view is determined based on the number of cameras, and the processing time for the vertical frame view is determined to be no greater than a second preset processing time threshold.

[0077] It is worth mentioning that when deployed on edge devices, such as NVIDIA Jetson (an embedded AI computing platform), this application embodiment can use GPU to accelerate the feature matching and fusion process to ensure that the stitching latency is less than 100ms and output the stitching result as a seamless head-up shelf image to facilitate semantic segmentation in subsequent steps.

[0078] In this embodiment, after obtaining the stitched image, the robot uses an algorithm deployed on the robot's edge to perform semantic segmentation on the stitched image. This involves identifying and classifying different elements in the image (such as goods, shelves, and people) to facilitate subsequent out-of-stock detection and other analyses. Based on the image semantic segmentation results, the robot can then perform out-of-stock detection on the shelves. If the quantity of a certain item is detected to be below a preset threshold or is completely missing, the item is marked as out of stock, and an out-of-stock detection result is generated.

[0079] It's worth noting that the aforementioned semantic segmentation model is a lightweight segmentation network (DeepLabv3plus-MobileNetV2), and the model's input is a stitched image of the shelf, with the output being the semantic category of each pixel (including: product, shelf, background, people, etc.). Furthermore, the training dataset for training the semantic segmentation model consists of pre-labeled shelf images, and data augmentation includes rotation, scaling, and lighting variations to improve generalization. In addition, in one specific implementation, when the model performs inference on the edge, computation can be reduced through quantization (INT8 precision) and pruning to keep the segmentation latency within 200ms.

[0080] Furthermore, embodiments of this application can utilize a stockout detection algorithm and extract individual products from the product area based on segmentation results (e.g., through connected component analysis). For each product category (e.g., beverages, snacks), the number of pixels within the area is counted, and the obtained statistical results are set as a product existence indicator. In addition, the stockout detection logic is as follows: a product quantity threshold is set (e.g., when the product area is less than 30% of the historical average, it is determined to be stockout), and the relationship between the current product quantity and the product quantity threshold is determined to determine whether the product is stockout, or to trigger an alarm when the product is completely missing. The algorithm's output includes the stockout location, product type, and confidence level.

[0081] Specifically, performing out-of-stock detection on the vertical frame view and obtaining the out-of-stock detection result can include: using a preset segmentation network to perform semantic segmentation on the vertical frame view to obtain the target semantic segmentation result; the target semantic segmentation result includes the product type, shelf type, background type, and personnel interference type corresponding to the vertical frame view; extracting the product category corresponding to each product in the product area based on the target semantic segmentation result, and setting the number of pixels in the product area as the product existence index, and then determining whether the product existence index is greater than a preset product area threshold; if the product existence index is greater than the preset product area threshold, then the status corresponding to the product is set to in stock; if the product existence index is not greater than the preset product area threshold, then the status corresponding to the product is set to out-of-stock, and the out-of-stock location, product type, and confidence level corresponding to the product are sent to the current user interface.

[0082] It is worth mentioning that, in this embodiment, the detection results are displayed in real time on the robot interface and uploaded to the store management system. The entire process is controlled by a loop structure, and the robot repeats the inspection operation at preset intervals (e.g., every 30 minutes) after each inspection to ensure completion. This ensures that the goods in the store are always kept in sufficient inventory. Specifically, after performing out-of-stock detection on the vertical frame view and obtaining the out-of-stock detection result, the process may further include: determining the inspection time interval based on production needs, and calling the inspection robot to re-trigger the process at each inspection time interval to collect images of the current shelf at preset points using several cameras on the inspection robot; wherein, the processing delay time corresponding to the out-of-stock detection result is determined not to be greater than a preset delay threshold.

[0083] As can be seen from the above, before performing real-time shelf out-of-stock detection based on the inspection robot, the embodiments of this application first need to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset points, and determine a training dataset based on the obtained corrected images, so as to train an initial neural network model using the training dataset to obtain a target neural network model. Then, several cameras on the inspection robot are used to collect current shelf images at preset points. Next, the target neural network model is used to process the current shelf images to obtain the distortion flow field prediction results. Then, a corrected image is generated using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distortion flow field prediction results, and the feature points and descriptors of the corrected image are determined. Then, feature points of images adjacent to the corrected image are matched based on the feature points and descriptors to obtain target matching point pairs. Finally, a random sampling consensus algorithm is used to determine the homography matrix based on the target matching point pairs, so as to align each corrected image to the same coordinate system using the homography matrix to obtain a transformed image. The transformed image is then processed using a multi-band fusion algorithm to obtain a vertical frame view, and then out-of-stock detection is performed on the vertical frame view to obtain the out-of-stock detection results. This improves the efficiency of detecting out-of-stock items on shelves during real-time shelf out-of-stock detection based on inspection robots, thereby enhancing the efficiency of the production process.

[0084] Accordingly, see Figure 5 As shown, this application also provides a real-time shelf out-of-stock detection device based on an inspection robot, including:

[0085] The current shelf image acquisition module 11 is used to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset points, and to determine a training dataset based on the obtained corrected images, so as to train an initial neural network model using the training dataset to obtain a target neural network model, and then use several cameras on the inspection robot to collect current shelf images at the preset points.

[0086] The matching point pair determination module 12 is used to process the current shelf image using the target neural network model to obtain the distortion flow field prediction result, then use a preset interpolation algorithm and generate a corrected image based on the distortion estimation parameters corresponding to the distortion flow field prediction result, and determine the feature points and descriptors of the corrected image, and then match the feature points of the image adjacent to the corrected image based on the feature points and the descriptors to obtain the target matching point pair;

[0087] The out-of-stock detection result determination module 13 is used to determine the homography matrix based on the target matching point pair using a random sampling consistency algorithm, so as to align each of the corrected images to the same coordinate system using the homography matrix to obtain the transformed image, and to process the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, and then to perform out-of-stock detection on the vertical frame view to obtain the out-of-stock detection result.

[0088] In some specific embodiments, the current shelf image acquisition module 11 may specifically include:

[0089] The image correction determination unit is used to determine the shelf straight line element corresponding to the historical shelf image, set the shelf straight line element as a geometric constraint, and then use a preset correction tool to perform distortion correction on the historical shelf image based on the geometric constraint to obtain a corrected image; wherein, the size of the overlapping area between the acquisition areas of each camera is not less than a preset overlapping area threshold.

[0090] The shelf image acquisition unit is used to control the inspection robot to move to a preset point in the target store using the position sensor in the inspection robot, and then use several cameras arranged vertically on the column of the inspection robot to acquire the current shelf image corresponding to the preset point based on a preset resolution and a preset frame rate.

[0091] In some specific embodiments, the current shelf image acquisition module 11 may specifically include:

[0092] A radial distortion coefficient determination unit is used to determine the distance from the midpoint of the plane in the corrected image to the center of the optical axis, and to determine a number of first radial distortion coefficients for determining the distortion morphology of the central region of the corrected image and a number of second radial distortion coefficients for determining the fitting accuracy of the edge region of the corrected image.

[0093] The training dataset determination unit is used to synthesize images using a preset distortion formula and based on each of the first radial distortion coefficients, each of the second radial distortion coefficients, and the corrected image to obtain distorted images, and to construct a training dataset based on each of the corrected images and each of the distorted images; each of the distorted images corresponds to a different degree of distortion;

[0094] The model training unit is used to construct an initial neural network model with an encoder and decoder architecture and residual connections, and to train the initial neural network model using the training dataset to obtain a target neural network model.

[0095] In some specific embodiments, the matching point pair determination module 12 may specifically include:

[0096] The distortion estimation parameter determination unit is used to process the current shelf image using the target neural network model and based on a preset loss function to obtain a distortion flow field prediction result, and then use a preset parameter estimation algorithm to determine distortion estimation parameters based on the distortion flow field prediction result, so as to generate a corrected image using a preset bilinear interpolation algorithm based on the distortion flow field prediction result and the distortion estimation parameters; wherein, the duration of the corrected image is determined to be no greater than a first preset processing time threshold;

[0097] The feature point generation unit is used to extract a first SIFT feature point and a first ORB feature point from the corrected image, generate a descriptor based on the first SIFT feature point and the first ORB feature point, determine the adjacent images adjacent to the corrected image, and determine the second SIFT feature point and the second ORB feature point corresponding to the adjacent images.

[0098] The feature point matching unit is used to match the first SIFT feature point with the second SIFT feature point and the first ORB feature point with the second ORB feature point using a preset high-dimensional space nearest neighbor search technology and the descriptor to obtain initial matching point pairs, and to filter the initial matching point pairs based on the geometric consistency of the shelf scene to obtain target matching point pairs.

[0099] In some specific embodiments, the shortage detection result determination module 13 may specifically include:

[0100] The image coordinate system alignment unit is used to estimate the target matching point pairs using the RANSAC algorithm to obtain the homography matrix and determine the coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment of the corrected images using the homography matrix and based on the coordinate points to obtain the transformed image.

[0101] The frequency band brightness transition unit is used to decompose the transformed image into frequency bands of different spatial frequencies, and to perform brightness transition processing on the low-frequency frequency band part that meets the preset low-frequency conditions using a preset smooth weighted average algorithm to obtain the first processing result.

[0102] The processing result reconstruction and fusion unit is used to directly superimpose or partially retain the high-frequency band portion that meets the preset high-frequency conditions to obtain a second processing result, and to reconstruct and fuse the first processing result and the second processing result using a multi-band fusion algorithm to obtain a vertical frame view; wherein, the resolution corresponding to the vertical frame view is a resolution determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined not to be greater than a second preset processing time threshold.

[0103] In some specific embodiments, the shortage detection result determination module 13 may specifically include:

[0104] The semantic segmentation result determination unit is used to perform semantic segmentation on the vertical frame view using a preset segmentation network to obtain a target semantic segmentation result; the target semantic segmentation result includes the product type, shelf type, background type, and personnel interference type corresponding to the vertical frame view;

[0105] The product category determination unit is used to extract the product category corresponding to each product in the product region based on the target semantic segmentation result, and set the number of pixels in the product region as the product existence index, and then determine whether the product existence index is greater than a preset product area threshold.

[0106] The indicator judgment unit is used to set the status of the product to "in stock" if the product existence indicator is greater than the preset product area threshold, and to set the status of the product to "out of stock" if the product existence indicator is not greater than the preset product area threshold, and to send the out-of-stock location, product type and confidence level corresponding to the product to the current user interface.

[0107] In some specific embodiments, the real-time shelf out-of-stock detection device based on the inspection robot may further include:

[0108] The inspection interval determination unit is used to determine the inspection interval based on production needs, and to call the inspection robot to re-trigger the step of using several cameras on the inspection robot to collect the current shelf image of the preset point every inspection interval; wherein, it is determined that the processing delay time corresponding to the out-of-stock detection result is not greater than a preset delay threshold.

[0109] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned real-time shelf shortage detection method based on an inspection robot. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0113] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A real-time shelf shortage detection method based on an inspection robot, characterized in that, include: Historical shelf images are corrected based on the straight-line elements of the shelves corresponding to the shelves at preset locations. A training dataset is determined based on the obtained corrected images to train an initial neural network model to obtain a target neural network model. Then, several cameras on an inspection robot are used to collect current shelf images at the preset locations. The process of determining the training dataset based on the obtained corrected images to train the initial neural network model to obtain the target neural network model includes: determining the distance from the midpoint of the plane in the corrected image to the center of the optical axis, and determining several first radial distortion coefficients for determining the distortion morphology of the central region of the corrected image and several second radial distortion coefficients for determining the fitting accuracy of the edge region of the corrected image; using a preset distortion formula and based on each of the first radial distortion coefficients, each of the second radial distortion coefficients, and the corrected image, image synthesis is performed to obtain a distorted image, and a training dataset is constructed based on each of the corrected images and each of the distorted images; the degree of distortion corresponding to each of the distorted images is different; an initial neural network model with an encoder and decoder architecture and including residual connections is constructed, and the initial neural network model is trained using the training dataset to obtain the target neural network model. The target neural network model is used to process the current shelf image to obtain the distortion flow field prediction result. Then, a corrected image is generated using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distortion flow field prediction result. The feature points and descriptors of the corrected image are determined. Then, based on the feature points and descriptors, the feature points of the images adjacent to the corrected image are matched to obtain the target matching point pair. A homography matrix is ​​determined based on the target matching point pairs using a random sampling consistency algorithm. The homography matrix is ​​then used to align each of the corrected images to the same coordinate system to obtain a transformed image. A multi-band fusion algorithm is then used to process the transformed image to obtain a vertical frame view. Finally, a shortage detection is performed on the vertical frame view to obtain a shortage detection result.

2. The real-time shelf shortage detection method based on an inspection robot according to claim 1, characterized in that, The process of correcting historical shelf images based on the shelf line elements corresponding to the shelves at preset locations, and determining the training dataset based on the obtained corrected images, includes: The shelf line elements corresponding to the historical shelf images are determined, and the shelf line elements are set as geometric constraints. Then, the historical shelf images are distorted using a preset correction tool based on the geometric constraints to obtain a corrected image. The overlapping area between the acquisition areas of each camera is not less than a preset overlapping area threshold. In the target store, the inspection robot is controlled by the position sensor in the inspection robot to move to the preset point. Then, several cameras arranged vertically on the column of the inspection robot are used to collect the current shelf image corresponding to the preset point based on the preset resolution and preset frame rate.

3. The real-time shelf shortage detection method based on an inspection robot according to claim 1, characterized in that, The process involves processing the current shelf image using the target neural network model to obtain a distorted flow field prediction result. Then, a corrected image is generated using a preset interpolation algorithm and based on the distortion estimation parameters corresponding to the distorted flow field prediction result. Feature points and descriptors of the corrected image are determined. Finally, feature points of images adjacent to the corrected image are matched based on the feature points and descriptors to obtain target matching point pairs, including: The target neural network model is used to process the current shelf image based on a preset loss function to obtain a distorted flow field prediction result. Then, a preset parameter estimation algorithm is used to determine distortion estimation parameters based on the distorted flow field prediction result. A preset bilinear interpolation algorithm is used to generate a corrected image based on the distorted flow field prediction result and the distortion estimation parameters. The duration of the corrected image is determined to be no greater than a first preset processing time threshold. First SIFT feature points and first ORB feature points are extracted from the corrected image, and a descriptor is generated based on the first SIFT feature points and first ORB feature points. Then, neighboring images adjacent to the corrected image are determined, and second SIFT feature points and second ORB feature points corresponding to the neighboring images are determined. Using a preset high-dimensional spatial nearest neighbor search technique and the descriptor, the first SIFT feature point and the second SIFT feature point, and the first ORB feature point and the second ORB feature point are matched to obtain initial matching point pairs. The initial matching point pairs are then filtered based on the geometric consistency of the shelf scene to obtain target matching point pairs.

4. The real-time shelf shortage detection method based on an inspection robot according to claim 1, characterized in that, The process of using a random sampling consensus algorithm to determine a homography matrix based on the target matching point pairs, aligning each of the corrected images to the same coordinate system using the homography matrix to obtain a transformed image, and then processing the transformed image using a multi-band fusion algorithm to obtain a vertical frame view includes: The RANSAC algorithm is used to estimate the target matching point pairs to obtain the homography matrix, and the coordinate points corresponding to each of the corrected images are determined. The coordinate system of the corrected images is then aligned using the homography matrix and based on the coordinate points to obtain the transformed image. The transformed image is decomposed into frequency bands of different spatial frequencies, and a preset smoothing weighted average algorithm is used to perform brightness transition processing on the low-frequency band portion that meets the preset low-frequency conditions to obtain the first processing result. The high-frequency band portion that meets the preset high-frequency conditions is directly superimposed or partially retained to obtain a second processing result. The first processing result and the second processing result are then reconstructed and fused using a multi-band fusion algorithm to obtain a vertical frame view. The resolution corresponding to the vertical frame view is determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined to be no greater than a second preset processing time threshold.

5. The real-time shelf shortage detection method based on an inspection robot according to claim 1, characterized in that, The process of performing out-of-stock detection on the vertical frame view to obtain out-of-stock detection results includes: The vertical frame view is semantically segmented using a preset segmentation network to obtain a target semantic segmentation result; the target semantic segmentation result includes the product type, shelf type, background type, and personnel interference type corresponding to the vertical frame view; Based on the target semantic segmentation result, extract the product category corresponding to each product in the product area, and set the number of pixels in the product area as the product existence index. Then determine whether the product existence index is greater than the preset product area threshold. If the product availability index is greater than the preset product area threshold, the product's status is set to "in stock". If the product availability index is not greater than the preset product area threshold, the product's status is set to "out of stock". The out-of-stock location, product type, and confidence level corresponding to the product are then sent to the current user interface.

6. The real-time shelf shortage detection method based on an inspection robot according to claim 1, characterized in that, After performing out-of-stock detection on the vertical frame view and obtaining the out-of-stock detection result, the method further includes: The inspection interval is determined based on production needs, and the inspection robot is invoked to re-trigger the step of using several cameras on the inspection robot to collect the current shelf image of the preset point every inspection interval; wherein, the processing delay time corresponding to the out-of-stock detection result is determined to be no greater than a preset delay threshold.

7. A real-time shelf shortage detection device based on an inspection robot, characterized in that, include: The current shelf image acquisition module is used to correct historical shelf images based on the shelf line elements corresponding to the shelves at preset locations, and to determine a training dataset based on the obtained corrected images. This training dataset is then used to train an initial neural network model to obtain a target neural network model. Several cameras on an inspection robot are then used to acquire current shelf images at the preset locations. The process of determining the training dataset based on the obtained corrected images to train the initial neural network model to obtain the target neural network model includes: determining the distance from the midpoint of the plane in the corrected image to the center of the optical axis; determining several first radial distortion coefficients for determining the distortion morphology of the central region of the corrected image and several second radial distortion coefficients for determining the fitting accuracy of the edge region of the corrected image; using a preset distortion formula and based on each of the first radial distortion coefficients, each of the second radial distortion coefficients, and the corrected image, performing image synthesis to obtain a distorted image; and constructing a training dataset based on each of the corrected images and each of the distorted images; each distorted image corresponds to a different degree of distortion; constructing an initial neural network model with an encoder and decoder architecture and including residual connections; and training the initial neural network model using the training dataset to obtain the target neural network model. The matching point pair determination module is used to process the current shelf image using the target neural network model to obtain the distortion flow field prediction result, then use a preset interpolation algorithm and generate a corrected image based on the distortion estimation parameters corresponding to the distortion flow field prediction result, and determine the feature points and descriptors of the corrected image, and then match the feature points of the image adjacent to the corrected image based on the feature points and the descriptors to obtain the target matching point pair; The out-of-stock detection result determination module is used to determine the homography matrix based on the target matching point pairs using a random sampling consistency algorithm, so as to align each of the corrected images to the same coordinate system using the homography matrix to obtain the transformed image, and to process the transformed image using a multi-band fusion algorithm to obtain a vertical frame view, and then to perform out-of-stock detection on the vertical frame view to obtain the out-of-stock detection result.

8. The real-time shelf shortage detection device based on an inspection robot according to claim 7, characterized in that, The out-of-stock detection result determination module includes: The image coordinate system alignment unit is used to estimate the target matching point pairs using the RANSAC algorithm to obtain the homography matrix and determine the coordinate points corresponding to each of the corrected images, so as to perform coordinate system alignment of the corrected images using the homography matrix and based on the coordinate points to obtain the transformed image. The frequency band brightness transition unit is used to decompose the transformed image into frequency bands of different spatial frequencies, and to perform brightness transition processing on the low-frequency frequency band part that meets the preset low-frequency conditions using a preset smooth weighted average algorithm to obtain the first processing result. The processing result reconstruction and fusion unit is used to directly superimpose or partially retain the high-frequency band portion that meets the preset high-frequency conditions to obtain a second processing result, and to reconstruct and fuse the first processing result and the second processing result using a multi-band fusion algorithm to obtain a vertical frame view; wherein, the resolution corresponding to the vertical frame view is a resolution determined based on the number of cameras, and the processing time corresponding to the vertical frame view is determined not to be greater than a second preset processing time threshold.

9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the real-time shelf shortage detection method based on an inspection robot as described in any one of claims 1 to 6.

Images