A shelf out-of-stock detection method, device, equipment and medium

Abstract

The application discloses a shelf out-of-stock detection method, device and equipment and a medium, and relates to the technical field of shelf detection. The method comprises the following steps: acquiring multiple local images of a shelf, detecting the images for goods, empty spaces and inventory units, and outputting detection results; splicing the images and fusing the detection results to generate unified detection data and convert the unified detection data into a structured shelf display sequence according to a physical result of the shelf. If there is an out-of-stock, the missing goods are determined by analyzing the information of the goods around the out-of-stock position. When the missing goods cannot be determined, the historical display data of the shelf are called and combined with the structured shelf display sequence and the position intersection area to determine the missing goods. Thus, the problem that a traditional method relies on a fixed template and is invalid when the layout changes is overcome, high-precision positioning of an out-of-stock area and accurate identification of missing goods are realized in a scene where the shelf layout frequently and dynamically changes, the robustness and continuous availability of the system are improved, manual intervention is reduced, and long-term maintenance costs are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of shelf detection, in particular to a shelf out-of-stock detection method, device, equipment and medium. BACKGROUND

[0002] In the real scenario of frequent dynamic changes of shelf commodity display layout, the traditional out-of-stock detection method has obvious limitations. It is difficult to effectively, accurately and dynamically complete the out-of-stock location positioning and specific identification of the missing commodity. Moreover, it is seriously dependent on fixed shelf drawings, price tag templates or standard shelf drawings, fixed price tag shelves as detection templates. Once the layout is changed due to promotion, price adjustment, commodity repositioning or restocking, the static template will be invalid immediately, which makes the system unable to accurately locate the out-of-stock location and accurately identify the missing commodity, resulting in a large number of false positives and false negatives. At the same time, this kind of method also needs manual maintenance and update of the detection template, which has high operation and maintenance cost and is easy to cause interruption of the automatic process. SUMMARY

[0003] The purpose of the present application is to provide a shelf out-of-stock detection method, device, equipment and medium, which can realize high-precision positioning of the out-of-stock area and accurate identification of the missing commodity in the scenario of frequent dynamic changes of shelf layout, improve the robustness and continuous availability of the system, reduce manual intervention, and reduce long-term maintenance cost.

[0004] In order to solve the above technical problems, the present application provides a shelf out-of-stock detection method, comprising:

[0005] Obtaining multiple local images of the shelf, and sequentially detecting commodities, empty positions and inventory units in each local image to output corresponding detection results;

[0006] Splicing the local images and fusing the detection results of each local image to generate unified detection data;

[0007] According to the physical structure of the shelf, the unified detection data is converted into a structured shelf display sequence;

[0008] When there is an out-of-stock condition in the structured shelf display sequence, the commodity information around the out-of-stock location is analyzed, and the commodity corresponding to the out-of-stock location is determined according to the analysis result;

[0009] When the commodity corresponding to the out-of-stock location cannot be determined, the historical display data of the shelf is retrieved and combined with the sorting and position intersection area of the structured shelf display sequence to delimit the commodity corresponding to the out-of-stock location.

[0010] In order to solve the above technical problems, the present application also provides a shelf out-of-stock detection device, comprising:

[0011] An image detection module is configured to acquire multiple local images of the shelf and sequentially detect commodities, empty spaces and inventory units in each of the local images and output corresponding detection results.

[0012] A splicing and fusion module is configured to splice the local images and fuse the detection results of each of the local images to generate unified detection data.

[0013] A sequence conversion module is configured to convert the unified detection data into a structured shelf display sequence according to the physical structure of the shelf.

[0014] A shortage analysis module is configured to analyze commodity information around a shortage space when there is a shortage condition in the structured shelf display sequence and determine a commodity corresponding to the shortage space according to an analysis result.

[0015] A retrieval and demarcation module is configured to retrieve historical shelf display data and combine an order and a position intersection area of the structured shelf display sequence to demarcate a commodity corresponding to the shortage space when the commodity corresponding to the shortage space cannot be determined.

[0016] To solve the above technical problems, the present application further provides an electronic device comprising:

[0017] A memory is configured to store a computer program.

[0018] A processor is configured to implement the steps of the shelf shortage detection method when the computer program is executed.

[0019] To solve the above technical problems, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the shelf shortage detection method.

[0020] As can be seen from the above technical solutions, the shelf shortage detection method provided by the present application comprises the following steps: acquiring multiple local images of a shelf and sequentially detecting commodities, empty spaces and inventory units in each of the local images and outputting corresponding detection results; splicing the local images and fusing the detection results of each of the local images to generate unified detection data; converting the unified detection data into a structured shelf display sequence according to the physical structure of the shelf; analyzing commodity information around a shortage space when there is a shortage condition in the structured shelf display sequence and determining a commodity corresponding to the shortage space according to an analysis result; and retrieving historical shelf display data and combining an order and a position intersection area of the structured shelf display sequence to demarcate a commodity corresponding to the shortage space when the commodity corresponding to the shortage space cannot be determined.

[0021] The beneficial effects of the present application are that the above-mentioned shelf out-of-stock detection method provided by the present application combines the commodity, empty position and inventory unit detection results of multiple partial images with image splicing and information fusion, generates a structured shelf display sequence, and performs decision analysis by fusing the information of the commodities around the out-of-stock position and historical display data, effectively overcoming the limitations of traditional methods that rely on fixed templates and become invalid when the layout changes, realizing high-precision positioning of the out-of-stock area and accurate identification of the missing commodities in the scene of frequent dynamic changes of shelf layout, improving the robustness of the system to dynamic shelves and the continuous availability of real retail scenarios, at the same time, constructing an end-to-end automated process from image acquisition, processing to analysis and decision-making, forming a closed-loop learning ability, significantly reducing manual intervention and reducing long-term maintenance costs of the system.

[0022] In addition, the present application also provides a corresponding shelf out-of-stock detection device, electronic equipment and computer readable storage medium for the shelf out-of-stock detection method, which have the same or corresponding technical features as the above-mentioned shelf out-of-stock detection method, and the effects are the same as above. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 The flowchart of the shelf out-of-stock detection method provided by the present application is shown in the figure.

[0025] Figure 2 The structural schematic diagram of the shelf out-of-stock detection device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0027] It should be noted that in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, not to describe a specific order or sequence.

[0028] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0029] In conjunction with the specific application environment architecture or specific hardware architecture on which the shelf out-of-stock detection method is executed, the specific application environment architecture or specific hardware architecture is described herein.

[0030] The embodiment of the present application provides a shelf out-of-stock detection method, and the method is described in detail in conjunction with the execution process of the shelf out-of-stock detection method. Figure 1 The flowchart of the shelf out-of-stock detection method provided by the embodiment of the present application is shown in Figure 1 The method comprises the following steps.

[0031] S101, acquiring multiple local images of a shelf, and sequentially detecting commodities, empty spaces and inventory units for each local image to output corresponding detection results.

[0032] In implementation, the present application can collect multiple local images of the shelf through robot vision inspection. The local image can be specifically an original image obtained by the inspection robot at a single shooting point and a single perspective, which only contains a part of the area of the shelf. By collecting multiple local images, complete coverage of the entire shelf can be achieved, and detection omission caused by the limitation of a single image perspective can be avoided. The detection of commodities, empty spaces and inventory units for each local image can accurately extract key information of each area, and provide detailed original data support for subsequent image stitching, data fusion and structured sequence generation.

[0033] S102, stitching the local images and fusing the detection results of the local images to generate unified detection data.

[0034] It should be noted that the present application can integrate the dispersed shelf area into a complete panoramic perspective by stitching the local images, thereby solving the problem of the limited field of view of a single local image. Moreover, fusing the detection results of the local images can summarize the key information of commodities, empty spaces and inventory units, eliminate repeated detection or detection loss, and finally generate unified detection data.

[0035] S103, transform the unified detection data into a structured shelf display sequence according to the physical structure of the shelf.

[0036] It can be understood that the present application reorganizes the unified detection data in combination with the physical structure of the shelf (such as vertical levels and horizontal shelf arrangement), can reorganize the scattered commodity and empty position detection information according to the actual layout logic of the shelf, and forms an orderly structured display sequence. The sequence can clearly present the specific position relationship of each commodity and empty position on the shelf.

[0037] S104, when there is an out-of-stock condition in the structured shelf display sequence, analyze the commodity information around the out-of-stock position, and determine the commodity corresponding to the out-of-stock position according to the analysis result.

[0038] It should be noted that when the present application detects an out-of-stock condition, the commodity information around the out-of-stock position (such as the left and right sides) can be analyzed, and the determination range can be narrowed down through the continuity rule of shelf commodity display. This reasoning method based on spatial neighborhood can quickly match the commodity corresponding to the out-of-stock position.

[0039] S105, when the commodity corresponding to the out-of-stock position cannot be determined, the historical display data of the shelf is called and combined with the sorting and position intersection area of the structured shelf display sequence to determine the commodity corresponding to the out-of-stock position.

[0040] In implementation, when the commodity corresponding to the out-of-stock position cannot be determined through the surrounding commodity information, the historical display data of the shelf is used, combined with the sorting logic of the structured display and the comparison of the position intersection area, the regularity of the historical display can be used to make up for the lack of current information. This backtracking determination method that integrates historical data further improves the accuracy and comprehensiveness of out-of-stock commodity identification, ensuring that even in complex display change scenarios, the out-of-stock commodity can be accurately determined.

[0041] In the above shelf out-of-stock detection method provided by the embodiment of the present application, the commodity, empty position and inventory unit detection results of multiple local images are combined with image stitching and information fusion to generate a structured shelf display sequence, and the surrounding commodity information of the out-of-stock position and historical display data are fused for decision analysis, effectively overcoming the limitation that the traditional method relies on fixed templates and is invalid when the layout changes. The present application realizes high-precision positioning of the out-of-stock area and accurate identification of the missing commodity in the scene of frequent dynamic changes of shelf layout, improves the robustness of the system to dynamic shelves and the continuous availability of real retail scenarios, at the same time, constructs an end-to-end automated process from image acquisition, processing to analysis and decision making, forms a closed-loop learning ability, significantly reduces manual intervention, and reduces long-term maintenance cost of the system.

[0042] Further, in the above-mentioned shelf out-of-stock detection method provided by the embodiments of the present application, in specific implementation, the step S101 of acquiring multiple local images of the shelf and sequentially detecting the goods, the empty space and the inventory unit in each local image can specifically include: acquiring the local images of the shelf collected by the inspection robot from multiple points; positioning the detection frame in which each real object of the goods is located in each local image, and acquiring the confidence degree that the goods are in the detection frame to obtain a first confidence degree; when the first confidence degree is within a first set range, performing coarse-grained classification on the goods and adding a corresponding category label; positioning the empty frame in which there is no goods on the shelf in each local image, and acquiring the confidence degree that the empty frame belongs to the goods empty space to obtain a second confidence degree; when the second confidence degree is within a second set range, it is determined that the empty frame definitely exists the goods empty space; performing inventory unit identification on the goods in the detection frame to obtain the inventory unit identification result of the goods, and acquiring the confidence degree of the inventory unit identification result to obtain a third confidence degree; when the second confidence degree is within a third set range, it is determined that the inventory unit identification result is valid.

[0043] In implementation, the present application can collect local images of multiple points of the shelf by the inspection robot to ensure that the area shot contains the entire shelf. And a target detection model based on deep learning (such as YOLO or Faster R-CNN) is used to detect the goods, identify the empty space and identify the inventory unit (Stock Keeping Unit, SKU) in each local image. The detection result includes a bounding box, a confidence degree and a category label. Among them, the goods detection will locate the rectangular detection frame of the real object of the goods, output the first confidence degree that the goods are in the detection frame, and complete the coarse-grained classification of the goods and add the category label when the confidence degree meets the set range; the empty space identification will locate the empty frame without goods, output the second confidence degree that the empty frame is the goods empty space, and determine that there is a goods empty space when the confidence degree meets the standard; the inventory unit identification identifies the goods in the detection frame, outputs the identification result and the third confidence degree, and confirms that the identification result is valid when the confidence degree meets the requirement. This process ensures the accuracy of the information extraction of goods, empty space and inventory unit through multi-dimensional detection and confidence checking, provides high-quality data support for subsequent shelf display analysis and out-of-stock determination, improves the automation and intelligence level of shelf inspection, and reduces the error of manual intervention.

[0044] Further, in the above-mentioned shelf out-of-stock detection method provided by the embodiments of the present application, in specific implementation, the step S102 of splicing the local images and fusing the detection results of each local image to generate unified detection data can specifically include: splicing the local images to form a panoramic image of the shelf; in the splicing process, calculating and recording the coordinate transformation relationship of each local image to the panoramic image of the shelf formed by splicing; according to the recorded coordinate transformation relationship, uniformly converting the detection results of each local image to the global coordinate system of the panoramic image of the shelf to generate unified detection data.

[0045] In implementation, the local images of each point are spliced by image splicing. Specifically, the local images of the shelf collected by the robot from multiple points can be spliced into a complete shelf panoramic image by using the image splicing. The image splicing can include: obtaining a first image and a second image to be spliced, and extracting feature points from the first image and the second image respectively; matching the feature points extracted from the first image and the second image to obtain a matching point set of the first image and a matching point set of the second image; clustering the matching point set of the first image and the matching point set of the second image respectively, and verifying the spatial internal consistency of the matching points by cross screening; extracting the optimal matching point pair from the matching point pairs that pass the verification by using displacement consistency, generating an optimal matching point subset containing the feature points corresponding to each other in the first image and the second image; and splicing the first image and the second image by using the optimal matching point subset. The above image splicing method can also be other methods, which will not be described here.

[0046] Next, the coordinate transformation relationship from the local image to the panoramic image is calculated and recorded. All detection results identified on each local image are uniformly mapped to the global coordinate system of the panoramic image according to the above coordinate transformation relationship to generate unified detection data. In this way, the global integration and spatial normalization of the shelf detection data are realized, which provides a complete and consistent global data basis for subsequent structured arrangement sequence generation and out-of-stock analysis, and improves the comprehensiveness and accuracy of shelf detection.

[0047] Further, in specific implementation, after the detection results of each local image are uniformly converted to the global coordinate system of the panoramic image of the shelf, it can further include: in the global coordinate system of the panoramic image of the shelf, performing deduplication processing on repeated detection boxes from different local images but pointing to the same commodity or the same empty position; screening out a detection box that best represents the same commodity or the same empty position as an optimal detection box, and updating the confidence of the optimal detection box in combination with the confidence of each repeated detection box, and outputting the unified detection result after deduplication at the panoramic image level.

[0048] In implementation, in the global coordinate system, the present application can perform deduplication (such as using a non-maximum suppression algorithm) on repeated detection boxes from different local images but pointing to the same physical target (such as the same commodity or the same empty position), and merge them into an optimal detection box, and update its confidence. Finally, a unified detection result set after deduplication at the panoramic image level is output, which effectively eliminates the redundant information of multiple local image detection, improves the accuracy and uniqueness of the detection result, provides high-quality and non-repeated input data for subsequent structured processing, and ensures the efficient development of the subsequent analysis process.

[0049] Further, in the specific implementation, in the shelf out-of-stock detection method provided by the embodiments of the present application, the step S103 converts the unified detection data into a structured shelf display sequence according to the physical structure of the shelf, and specifically can include: extracting the center points of all the detection boxes in the unified detection data, and determining the coordinates of the center points in the panoramic image coordinate system; for the longitudinal coordinates of the center points, clustering analysis is performed by using a clustering algorithm to obtain a clustering result; according to the clustering result, the center points with similar longitudinal coordinate values are divided into the same group, each group corresponds to a vertical layer of the shelf, and each detection target is assigned a corresponding layer number to determine the vertical layer level of each detection target; in each vertical layer, all the detection targets in the vertical layer are sorted according to the order of the horizontal coordinates of the center points of the detection boxes to restore the arrangement order of the goods on the shelf layer; the sorted detection target list is checked one by one to check adjacent detection boxes; if the intersection and union ratio of two detection boxes in the horizontal direction exceeds a preset threshold, it is determined that the goods corresponding to the two detection boxes are stacked on the same physical shelf, and the physical shelf where the goods corresponding to the two detection boxes are located is merged; if the intersection and union ratio of two detection boxes in the horizontal direction does not exceed the preset threshold, the goods or empty space corresponding to each detection box occupies an independent shelf; and the structured shelf display sequence is output.

[0050] In implementation, the present application converts the shelf area of multiple points into a structured shelf display sequence by combining visual detection results with image stitching and information fusion. The specific steps can include: the first step is vertical layering: the center point coordinates of all detection boxes in the panoramic image coordinate system after fusion are extracted, and the cluster analysis is performed on the vertical coordinates (Y coordinates) of the center points, such as using the density-based spatial clustering algorithm with noise application (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) or K-means algorithm. According to the clustering results, the center points with similar vertical coordinates are grouped, each group corresponds to a vertical layer of the shelf, and all detection targets are assigned with layer numbers. The second step is horizontal sorting and allocation of goods positions: in each vertical layer, the center points of the detection boxes are sorted from left to right according to the horizontal coordinates (X coordinates). The adjacent detection boxes after sorting are checked, if the intersection area in the horizontal direction exceeds a preset threshold (such as 0.3), it is determined that the stacked goods on the same physical goods position, and they are merged and attributed to a goods position; otherwise, each detection box occupies an independent goods position. The third step is sequence output: a structured shelf display sequence is finally output, which clearly shows the distribution of each layer of goods position, the type of goods in each goods position or the state of empty position, and each detection box contains structured information such as category, layer number, goods position serial number, and whether it is stacked or not, and clearly presents the arrangement order of goods in each layer and the position of empty position. The above process converts the scattered detection data into a structured display sequence through vertical layering and horizontal goods position allocation, which can clearly present the distribution of each layer of goods position, the information of goods and the position of empty position, and provides regular and intuitive data support for subsequent out-of-stock identification, inventory counting and other operations, which significantly improves the refinement and efficiency of shelf management.

[0051] Further, in the above shelf out-of-stock detection method provided by the embodiments of the present application, the step S104 analyzes the information of goods around the out-of-stock goods position, and determines the goods corresponding to the out-of-stock goods position according to the analysis result, which can specifically include: screening all goods positions marked as empty from the structured shelf display sequence, and determining the position of each out-of-stock goods position in the shelf; obtaining the information of goods inventory units of the adjacent goods positions on both sides of each out-of-stock goods position in the same vertical layer in the horizontal direction; if the goods positions on both sides correspond to the same goods inventory unit, it is determined that the goods corresponding to the out-of-stock goods position are the goods corresponding to the same inventory unit on both sides; if only one side of the goods position has goods, or the goods positions on both sides have goods but the inventory units are different, it is determined that the goods corresponding to the out-of-stock goods position cannot be determined.

[0052] In implementation, based on the latest structured shelf display sequence, if the batch is not out-of-stock, it is saved as the latest shelf display history; if there is out-of-stock, spatial position analysis is carried out, that is, all the positions marked as vacancy are screened and their shelf positions are determined, the commodity inventory unit information of the adjacent positions on both sides of each out-of-stock position in the same vertical layer is obtained, if the positions on both sides are the same commodity inventory unit, it is determined that the out-of-stock position corresponds to the commodity; if only one side has a commodity, the positions on both sides are different commodities, or both sides have commodities but the inventory units are different, it is determined that the corresponding commodity of the out-of-stock position cannot be determined. This process determines the out-of-stock state first and carries out targeted spatial position analysis, quickly determines the out-of-stock commodity through the adjacent relationship of the positions, improves the efficiency of out-of-stock identification, and provides timely reference for subsequent replenishment decision.

[0053] Further, in the above shelf out-of-stock detection method provided by the embodiments of the application, step S105 calls the shelf historical display data and combines the ordering of the structured shelf display sequence and the intersection area of the positions to determine the commodity corresponding to the out-of-stock position, which can specifically include: calling the shelf historical display data, obtaining the commodity inventory unit information of each position of the shelf in the historical full-stock state and the corresponding detection frame position information; determining the position of the current out-of-stock position in the historical full-stock display data; calculating the intersection union ratio of the vacancy frame in which the position corresponding to the current out-of-stock position is located and each position detection frame in the historical full-stock display data, finding the historical position with the highest intersection union ratio with the boundary frame of the current out-of-stock position; determining the commodity inventory unit corresponding to the current out-of-stock position according to the ordering rule defined by the structured shelf display sequence and the found historical position, and dividing the commodity corresponding to the out-of-stock position.

[0054] In implementation, when the commodity corresponding to the out-of-stock position cannot be determined, the physical position information of the current out-of-stock position is compared with the historical full-stock display position, the ordering rule defined by the structured shelf display sequence is combined, and the intersection union ratio of the boundary frame of the current out-of-stock position and the boundary frame of each position in the historical full-stock display is calculated to determine the commodity inventory unit corresponding to the out-of-stock position. This method adopts a decision logic combining space and time sequence, first infers the missing commodity by analyzing the adjacent commodity of the current batch out-of-stock position, and then further compares the recent historical display data when the adjacent information is unknown, so that the system can adapt to the dynamic change scene of the shelf, improve the comprehensiveness and accuracy of the identification of the out-of-stock commodity, and effectively cope with the out-of-stock determination demand under complex display changes.

[0055] It should be noted that the traditional out-of-stock detection method relies heavily on fixed shelf drawings or price tag templates. Once there is a layout change such as promotion, price adjustment or commodity transposition, the system will not work normally due to the invalidation of the template. The present application completely abandons the static template, generates a structured shelf display sequence in real time for dynamic analysis, so that the system can adapt to the frequent changes of the shelf layout, greatly improving the robustness and continuous availability in the real retail scene. In addition, the traditional out-of-stock detection method needs manual maintenance and update of the detection template, and the operation and maintenance cost is high and the automatic process is easy to interrupt. The present application realizes end-to-end automation from image acquisition, processing to analysis and decision-making, and can automatically update the no-out-of-stock state to the historical benchmark, forming a closed-loop learning ability, greatly reducing the need for manual intervention and reducing the long-term maintenance cost of the system.

[0056] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0057] The embodiment of the present application also provides a shelf out-of-stock detection device. Figure 2 The structure diagram of the shelf out-of-stock detection device provided by the embodiment of the present application is shown in the figure. Figure 2 As shown in the figure, the device comprises:

[0058] An image detection module 10 is used to acquire multiple local images of the shelf, and sequentially detect the goods, empty positions and inventory units of each local image, and output the corresponding detection results;

[0059] A splicing and fusion module 11 is used to splice the local images and fuse the detection results of each local image to generate unified detection data;

[0060] A sequence conversion module 12 is used to convert the unified detection data into a structured shelf display sequence according to the physical structure of the shelf;

[0061] An out-of-stock analysis module 13 is used to analyze the goods information around the out-of-stock position when there is an out-of-stock condition in the structured shelf display sequence, and determine the goods corresponding to the out-of-stock position according to the analysis result;

[0062] A call and demarcation module 14 is used to call the historical display data of the shelf and combine the sorting and position intersection area of the structured shelf display sequence when the goods corresponding to the out-of-stock position cannot be determined, and demarcate the goods corresponding to the out-of-stock position.

[0063] In the shelf out-of-stock detection device provided by the embodiment of the present application, the detection results of the multiple local images of the goods, the empty positions and the inventory units are combined with image stitching and information fusion through the interaction of the five modules to generate a structured shelf display sequence, and the surrounding goods information and historical display data of the out-of-stock positions are fused for decision analysis, thereby effectively overcoming the limitation that the traditional method is invalid when the layout changes and realizing high-precision positioning of the out-of-stock area and accurate identification of the missing goods in the scenario of frequent dynamic changes of the shelf layout, improving the robustness of the system to the dynamic shelf and the continuous usability of the real retail scene, simultaneously constructing an end-to-end automated process from image acquisition, processing to analysis and decision, forming a closed-loop learning capability, significantly reducing manual intervention and reducing the long-term maintenance cost of the system.

[0064] Since the embodiments of the shelf out-of-stock detection device part correspond to the embodiments of the shelf out-of-stock detection method part, the description of the features in the embodiments corresponding to the shelf out-of-stock detection device can be referred to the related description of the embodiments corresponding to the shelf out-of-stock detection method, which will not be repeated here. And has the same beneficial effects as the above-mentioned shelf out-of-stock detection method.

[0065] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0066] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above-mentioned shelf out-of-stock detection method embodiments when running.

[0067] In an exemplary embodiment, the above-mentioned computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0068] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0069] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0070] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be embodied in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples disclosed herein have been described generally in terms of their functionality, without reference to the corresponding structure. Those skilled in the art will appreciate that the described functionality can be implemented in either hardware or software, or any combination thereof, depending on the particular application and design constraints. It will also be appreciated that the described functionality can be implemented using different amounts of hardware and software depending on the particular application and design constraints. For example, where long battery life is required, more functionality can be implemented in hardware, while where short battery life is acceptable, more functionality can be implemented in software.

[0071] The above provides a detailed introduction to the shelf out-of-stock detection method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above example description is only applicable to help understand the method and core idea of the present application. It should be pointed out that for ordinary skilled persons in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method for detecting out-of-stock items on shelves, characterized in that, The method comprises the following steps: acquiring multiple local images of a shelf and sequentially detecting commodities, empty spaces and inventory units in each of the local images using a deep learning-based target detection model to output corresponding detection results; stitching the local images and fusing the detection results of each of the local images to generate unified detection data; extracting the center points of all detection boxes in the unified detection data and determining the coordinates of the center points in a panoramic image coordinate system; performing clustering analysis on the longitudinal coordinates of the center points using a clustering algorithm to obtain clustering results; according to the clustering results, dividing the center points with similar longitudinal coordinate values into the same group, each group corresponding to a vertical layer of the shelf, and assigning a corresponding layer number to all detection targets to determine the vertical layer level of each detection target; in each vertical layer, all detection targets in the vertical layer are sorted in the order of the horizontal coordinates of the center points of the detection boxes to restore the arrangement order of commodities on the shelf layer; the sorted detection target list is checked one by one to determine whether two detection boxes correspond to commodities stacked on the same physical shelf according to the intersection-over-union of the two detection boxes in the horizontal direction; if the intersection-over-union exceeds a preset threshold, the physical shelf on which the two detection boxes correspond to the commodities is merged; if the intersection-over-union does not exceed the preset threshold, each detection box corresponds to a commodity or an empty space occupying an independent shelf; outputting a structured shelf arrangement sequence; when there is an out-of-stock condition in the structured shelf arrangement sequence, analyzing the commodity information around the out-of-stock shelf to determine the commodity corresponding to the out-of-stock shelf according to the analysis results; when the commodity corresponding to the out-of-stock shelf cannot be determined, retrieving historical shelf arrangement data and combining the sorting and position intersection area of the structured shelf arrangement sequence to determine the commodity corresponding to the out-of-stock shelf.

2. The shelf out detection method of claim 1, wherein, acquiring multiple local images of a shelf and sequentially detecting commodities, empty spaces and inventory units in each of the local images, comprising: acquiring local images of a shelf collected by a patrol robot from multiple points; locating each detection box in which a commodity real object is located in each of the local images and obtaining the confidence level of the commodity in the detection box to obtain a first confidence level; when the first confidence level is within a first set range, the commodity is classified in a coarse-grained manner and a corresponding category label is added; locating the empty frame in each of the local images in which there is no commodity on the shelf and obtaining the confidence level of the empty frame belonging to a commodity empty space to obtain a second confidence level; when the second confidence level is within a second set range, it is determined that there is a commodity empty space in the empty frame; performing inventory unit identification on the commodity in the detection box to obtain an inventory unit identification result of the commodity and obtaining the confidence level of the inventory unit identification result to obtain a third confidence level; when the third confidence level is within a third set range, it is determined that the inventory unit identification result is valid.

3. The shelf out detection method of claim 1, wherein, stitching the local images and fusing the detection results of each of the local images to generate unified detection data, comprising: stitching the local images to form a panoramic image of the shelf; during the stitching process, calculating and recording the coordinate transformation relationship of each of the local images to the stitched panoramic image of the shelf; According to the recorded coordinate transformation relationship, the detection results of each of the local images are uniformly converted into a global coordinate system of the shelf panoramic image to generate unified detection data.

4. The shelf out detection method of claim 3, wherein, After the detection results of each of the local images are uniformly converted into the global coordinate system of the shelf panoramic image, the method further includes: In the global coordinate system of the shelf panoramic image, repeated detection boxes from different local images but pointing to the same commodity or the same empty space are processed to remove the repetitions; A detection box that best represents the same commodity or the same empty space is screened out as an optimal detection box, and the confidence of the optimal detection box is updated in combination with the confidence of each repeated detection box, and the unified detection result after the removal of the repetitions at the panoramic image level is output.

5. The shelf out detection method of claim 1, wherein, Analyzing commodity information around the out-of-stock location, determining the commodity corresponding to the out-of-stock location according to the analysis result, including: Filtering all locations marked as empty from the structured shelf display sequence, and determining the location of each out-of-stock location in the shelf; Obtaining commodity inventory unit information of two adjacent locations on the horizontal direction of each out-of-stock location in the same vertical layer; If the two adjacent locations correspond to the same commodity inventory unit, it is determined that the commodity corresponding to the out-of-stock location is the commodity corresponding to the same inventory unit on the two sides; if only one side has a commodity, or the two sides have commodities but the inventory units are different, it is determined that the commodity corresponding to the out-of-stock location cannot be determined.

6. The shelf out detection method of claim 1, wherein, Retrieving historical display data of the shelf and combining the order and position intersection area of the structured shelf display sequence to determine the commodity corresponding to the out-of-stock location, including: Retrieving historical display data of the shelf, obtaining commodity inventory unit information and corresponding detection box position information of each location of the shelf in the historical full-stock state; Determining the location of the current out-of-stock location corresponding to the historical full-stock display data; Calculating the intersection union ratio of the empty box where the location corresponding to the current out-of-stock location is located and each location detection box in the historical full-stock display data, and finding the historical location with the highest intersection union ratio with the boundary box of the current out-of-stock location; According to the order rule defined by the structured shelf display sequence and the found historical location, determining the commodity inventory unit corresponding to the current out-of-stock location, and determining the commodity corresponding to the out-of-stock location.

7. A shelf out-of-stock detection apparatus characterized by comprising: The method includes: An image detection module configured to obtain multiple local images of a shelf, and sequentially detect commodities, empty spaces, and inventory units of each of the local images using a target detection model based on deep learning, and output corresponding detection results; A splicing and fusion module configured to splice the local images, and fuse the detection results of each of the local images to generate unified detection data; A sequence conversion module configured to extract center points of all detection boxes in the unified detection data, and determine coordinates of the center points in a panoramic image coordinate system; and perform clustering analysis on the vertical coordinates of the center points using a clustering algorithm to obtain clustering results. According to the clustering result, the center points with similar vertical coordinate values are divided into the same group, each group corresponds to a vertical layer of the shelf, and each detection target is assigned a corresponding layer number to determine the vertical layer level of each detection target; in each vertical layer, all detection targets in the vertical layer are sorted according to the order of the horizontal coordinates of the center points of the detection boxes, and the arrangement order of the goods on the shelf layer is restored; the sorted detection target list is checked one by one, and the adjacent detection boxes are checked; if the intersection-over-union of two detection boxes in the horizontal direction exceeds a preset threshold, it is determined that the goods corresponding to the two detection boxes are stacked on the same physical shelf, and the physical shelf where the goods corresponding to the two detection boxes are located is merged; if the intersection-over-union of two detection boxes in the horizontal direction does not exceed the preset threshold, the goods or empty space corresponding to each detection box occupies an independent shelf; outputting a structured shelf arrangement sequence; an out-of-stock analysis module for analyzing the goods information around the out-of-stock shelf when there is an out-of-stock condition in the structured shelf arrangement sequence, and determining the goods corresponding to the out-of-stock shelf according to the analysis result; a retrieval and demarcation module for retrieving historical shelf arrangement data and combining the intersection area of the ordering and position of the structured shelf arrangement sequence when the goods corresponding to the out-of-stock shelf cannot be determined, and demarcating the goods corresponding to the out-of-stock shelf.

8. An electronic device, comprising: comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the shelf out-of-stock detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the shelf out-of-stock detection method according to any one of claims 1 to 6.

Images