Goods shelf stockout detection method, device and equipment and medium

Abstract

The invention discloses a shelf stockout detection method, device and equipment and a medium, and relates to the technical field of shelf detection, and the method comprises the steps: obtaining a plurality of local images of a shelf, carrying out the detection of commodities, vacancies and inventory units of the images, and outputting a detection result; splicing the images and fusing the detection results, generating unified detection data, and converting the unified detection data into a structured shelf display sequence according to shelf physical results. If the goods are out of stock, determining the lacked goods by analyzing the information of the goods around the out-of-stock goods location; if not, the historical shelf display data is called, and the missing commodities are delimited in combination with the structured shelf display sequence and the position intersection area. In this way, the problems that a traditional method depends on a fixed template, and layout changes are invalid are solved, high-precision positioning of a stockout area and accurate recognition of missing commodities under the scene that the shelf layout frequently and dynamically changes are achieved, the robustness and continuous availability of a system are improved, manual intervention is reduced, and the long-term maintenance cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of shelf detection, in particular to a shelf out-of-stock detection method, device, equipment and medium. BACKGROUND

[0002] In the real scene of frequent dynamic changes of shelf commodity display layout, the traditional out-of-stock detection method has obvious limitations, not only difficult to effectively, accurately and dynamically complete the out-of-stock location positioning and specific identification of the missing commodity, but also seriously dependent on fixed shelf drawings, price tag templates or standard shelf drawings, fixed price tag shelves as detection templates. Once the layout is changed due to promotion, price adjustment, commodity repositioning or restocking, the static template will be invalid immediately, which makes the system unable to accurately locate the out-of-stock location and accurately identify the missing commodity, resulting in a large number of false positives and false negatives. At the same time, this kind of method also needs manual maintenance and update of detection template, which has high operation and maintenance cost and is easy to cause interruption of automatic process. SUMMARY

[0003] The purpose of the present application is to provide a shelf out-of-stock detection method, device, equipment and medium, which can realize high-precision positioning of the out-of-stock area and accurate identification of the missing commodity in the scene of frequent dynamic changes of shelf layout, improve the robustness and continuous availability of the system, reduce manual intervention, and reduce long-term maintenance cost.

[0004] In order to solve the above technical problems, the present application provides a shelf out-of-stock detection method, comprising: obtaining multiple local images of a shelf, and sequentially detecting commodities, empty positions and inventory units of each of the local images to output corresponding detection results; stitching the local images and fusing the detection results of each of the local images to generate unified detection data; converting the unified detection data into a structured shelf display sequence according to the physical structure of the shelf; when there is an out-of-stock condition in the structured shelf display sequence, analyzing the commodity information around the out-of-stock location, and determining the commodity corresponding to the out-of-stock location according to the analysis result; when the commodity corresponding to the out-of-stock location cannot be determined, retrieving historical display data of the shelf and combining the ordering and position intersection area of the structured shelf display sequence to delimit the commodity corresponding to the out-of-stock location.

[0005] In order to solve the above technical problems, the present application also provides a shelf out-of-stock detection device, comprising: an image detection module, configured to obtain multiple local images of a shelf, and sequentially detect commodities, empty positions and inventory units of each of the local images to output corresponding detection results; The splicing and fusing module is configured to splice the local images and fuse the detection results of the local images to generate unified detection data. The sequence conversion module is configured to convert the unified detection data into a structured shelf display sequence according to the physical structure of the shelf. The out-of-stock analysis module is configured to analyze the commodity information around the out-of-stock shelf position when the structured shelf display sequence has an out-of-stock condition, and determine the commodity corresponding to the out-of-stock shelf position according to the analysis result. The call and demarcation module is configured to call the historical shelf display data and combine the sorting and position intersection area of the structured shelf display sequence to demarcate the commodity corresponding to the out-of-stock shelf position when the commodity corresponding to the out-of-stock shelf position cannot be determined.

[0006] To solve the above technical problems, the present application further provides an electronic device, comprising: A memory configured to store a computer program. A processor configured to execute the computer program to implement the steps of the shelf out-of-stock detection method.

[0007] To solve the above technical problems, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the shelf out-of-stock detection method.

[0008] As can be seen from the above technical solutions, the shelf out-of-stock detection method provided by the present application comprises: acquiring multiple local images of a shelf, and sequentially detecting commodities, empty positions and inventory units in each local image to output corresponding detection results; splicing the local images and fusing the detection results of the local images to generate unified detection data; converting the unified detection data into a structured shelf display sequence according to the physical structure of the shelf; analyzing the commodity information around the out-of-stock shelf position when the structured shelf display sequence has an out-of-stock condition, and determining the commodity corresponding to the out-of-stock shelf position according to the analysis result; and calling the historical shelf display data and combining the sorting and position intersection area of the structured shelf display sequence to demarcate the commodity corresponding to the out-of-stock shelf position when the commodity corresponding to the out-of-stock shelf position cannot be determined.

[0009] The application has the beneficial effects that the above-mentioned shelf out-of-stock detection method provided by the application combines the commodity, empty position and inventory unit detection results of multiple partial images with image splicing and information fusion to generate a structured shelf display sequence, and performs decision analysis by fusing the commodity information around the out-of-stock position and historical display data, effectively overcoming the limitation that the traditional method is invalid when the layout changes, realizing high-precision positioning of the out-of-stock area and accurate identification of the missing commodity in the scene of frequent dynamic changes of shelf layout, improving the robustness of the system to dynamic shelves and the continuous usability of the real retail scene, at the same time, constructing an end-to-end automated process from image acquisition, processing to analysis and decision, forming a closed-loop learning ability, significantly reducing manual intervention and reducing long-term maintenance costs of the system.

[0010] In addition, the application also provides a corresponding shelf out-of-stock detection device, electronic equipment and computer readable storage medium for the shelf out-of-stock detection method, which have the same or corresponding technical features as the above-mentioned shelf out-of-stock detection method, and the effects are the same as above. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 The flowchart of the shelf out-of-stock detection method provided by the embodiments of the application; Figure 2 The structural schematic diagram of the shelf out-of-stock detection device provided by the embodiments of the application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0014] It should be noted that in the description of the application, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the application are used to distinguish similar objects, not to describe a specific order or sequence.

[0015] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0016] In conjunction with the specific application environment architecture or the specific hardware architecture on which the execution of the shelf out-of-stock detection method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0017] Embodiments of the present application provide a shelf out-of-stock detection method, and the method is described in detail in conjunction with the execution process of the shelf out-of-stock detection method. Figure 1 The flowchart of the shelf out-of-stock detection method provided by the embodiments of the present application is shown in Figure 1 The method comprises the following steps: S101, a plurality of local images of a shelf are acquired, and detection of commodities, empty spaces and inventory units is performed on each local image in sequence, and corresponding detection results are output.

[0018] In implementation, the present application can collect a plurality of local images of the shelf through robot vision inspection. The local image can be specifically an original image obtained by the inspection robot at a single shooting point and a single perspective, which only contains a part of the area of the shelf. Through the collection of a plurality of local images, complete coverage of the entire shelf can be achieved, and detection omission caused by the limitation of a single image perspective can be avoided; the detection of commodities, empty spaces and inventory units is performed on each local image respectively, which can accurately extract key information of each area, and provide detailed original data support for subsequent image stitching, data fusion and structured sequence generation.

[0019] S102, the local images are stitched, and the detection results of the local images are fused to generate unified detection data.

[0020] It should be noted that, by stitching the local images, the present application can integrate the dispersed shelf area into a complete panoramic perspective, and solve the problem of the limited field of view of a single local image. Moreover, by fusing the detection results of the local images, key information of commodities, empty spaces and inventory units can be summarized, and the situation of repeated detection or missing detection can be eliminated, and finally the unified detection data is generated.

[0021] S103, the unified detection data is converted into a structured shelf display sequence according to the physical structure of the shelf.

[0022] It can be understood that, by combining the physical structure of the shelf (such as vertical levels and horizontal shelf arrangement), the present application can reorganize the dispersed commodity and empty space detection information according to the actual layout logic of the shelf, and form an orderly structured display sequence. The sequence can clearly present the specific positional relationship of each commodity and empty space on the shelf.

[0023] S104, when there is an out-of-stock situation in the structured shelf display sequence, then the commodity information around the out-of-stock location is analyzed, and the commodity corresponding to the out-of-stock location is determined according to the analysis result.

[0024] It should be noted that when the out-of-stock situation is detected, the commodity information around the out-of-stock location (such as the left and right sides) is analyzed, and the determination range can be narrowed down through the continuity rule of shelf commodity display. This reasoning method based on spatial neighborhood can quickly match the commodity corresponding to the out-of-stock location.

[0025] S105, when the commodity corresponding to the out-of-stock location cannot be determined, the historical display data of the shelf is called and combined with the sorting and position intersection area of the structured shelf display sequence to determine the commodity corresponding to the out-of-stock location.

[0026] In implementation, when the commodity corresponding to the out-of-stock location cannot be determined through the surrounding commodity information, the historical display data of the shelf is used, combined with the sorting logic of the structured display and the comparison of the intersection area of the location, to use the regularity of the historical display to make up for the lack of current information. This backtracking determination method that integrates historical data further improves the accuracy and comprehensiveness of out-of-stock commodity identification, ensuring accurate determination of out-of-stock commodities even in complex display change scenarios.

[0027] The above shelf out-of-stock detection method provided by the embodiment of the present application combines the commodity, empty location and inventory unit detection results of multiple local images with image stitching and information fusion to generate a structured shelf display sequence, and integrates the surrounding commodity information of the out-of-stock location and historical display data for decision analysis, effectively overcoming the limitations of traditional methods that rely on fixed templates and become invalid when the layout changes. The present application realizes high-precision positioning of the out-of-stock area and accurate identification of the missing commodity in the scenario of frequent dynamic changes of shelf layout, improves the robustness of the system to dynamic shelves and the continuous availability of real retail scenarios, at the same time, constructs an end-to-end automated process from image acquisition, processing to analysis and decision making, forms a closed-loop learning ability, significantly reduces manual intervention, and reduces long-term maintenance cost of the system.

[0028] Further, in the above-mentioned shelf out-of-stock detection method provided by the embodiments of the present application, in specific implementation, the step S101 of acquiring multiple local images of the shelf and sequentially detecting the goods, the empty space and the inventory unit in each local image can specifically include: acquiring the local images of the shelf collected by the inspection robot from multiple points; positioning the detection frame in which each real object of the goods is located in each local image, and acquiring the confidence degree that the goods are in the detection frame to obtain a first confidence degree; when the first confidence degree is within a first set range, performing coarse-grained classification on the goods and adding a corresponding category label; positioning the empty frame in which there is no goods on the shelf in each local image, and acquiring the confidence degree that the empty frame belongs to the goods empty space to obtain a second confidence degree; when the second confidence degree is within a second set range, it is determined that the empty frame definitely exists the goods empty space; performing inventory unit identification on the goods in the detection frame to obtain the inventory unit identification result of the goods, and acquiring the confidence degree of the inventory unit identification result to obtain a third confidence degree; when the second confidence degree is within a third set range, it is determined that the inventory unit identification result is valid.

[0029] In implementation, the present application can collect local images of multiple points of the shelf by the inspection robot to ensure that the area of the image contains the entire shelf. A target detection model based on deep learning (such as YOLO or Faster R-CNN) is used to detect goods, identify empty space and identify Stock Keeping Unit (SKU) in each local image. The detection result includes a bounding box, a confidence degree and a category label. Among them, the goods detection will locate the rectangular detection frame of the real object of the goods, output the first confidence degree that the goods are in the detection frame, and complete the coarse-grained classification of the goods and add the category label when the confidence degree meets the set range; the empty space identification will locate the empty frame without goods, output the second confidence degree that the empty frame is the goods empty space, and determine that there is a goods empty space when the confidence degree meets the standard; the inventory unit identification identifies the goods in the detection frame, outputs the identification result and the third confidence degree, and confirms that the identification result is valid when the confidence degree meets the requirement. This process ensures the accuracy of the extraction of goods, empty space and inventory unit information through multi-dimensional detection and confidence checking, provides high-quality data support for subsequent shelf display analysis and out-of-stock determination, improves the automation and intelligence level of shelf inspection, and reduces the error of manual intervention.

[0030] Further, in the above-mentioned shelf out-of-stock detection method provided by the embodiments of the present application, in specific implementation, the step S102 of splicing the local images and fusing the detection results of each local image to generate unified detection data can specifically include: splicing the local images to form a panoramic image of the shelf; in the splicing process, calculating and recording the coordinate transformation relationship of each local image to the panoramic image of the shelf formed by splicing; according to the recorded coordinate transformation relationship, converting the detection results of each local image to the global coordinate system of the panoramic image of the shelf to generate unified detection data.

[0031] In implementation, the local images of each point are spliced by image splicing. Specifically, the local images of the shelf collected by the robot from multiple points can be spliced into a complete shelf panoramic image by using the image splicing. The image splicing can include: obtaining a first image and a second image to be spliced, and extracting feature points from the first image and the second image respectively; matching the feature points extracted from the first image and the second image to obtain a matching point set of the first image and a matching point set of the second image; clustering the matching point set of the first image and the matching point set of the second image respectively, and verifying the spatial internal consistency of the matching points by cross screening; extracting the optimal matching point pair from the matching point pairs that pass the verification by using displacement consistency, generating an optimal matching point subset containing the feature points corresponding to each other in the first image and the second image; and splicing the first image and the second image by using the optimal matching point subset. The above image splicing method can also be other methods, which will not be described here.

[0032] Next, the coordinate transformation relationship from the local image to the panoramic image is calculated and recorded. All detection results identified on each local image are uniformly mapped to the global coordinate system of the panoramic image according to the above coordinate transformation relationship to generate unified detection data. In this way, the global integration and spatial normalization of the shelf detection data are realized, which provides a complete and consistent global data basis for subsequent structured arrangement sequence generation and out-of-stock analysis, and improves the comprehensiveness and accuracy of shelf detection.

[0033] Further, in specific implementation, after the detection results of each local image are uniformly converted to the global coordinate system of the panoramic image of the shelf, it can further include: in the global coordinate system of the panoramic image of the shelf, performing deduplication processing on repeated detection boxes from different local images but pointing to the same commodity or the same empty position; screening out a detection box that best represents the same commodity or the same empty position as an optimal detection box, and updating the confidence of the optimal detection box in combination with the confidence of each repeated detection box, and outputting the unified detection result after deduplication at the panoramic image level.

[0034] In implementation, in the global coordinate system, the present application can perform deduplication (such as using a non-maximum suppression algorithm) on repeated detection boxes from different local images but pointing to the same physical target (such as the same commodity or the same empty position), and merge them into an optimal detection box, and update its confidence. Finally, a unified detection result set after deduplication at the panoramic image level is output, which effectively eliminates the redundant information of multiple local image detection, improves the accuracy and uniqueness of the detection result, provides high-quality and non-repeated input data for subsequent structured processing, and ensures the efficient development of the subsequent analysis process.

[0035] Further, in the specific implementation, in the shelf out-of-stock detection method provided by the embodiments of the present application, the step S103 converts the unified detection data into a structured shelf display sequence according to the physical structure of the shelf, and specifically can include: extracting the center points of all the detection boxes in the unified detection data, and determining the coordinates of the center points in the panoramic image coordinate system; for the longitudinal coordinates of the center points, clustering analysis is performed by using a clustering algorithm to obtain a clustering result; according to the clustering result, the center points with similar longitudinal coordinate values are divided into the same group, each group corresponds to a vertical layer of the shelf, and each detection target is assigned a corresponding layer number to determine the vertical layer level of each detection target; in each vertical layer, all the detection targets in the vertical layer are sorted according to the order of the horizontal coordinates of the center points of the detection boxes to restore the arrangement order of the goods on the shelf layer; the sorted detection target list is checked one by one to check adjacent detection boxes; if the intersection and union ratio of two detection boxes in the horizontal direction exceeds a preset threshold, it is determined that the goods corresponding to the two detection boxes are stacked on the same physical shelf, and the physical shelf where the goods corresponding to the two detection boxes are located is merged; if the intersection and union ratio of two detection boxes in the horizontal direction does not exceed the preset threshold, the goods or empty space corresponding to each detection box occupies an independent shelf; and the structured shelf display sequence is output.

[0036] In implementation, the present application converts the shelf area of multiple points into a structured shelf display sequence by combining visual detection results with image stitching and information fusion. The specific steps can include: the first step is vertical layering: the center point coordinates of all detection boxes in the panoramic image coordinate system after fusion are extracted, and the cluster analysis is performed on the vertical coordinates (Y coordinates) of the center points, such as using the density-based spatial clustering algorithm with noise application (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) or K-means algorithm. According to the clustering results, the center points with similar vertical coordinates are grouped, each group corresponds to a vertical layer of the shelf, and all detection targets are assigned with layer numbers. The second step is horizontal sorting and allocation of goods positions: in each vertical layer, the center points of the detection boxes are sorted from left to right according to the horizontal coordinates (X coordinates). The adjacent detection boxes after sorting are checked, if the intersection area in the horizontal direction exceeds a preset threshold (such as 0.3), it is determined that the stacked goods on the same physical goods position, and they are merged and attributed to a goods position; otherwise, each detection box occupies an independent goods position. The third step is sequence output: a structured shelf display sequence is finally output, which clearly shows the distribution of each layer of goods position, the type of goods in each goods position or the state of empty position, and each detection box contains structured information such as category, layer number, goods position serial number, and whether it is stacked or not, and clearly presents the arrangement order of goods in each layer and the position of empty position. The above process converts the scattered detection data into a structured display sequence through vertical layering and horizontal goods position allocation, which can clearly present the distribution of each layer of goods position, the information of goods and the position of empty position, and provides regular and intuitive data support for subsequent out-of-stock identification, inventory counting and other operations, which significantly improves the refinement and efficiency of shelf management.

[0037] Further, in the above shelf out-of-stock detection method provided by the embodiments of the present application, the step S104 analyzes the information of goods around the out-of-stock goods position, and determines the goods corresponding to the out-of-stock goods position according to the analysis result, which can specifically include: screening all goods positions marked as empty from the structured shelf display sequence, and determining the position of each out-of-stock goods position in the shelf; obtaining the information of goods inventory units of the adjacent goods positions on both sides of each out-of-stock goods position in the same vertical layer in the horizontal direction; if the goods positions on both sides correspond to the same goods inventory unit, it is determined that the goods corresponding to the out-of-stock goods position are the goods corresponding to the same inventory unit on both sides; if only one side of the goods position has goods, or the goods positions on both sides have goods but the inventory units are different, it is determined that the goods corresponding to the out-of-stock goods position cannot be determined.

[0038] In implementation, based on the latest structured shelf display sequence, if the batch is not out-of-stock, it is saved as the latest shelf display history; if there is out-of-stock, spatial position analysis is carried out, that is, all the positions marked as vacancy are screened and their shelf positions are determined, the commodity inventory unit information of the adjacent positions on both sides of each out-of-stock position in the same vertical layer is obtained, if the positions on both sides are the same commodity inventory unit, it is determined that the out-of-stock position corresponds to the commodity; if only one side has a commodity, the positions on both sides are different commodities, or both sides have commodities but the inventory units are different, it is determined that the corresponding commodity of the out-of-stock position cannot be determined. This process determines the out-of-stock state first and carries out targeted spatial position analysis, quickly determines the out-of-stock commodity through the adjacent relationship of the positions, improves the efficiency of out-of-stock identification, and provides timely reference for subsequent replenishment decision.

[0039] Further, in the above shelf out-of-stock detection method provided by the embodiments of the application, step S105 calls the shelf historical display data and combines the ordering of the structured shelf display sequence and the intersection region of the positions to determine the commodity corresponding to the out-of-stock position, which can specifically include: calling the shelf historical display data, obtaining the commodity inventory unit information of each position of the shelf in the historical full-stock state and the corresponding detection frame position information; determining the position of the current out-of-stock position corresponding to the historical full-stock display data; calculating the intersection union ratio of the empty frame in which the position corresponding to the current out-of-stock position is located and each position detection frame in the historical full-stock display data, finding the historical position with the highest intersection union ratio with the boundary frame of the current out-of-stock position; determining the commodity inventory unit corresponding to the current out-of-stock position according to the ordering rule defined by the structured shelf display sequence and the found historical position, and dividing the commodity corresponding to the out-of-stock position.

[0040] In implementation, when the commodity corresponding to the out-of-stock position cannot be determined, the physical position information of the current out-of-stock position is compared with the historical full-stock display position, the ordering rule defined by the structured shelf display sequence is combined, and the intersection union ratio of the boundary frame of the current out-of-stock position and the boundary frame of each position in the historical full-stock display is used to determine the commodity inventory unit corresponding to the out-of-stock position. This method uses a decision logic combining space and time sequence, first infers the missing commodity by analyzing the adjacent commodity of the current batch out-of-stock position, and then further compares the recent historical display data when the adjacent information is unknown, so that the system can adapt to the dynamic change scene of the shelf, improve the comprehensiveness and accuracy of the identification of the out-of-stock commodity, and effectively cope with the out-of-stock determination demand under complex display changes.

[0041] It should be noted that the traditional out-of-stock detection method relies heavily on fixed shelf drawings or price tag templates. Once there is a layout change such as promotion, price adjustment or commodity transposition, the system will not work normally due to the invalidation of the template. The present application completely abandons the static template, generates a structured shelf display sequence in real time for dynamic analysis, so that the system can adapt to the frequent changes of the shelf layout, greatly improving the robustness and continuous availability in the real retail scene. In addition, the traditional out-of-stock detection method needs manual maintenance and update of the detection template, and the operation and maintenance cost is high and the automatic process is easy to interrupt. The present application realizes end-to-end automation from image acquisition, processing to analysis and decision-making, and can automatically update the no-stock state to the historical benchmark, forming a closed-loop learning ability, greatly reducing the need for manual intervention and reducing the long-term maintenance cost of the system.

[0042] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0043] The embodiment of the present application also provides a shelf out-of-stock detection device. Figure 2 The structural schematic diagram of the shelf out-of-stock detection device provided by the embodiment of the present application. The embodiment is based on the angle of function modules, as shown in the figure, the device comprises: Figure 2 An image detection module 10 is used to acquire multiple local images of the shelf, and sequentially detect the goods, empty positions and inventory units of each local image, and output the corresponding detection results; A splicing and fusion module 11 is used to splice the local images and fuse the detection results of each local image to generate unified detection data; A sequence conversion module 12 is used to convert the unified detection data into a structured shelf display sequence according to the physical structure of the shelf; An out-of-stock analysis module 13 is used to analyze the commodity information around the out-of-stock position when there is an out-of-stock condition in the structured shelf display sequence, and determine the commodity corresponding to the out-of-stock position according to the analysis result; A call and demarcation module 14 is used to call the historical display data of the shelf and combine the sorting and position intersection area of the structured shelf display sequence when the commodity corresponding to the out-of-stock position cannot be determined, and demarcate the commodity corresponding to the out-of-stock position.

[0044] ​In the shelf out-of-stock detection device provided by the embodiment of the present application, the detection results of the multiple local images of the goods, the empty positions and the inventory units are combined with image stitching and information fusion through the interaction of the five modules to generate a structured shelf display sequence, and the surrounding goods information and historical display data of the out-of-stock position are fused for decision analysis, thereby effectively overcoming the limitation that the traditional method is invalid when the layout changes and realizing high-precision positioning of the out-of-stock area and accurate identification of the missing goods in the scenario of frequent dynamic changes of the shelf layout, improving the robustness of the system to the dynamic shelf and the continuous usability of the real retail scene, simultaneously constructing an end-to-end automated process from image acquisition, processing to analysis and decision, forming a closed-loop learning capability, significantly reducing manual intervention and reducing the long-term maintenance cost of the system.

[0045] Since the embodiments of the shelf out-of-stock detection device part correspond to the embodiments of the shelf out-of-stock detection method part, the description of the features in the embodiments corresponding to the shelf out-of-stock detection device can be referred to the related description of the embodiments corresponding to the shelf out-of-stock detection method, which will not be repeated here. And has the same beneficial effects as the above-mentioned shelf out-of-stock detection method.

[0046] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0047] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above-mentioned shelf out-of-stock detection method embodiments when running.

[0048] In an exemplary embodiment, the above-mentioned computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0049] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0050] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in any of the above-mentioned shelf out-of-stock detection method embodiments.

[0051] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be implemented in electronic hardware, computer software, or both. As described above, the disclosure is directed to each individual feature, hardware and software, and method steps of the disclosed examples. Accordingly, the disclosure can be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which can collectively be referred to as "software process" herein. Furthermore, the disclosure can take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium that can be executed by a computer. The disclosure can also take the form of computer program code embodied in the hardware, such as application specific integrated circuit(s) or field programmable gate array(s).

[0052] The above provides a detailed introduction to the shelf out-of-stock detection method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A method for detecting out-of-stock items on shelves, characterized in that, include: Acquire multiple partial images of the shelf, and sequentially detect the goods, empty spaces, and inventory units in each partial image, and output the corresponding detection results; The local images are stitched together, and the detection results of each local image are merged to generate unified detection data; Based on the physical structure of the shelves, the unified detection data is transformed into a structured shelf display sequence; When there is a stockout in the structured shelf display sequence, the product information around the stockout location is analyzed, and the product corresponding to the stockout location is determined based on the analysis results. When the product corresponding to the out-of-stock location cannot be determined, historical shelf display data is retrieved and combined with the sorting and positional intersection area of ​​the structured shelf display sequence to identify the product corresponding to the out-of-stock location.

2. The shelf shortage detection method according to claim 1, characterized in that, Acquire multiple partial images of the shelf, and sequentially detect goods, empty spaces, and inventory units in each partial image, including: Acquire partial images of the shelves collected by the inspection robot from multiple locations; Locate the detection box containing each product in each of the local images, and obtain the confidence level that the product is within the detection box to obtain a first confidence level; when the first confidence level is within a first set range, perform coarse-grained classification on the product and add the corresponding category label; Locate empty frames without products on the shelves in each of the local images, and obtain the confidence level of whether the empty frame belongs to an empty product position to obtain a second confidence level; when the second confidence level is within a second set range, it is determined that there is an empty product position in the empty frame; The inventory unit of the product is identified within the detection box to obtain the inventory unit identification result. The credibility of the inventory unit identification result is then obtained to obtain the third confidence level. When the second confidence level is within the third set range, the inventory unit identification result is deemed valid.

3. The shelf shortage detection method according to claim 1, characterized in that, The local images are stitched together, and the detection results of each local image are fused to generate unified detection data, including: The local images are stitched together to form a panoramic view of the shelf. During the stitching process, the coordinate transformation relationship between each local image and the stitched panoramic image of the shelf is calculated and recorded; Based on the recorded coordinate transformation relationship, the detection results of each local image are uniformly transformed into the global coordinate system of the shelf panoramic image to generate unified detection data.

4. The shelf shortage detection method according to claim 3, characterized in that, After uniformly transforming the detection results of each local image to the global coordinate system of the shelf panorama, the method further includes: In the global coordinate system of the shelf panorama, duplicate detection boxes from different local images that point to the same product or the same empty space are deduplicated. The detection box that best represents the same product or the same empty space is selected as the optimal detection box. The confidence of the optimal detection box is updated by combining the confidence of each duplicate detection box, and the unified detection result after deduplication is output at the panoramic image level.

5. The shelf shortage detection method according to claim 1, characterized in that, Based on the physical structure of the shelves, the unified detection data is transformed into a structured shelf display sequence, including: Extract the center points of all detection boxes in the unified detection data and determine the coordinates of the center points in the panoramic coordinate system; Clustering analysis is performed using a clustering algorithm based on the ordinate of the center point to obtain the clustering results; Based on the clustering results, the center points with similar vertical coordinate values ​​are divided into the same group, each group corresponds to a vertical layer of the shelf, and all detection targets are assigned a corresponding layer number to determine the vertical layer of each detection target. Within each vertical layer, all detection targets within the vertical layer are sorted according to the order of the horizontal coordinates of the center point of the detection frame, thus restoring the placement order of the goods on the shelf layer. For the sorted list of detection targets, check adjacent detection boxes one by one; if the intersection-union ratio of two detection boxes in the horizontal direction exceeds the preset threshold, it is determined that the products corresponding to the two detection boxes are products stacked on the same physical storage location, and the physical storage locations of the products corresponding to the two detection boxes are merged; if the intersection-union ratio of two detection boxes in the horizontal direction does not exceed the preset threshold, the products or empty spaces corresponding to each detection box occupy an independent storage location. Output a structured shelf display sequence.

6. The shelf shortage detection method according to claim 1, characterized in that, Analyze the product information around the out-of-stock location, and determine the products corresponding to the out-of-stock location based on the analysis results, including: From the structured shelf display sequence, select all the empty locations marked as empty, and determine the position of each empty location in the shelf; Obtain the inventory unit information of the goods in the horizontal direction of the two adjacent storage locations on the same vertical layer for each out-of-stock location; If the storage locations on both sides correspond to the same inventory unit, then the out-of-stock storage location is determined to be the same inventory unit on both sides; if only one storage location has the product, or if both storage locations have the product but different inventory units, then the out-of-stock storage location cannot be determined.

7. The shelf shortage detection method according to claim 1, characterized in that, By retrieving historical shelf display data and combining it with the sorting and positional intersection areas of the structured shelf display sequence, the products corresponding to the out-of-stock locations are identified, including: Retrieve historical shelf display data to obtain information on the inventory units of each shelf location under historical full-stock conditions, as well as the corresponding detection box position information; Determine the location of the currently out-of-stock item in the historical data of fully stocked displays; Calculate the intersection-union ratio (IoU) between the empty frame of the current out-of-stock location and the detection frames of each location in the historical full-stock display data, and find the historical location with the highest IoU with the current out-of-stock location's bounding box; Based on the sorting rules defined in the structured shelf display sequence and the found historical stock locations, determine the inventory unit of the goods corresponding to the current out-of-stock location and classify the goods corresponding to the out-of-stock location.

8. A shelf out-of-stock detection device, characterized in that, include: The image detection module is used to acquire multiple partial images of the shelf, and sequentially detect the goods, empty spaces, and inventory units in each partial image, and output the corresponding detection results. The stitching and fusion module is used to stitch together the local images and fuse the detection results of each local image to generate unified detection data; The sequence conversion module is used to convert the unified detection data into a structured shelf display sequence based on the physical structure of the shelf. The out-of-stock analysis module is used to analyze the product information around the out-of-stock location when there is an out-of-stock situation in the structured shelf display sequence, and determine the product corresponding to the out-of-stock location based on the analysis results; The delineation module is used to retrieve historical shelf display data and combine it with the sorting and positional intersection area of ​​the structured shelf display sequence to delineate the product corresponding to the out-of-stock location when the product corresponding to the out-of-stock location cannot be determined.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the shelf shortage detection method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the shelf shortage detection method as described in any one of claims 1 to 7.

Images