Display analysis method, device, electronic device and storage medium

Abstract

The present invention belongs to the field of information technology and discloses a display analysis method, device, electronic device and storage medium. The display analysis method includes: obtaining an image of a basket for accommodating goods in a horizontal freezer; performing basket detection and basket correction on the basket image to obtain a corrected image; identifying the display information of the goods on the corrected image to obtain a product display identification result, wherein the display information includes: type, location and quantity; and obtaining a display analysis result based on a preset standard display diagram and the product display identification result. The display analysis device includes: an acquisition module, a first obtaining module, a second obtaining module and a third obtaining module. The above technical solution solves the problems of high labor cost, low frequency and difficult quality assurance in manual inspection in existing display inspection methods.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a display analysis method, device, electronic equipment and storage medium. Background Art

[0002] In recent years, smart retail has rapidly developed. Freezers are the ultimate platform for companies to showcase their products to consumers. Standardized displays are not only aesthetically pleasing but also attract consumer attention and boost sales. However, companies have limited means to manage and access information about these terminal freezers. Existing display inspections rely solely on manual on-site verification, a process that consumes significant human resources, is infrequent, and lacks quality assurance. Summary of the Invention

[0003] To address the aforementioned issues, the present invention provides, on one hand, a display analysis method, comprising: acquiring an image of a basket for accommodating goods within a chest freezer; performing basket detection and basket correction on the basket image to obtain a corrected image; identifying display information of goods in the corrected image to obtain a product display recognition result, wherein the display information includes type, location, and quantity; and obtaining a display analysis result based on a preset standard display diagram and the product display recognition result.

[0004] In the display analysis method described above, optionally, obtaining an image of a basket for accommodating goods includes: obtaining a first basket image when a chest freezer door opening is sensed; obtaining a second basket image when a chest freezer door closing is sensed, wherein the shooting angles corresponding to the second basket image and the shooting angles corresponding to the first basket image are respectively arranged on opposite sides of the chest freezer; and splicing the first basket image and the second basket image to obtain the basket image.

[0005] In the display analysis method as described above, optionally, before the first basket image and the second basket image are spliced ​​together to obtain the basket image, the method further includes: determining a first similarity between the first basket image acquired this time and the first basket image acquired last time, and a second similarity between the second basket image acquired this time and the second basket image acquired last time; if both the first similarity and the second similarity are lower than a preset similarity threshold, executing the step of splicing the first basket image and the second basket image to obtain the basket image.

[0006] In the display analysis method as described above, optionally, performing basket detection and basket correction on the basket image to obtain a corrected image includes: performing basket detection on the basket image using a target detection algorithm to obtain a basket target image; and correcting the basket target image using an affine transformation algorithm to obtain a corrected image.

[0007] On the other hand, a display analysis device is provided, comprising: an acquisition module for acquiring an image of a basket for accommodating goods in a horizontal freezer; a first acquisition module for performing basket detection and basket correction on the basket image to obtain a corrected image; a second acquisition module for identifying display information of goods in the corrected image to obtain a product display identification result, wherein the display information includes type, location, and quantity; and a third acquisition module for obtaining a display analysis result based on a preset standard display diagram and the product display identification result.

[0008] In the display analysis device as described above, optionally, the acquisition module includes: a first acquisition unit, configured to acquire a first cargo basket image when sensing that the chest freezer door is open; a second acquisition unit, configured to acquire a second cargo basket image when sensing that the chest freezer door is closed, wherein the shooting angles corresponding to the second cargo basket image and the shooting angles corresponding to the first cargo basket image are respectively arranged on opposite sides of the chest freezer; and a splicing unit, configured to splice the first cargo basket image and the second cargo basket image to obtain the cargo basket image.

[0009] In the display analysis device as described above, optionally, before the splicing unit splices the first basket image and the second basket image to obtain the basket image, the acquisition module further includes: a determination unit, used to determine a first similarity between the first basket image acquired this time and the first basket image acquired last time, and a second similarity between the second basket image acquired this time and the second basket image acquired last time; and a jump unit, used to execute the step of splicing the first basket image and the second basket image to obtain the basket image if both the first similarity and the second similarity are lower than a preset similarity threshold.

[0010] In the display analysis device as described above, optionally, the first obtaining module includes: a first obtaining unit, used to perform basket detection on the basket image using a target detection algorithm to obtain a basket target image; and a second obtaining unit, used to correct the basket target image using an affine transformation algorithm to obtain a corrected image.

[0011] On the other hand, an electronic device is provided, comprising: a processor and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned display analysis method.

[0012] On the other hand, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned display analysis method.

[0013] The technical solution provided by the embodiment of the present invention has the following beneficial effects:

[0014] By acquiring an image of a basket for accommodating goods in a horizontal freezer, performing basket detection and basket correction on the basket image to obtain a corrected image, and then identifying the display information of the goods in the corrected image to obtain a product display recognition result, a display analysis result is obtained based on a preset standard display diagram and the product display recognition result. This solves the problems of high labor cost, low frequency and difficult quality assurance in manual inspections in existing display inspection methods, can promptly inform management personnel of the display status, improve management efficiency, and provide more and more accurate information for management decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic flow chart of a display analysis method provided by an embodiment of the present invention;

[0016] Figure 2 A schematic structural diagram of a display analysis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0018] See also Figure 1 , an embodiment of the present invention provides a display analysis method, which includes but is not limited to the following steps:

[0019] Step 101: Acquire an image of a basket for storing goods in a horizontal freezer.

[0020] There are usually multiple baskets in a horizontal freezer, and each basket contains different goods so that customers can choose according to their needs. As different customers make purchases, the display of goods in the horizontal freezer will change. In order to better inspect the display of goods in the horizontal freezer, this method adopts a display analysis method based on computer vision. In this method, it is necessary to obtain an image of the basket. When the basket is not empty, the basket image will contain goods. In actual applications, a camera can be installed on the horizontal freezer to capture the image of the basket. The trigger condition for image acquisition can be a timed trigger or a door opening and closing perception trigger. This embodiment does not specifically limit the content of the trigger condition. By setting the trigger condition, it is possible to obtain images by taking pictures, which can reduce the amount of image processing and reduce power consumption.

[0021] Specifically, a chest freezer typically has sliding doors arranged in pairs, for example, a left sliding door and a right sliding door. An image capture device, such as a camera, is provided on the inner surface of one door of the chest freezer. This allows the image capture device to move as the side door is pushed or pulled, thereby being located at different positions within the chest freezer and photographing baskets at different positions. In practical applications, an image capture device can be provided on the inner surface of the right sliding door. When the right sliding door is moved to the left and opened, the image capture device in the open position photographs the basket, obtaining a first basket image. The first basket image corresponds to the basket image on the left side of the chest freezer. When the right sliding door is moved to the right and closed, the image capture device in the closed position photographs the basket, obtaining a second basket image. The second basket image corresponds to the basket image on the right side of the chest freezer. The combination of the second basket image and the first basket image covers all basket images within the chest freezer. Because the first and second basket images alone cannot fully represent the baskets within the chest freezer, they must be spliced ​​together to produce a complete basket image. This image represents the complete freezer basket image. During splicing, the overlapping portions between the first and second basket images are removed, resulting in a complete freezer basket image (or display image). Before splicing, to improve recognition accuracy, poor quality images can be removed by assessing image characteristics such as brightness and clarity.

[0022] The opening and closing of the cabinet door can be determined by sensing using a sensor, that is, by capturing images based on motion information, which can reduce power consumption and repeated recognition. The sensor can be a pyroelectric sensor and a ranging sensor. The pyroelectric sensor is used to sense changes in the thermal radiation of the chest freezer. When the cabinet door of the chest freezer is opened or closed, the thermal radiation changes. The ranging sensor is used to sense the distance between the image acquisition device and the inner wall (such as the right wall or the left wall) in the width direction of the chest freezer. When a customer pushes and pulls the cabinet door to open the chest freezer while shopping, the pyroelectric sensor will sense the change in thermal radiation. When the distance change measured by the ranging sensor is greater than a preset distance threshold, it indicates that the cabinet door on one side where the image acquisition device is installed is pushed and pulled to the other side. At this time, the image acquisition device starts to capture the image inside the chest freezer, which is called the first basket image. The image corresponds to the cabinet door on the other side. When the customer finishes shopping and pushes or pulls the door to close the chest freezer, the pyroelectric sensor senses a change in thermal radiation. When the distance change measured by the ranging sensor is greater than a preset distance threshold, it indicates that the door on the side where the image acquisition device is installed has been reset. At this time, the image acquisition device starts to capture an image inside the chest freezer, which is called the second basket image. This image corresponds to the door on that side.

[0023] In actual applications, it is possible that a customer opens the door of a chest freezer but does not make any purchases. To reduce the image processing workload, the method further includes the following steps:

[0024] Determine a first similarity between the first cargo basket image acquired this time and the first cargo basket image acquired last time, and a second similarity between the second cargo basket image acquired this time and the second cargo basket image acquired last time. If both the first similarity and the second similarity are lower than a preset similarity threshold, execute a step of splicing the first cargo basket image and the second cargo basket image to obtain a cargo basket image. If both the first similarity and the second similarity are not lower than a preset similarity threshold, do not execute the subsequent step, that is, calculate the similarity between the image taken this time and the image taken last time by comparison. If the similarity is high, it is considered that the display has not changed within the time period (the time corresponding to the last photographed image and the time corresponding to the current photographed image), and no processing is performed in the next stage.

[0025] Step 102: Perform basket detection and basket correction on the basket image to obtain a corrected image.

[0026] The basket is used as the target. A target detection algorithm is used to detect the basket image and locate each basket region. This generates a target basket image containing only the basket region, which typically contains merchandise. This embodiment does not limit the specifics of the target detection algorithm; it may employ a deep neural network-based target detection algorithm. The image acquisition device is positioned near the inner wall (rear wall) along the length of the chest freezer, with the camera angled downward. Due to the camera's angle, merchandise in the captured image may be distorted due to distortion, significantly deviating from its true proportions. This increases the difficulty and reduces the accuracy of merchandise recognition. To address this issue, the target basket image is corrected to generate a corrected image that restores the distorted basket (or merchandise) to its true proportions. The correction method employed is an affine transformation algorithm. In practical applications, the shape of the basket in the image may not be a rectangle. Affine transformation can be used to obtain a physically proportional rectangle.

[0027] Step 103 : Recognize the display information of the commodity on the corrected image to obtain a commodity display recognition result.

[0028] Product display information includes: product type, product location, and product quantity. When identifying product types, image classification algorithms can be used. Because this type of algorithm is difficult to expand (adding new products requires a long period of time) and has a large data requirement, image retrieval algorithms can be used when identifying product types. This algorithm only requires a small amount of data and can easily add new products. Image retrieval algorithms typically extract a feature representation corresponding to the image to be identified, which is generally a vector. The feature vector is then searched for similar images in a preset product library using Euclidean distance or cosine distance. New products only require a small number of images to be registered in the product library to support retrieval and identification.

[0029] The display location of goods is used to represent the baskets in which they are stored; that is, the specific items contained in each basket are determined. After identifying the baskets and the types of goods, the display location of the goods can be determined. The quantity of goods, or the inventory level, can be expressed as a percentage, for example, the percentage of goods remaining in the basket compared to the total number of goods when the basket is fully stocked. In applications, several quantity thresholds can be defined. When there are four quantity thresholds, the four quantity thresholds can be: 25%, 50%, 75%, and 100%. The corresponding levels for each quantity threshold are: 0-25%, 26%-50%, 51%-75%, and 76%-100%, respectively. The levels can be named: Empty, Out of Stock, Half Full, and Full. The numerical values ​​of the quantity thresholds can be determined based on specific operational conditions and are not specifically defined in this embodiment. Deep learning algorithms can be used to identify the quantity of goods.

[0030] Step 104: Obtain display analysis results based on the preset standard display diagram and the product display recognition results.

[0031] The standard planogram is pre-determined and includes preset product placements and quantities. By comparing the product display identification results with the standard planogram, a display analysis is generated, including whether the product placement meets the standards (i.e., whether the current baskets contain the correct items) and whether inventory is sufficient. If the display analysis indicates an incorrect display or insufficient inventory, management is promptly notified. The inventory level (percentage) of each basket is identified, and if the inventory level falls below a preset value, the item is considered out of stock.

[0032] By acquiring an image of a basket for accommodating goods, performing basket detection and basket correction on the basket image to obtain a corrected image, and identifying the display information of the goods in the corrected image to obtain a product display recognition result, a display analysis result is obtained based on a preset standard display diagram and the product display recognition result. This solves the problems of high labor cost, low frequency and difficult quality assurance in manual inspection in existing display inspection methods, can promptly inform management personnel of the status of the display, improve management efficiency, and provide more and more accurate information for management decisions.

[0033] See also Figure 2 The embodiment of the present invention provides a display analysis device for executing the display analysis method of the above embodiment, which includes: an acquisition module 201, a first obtaining module 202, a second obtaining module 203 and a third obtaining module 204.

[0034] Specifically, the acquisition module 201 is used to acquire an image of a basket containing merchandise within a chest freezer. The first acquisition module 202 is used to perform basket detection and basket correction on the basket image to obtain a corrected image. The second acquisition module 203 is used to identify merchandise display information in the corrected image to obtain a merchandise display recognition result. The display information includes type, location, and quantity. The third acquisition module 204 is used to obtain a display analysis result based on a preset standard planogram and the merchandise display recognition result.

[0035] Optionally, the acquisition module 201 includes a first acquisition unit, a second acquisition unit, and a splicing unit. The first acquisition unit is configured to acquire a first basket image when sensing that the chest freezer door is open. The second acquisition unit is configured to acquire a second basket image when sensing that the chest freezer door is closed, with the second basket image and the first basket image corresponding to shooting angles located on opposite sides of the chest freezer. The splicing unit is configured to splice the first basket image and the second basket image to generate a basket image.

[0036] Optionally, before the splicing unit splices the first and second basket images to obtain the basket image, the acquisition module 201 further includes a determination unit and a jump unit. The determination unit is configured to determine a first similarity between the currently acquired first basket image and the previously acquired first basket image, and a second similarity between the currently acquired second basket image and the previously acquired second basket image. The adjustment unit is configured to execute the step of splicing the first and second basket images to obtain the basket image if both the first and second similarities are below a preset similarity threshold.

[0037] Optionally, the first obtaining module 202 includes a first obtaining unit and a second obtaining unit. The first obtaining unit is configured to perform basket detection on the basket image using a target detection algorithm to obtain a basket target image. The second obtaining unit is configured to correct the basket target image using an affine transformation algorithm to obtain a corrected image.

[0038] It should be noted that the display analysis device provided in the above embodiment is merely illustrated by the division of the aforementioned functional modules when performing display analysis. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the display analysis device provided in the above embodiment and the display analysis method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be detailed here.

[0039] One embodiment of the present invention provides an electronic device, comprising: a memory and a processor. The processor is connected to the memory and is configured to execute the above-mentioned display analysis method based on the instructions stored in the memory. The number of processors can be one or more, and the processor can be single-core or multi-core. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory can be an example of the computer-readable medium described below.

[0040] One embodiment of the present invention provides a computer-readable storage medium having stored thereon at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the above-mentioned display analysis method. Computer-readable storage media include: permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0041] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A display analysis method, characterized in that: include: Acquire an image of a basket for accommodating goods in a chest freezer, wherein an image acquisition device is provided on an inner surface of a door of one side of the chest freezer; The obtaining of an image of a basket for containing goods in a horizontal freezer includes: When it is sensed that the chest freezer door is opened, and a change in the distance between the image acquisition device and the inner side wall of the chest freezer in the width direction measured by the distance measuring sensor is greater than a preset distance threshold, it is determined that one side of the chest freezer door where the image acquisition device is located has been pushed or pulled to the other side, and a first cargo basket image is acquired, where the first cargo basket image corresponds to the other side of the chest freezer door; When the chest freezer door is sensed to be closed and a change in the distance between the image acquisition device and the inner side wall of the chest freezer in the width direction measured by the ranging sensor is less than a preset distance threshold, it is determined that the door of the chest freezer on which the image acquisition device is provided is reset, and a second cargo basket image is acquired, the second cargo basket image corresponding to the door of the chest freezer, and the shooting angle corresponding to the second cargo basket image and the shooting angle corresponding to the first cargo basket image are respectively arranged on opposite sides of the chest freezer; splicing the first cargo basket image and the second cargo basket image to obtain the cargo basket image; Performing cargo basket detection and cargo basket correction on the cargo basket image to obtain a corrected image; Identifying product display information on the corrected image to obtain a product display identification result, wherein the display information includes: type, location, and quantity; The display analysis result is obtained according to the preset standard display diagram and the product display identification result.

2. The display analysis method according to claim 1, characterized in that: Before the first cargo basket image and the second cargo basket image are stitched together to obtain the cargo basket image, the method further includes: Determining a first similarity between the first cargo basket image acquired this time and the first cargo basket image acquired last time, and determining a second similarity between the second cargo basket image acquired this time and the second cargo basket image acquired last time; If both the first similarity and the second similarity are lower than a preset similarity threshold, the step of splicing the first cargo basket image and the second cargo basket image is performed to obtain the cargo basket image.

3. The display analysis method according to claim 1, characterized in that: The performing cargo basket detection and cargo basket correction on the cargo basket image to obtain a corrected image includes: Performing basket detection on the basket image using a target detection algorithm to obtain a basket target image; The cargo basket target image is corrected using an affine transformation algorithm to obtain a corrected image.

4. A display analysis device, characterized in that: include: an acquisition module, configured to acquire an image of a basket for accommodating goods in a chest freezer, wherein an image acquisition device is provided on an inner surface of a door of one side of the chest freezer; The acquisition module is specifically configured to, when sensing that the chest freezer door is opened and a change in the distance between the image acquisition device and the inner side wall of the chest freezer in the width direction measured by the distance measuring sensor is greater than a preset distance threshold, determine that one side of the chest freezer door on which the image acquisition device is provided is pushed or pulled to the other side, and acquire a first cargo basket image, wherein the first cargo basket image corresponds to the other side of the chest freezer door; When the chest freezer door is sensed to be closed and a change in the distance between the image acquisition device and the inner side wall of the chest freezer in the width direction measured by the ranging sensor is less than a preset distance threshold, it is determined that the door of the chest freezer on which the image acquisition device is provided is reset, and a second cargo basket image is acquired, the second cargo basket image corresponding to the door of the chest freezer, and the shooting angle corresponding to the second cargo basket image and the shooting angle corresponding to the first cargo basket image are respectively arranged on opposite sides of the chest freezer; splicing the first cargo basket image and the second cargo basket image to obtain the cargo basket image; A first obtaining module is used to perform cargo basket detection and cargo basket correction on the cargo basket image to obtain a corrected image; The second obtaining module performs commodity display information recognition on the corrected image to obtain commodity display recognition results, wherein the display information includes: type, location and quantity; The third obtaining module obtains the display analysis result according to the preset standard display diagram and the product display recognition result.

5. The display analysis device according to claim 4, characterized in that: Before the splicing unit splices the first cargo basket image and the second cargo basket image to obtain the cargo basket image, the acquisition module further includes: a determining unit, configured to determine a first similarity between the first cargo basket image acquired this time and the first cargo basket image acquired last time, and a second similarity between the second cargo basket image acquired this time and the second cargo basket image acquired last time; A jump unit is configured to execute the step of splicing the first cargo basket image and the second cargo basket image to obtain the cargo basket image if both the first similarity and the second similarity are lower than a preset similarity threshold.

6. The display analysis device according to claim 4, characterized in that: The first obtaining module includes: a first obtaining unit, configured to perform basket detection on the basket image using a target detection algorithm to obtain a basket target image; The second obtaining unit is used to correct the cargo basket target image using an affine transformation algorithm to obtain a corrected image.

7. An electronic device, characterized in that: The electronic device includes: a processor and a memory for storing executable instructions of the processor; The processor is configured to execute the display analysis method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the display analysis method according to any one of claims 1 to 3.

Images