Out-of-stock commodity identification method and device, storage medium and electronic device

Abstract

The embodiment of the invention provides an out-of-stock commodity identification method and device, a storage medium and an electronic device.The method comprises the steps that a first image obtained by shooting a target area according to a preset time interval is obtained, the target area is used for containing a first tray, the first tray is used for containing commodities, and the first tray is used for containing the commodities; a plurality of first trays are arranged; under the condition that the first image meets a preset condition, determining a first area of a commodity included in a first tray included in the first image; determining a second area less than a predetermined threshold included in the first area; and executing an alarm operation under the condition of determining that the first tray is lack of the commodity based on the second area. Through the method and the device, the problem of low commodity stockout determination efficiency is solved, and the effect of improving the commodity stockout determination efficiency is further achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computers, and more particularly, to a method, apparatus, storage medium, and electronic device for identifying out-of-stock items. Background Art

[0002] In the related art, due to the large variety, different shapes, and non-obvious features of items, out-of-stock monitoring is somewhat difficult. Traditional out-of-stock monitoring methods usually rely on manual inspections to determine whether an item is out of stock.

[0003] It can be seen that there is a problem of low efficiency in determining out-of-stock items in the related art.

[0004] To address the above problems existing in the related art, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present invention provide a method, apparatus, storage medium, and electronic device for identifying out-of-stock items, so as to at least address the problem of low efficiency in determining out-of-stock items in the related art.

[0006] According to an embodiment of the present invention, a method for identifying out-of-stock items is provided, including: obtaining a first image captured of a target area at a predetermined time interval, where the target area is for placing a first tray for holding items, and the number of the first trays is multiple; determining a first area of the items included in the first trays included in the first image when the first image meets a predetermined condition; determining a second area smaller than a predetermined threshold in the first area; and performing an alarm operation when it is determined that an item is missing from the first tray based on the second area.

[0007] In an exemplary embodiment, before performing the warning operation, the method further includes one of the following: determining a third area included in the second area and belonging to the same target tray, and determining that there is a shortage of goods in the target tray when the number of the third areas is greater than or equal to a first preset number, where the target tray is any one of the trays included in the first tray; determining a third area included in the second area and belonging to the same target tray, dividing the third area according to the shooting angle to obtain area categories, and determining that there is a shortage of goods in the target tray when the number of area types is greater than a second preset number, where the target tray is any one of the trays included in the first tray; determining a third area included in the second area and belonging to the same target tray, dividing the third area according to the shooting angle to obtain area categories, and determining that there is a shortage of goods in the target tray when the number of area types is the same as the number of shooting angles, where the target tray is any one of the trays included in the first tray.

[0008] In an exemplary embodiment, the method further includes: when the first image includes a second image that does not meet the predetermined condition, controlling the imaging device to capture the target area to obtain a third image; and replacing the third image included in the first image with the third image.

[0009] In an exemplary embodiment, before determining the first area of the goods included in the first tray included in the first image, the method further includes: determining the illumination intensity of the first image; performing filtering processing on the first image according to the illumination intensity to obtain a fourth image; enhancing the contrast of the fourth image to obtain a fifth image; and determining that the first image meets the predetermined condition when the fifth image meets the predetermined condition.

[0010] In an exemplary embodiment, performing the warning operation includes: determining the out-of-stock time when it is determined that there is a shortage of goods in the first tray based on the second area; determining the time difference between the current time and the out-of-stock time; displaying a warning message on a target interface when the time difference is greater than a first threshold; sending the warning message to a target account when the time difference is greater than a second threshold; and voice-playing the warning message when the time difference is greater than a third threshold; where the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

[0011] In an exemplary embodiment, performing an alarm operation includes: identifying a target barcode of a first tray included in the first picture to obtain barcode information, where the target barcode is placed in a target area of the first tray, and the size of the target barcode and the size of the first tray satisfy a preset proportional relationship; identifying a target code in the first tray included in the first picture to obtain code information, where the target code is placed in the target area of the first tray; determining a target commodity placed in the first tray based on the barcode information and / or the code information; generating an alarm information based on the commodity information of the target commodity; and performing the alarm operation based on the alarm information.

[0012] In an exemplary embodiment, determining a first area of a commodity included in the first tray included in the first image includes: using a multimodal model to determine the first area of the commodity included in the first tray included in the first image.

[0013] According to another embodiment of the present invention, there is provided an identification device for out-of-stock commodities, including: an acquisition module, configured to acquire a first image obtained by photographing a target area at a predetermined time interval, where the target area is used to place a first tray, the first tray is used to hold commodities, and the number of the first trays is multiple; a first determination module, configured to determine a first area of a commodity included in the first tray included in the first image when the first image meets a predetermined condition; a second determination module, configured to determine a second area smaller than a predetermined threshold included in the first area; and an execution module, configured to perform an alarm operation when it is determined that the first tray lacks commodities based on the second area.

[0014] According to still another embodiment of the present invention, there is further provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0015] According to still another embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to still another embodiment of the present invention, there is further provided a computer program product, including a computer program, where the steps of the methods described in various embodiments of the present application are implemented when the computer program is executed by a processor.

[0017] Through the present invention, a first image obtained by photographing a target area at a predetermined time interval can be acquired, where the target area can be used to place a plurality of first trays for holding commodities. When the first image meets a predetermined condition, the first area of the commodities in the first tray in the first image can be determined, as well as a second area in the first area that is smaller than a predetermined threshold. Whether there is a shortage of commodities in the first tray can be determined according to the second area, and in the case where a shortage of commodities is determined, an alarm operation can be performed. Since the present invention can confirm the second area in the first area of the commodities in the first tray that is smaller than the predetermined threshold through the first image obtained by photographing at a predetermined time interval, and then determine the first tray with a shortage of commodities based on the second area, the problem of low efficiency in determining out-of-stock commodities in the related art can be solved, and the effect of improving the efficiency of determining out-of-stock commodities can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a hardware structure block diagram of a mobile terminal for a method of identifying out-of-stock commodities according to an embodiment of the present invention;

[0019] Figure 2 is a flowchart of a method of identifying out-of-stock commodities according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a target area according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a target barcode of a first tray according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a method of identifying out-of-stock commodities according to a specific embodiment of the present invention;

[0023] Figure 6 is a structure block diagram of a device for identifying out-of-stock commodities according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The embodiments of the present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal for a method of identifying out-of-stock commodities according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more (Figure 1 Only one processor 102 is shown (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only illustrative and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than those shown in Figure 1 or have a different configuration from that shown in Figure 1 shown.

[0027] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the method for identifying out-of-stock items in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely provided with respect to the processor 102, and these remote memories can be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for identifying out-of-stock items is provided. Figure 2 is a flowchart of the method for identifying out-of-stock items according to the embodiments of the present invention, as shown in Figure 2 shown, and the process includes the following steps:

[0030] Step S202, obtain a first image obtained by photographing a target area at a predetermined time interval, where the target area is used to place a first tray, the first tray is used to hold items, and the number of the first trays is multiple;

[0031] Step S204, when the first image meets a predetermined condition, determine a first area of the commodities included in the first tray included in the first image;

[0032] Step S206, determine a second area smaller than a predetermined threshold included in the first area;

[0033] Step S208, when it is determined that a commodity is missing from the first tray based on the second area, perform an alarm operation.

[0034] In the above embodiment, a monitoring device can be used to capture the target area in real time at a predetermined time interval to obtain a first image, that is, the first image can be multiple frames of images, which can be 3 frames of images, or can also be 5 frames of images, etc. Among them, the predetermined time interval can be a fixed time interval (for example: every 30s, 1min or 30min, etc.), or can also be a dynamic time interval (for example: set according to characteristic factors such as sales peak and valley values, pedestrian flow, holidays, etc., the shooting interval can be shortened during the sales peak period and extended during the low peak period), or can also be triggered to reduce or increase the time interval under certain specific events (for example: detecting that an employee is restocking in the target area, a customer stays in the target area for a long time, etc.). The target area can be understood as an area where multiple first trays are placed. Figure 3 is a schematic diagram of the target area according to an embodiment of the present invention, as Figure 3 shown, the target area includes multiple first trays, and each first tray contains a commodity.

[0035] In the above embodiments, after the first image is obtained, not all of the first images will be used for analysis. First, it can be determined whether the first image obtained by photographing the target area meets a predetermined condition, and only the first image that meets the preset condition can be used for subsequent analysis. Among them, the preset condition can be image clarity, lighting condition, perspective stability, image quality, and background and interference removal, etc. In the first image that meets the first preset condition, the first area occupied by the commodities placed in the first tray can be determined. After calculating the first area, the second area smaller than a predetermined threshold included in the first area can be further analyzed. Among them, the predetermined threshold can be set based on business requirements, and can be a commodity coverage threshold (that is, the commodity area in the first tray is smaller than this threshold), a commodity contour integrity threshold (that is, the commodity contour is incomplete or broken by more than a certain ratio), or a commodity height or volume change threshold (that is, the change range of the commodity height or volume exceeds this threshold), a commodity weight threshold (that is, the weight of the commodity is less than this threshold), etc. Example: When the preset threshold is the commodity coverage rate, the commodities with a commodity area ratio less than 50% in the first tray can be determined as out of stock. It should be noted that 50% here is only an exemplary illustration, and the present invention is not limited thereto. When it is determined that there are missing commodities in the first tray through the second area (as long as there is one first tray lacking commodities, it is considered that the target area is out of stock), an alarm operation can be executed to notify relevant personnel to replenish the stock or conduct an inspection.

[0036] Through the present invention, the first image obtained by photographing the target area at a predetermined time interval can be obtained, where the target area can be used to place a plurality of first trays for holding commodities. When the first image meets the predetermined condition, the first area of the commodities in the first tray in the first image and the second area smaller than the predetermined threshold in the first area can be determined. Whether there are missing commodities in the first tray can be determined according to the second area, and when it is determined that there are missing commodities, the alarm operation can be executed. Since the present invention can confirm the second area smaller than the predetermined threshold of the first area of the commodities in the first tray through the first image obtained by photographing at a predetermined time interval, and then determine the first tray with missing commodities based on the second area, the problem of low efficiency in determining out-of-stock commodities in the related art can be solved, and the effect of improving the efficiency of determining out-of-stock commodities can be achieved.

[0037] Optionally, the execution subject of the above steps can be a background processor, a terminal, or a server, or other devices with similar processing capabilities, or a machine at least integrated with an image acquisition device and a data processing device. Among them, the image acquisition device can include a graphic acquisition module such as a camera, and the data processing device can include a terminal such as a computer or a mobile phone, but is not limited thereto.

[0038] In an exemplary embodiment, before performing the alarm operation, the method further includes one of the following: determining a third area belonging to the same target tray included in the second area, and determining that there is a shortage of goods in the target tray when the number of the third areas is greater than or equal to a first preset number, where the target tray is any one of the trays included in the first tray; determining a third area belonging to the same target tray included in the second area, dividing the third area according to the shooting angle to obtain area categories, and determining that there is a shortage of goods in the target tray when the number of area types is greater than a second preset number, where the target tray is any one of the trays included in the first tray; determining a third area belonging to the same target tray included in the second area, dividing the third area according to the shooting angle to obtain area categories, and determining that there is a shortage of goods in the target tray when the number of area types is the same as the number of shooting angles, where the target tray is any one of the trays included in the first tray.

[0039] In the above embodiment, in order to improve the accuracy and reliability of out-of-stock judgment, the second tray with missing goods can be subjected to timing inspection to avoid false alarms caused by accidental occlusion or poor image quality, and the alarm operation is performed when the inspection result is met. That is, after it is determined in the second area that there is indeed a shortage of goods in the first tray, the tray with defective goods in the first tray can be further inspected: First, the third area of the same tray (i.e., the above-mentioned target tray) in multiple first images can be determined. When the third area of a certain target tray is included in multiple first images, the number of the third areas of the target tray can be confirmed. When the above number is greater than or equal to the first preset number, it can be determined that there is a shortage of goods in this target tray. Among them, the first preset number can be determined according to the number of first images, and can be one-half or three-fourths of the number of first images, or 3 frames of first images, 5 frames of first images, etc. Example: When there are 4 frames of first images, the situation where out-of-stock goods are detected in the target tray for 3 consecutive frames can be determined as out-of-stock, or the target tray with 2 occurrences (i.e., the number of the third areas of the above-mentioned target tray) can be determined as out-of-stock, so as to perform the alarm operation.

[0040] In the above embodiments, in addition to performing timing verification, cross-perspective verification can also be carried out to avoid misjudgment caused by the limitations of a single perspective. That is, for the area covered by multiple camera devices, the images in the third area at the same shooting angle can be divided from the images at different shooting angles, and multiple area categories can be obtained. When the number of area types (different out-of-stock shapes or positions) is greater than the second preset number, it can be determined that there are missing items in this target tray. The second preset number can be determined according to the number of different shooting angles. For example, if there are 5 different shooting angles, a target tray with 3 area types can be determined as out of stock. In this embodiment, when the number of area types is the same as the number of shooting angles (that is, out-of-stock situations are detected at each shooting angle), it can be explained that there are out-of-stock situations in the target tray no matter from which angle it is observed, and thus an alarm operation is executed.

[0041] In an exemplary embodiment, the method further includes: when the first image includes a second image that does not meet the predetermined condition, controlling the camera device to capture the target area to obtain a third image; and replacing the third image included in the first image with the third image.

[0042] In the above embodiments, after obtaining the first image, conditions for occlusion and light change can be added to the model. First, it is determined whether it contains a second image that does not meet the predetermined condition. If it is detected that there is a second image in the first image that does not meet the predetermined condition, the angle, light, or focal length of the camera device can be adjusted, and then the camera device is controlled to capture the target area again to obtain a new third image. Once a third image with higher quality is obtained, it can be used to replace the second image in the first image that does not meet the predetermined condition, which can ensure that all images used for subsequent analysis meet sufficient quality standards, thereby improving the accuracy and stability of commodity out-of-stock detection.

[0043] In an exemplary embodiment, before determining the first area of the commodity included in the first tray included in the first image, the method further includes: determining the light intensity of the first image; performing filtering processing on the first image according to the light intensity to obtain a fourth image; enhancing the contrast of the fourth image to obtain a fifth image; and when the fifth image meets the predetermined condition, determining that the first image meets the predetermined condition.

[0044] In the above embodiments, to ensure the image quality, the first image can be preprocessed before analyzing it: First, the brightness distribution of the pixels in the first image can be judged to determine the average illumination intensity of the image. An appropriate filtering algorithm can be selected to adaptively denoise according to the detected illumination intensity value. For example, the Non-local Means algorithm can automatically adjust the filtering parameters according to the illumination intensity. If the illumination is too strong, the Gaussian blur filtering algorithm can also be used; if the illumination is insufficient, the sharpening filter that enhances details can also be used. Through the filtering process, the noise in the image can be removed, the clarity and detail visibility of the image can be improved, and thus a higher-quality fourth image can be obtained.

[0045] In the above embodiments, for the denoised fourth image, contrast enhancement can be performed through various algorithms, such as histogram equalization, CLAHE (Contrast Limited Adaptive Histogram Equalization), etc. The enhancement of the contrast can highlight the features of the commodities in the image, and a fifth image that is more suitable for recognition and analysis can be obtained. If the fifth image after filtering and contrast enhancement meets the predetermined conditions, it can be considered that the original first image is suitable for subsequent recognition and analysis after appropriate preprocessing. Therefore, the fifth image can be directly used for the detection and judgment of out-of-stock commodities without discarding or reshooting the first image. By adaptively adjusting the parameters of filtering and contrast enhancement, the image quality problems under different illumination conditions can be overcome, and the robustness and accuracy of recognition can be improved. Only on the premise of ensuring the image quality will the system make a judgment on out-of-stock commodities, thus avoiding misrecognition or missed recognition caused by poor image quality.

[0046] In an exemplary embodiment, performing the warning operation includes: determining the out-of-stock time when a commodity is missing in the first tray based on the second area; determining the time difference between the current time and the out-of-stock time; displaying a warning message on a target interface when the time difference is greater than a first threshold; sending the warning message to a target account when the time difference is greater than a second threshold; and voice-playing the warning message when the time difference is greater than a third threshold; where the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

[0047] In the above embodiments, since directly displaying reminders in the management software is a basic operation, but the relevant employees cannot keep staring at the management software all the time. To avoid ignoring important warning messages, different hierarchical warning protocols can be set according to the out-of-stock time of the commodities. The out-of-stock time of the out-of-stock commodities in the first tray determined by the second area can be recorded and compared with the current server time to calculate the time difference from when the commodity starts to be out of stock to the current time. Three different thresholds can be set for different time differences to trigger warning operations at different levels. Among them, the first threshold can be 30 minutes, 45 minutes, etc., the second threshold can be 60 minutes, 80 minutes, etc., and the third threshold can be 120 minutes, 150 minutes, etc. The present invention does not limit this.

[0048] In the above embodiments, if the time difference is greater than the first threshold, out-of-stock warning information can be displayed in the target interface to remind relevant personnel to check and confirm the out-of-stock status of the tray. If the time difference is greater than the second threshold, the out-of-stock warning information can be further sent to the specified target account by means of SMS, email, APP push, etc., to ensure that even if the operator does not immediately view the interface, the out-of-stock notice can be received in time. If the time difference is greater than the third threshold, the voice broadcast function can be activated to play the out-of-stock warning information through the in-store broadcast system to ensure that everyone can realize the severity of the out-of-stock situation and take emergency replenishment measures. Through the hierarchical warning strategy, different intensity reminder measures can be taken according to the length of the out-of-stock time, which not only avoids the interference caused by frequent warnings but also ensures that relevant personnel can be notified in time during long-term out-of-stock situations, avoiding omissions in inventory management and deterioration of the customer experience.

[0049] In an exemplary embodiment, performing the warning operation includes: identifying the target barcode of the first tray included in the first picture to obtain barcode information, wherein the target barcode is placed in the target area of the first tray, and the size of the target barcode and the size of the first tray satisfy a preset proportional relationship; identifying the target code in the first tray included in the first picture to obtain code information, wherein the target code is placed in the target area of the first tray; determining the target commodity placed in the first tray based on the barcode information and / or the code information; generating warning information based on the commodity information of the target commodity; and performing the warning operation based on the warning information.

[0050] In the above embodiments, each first tray corresponds to a unique target barcode. The target barcode is associated with the bulk goods in the first tray one-to-one, and both the target barcode and the corresponding commodity information can be stored in the commodity information database, which can solve the problem that it is difficult to identify the out-of-stock of bulk goods due to the limitations of hardware shooting conditions and the unclear characteristics of bulk goods. Among them, the target barcode can be a handwritten or printed digital code and can be placed at an obvious position on the appearance of the first tray. Figure 4 is a schematic diagram of the target barcode of the first tray according to an embodiment of the present invention. As Figure 4 shown, the target barcode can be placed on the upper edge of the front side edge, the corner of the tray, the side of the tray of the first tray, and can also be a signboard or a standing sign at one side or corner of the tray, the commodity packaging, etc. Example: It can be 5 cm from the front side edge of the tray in the direction facing the aisle and the camera device, and can also be on the commodity packaging of some packaged goods, but not limited thereto. The target barcode can be generated according to the target area code (2 digits), the shelf number (3 digits), and the tray serial number (2 digits). Example: AB00105 represents the 5th tray of shelf 001 in group B of area A. In addition, the coding size and the tray size satisfy a preset proportional relationship. Among them, the preset relationship can be the visual recognition principle of d (coding length) ≥ 0.05L (L is the length of the long side of the tray). The commodity information database can be managed by relevant personnel by scanning the code with a mobile terminal, that is, relevant personnel can use a mobile terminal (example: tablet computer, mobile phone) to scan the target barcode, select the SKU (Stock Keeping Unit) identifier of the commodity and automatically write it into the commodity information database. When the SKU is switched, it can be forced to require re-scanning for confirmation, and after the end of daily business, the data of the ERP (Enterprise Resource Planning) system can be automatically compared to mark abnormal associated items. In addition, a dynamic verification system can also be set: by adding a weight sensor to detect the total weight of the tray and performing a ±10% threshold verification with the standard weight of the commodity.

[0051] In the above embodiments, a dual identifier may also be set to prevent the target barcode from being damaged or unrecognizable. That is, in addition to the target barcode, a target code (e.g., QR code, two-dimensional code, etc.) with a contrast ratio of ≥ 70% to the chassis background color can be printed synchronously as a backup identification. After the target barcode or target code is recognized, barcode information or code information can be obtained. By matching the barcode information or code information with the commodity information database, the specific target commodity placed in the first tray can be determined. Once the missing commodity (i.e., the above commodity information) is confirmed, an alarm message can be generated and different levels of alarm operations can be performed according to different time differences. Among them, the alarm message may include "Tray No. x", "Out of stock", or "In stock". The execution of the alarm operation ensures that the out-of-stock warning can be noticed in time, so as to quickly take replenishment measures, reduce the loss of sales opportunities and improve customer satisfaction.

[0052] In an exemplary embodiment, determining the first area of the commodity included in the first tray included in the first image includes: using a multimodal model to determine the first area of the commodity included in the first tray included in the first image.

[0053] In the above embodiments, a large model that supports multimodal input (such as images, texts, etc.) can be selected and trained for out-of-stock recognition. For example, multimodal large language models (MLLMs) can use industry closed-source models or open-source models, and basic models (such as Qianwen QWEN-VL, Zhipu GLM-4V, etc.) may not require custom training.

[0054] In the above embodiments, prompt words can be set first, and the template of the prompt words is as follows:

[0055] Bash

[0056] # Role

[0057] You are a retail replenishment reminder assistant. You will receive pictures of the display of bulk goods, and the bulk goods are placed in trays. You need to judge whether the goods in the tray are out of stock.

[0058] # Judgment rule

[0059] When judging out of stock, please check the numbers of all visible trays in the picture and the overall vacant part.

[0060] The standard for vacancy is: the goods in a single tray are not fully filled, and the area of the tray bottom plate exposed exceeds 50% of a single tray. (The out-of-stock business standard is set according to business requirements. For the impact on the recognition result after the standard is adjusted, corresponding adjustments and explanations are made through prompt words to restrict the recognition and judgment of the large model)

[0061] Judge each tray. If more than 1 tray is found to be vacant, it is considered that the shelf is out of stock; otherwise, it is considered that the shelf is not out of stock.

[0062] # Output requirements

[0063] If the shelf is out of stock, return 'Tray No. x' and 'Out of stock'.

[0064] If the shelf is not out of stock, only return 'Not out of stock'.

[0065] # Output format

[0066] "Tray No. x""Out of stock"

[0067] "Not out of stock"

[0068] In the above embodiment, the responsibilities of the model role are defined as a "Retail Replenishment Reminder Assistant". After receiving the display image of unpackaged goods placed in trays (i.e., the above first image), it is necessary to judge whether the goods in the trays are out of stock. When judging out of stock, the target barcodes, target codes, and overall vacant parts (i.e., the above first area) of all the first trays in the first image can be checked. If the goods in a certain first tray are not fully filled and the area of the tray bottom plate exposed exceeds the predetermined threshold of a single tray, it is considered that the above first tray is out of stock (Judge each tray. If more than 1 tray is found to be vacant, it is considered out of stock; otherwise, it is considered that the shelf is not out of stock). Among them, the predetermined threshold can be changed by adjusting the prompt words, which can restrict the recognition and judgment of the model. If there is a first tray out of stock, 'Tray No. x' and 'Out of stock' can be output; if there is no first tray out of stock, "Not out of stock" can be output.

[0070] In the above embodiment, for the situation where there is a deviation between the recognition result of the model and the actual scenario, these data can be marked as "suspected false alarms" and relevant information can be recorded, such as: tray number, capture time, recognized product information, etc. In order to enable the model to learn from limited labeled data and improve the recognition logic, the few-shot method can be adopted. As the input of few-shot learning, it is used for fine-tuning or optimization of the model. By adding new labeled data in real time or regularly, the model can continuously learn and adapt to the changes in the retail environment, improving the accuracy and robustness of recognition. The recognition result of the optimized model will be fed back to the system in real time for subsequent out-of-stock detection. At the same time, the review result will also be updated to the product information database to ensure that the product information in the system is the latest and most accurate.

[0071] The method for identifying out-of-stock products will be described below in conjunction with specific embodiments. Figure 5 It is a schematic diagram of the method for identifying out-of-stock products according to a specific embodiment of the present invention. The method includes:

[0072] Step S502: The unique code of the tray corresponds to the commodity.

[0073] Step S504: Start.

[0074] Step S506: The camera captures frames in real time and performs image preprocessing.

[0075] Step S508: Identify using the multi-modal large model MLLMs.

[0076] Step S510: Descriptions such as prompts and out-of-stock business standards.

[0077] Step S512: Is there a shortage? If yes, execute Step S514; if no, execute Step S516.

[0078] Step S514: Push out-of-stock reminders.

[0079] Step S516: End.

[0080] In the above embodiments, out-of-stock detection is performed through multi-modal recognition technology, which can more accurately determine whether there is a shortage of commodities in the tray, reducing false alarms and missed reports. Prompting for replenishment based on the commodity information corresponding to the number can quickly locate out-of-stock commodities, improve the replenishment efficiency, and timely replenishment can avoid the loss of customers due to out-of-stock, enhancing customer satisfaction and loyalty. In addition, through automated and intelligent out-of-stock monitoring, the cost of manual monitoring can be reduced and the operation efficiency can be improved.

[0081] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0082] In this embodiment, an out-of-stock commodity identification device is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0083] Figure 6It is a structural block diagram of an out-of-stock item identification device according to an embodiment of the present invention. As Figure 6 shown, the device includes

[0084] An acquisition module 62, configured to acquire a first image obtained by photographing a target area at a predetermined time interval, wherein the target area is used to place a first tray, the first tray is used to hold items, and the number of the first trays is multiple;

[0085] A first determination module 64, configured to determine a first area of the items included in the first tray included in the first image when the first image meets a predetermined condition;

[0086] A second determination module 66, configured to determine a second area included in the first area that is less than a predetermined threshold;

[0087] An execution module 68, configured to perform an alarm operation when it is determined that an item is missing from the first tray based on the second area.

[0088] In an exemplary embodiment, before performing the alarm operation, the device can be used for one of the following: determining a third area included in the second area that belongs to the same target tray, and determining that an item is missing from the target tray when the number of the third areas is greater than or equal to a first preset number, wherein the target tray is any one of the first trays included; determining a third area included in the second area that belongs to the same target tray, dividing the third area according to the shooting angle to obtain area categories, and determining that an item is missing from the target tray when the number of area types is greater than a second preset number, wherein the target tray is any one of the first trays included; determining a third area included in the second area that belongs to the same target tray, dividing the third area according to the shooting angle to obtain area categories, and determining that an item is missing from the target tray when the number of area types is the same as the number of shooting angles, wherein the target tray is any one of the first trays included.

[0089] In an exemplary embodiment, the device can also be used for: when the first image includes a second image that does not meet the predetermined condition, controlling an imaging device to photograph the target area to obtain a third image; and replacing the third image included in the first image with the third image.

[0090] In an exemplary embodiment, the device may also be used to, before determining the first area of the commodity included in the first tray included in the first image: determine the illumination intensity of the first image; perform filtering processing on the first image according to the illumination intensity to obtain a fourth image; enhance the contrast of the fourth image to obtain a fifth image; and determine that the first image meets the predetermined condition when the fifth image meets the predetermined condition.

[0091] In an exemplary embodiment, the execution module 68 may implement the execution of the alarm operation in the following manner: determine the out-of-stock time when it is determined that the commodity is missing from the first tray based on the second area; determine the time difference between the current time and the out-of-stock time; display an alarm message on the target interface when the time difference is greater than a first threshold; send the alarm message to the target account when the time difference is greater than a second threshold; and broadcast the alarm message by voice when the time difference is greater than a third threshold; wherein, the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

[0092] In an exemplary embodiment, the execution module 68 may implement the execution of the alarm operation in the following manner: identify the target barcode of the first tray included in the first picture to obtain barcode information, wherein the target barcode is placed in the target area of the first tray, and the size of the target barcode and the size of the first tray satisfy a preset proportional relationship; identify the target code in the first tray included in the first picture to obtain code information, wherein the target code is placed in the target area of the first tray; determine the target commodity placed in the first tray based on the barcode information and / or the code information; generate an alarm message based on the commodity information of the target commodity; and perform the alarm operation based on the alarm message.

[0093] In an exemplary embodiment, the first determination module 64 may implement the determination of the first area of the commodity included in the first tray included in the first image in the following manner: use a multimodal model to determine the first area of the commodity included in the first tray included in the first image.

[0094] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: all the above-mentioned modules are located in the same processor; or, the above-mentioned various modules are respectively located in different processors in any combination form.

[0095] An embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0096] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0097] An embodiment of the present invention also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0098] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. Among them, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0099] An embodiment of the present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the methods in various embodiments of the present application are implemented.

[0100] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0101] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0102] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying out-of-stock goods, characterized in that: include: Acquire a first image obtained by photographing a target area at a predetermined time interval, wherein the target area is used to place a first tray, the first tray is used to hold commodities, and there are multiple first trays; In a case where the first image satisfies a predetermined condition, determining a first area of ​​the commodity included in the first tray included in the first image; determining a second area included in the first area and smaller than a predetermined threshold; When it is determined based on the second area that there is a lack of merchandise in the first tray, an alarm operation is performed.

2. The method according to claim 1, characterized in that Before performing the alarm operation, the method further includes one of the following: Determine a third area included in the second area and belonging to the same target pallet, and if the number of the third areas is greater than or equal to the first preset number, determine that the target pallet is missing goods, wherein the target pallet is any one of the first pallets; Determine a third area included in the second area and belonging to the same target pallet, divide the third area according to the shooting angle to obtain area categories, and determine that the target pallet is missing goods when the number of area types is greater than a second preset number, wherein the target pallet is any one of the first pallets; Determine a third area included in the second area and belonging to the same target pallet, divide the third area according to the shooting angle to obtain area categories, and when the number of the area types is the same as the number of the shooting angles, determine that goods are missing from the target pallet, wherein the target pallet is any one of the pallets included in the first pallet.

3. The method according to claim 1, characterized in that: The method further comprises: When the first image includes the second image that does not meet the predetermined condition, controlling the camera to shoot the target area to obtain a third image; The third image included in the first image is replaced with the third image.

4. The method according to claim 1, characterized in that Before determining a first area of ​​the commodity included in the first pallet included in the first image, the method further includes: determining a light intensity of the first image; Performing filtering processing on the first image according to the light intensity to obtain a fourth image; enhancing the contrast of the fourth image to obtain a fifth image; In a case where the fifth image satisfies the predetermined condition, it is determined that the first image satisfies the predetermined condition.

5. The method according to claim 1, characterized in that Executing alarm operations includes: determining a stock-out time of a lack of goods in the first pallet based on the second area; Determine the time difference between the current time and the out-of-stock time; When the time difference is greater than a first threshold, displaying an alarm message on the target interface; When the time difference is greater than a second threshold, sending an alarm message to a target account; When the time difference is greater than a third threshold, the alarm information is broadcasted by voice; The first threshold is smaller than the second threshold, and the second threshold is smaller than the third threshold.

6. The method according to claim 1, characterized in that Executing alarm operations includes: Identify a target barcode of a first pallet included in the first image to obtain barcode information, wherein the target barcode is placed in a target area of ​​the first pallet, and a size of the target barcode satisfies a preset proportional relationship with a size of the first pallet; Identify a target code in a first tray included in the first picture to obtain coding information, wherein the target code is placed in a target area of ​​the first tray; Determine the target commodity contained in the first tray based on the barcode information and / or the coding information; generating warning information based on the commodity information of the target commodity; The warning operation is performed based on the warning information.

7. The method according to claim 1, characterized in that Determining a first area of ​​the commodity included in the first tray included in the first image includes: A first area of ​​the commodity included in the first pallet included in the first image is determined using a multimodal model.

8. A device for identifying out-of-stock goods, characterized in that: include: an acquisition module, configured to acquire a first image obtained by photographing a target area at a predetermined time interval, wherein the target area is used to place a first tray, the first tray is used to hold commodities, and the number of the first trays is multiple; A first determining module, configured to determine a first area of ​​the commodity included in the first tray included in the first image if the first image satisfies a predetermined condition; A second determining module, configured to determine a second area included in the first area and smaller than a predetermined threshold; An execution module is used to execute an alarm operation when it is determined based on the second area that there is a lack of goods in the first tray.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images