Method, device and equipment for identifying vehicle-mounted unmanned vending cabinet based on dynamic and static combination and storage medium

Abstract

The invention provides a method, device and equipment for recognizing a vehicle-mounted unmanned vending cabinet based on dynamic and static combination and a storage medium, and the method comprises the steps: recognizing a commodity image of a to-be-recognized vehicle-mounted unmanned vending cabinet through a preset static image recognition model, and obtaining a static recognition result, the static recognition result comprises the types and the number of commodities before door opening and the types and the number of commodities after door closing of the to-be-recognized vehicle-mounted unmanned vending cabinet; the commodity change condition and the hand change condition of the user after the door of the to-be-recognized vehicle-mounted unmanned vending cabinet is opened are recognized through a preset dynamic image recognition model, a dynamic recognition result is obtained, and the dynamic recognition result comprises the hand action change and the variety and number change of the commodities; and fusing the static identification result and the dynamic identification result, and identifying the sales and replenishment conditions of the vehicle-mounted unmanned vending cabinet. Through the method, the effect of accurately and timely identifying the selling and replenishment conditions of the vehicle-mounted unmanned vending cabinet can be achieved.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and more specifically, to a method, device, equipment and storage medium for identifying an unmanned vehicle vending machine based on a combination of static and dynamic recognition. Background Art

[0002] With the development of autonomous driving technology and the Internet of Things technology, vehicle-mounted unmanned vending machines, as a new retail format, are attracting more and more attention. Traditional vehicle-mounted unmanned vending machines mainly rely on RFID (radio frequency identification) technology or image processing technology for product identification and transaction confirmation.

[0003] Although traditional static recognition methods work well when products are placed stably, they are prone to recognition delays or errors in dynamic environments (such as when customers quickly select products or products are blocked). RFID technology has problems such as high cost and susceptibility to interference, while single image recognition technology is prone to misidentification or missed recognition in dynamic environments.

[0004] Therefore, how to accurately and timely identify the sales and replenishment status of vehicle-mounted unmanned vending machines is a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method for identifying an unmanned vehicle-mounted vending machine based on a combination of dynamic and static aspects. Through the technical solution of the embodiments of the present application, the effect of accurately and timely identifying the sales and replenishment status of the unmanned vehicle-mounted vending machine can be achieved.

[0006] In the first aspect, an embodiment of the present application provides a method for identifying a vehicle-mounted unmanned vending machine based on a combination of static and dynamic identification, comprising: identifying the product image of the vehicle-mounted unmanned vending machine to be identified through a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the product before the vehicle-mounted unmanned vending machine to be identified opens the door and the type and quantity of the product after the door is closed; identifying the changes in the product and the changes in the user's hands after the vehicle-mounted unmanned vending machine to be identified opens the door through a preset dynamic image recognition model to obtain a dynamic recognition result, wherein the dynamic recognition result includes changes in hand movements and changes in the types and quantities of products; fusing the static recognition result and the dynamic recognition result to identify the sales and replenishment situation of the vehicle-mounted unmanned vending machine.

[0007] In the above embodiments of the present application, static image recognition models and dynamic image recognition models are used to identify the types, quantities, and corresponding changes in goods and hand changes during the sales and replenishment process, so that static recognition results and dynamic recognition results can be obtained. By combining the two recognition results, the sales and replenishment situation is finally determined. Through dual recognition, the effect of accurately and timely identifying the sales and replenishment situation of the vehicle-mounted unmanned vending machine can be achieved.

[0008] In some embodiments, before identifying the product image of the vehicle-mounted unmanned vending machine to be identified through a preset static image recognition model, it also includes: constructing a data set containing multiple product images and different user hand images, wherein the multiple product images and different user hand images respectively include images of different angles, images of different lighting conditions, and images of different background complexities; training a basic target detection model with the multiple product images in the data set to obtain a static image recognition model; training a basic target detection model with the different user hand images and multiple product images in the data set to obtain a dynamic image recognition model.

[0009] In the above embodiments of the present application, a basic target detection model can be trained through a data set constructed by different product images and hand images to obtain a static image recognition model and a dynamic recognition model, which are used to accurately perform dual recognition of the sales and replenishment status of unmanned vending machines to improve recognition accuracy.

[0010] In some embodiments, a basic target detection model is trained using multiple product images in a data set to obtain a static image recognition model, including: adding hyperparameters of the basic target detection model, and optimizing the basic target detection model using an attention mechanism and a spatiotemporal correlation analysis algorithm to obtain an optimized target detection model; training the optimized target detection model using multiple product images to obtain a static image recognition model; training the basic target detection model using different user hand images and multiple product images in a data set to obtain a dynamic image recognition model, including: training the optimized target detection model using different user hand images and multiple product images to obtain a dynamic image recognition model.

[0011] In the above embodiments of the present application, the target price detection model is optimized by introducing the attention mechanism and the spatiotemporal correlation analysis algorithm. The dynamic sample and static sample training of the target detection model can respectively obtain a dynamic image recognition model and a static image recognition model. The model trained by introducing the attention mechanism and the spatiotemporal correlation analysis algorithm can improve the accuracy of the model in identifying the sales and replenishment status of the goods.

[0012] In some embodiments, the basic target detection model is optimized by using the attention mechanism and the spatiotemporal correlation analysis algorithm to obtain an optimized target detection model, including: inserting an attention module and a spatiotemporal correlation analysis algorithm into the feature pyramid or path aggregation network of the basic target detection model to obtain an optimized target detection model.

[0013] In the above embodiments of the present application, the introduction of the attention mechanism and the model trained by the spatiotemporal correlation analysis algorithm can improve the accuracy of the model in identifying the sales and replenishment status of commodities.

[0014] In some embodiments, the static recognition results and the dynamic recognition results are integrated to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, including: determining whether the static recognition result and the dynamic recognition result are consistent; when it is determined that the static recognition result and the dynamic recognition result are consistent, selecting any one of the static recognition result and the dynamic recognition result as the final result, and determining the sales and replenishment status based on the final result; when it is determined that the static recognition result and the dynamic recognition result are inconsistent, determining the final sales result by post-processing the results, and determining the sales and replenishment status based on the final sales result.

[0015] In the above embodiment of the present application, when it is determined that the static recognition result and the dynamic recognition result are consistent, the sales and replenishment situation can be directly determined through the recognition result. When it is determined that the static recognition result and the dynamic recognition result are inconsistent, the recognition result can be further processed to obtain the final sales and replenishment situation.

[0016] In some embodiments, the final sales result is determined by post-processing the results, including: multiple recognition results are obtained by multiple recognition of product images through a static image recognition model or multiple recognition of changes in products and changes in user hands after the vehicle-mounted unmanned vending machine is opened through a dynamic image recognition model; multiple recognition results are calculated by majority voting or weighted summation to obtain the final sales result.

[0017] In the above embodiments of the present application, the recognition results can be further processed by majority voting or weighted summation to accurately obtain the final sales and replenishment situation.

[0018] In some embodiments, after fusing the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, it also includes: determining the changes in goods before and after the replenishment of the goods through the static recognition results; and updating the product information of the vending system through the product changes.

[0019] In the above-mentioned embodiment of the present application, when the user replenishes the goods, the replenishment status can be directly determined through the static recognition results, and the product information in the system can be updated in time.

[0020] In some embodiments, after fusing the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, it also includes: displaying the sales items and corresponding prices on the display screen according to the sales and replenishment status, and prompting the user to pay.

[0021] In the above embodiment of the present application, after the sale, the payment page can be displayed according to the sales situation, so as to facilitate users to pay quickly.

[0022] In a second aspect, an embodiment of the present application provides a device for identifying an unmanned vehicle vending machine based on a combination of motion and stillness, including:

[0023] The first recognition module is used to recognize the commodity image of the vehicle-mounted unmanned vending cabinet to be recognized through a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the door is opened and the type and quantity of the commodity after the door is closed;

[0024] The second recognition module is used to recognize the changes of the goods and the changes of the user's hands after the door of the unmanned vending machine to be recognized is opened through a preset dynamic image recognition model, and obtain dynamic recognition results, wherein the dynamic recognition results include changes in hand movements and changes in the types and quantities of goods;

[0025] The third recognition module is used to integrate the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

[0026] Optionally, the device further comprises:

[0027] A training module, used for the first recognition module to construct a data set including multiple product images and different user hand images before recognizing the product image of the vehicle-mounted unmanned vending machine to be recognized through a preset static image recognition model, wherein the multiple product images and different user hand images respectively include images of different angles, images of different lighting conditions, and images of different background complexities;

[0028] The basic object detection model is trained using multiple product images in the dataset to obtain a static image recognition model.

[0029] The basic object detection model is trained using different user hand images and multiple product images in the dataset to obtain a dynamic image recognition model.

[0030] Optionally, the training module is specifically used for:

[0031] Add the hyperparameters of the basic target detection model, and optimize the basic target detection model through the attention mechanism and spatiotemporal correlation analysis algorithm to obtain the optimized target detection model;

[0032] The optimized object detection model is trained using a variety of product images to obtain a static image recognition model;

[0033] The basic object detection model is trained by using different user hand images and multiple product images in the dataset to obtain a dynamic image recognition model, including:

[0034] The optimized object detection model is trained using different user hand images and multiple product images to obtain a dynamic image recognition model.

[0035] Optionally, the training module is specifically used for:

[0036] Insert the attention module and spatiotemporal correlation analysis algorithm into the feature pyramid or path aggregation network of the basic target detection model to obtain an optimized target detection model.

[0037] Optionally, the third identification module is specifically used for:

[0038] Determine whether the static recognition result and the dynamic recognition result are consistent;

[0039] When it is determined that the static recognition result and the dynamic recognition result are consistent, either the static recognition result or the dynamic recognition result is selected as the final result, and the sales and replenishment status is determined according to the final result;

[0040] When it is determined that the static recognition result and the dynamic recognition result are inconsistent, the final sales result is determined by post-processing the results, and the sales and replenishment status is determined based on the final sales result.

[0041] Optionally, the third identification module is specifically used for:

[0042] Recognize the product image multiple times through the static image recognition model or recognize the changes of the product and the user's hand after the vehicle-mounted unmanned vending machine is opened through the dynamic image recognition model to obtain multiple recognition results;

[0043] Multiple recognition results are calculated by majority voting or weighted summation to obtain the final sales result.

[0044] Optionally, the device further comprises:

[0045] An updating module, which is used for the third recognition module to identify the sales and replenishment of the vehicle-mounted unmanned vending machine by fusing the static recognition result and the dynamic recognition result, and to determine the change of the goods before and after the replenishment of the goods through the static recognition result;

[0046] Update the product information of the vending system according to product changes.

[0047] Optionally, the device further comprises:

[0048] The display module is used for displaying the items on sale and the corresponding prices on the display screen according to the sales and replenishment situation after the third recognition module integrates the static recognition result and the dynamic recognition result to identify the sales and replenishment situation of the vehicle-mounted unmanned vending machine, and prompting the user to pay.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0050] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.

[0051] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flow chart of a method for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of an implementation method for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application;

[0055] Figure 3 A schematic block diagram of a device for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a device for identifying an unmanned vehicle-mounted vending machine based on a combination of static and dynamic identification provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0058] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0059] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0060] RFID: Radio Frequency Identification is a technology that uses radio signals to identify specific targets and read and write related data without the need for mechanical or optical contact between the identification system and the specific target.

[0061] Yolov8 is a deep learning model for object detection. It is one of the latest versions of the Yolo (You Only Look Once) series of models. As an advanced model in the field of object detection, Yolov8 inherits the consistent efficiency and real-time performance of the Yolo series. The Yolo series is famous for its unique Single Shot MultiBoxDetector mechanism, which enables the model to simultaneously identify multiple objects in an image in a single forward pass, thereby achieving fast object detection.

[0062] The present application is applied to the scenario of image recognition in unmanned vending machines. The specific scenario is to use a model to identify static images and dynamic images in unmanned vending machines during the sales and replenishment process to determine the sales and replenishment status of the unmanned vending machines.

[0063] With the development of autonomous driving technology and the Internet of Things technology, unmanned car vending machines, as a new retail format, are attracting more and more attention. Traditional unmanned car vending machines mainly rely on RFID (radio frequency identification) technology or image processing technology for product identification and transaction confirmation. Although traditional static recognition methods perform well when the goods are placed stably, they are prone to recognition delays or errors in dynamic environments (such as customers quickly selecting goods, goods being blocked, etc.). RFID technology has problems such as high cost and susceptibility to interference, while single image recognition technology is prone to misidentification or missed recognition in dynamic environments.

[0064] To this end, the present application uses a preset static image recognition model to identify the commodity image of the vehicle-mounted unmanned vending machine to be identified, and obtains a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the vehicle-mounted unmanned vending machine to be identified opens the door and the type and quantity of the commodity after closing the door; uses a preset dynamic image recognition model to identify the changes in the commodity after the vehicle-mounted unmanned vending machine to be identified opens the door and the changes in the user's hands, and obtains a dynamic recognition result, wherein the dynamic recognition result includes changes in hand movements and changes in the type and quantity of the commodity; and integrates the static recognition result and the dynamic recognition result to identify the sales and replenishment situation of the vehicle-mounted unmanned vending machine. By using a static image recognition model and a dynamic image recognition model to identify the type, quantity, and corresponding changes and hand changes of the goods during the sales and replenishment process, a static recognition result and a dynamic recognition result can be obtained. By combining the two recognition results, the sales and replenishment situation is finally determined. Through double recognition, the effect of accurately and timely identifying the sales and replenishment situation of the vehicle-mounted unmanned vending machine can be achieved.

[0065] In the embodiment of the present application, the execution entity can be a vehicle-mounted unmanned vending machine device based on combined motion and stillness identification in a vehicle-mounted unmanned vending machine system based on combined motion and stillness identification. In actual applications, the vehicle-mounted unmanned vending machine device based on combined motion and stillness identification can be an electronic device such as a terminal device and a server, and there is no limitation here.

[0066] Combine the following Figure 1 The method for identifying an unmanned vehicle-mounted vending machine based on a combination of motion and stillness in an embodiment of the present application is described in detail.

[0067] Please see Figure 1 , Figure 1 A flowchart of a method for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application is shown in FIG. Figure 1 The method for identifying an unmanned vehicle vending machine based on a combination of motion and stillness includes:

[0068] Step 110: Identify the commodity image of the vehicle-mounted unmanned vending machine to be identified through a preset static image recognition model to obtain a static recognition result.

[0069] The static recognition result includes the type and quantity of the merchandise before the door of the unmanned vending machine to be identified is opened, and the type and quantity of the merchandise after the door is closed. It may also include information such as the color, position and angle of the merchandise before the door is opened, and the color, position and angle of the merchandise after the door is opened. The merchandise image may include an image of the merchandise and an image of the environment and objects surrounding the merchandise.

[0070] In some embodiments of the present application, before the commodity image of the vehicle-mounted unmanned vending machine to be identified is identified by a preset static image recognition model, Figure 1The method shown also includes: constructing a data set including multiple product images and different user hand images, wherein the multiple product images and different user hand images respectively include images at different angles, images under different lighting conditions, and images with different background complexities; training a basic target detection model using multiple product images in the data set to obtain a static image recognition model; training a basic target detection model using different user hand images and multiple product images in the data set to obtain a dynamic image recognition model.

[0071] In the above process, the present application can train the basic target detection model through the data set constructed by different product images and hand images to obtain a static image recognition model and a dynamic recognition model, which are used to accurately perform dual recognition of the sales and replenishment status of unmanned vending machines to improve the recognition accuracy.

[0072] Among them, the target detection model can be YOLOv8, and data enhancement technology can be used in the training process to improve the generalization ability of the model; during the training process, appropriate hyperparameters such as learning rate and batch size are set to optimize the model performance. Combining the attention mechanism with spatiotemporal correlation analysis in the target tracking of YOLOv8 can improve the target detection and tracking accuracy of the model in complex scenes. The specific network structure can be MobileNet or EfficientNet. The images in the data set can be images of uniform products such as the front and side or uniform hand at different angles, and each image is annotated. The annotation information can be at least one of the information such as type, quantity, color, position and angle, and the image boundary information and key point information can also be annotated, such as key points such as bottle caps and bottle bodies. The data set includes various types of commodities, such as food, beverages, daily necessities, etc., and includes commodity images taken under different lighting conditions, including simple solid color backgrounds and complex daily environment background images, as well as images obtained by rotating, scaling, translating and other geometric transformations of the images, including images with Gaussian noise and salt and pepper noise added, which can improve the robustness of the model.

[0073] In some embodiments of the present application, a basic target detection model is trained using multiple product images in a data set to obtain a static image recognition model, including: adding hyperparameters of the basic target detection model, and optimizing the basic target detection model using an attention mechanism and a spatiotemporal correlation analysis algorithm to obtain an optimized target detection model; training the optimized target detection model using multiple product images to obtain a static image recognition model; training the basic target detection model using different user hand images and multiple product images in a data set to obtain a dynamic image recognition model, including: training the optimized target detection model using different user hand images and multiple product images to obtain a dynamic image recognition model.

[0074] In the above process, the present application optimizes the target price detection model by introducing the attention mechanism and the spatiotemporal correlation analysis algorithm. The dynamic sample and static sample training of the target detection model can respectively obtain the dynamic image recognition model and the static image recognition model. The model trained by introducing the attention mechanism and the spatiotemporal correlation analysis algorithm can improve the accuracy of the model in identifying the sales and replenishment status of the goods.

[0075] Among them, the attention mechanism allows the model to focus on specific parts of the input, thereby improving the performance and robustness of the model. In the target detection task, it can enhance the model's sensitivity to the target area and reduce the impact of background noise. After integrating the attention mechanism, the specific recognition process of the model is as follows: YOLOv8 uses the backbone network (Backbone) to extract image features, and the attention module can be added on this basis. Insert an attention module into the feature pyramid network (FPN) or PANet of YOLOv8, such as the window multi-head self-attention (W-MSA), SOCA attention mechanism, LSK attention mechanism, etc., to adaptively adjust the channel and spatial information weights in the feature map. Fuse the features generated by the attention module with the original features, for example, using element-by-element multiplication (Hadamard Product) or residual connection (ResidualConnection) to achieve feature fusion.

[0076] Optionally, spatiotemporal correlation analysis is used to process target detection and tracking between consecutive frames. By analyzing the continuity of targets in time and space, the accuracy and stability of target tracking can be improved. The specific implementation steps after adding spatiotemporal correlation analysis are as follows: Obtain information such as the position and category of the target in the current frame. Extract feature representations for each detected target, which can be depth features, appearance features, or motion features. Use the target state of the previous frame to predict the possible position of the target in the current frame, and use methods such as Kalman filtering, Hungarian algorithm, DeepSORT, etc. to perform target matching and tracking ID assignment. Fusion of target information between consecutive frames can use models such as recursive neural network (RNN), time-recurrent neural network (LSTM), or recurrent neural network (GRU) to model time series data and enhance the understanding of target motion trends.

[0077] In some embodiments of the present application, a basic target detection model is optimized by an attention mechanism and a spatiotemporal correlation analysis algorithm to obtain an optimized target detection model, including: inserting an attention module and a spatiotemporal correlation analysis algorithm into a feature pyramid or a path aggregation network of the basic target detection model to obtain an optimized target detection model.

[0078] In the above process, the present application introduces the model trained by the attention mechanism and the spatiotemporal correlation analysis algorithm to improve the accuracy of the model in identifying the sales and replenishment status of goods.

[0079] Optionally, after optimizing the basic target detection model through the attention mechanism and the spatiotemporal correlation analysis algorithm to obtain the optimized target detection model, the model preprocessing and postprocessing are also included, as follows:

[0080] Preprocessing: Preprocess the input image, including scaling, normalization, etc., to ensure that the data format is compatible with YOLOv8. Use YOLOv8 with attention mechanism for target detection, and then perform spatiotemporal correlation analysis on consecutive frames to achieve target tracking.

[0081] Post-processing: Perform post-processing on the detection and tracking results, such as non-maximum suppression (NMS), tracking ID assignment, etc., to obtain the final tracking results. Evaluate the performance of the model, including detection accuracy, tracking accuracy, real-time performance, etc., and optimize the model based on the evaluation results, such as adjusting the parameters of the attention mechanism, improving the spatiotemporal correlation algorithm, etc.

[0082] Step 120: The preset dynamic image recognition model is used to identify the changes in the merchandise and the user's hands after the vehicle-mounted unmanned vending machine to be identified is opened, and a dynamic recognition result is obtained.

[0083] The dynamic recognition results include changes in hand movements, changes in the types and quantities of goods, and may also include changes in the position and angle of goods. Changes in goods include changes in information such as the position, angle, type and quantity of goods. Changes in hands include changes in hand shape, angle, position and movement path.

[0084] Optionally, a preset dynamic image recognition model is used to recognize changes in the merchandise and the user's hands after the vehicle-mounted unmanned vending machine to be recognized opens its door, and a dynamic recognition result is obtained, including:

[0085] The preset dynamic image recognition model is used to identify continuous product images and hand images after the door of the unmanned vending machine to be identified is opened, and a dynamic recognition result is obtained, wherein the continuous product images and hand images can be images obtained by shooting at intervals of a preset time (for example, 1s).

[0086] Step 130: Integrate the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

[0087] Among them, the sales and replenishment situation includes sales situation and replenishment situation. Sales situation includes the types and quantities of the sold goods. Replenishment situation includes information such as the quantity and types of the replenished goods.

[0088] In some embodiments of the present application, static recognition results and dynamic recognition results are integrated to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, including: determining whether the static recognition result and the dynamic recognition result are consistent; when it is determined that the static recognition result and the dynamic recognition result are consistent, selecting any one of the static recognition result and the dynamic recognition result as the final result, and determining the sales and replenishment status based on the final result; when it is determined that the static recognition result and the dynamic recognition result are inconsistent, determining the final sales result by post-processing the results, and determining the sales and replenishment status based on the final sales result.

[0089] In the above process, when it is determined that the static recognition result and the dynamic recognition result are consistent, the sales and replenishment situation can be directly determined through the recognition result. When it is determined that the static recognition result and the dynamic recognition result are inconsistent, the recognition result can be further processed to obtain the final sales and replenishment situation.

[0090] Anomaly detection: If there are products in the dynamic recognition results that are not recognized statically, further inspection is required to determine whether there are any misidentifications or omissions.

[0091] For example: if the result of dynamic recognition is consistent with the result of static recognition, the result of dynamic recognition is directly used as the final result. If the result of dynamic recognition is inconsistent with the result of static recognition, but the result of dynamic recognition is more reliable (such as multiple confirmations), the result of dynamic recognition is used. If the result of dynamic recognition conflicts with the result of static recognition, but it is not possible to determine which one is more reliable, the final result can be determined by majority voting or weighted average.

[0092] In some embodiments of the present application, the final sales result is determined by post-processing the results, including: multiple recognition results are obtained by multiple recognition of product images through a static image recognition model or multiple recognition of changes in products and changes in user hands after the vehicle-mounted unmanned vending machine is opened through a dynamic image recognition model; multiple recognition results are calculated by majority voting or weighted summation to obtain the final sales result.

[0093] In the above process, the present application can further process the identification results by majority voting or weighted summation to accurately obtain the final sales and replenishment situation.

[0094] The specific weighted sum weight can be set according to the demand or determined according to the number of recognition times.

[0095] In some embodiments of the present application, after the static recognition result and the dynamic recognition result are integrated to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, Figure 1 The method shown also includes: determining the change of goods before and after the replenishment of goods through the static recognition result; and updating the goods information of the vending system through the change of goods.

[0096] In the above process, when the user replenishes the goods, the replenishment status can be directly determined through the static recognition results, and the product information in the system can be updated in time.

[0097] In some embodiments of the present application, after the static recognition result and the dynamic recognition result are integrated to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, Figure 1 The method shown also includes: displaying the sale items and corresponding prices on the display screen according to the sale and replenishment situation, and prompting the user to pay.

[0098] In the above process, this application can display the payment page based on the sales situation after the sale, so that users can pay quickly.

[0099] In the above Figure 1 In the process shown, the present application identifies the commodity image of the unmanned vehicle vending machine to be identified through a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the unmanned vehicle vending machine to be identified opens the door and the type and quantity of the commodity after closing the door; identifies the changes in the commodity and the user's hand after the unmanned vehicle vending machine to be identified opens the door through a preset dynamic image recognition model to obtain a dynamic recognition result, wherein the dynamic recognition result includes the changes in hand movements and the types and quantities of the commodities; and integrates the static recognition result and the dynamic recognition result to identify the sales and replenishment of the unmanned vehicle vending machine. By identifying the types, quantities, corresponding changes and hand changes of the goods in the sales and replenishment process through the static image recognition model and the dynamic image recognition model, the static recognition result and the dynamic recognition result can be obtained. By combining the two recognition results, the sales and replenishment situation is finally determined. The double recognition can achieve the effect of accurately and timely identifying the sales and replenishment of the unmanned vehicle vending machine.

[0100] Combine the following Figure 2 The implementation method of the embodiment of the present application based on the combined dynamic and static identification of the vehicle-mounted unmanned vending machine is described in detail.

[0101] Please see Figure 2 , Figure 2 A flowchart of an implementation method for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application, such as Figure 2 The implementation method of identifying the vehicle-mounted unmanned vending machine based on the combination of motion and stillness includes:

[0102] Step 210: Train a static image recognition model and a dynamic image recognition model.

[0103] Specifically: construct a data set containing multiple product images and different user hand images, wherein the multiple product images and different user hand images include images at different angles, images under different lighting conditions, and images with different background complexities; train the basic target detection model with the multiple product images in the data set to obtain a static image recognition model; train the basic target detection model with the different user hand images and multiple product images in the data set to obtain a dynamic image recognition model.

[0104] Step 220: Recognize the product image and hand information of the vehicle-mounted unmanned vending machine to be identified by using a static image recognition model and a dynamic image recognition model respectively.

[0105] Specifically: The product image of the unmanned vehicle vending machine to be identified is identified through a preset static image recognition model to obtain a static recognition result. The changes in the product and the changes in the user's hands after the unmanned vehicle vending machine to be identified is identified through a preset dynamic image recognition model to obtain a dynamic recognition result.

[0106] Step 230: Integrate the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

[0107] Specifically: determine whether the static recognition result and the dynamic recognition result are consistent; when it is determined that the static recognition result and the dynamic recognition result are consistent, select any one of the static recognition result and the dynamic recognition result as the final result, and determine the sales and replenishment situation according to the final result; when it is determined that the static recognition result and the dynamic recognition result are inconsistent, determine the final sales result by post-processing the results, and determine the sales and replenishment situation according to the final sales result.

[0108] also, Figure 2 The specific methods and steps shown can be found in Figure 1 The method shown will not be described in detail here.

[0109] Previous article Figure 1-Figure 2 The method of identifying unmanned vending machines based on the combination of motion and static is described below. Figure 3-Figure 4 The invention describes a device for identifying an unmanned vending machine mounted on a vehicle based on a combination of motion and stillness.

[0110] Please refer to Figure 3 , is a schematic block diagram of a device 300 for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application. The device 300 may be a module, program segment or code on an electronic device. The device 300 is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device 300 can be found in the description below. To avoid repetition, the detailed description is appropriately omitted here.

[0111] Optionally, the device 300 includes:

[0112] The first recognition module 310 is used to recognize the commodity image of the vehicle-mounted unmanned vending machine to be recognized through a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the door is opened and the type and quantity of the commodity after the door is closed;

[0113] The second recognition module 320 is used to recognize the changes of the goods and the changes of the user's hands after the unmanned vending machine to be recognized is opened through a preset dynamic image recognition model, and obtain a dynamic recognition result, wherein the dynamic recognition result includes the changes of the hand movements and the changes of the types and quantities of the goods;

[0114] The third recognition module 330 is used to integrate the static recognition results and the dynamic recognition results to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

[0115] Optionally, the device further comprises:

[0116] A training module is used for constructing a data set containing multiple product images and hand images of different users before the first recognition module recognizes the product image of the vehicle-mounted unmanned vending machine to be recognized through a preset static image recognition model, wherein the multiple product images and hand images of different users respectively include images of different angles, images of different lighting conditions, and images of different background complexities; training a basic target detection model through the multiple product images in the data set to obtain a static image recognition model; training a basic target detection model through the hand images of different users and multiple product images in the data set to obtain a dynamic image recognition model.

[0117] Optionally, the training module is specifically used for:

[0118] Add hyperparameters of the basic target detection model, and optimize the basic target detection model through the attention mechanism and the spatiotemporal correlation analysis algorithm to obtain the optimized target detection model; train the optimized target detection model through a variety of product images to obtain a static image recognition model; train the basic target detection model through different user hand images and a variety of product images in the data set to obtain a dynamic image recognition model, including: train the optimized target detection model through different user hand images and a variety of product images to obtain a dynamic image recognition model.

[0119] Optionally, the training module is specifically used for:

[0120] Insert the attention module and spatiotemporal correlation analysis algorithm into the feature pyramid or path aggregation network of the basic target detection model to obtain an optimized target detection model.

[0121] Optionally, the third identification module is specifically used for:

[0122] Determine whether the static recognition result and the dynamic recognition result are consistent; when it is determined that the static recognition result and the dynamic recognition result are consistent, select any one of the static recognition result and the dynamic recognition result as the final result, and determine the sales and replenishment situation according to the final result; when it is determined that the static recognition result and the dynamic recognition result are inconsistent, determine the final sales result by post-processing the results, and determine the sales and replenishment situation according to the final sales result.

[0123] Optionally, the third identification module is specifically used for:

[0124] The static image recognition model is used to recognize the product image multiple times, or the dynamic image recognition model is used to recognize the changes in the product and the changes in the user's hands after the vehicle-mounted unmanned vending machine is opened, to obtain multiple recognition results; the multiple recognition results are calculated by majority voting or weighted summation to obtain the final sales result.

[0125] Optionally, the device further comprises:

[0126] The updating module is used for the third recognition module to identify the sales and replenishment of the vehicle-mounted unmanned vending machine by fusing the static recognition results and the dynamic recognition results, and then determine the changes in the goods before and after the replenishment of the goods through the static recognition results; and update the product information of the vending system according to the changes in the goods.

[0127] Optionally, the device further comprises:

[0128] The display module is used for displaying the items on sale and the corresponding prices on the display screen according to the sales and replenishment situation after the third recognition module integrates the static recognition result and the dynamic recognition result to identify the sales and replenishment situation of the vehicle-mounted unmanned vending machine, and prompting the user to pay.

[0129] Please refer to Figure 4 The structure schematic block diagram of a device for identifying an unmanned vehicle vending machine based on a combination of motion and stillness provided in an embodiment of the present application, the device may include a memory 410 and a processor 420. Optionally, the device may also include: a communication interface 430 and a communication bus 440. The device is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the description below.

[0130] Specifically, the memory 410 is used to store computer-readable instructions.

[0131] Processor 420 is used to process the readable instructions stored in the memory and can execute Figure 1 The steps in the method.

[0132] The communication interface 430 is used for signaling or data communication with other node devices, for example, for communication with a server or a terminal, or for communication with other device nodes, but the embodiments of the present application are not limited thereto.

[0133] The communication bus 440 is used to realize direct connection and communication among the above components.

[0134] The communication interface 430 of the device in the embodiment of the present application is used to communicate signals or data with other node devices. The memory 410 can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 410 can also be at least one storage device located away from the aforementioned processor. The memory 410 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 420, the electronic device executes the aforementioned Figure 1 The method process shown. The processor 420 can be used on the device 300 and is used to perform the functions in the present application. Exemplarily, the above-mentioned processor 420 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the embodiments of the present application are not limited thereto.

[0135] The embodiment of the present application also provides a readable storage medium, when the computer program is executed by a processor, Figure 1 The method process in the method embodiment shown is executed by the electronic device.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0137] In summary, the embodiments of the present application provide a method, device, electronic device and readable storage medium for identifying an unmanned vehicle vending machine based on a combination of static and dynamic recognition. The method includes: identifying the commodity image of the unmanned vehicle vending machine to be identified by a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the unmanned vehicle vending machine to be identified opens the door and the type and quantity of the commodity after the door is closed; identifying the changes in the commodity and the user's hand after the unmanned vehicle vending machine to be identified opens the door by a preset dynamic image recognition model to obtain a dynamic recognition result, wherein the dynamic recognition result includes the changes in hand movements and the types and quantities of commodities; fusing the static recognition result and the dynamic recognition result to identify the sales and replenishment of the unmanned vehicle vending machine. This method can achieve the effect of accurately and timely identifying the sales and replenishment of the unmanned vehicle vending machine.

[0138] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0140] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0143] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for identifying vehicle-mounted unmanned vending machines based on a combination of static and dynamic information, characterized in that: include: Identify the commodity image of the unmanned vehicle vending machine to be identified by a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the unmanned vehicle vending machine to be identified is opened and the type and quantity of the commodity after the door is closed; The preset dynamic image recognition model is used to identify the changes in the goods and the changes in the user's hands after the unmanned vehicle vending machine to be identified is opened, and a dynamic recognition result is obtained, wherein the dynamic recognition result includes the changes in hand movements and the changes in the types and quantities of goods; The static recognition result and the dynamic recognition result are integrated to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

2. The method according to claim 1, characterized in that Before identifying the commodity image of the vehicle-mounted unmanned vending machine to be identified by using a preset static image recognition model, the method further includes: Constructing a data set including multiple product images and different user hand images, wherein the multiple product images and the different user hand images respectively include images at different angles, images under different lighting conditions, and images with different background complexities; The basic object detection model is trained by using the multiple commodity images in the data set to obtain the static image recognition model; The basic object detection model is trained by using the different user hand images and the multiple commodity images in the data set to obtain the dynamic image recognition model.

3. The method according to claim 2, characterized in that The step of training a basic object detection model by using the multiple commodity images in the data set to obtain the static image recognition model includes: Adding hyperparameters of the basic target detection model, and optimizing the basic target detection model through an attention mechanism and a spatiotemporal correlation analysis algorithm to obtain an optimized target detection model; The optimized object detection model is trained by using the multiple commodity images to obtain the static image recognition model; The step of training the basic object detection model by using the different user hand images and the multiple commodity images in the data set to obtain the dynamic image recognition model includes: The optimized object detection model is trained by using the different user hand images and the multiple commodity images to obtain the dynamic image recognition model.

4. The method according to claim 3, characterized in that The method of optimizing the basic target detection model by using the attention mechanism and the spatiotemporal correlation analysis algorithm to obtain an optimized target detection model includes: An attention module and a spatiotemporal correlation analysis algorithm are inserted into the feature pyramid or path aggregation network of the basic target detection model to obtain the optimized target detection model.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing of the static recognition result and the dynamic recognition result to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine includes: Determining whether the static recognition result and the dynamic recognition result are consistent; When it is determined that the static recognition result and the dynamic recognition result are consistent, selecting any one of the static recognition result and the dynamic recognition result as the final result, and determining the sales and replenishment status according to the final result; When it is determined that the static recognition result and the dynamic recognition result are inconsistent, a final sales result is determined by post-processing the results, and the sales and replenishment status is determined according to the final sales result.

6. The method according to claim 5, characterized in that The method of determining the final sales result by post-processing the result includes: Recognize the commodity image multiple times by the static image recognition model or recognize the change of the commodity and the change of the user's hand after the vehicle-mounted unmanned vending machine is opened multiple times by the dynamic image recognition model to obtain multiple recognition results; The multiple recognition results are calculated by majority voting or weighted sum to obtain the final sales result.

7. The method according to any one of claims 1 to 4, characterized in that: After fusing the static recognition result and the dynamic recognition result to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine, the method further includes: Determine the change of the goods before and after the replenishment of the goods through the static recognition result; The commodity information of the vending system is updated according to the commodity changes.

8. A device for identifying unmanned vending machines on vehicles based on a combination of motion and stillness, characterized in that: include: The first recognition module is used to recognize the commodity image of the vehicle-mounted unmanned vending cabinet to be recognized through a preset static image recognition model to obtain a static recognition result, wherein the static recognition result includes the type and quantity of the commodity before the door is opened and the type and quantity of the commodity after the door is closed; The second recognition module is used to recognize the changes of the goods and the changes of the user's hands after the door of the unmanned vehicle vending machine to be recognized is opened through a preset dynamic image recognition model, and obtain a dynamic recognition result, wherein the dynamic recognition result includes the changes of the hand movements and the changes of the types and quantities of the goods; The third recognition module is used to integrate the static recognition result and the dynamic recognition result to identify the sales and replenishment status of the vehicle-mounted unmanned vending machine.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium, characterized in that: include: A computer program, when the computer program is run on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images