Lightweight commodity identification tracking system, method, storage medium, device, terminal

Abstract

The present application belongs to the technical field of neural network lightweight design and target detection tracking neural network fast inference framework, and discloses a lightweight commodity identification and tracking system, method, storage medium, equipment and terminal, which collects and labels commodity identification training data set through installation of picture collection facilities in actual intelligent vending scene; designs a backbone feature extraction network based on Ghost Module, designs a multi-scale commodity detection module based on RFP, and designs a multi-target detection and target occlusion identification module based on DIOU_NMS algorithm; adds a commodity tracking algorithm in the lightweight commodity identification network to analyze the behavior of the user, and discriminates the user's taking and abnormal operation based on continuous key frames. The present application provides a lightweight commodity identification and tracking fusion network, which can provide suggestion guidance for embedded application of network deployment in edge intelligent equipment in corresponding scenes.

Description

Technical Field

[0001] This invention belongs to the technical field of lightweight neural network design and fast inference framework for target detection and tracking neural networks, and particularly relates to a lightweight commodity recognition and tracking system and method deployed on edge intelligent devices. Background Technology

[0002] Product monitoring technologies designed for the unmanned retail sector can be divided into two categories: static product monitoring and recognition, and dynamic product monitoring and recognition. Static product recognition involves installing 1-2 cameras on each floor of the vending machine. Using target monitoring algorithms such as HOG, LBP, and DPM, it analyzes the foreground and background of the vending machine in a state where no products have been taken to obtain the product category and location. It then compares the changes in the quantity and type of products in the images before and after product removal to determine the product removal behavior. This type of product recognition and tracking technology relies on manually designed features for monitoring and recognition, resulting in less than ideal monitoring performance. Furthermore, installing multiple cameras on each floor increases the deployment cost of the product recognition system.

[0003] Subsequent developments in dynamic product recognition and tracking systems employ external cameras, utilizing target recognition and tracking technologies to identify and locate products in multiple consecutive frames of images, and maintaining and updating the products in subsequent frames. Existing solutions typically use YOLOv5, CenterNet, RetinaNet, and SSD as feature recognition network architectures to construct the product recognition network, learning product features through deep learning for identification. Simultaneously, filtering algorithms such as particle filtering, Kalman filtering, or median filtering are added to construct a product tracking module for product path tracking, verifying product retrieval behavior. This type of product recognition and tracking system is generally deployed on a large backend server, resulting in a large network footprint. The front-end vending machine captures images of product retrieval via camera, transmitting them over the network to the server for recognition. Compared to static product recognition systems, this approach offers significantly improved accuracy, but it is heavily influenced by network fluctuations. During network outages, images and videos are transmitted back to the server, awaiting the server's recognition and tracking feedback before being transmitted back to the vending machine, extending the recognition time throughout the shopping process and reducing the vending machine's efficiency.

[0004] Based on the aforementioned issues, in scenarios like unattended vending machines, which involve highly complex environments, sensitive commodity transactions, and a need to consider user experience, deploying an offline, front-end commodity recognition network and feeding back the recognition information to a back-end management server to complete the recognition and payment process has become an important solution to address the increased shopping time caused by network transmission. However, limitations in the computing and storage resources of the front-end industrial control computers and deployment costs necessitate a lightweight network design and acceleration to achieve higher accuracy, which presents a significant challenge for research.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0006] (1) Due to the high complexity of the environment and the sensitivity of commodity transactions in unattended vending scenarios, while also taking into account user experience, it is objectively difficult to deploy target detection algorithms.

[0007] (2) Static commodity recognition technology relies on manually designed features for monitoring and recognition. In comparison, the monitoring effect is not ideal, and installing multiple cameras on each floor increases the deployment cost of the commodity recognition system.

[0008] (3) The back-end product identification and tracking network is sensitive to network fluctuations. At the same time, under network fluctuations, it sends back images and videos to the server and waits for the server to generate identification and tracking feedback results before transmitting them to the cabinet through the network, which prolongs the identification time in the entire shopping process and reduces the efficiency of the vending machine.

[0009] The difficulty in solving the above problems and defects is as follows:

[0010] (1) The front-end product identification and tracking system is highly adaptable to the network, but it is limited by the computing and storage resources of the deployed edge intelligent devices. It is necessary to design the identification and tracking network in a lightweight manner. The lightweight design, which is mainly based on model compression and model acceleration, is itself very difficult.

[0011] (2) There are unfavorable identification situations in the sale of goods, such as small targets, multiple targets, and different degrees of obstruction of goods. It is necessary to optimize each link of the product identification and tracking network to improve the accuracy of product identification under unfavorable identification situations and meet the commercial scenarios of goods.

[0012] The significance of solving the above problems and defects is as follows:

[0013] (1) Expand the scope of use of unmanned vending machines. By deploying edge intelligent devices of commodity recognition and behavior tracking systems, the impact of network fluctuations on smart vending machines can be reduced, and vending machines can be placed in more areas.

[0014] (2) By optimizing the model, the design optimization is carried out to address the unfavorable recognition problem in the smart goods retail scenario, which greatly improves the recognition accuracy of smart vending machines and is of great significance to their large-scale commercial use.

[0015] (3) The contactless mode has become an application area with urgent development needs. This invention lays the foundation for the development and application research of the contactless retail field and provides experience guidance for the application research of contactless retail. Summary of the Invention

[0016] To overcome the problems existing in related technologies, the present invention discloses a lightweight product identification and tracking system and method deployed on edge intelligent devices, particularly relating to a lightweight product identification and tracking system, method, medium, device, and terminal deployed on edge intelligent devices. The technical solution is as follows:

[0017] This invention is implemented as follows: a lightweight product identification and tracking method deployed on edge intelligent devices, the lightweight product identification and tracking method deployed on edge intelligent devices comprising:

[0018] The training dataset for product recognition in a real-world smart vending scenario was collected and labeled by installing image acquisition facilities. A backbone feature extraction network was designed based on Ghost Module, a multi-scale product detection module was designed based on RFP, and a multi-target detection and target occlusion recognition module was designed based on the DIOU_NMS algorithm. A product tracking algorithm was added to the lightweight product recognition network to analyze user behavior and to distinguish user picking up and abnormal operations based on continuous keyframes.

[0019] In one embodiment, the lightweight product identification and tracking method deployed on an edge intelligent device includes the following steps:

[0020] Step 1: Collection and construction of targeted datasets for actual product sales scenarios;

[0021] Step 2: For the embedded application scenario of intelligent edge devices, a new lightweight feature extraction network is designed by adding a Ghost Module to the backbone feature extraction network.

[0022] Step 3: Design a multi-scale target monitoring enhancement module for goods based on RFP;

[0023] Step 4: Introduce and improve the DIOU_NMS algorithm to design a multi-target detection and occlusion recognition module for goods;

[0024] Step 5: Design a behavior tracking algorithm to be integrated into the product recognition network and establish a product recognition and tracking target detection network at the edge intelligent device.

[0025] Step six involves converting the constructed lightweight product identification and tracking network into an ONNX model, transforming the ONNX version of the lightweight product identification and tracking network model into a model inference engine for edge intelligent devices, and completing the identification of retail products and user tracking on the Jeston Nano.

[0026] In one embodiment, the targeted dataset collection and construction in step one includes:

[0027] During the data collection process, a network camera was installed at the top center of the vending machine to simulate the steps of picking up and putting back items. The video stream from the network camera was acquired via the RTSP protocol, and the video frames were saved. Frames showing 75%, 50%, and 25% occlusion during item handling, as well as frames showing small items, multiple items, and irrelevant items affecting the product, were selected proportionally and combined to form a targeted dataset for network training and detection. The LabelImg software was used to label the items in the saved video frames.

[0028] In one embodiment, step two, which involves adding a Ghost Module to the backbone feature extraction network of the object detection network architecture to design a new lightweight feature extraction network, includes:

[0029] The Ghost method generates a feature map in convolution, then generates a Ghost feature map through a simple mapping transformation, and finally concatenates them to form the Output.

[0030] The Output is generated and calculated as follows:

[0031] The formula for calculating the intrinsic map is as follows:

[0032] ;

[0033] in, This represents the input matrix after image convolution in the previous neural network layer. The convolution operation, representing a neural network, is generally represented by... and The convolution kernel is used to perform convolution operations on the image feature matrix. This represents the image feature matrix of the layer completed in this convolution operation.

[0034] The formula for calculating the Ghost feature map is as follows:

[0035] ;

[0036] in, This represents the feature matrix of the feature map generated by the ghost module operation, which is performed by... To achieve this, the feature matrices of each layer from the previous step... After a simple linear calculation, the mapping generates the ghost feature layer.

[0037] Adding a Ghost module to the backbone feature extraction network improves the speed of the resulting model. and model size compression ratio for:

[0038] ;

[0039] ;

[0040] In the above two formulas, Input the number of channels. The number of Ghost feature maps generated. This represents the original number of feature maps required. and This represents the length and width of the feature map after convolution. and The kernel size is denoted by ; the speedup of the model obtained by introducing the Ghost module is related to the proportion of Ghost graphs generated in the total output.

[0041] In one embodiment, the commodity multi-scale target monitoring enhancement module in step three consists of three structures:

[0042] (1) Initial features obtained by top-down feature extraction through RFP Top-down upsampling :

[0043] , , ;

[0044] in, This represents the feature data matrix of each feature layer after the convolutional layer. This represents the feature values ​​of each layer obtained by upsampling the input image; it is calculated by the forward pass of the neural network. get, It also includes a feature extraction network consisting of convolution and pooling layers performing convolution operations on the input image, where the feature parameters of each feature layer of the input image are magnified by 2 times for output. It is a convolutional layer obtained by convolving the input image of each layer. The number of channels is modified by convolution, so that the number of channels in each layer is proportional to the channel count. Consistency in Count the features of each layer in the neural network.

[0045] (2) To and Perform lateral connections on the corresponding feature layers to obtain features. :

[0046] ;

[0047] in It is and The characteristic matrices are added bit by bit.

[0048] (3) For horizontal links Perform cyclic feature acquisition, repeating steps (1) and (2) to obtain output features. :

[0049] ;

[0050] represent By repeating the upsampling, downsampling, and lateral connection operations in steps (1) and (2), the feature pyramid network is reused to obtain more small target feature details.

[0051] In one embodiment, the loss calculation for the improved Cluster_IoU in step four. :

[0052] ;

[0053] in, This represents the intersection-union ratio of the bounding boxes, focusing on the overlapping area of ​​bounding boxes in product target detection and recognition; Represents European distance. and This represents the center point of the two bounding boxes, which together measure the center distance of the normalized detection boxes; This represents the length of the diagonal of the two smallest defined bounding boxes. This indicates the similarity ratio between the anchor and the target bounding box in terms of length and width.

[0054] In one embodiment, the behavior tracking algorithm in step five consists of two parts: tracking and recognition, and behavior logic judgment. Before the sale of goods, the behavior tracking algorithm sets a baseline L, collects the edge points of the sales cabinet using four coordinate points, and constructs the baseline. During the product recognition analysis, the continuous keyframes identified by the network are input into a newly opened thread, where the distance S between the center point c of the continuous products and L is compared. When the sign of the distance changes, the picking and placing behavior is determined. When the distance S suddenly exceeds a threshold, the abnormal picking behavior of the user is determined. When S is 0 for a long time, it is determined that the user has not picked up the product or the recognition is incorrect.

[0055] In one embodiment, in step six, the constructed lightweight product identification and tracking network is converted into an ONNX model, and based on the edge intelligent device Jeston Nano, the ONNX version of the lightweight product identification and tracking network model is converted into a model inference engine on the edge intelligent device using TensorRT technology, and the identification and tracking of retail products are completed on the Jeston Nano.

[0056] Another object of the present invention is to provide a lightweight product identification and tracking system deployed on an edge intelligent device, which applies the aforementioned lightweight product identification and tracking method deployed on an edge intelligent device. The lightweight product identification and tracking system deployed on an edge intelligent device includes:

[0057] The data acquisition module is used for the collection and construction of targeted datasets in actual product sales scenarios;

[0058] The feature extraction network building module is designed for embedded application scenarios of intelligent edge devices, and adds a Ghost Module to the backbone feature extraction network to design a new lightweight feature extraction network.

[0059] The detection and recognition optimization module is used to optimize the target detection network for small target detection and recognition, and an enhanced module for multi-scale target detection of goods is designed based on RFP.

[0060] The target detection network construction module is used to introduce and improve the DIOU_NMS algorithm to design a multi-target detection and occlusion recognition module for goods; the behavior tracking algorithm is integrated into the goods recognition network to establish a goods recognition and tracking target detection network at the edge intelligent device.

[0061] The product identification and tracking module is used to convert the constructed lightweight product identification and tracking network into an ONNX model, convert the ONNX version of the lightweight product identification and tracking network model into a model inference engine on the edge intelligent device, and complete the identification and tracking of retail products on the Jeston Nano.

[0062] Another object of the present invention is to provide a user input program storage medium, wherein the stored computer program enables an electronic device to execute the lightweight commodity identification and tracking method deployed on an edge intelligent device.

[0063] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the lightweight product identification and tracking method deployed on an edge intelligent device.

[0064] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the lightweight product identification and tracking method deployed on an edge intelligent device.

[0065] Another object of the present invention is to provide an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the lightweight commodity identification and tracking system deployed on an edge intelligent device.

[0066] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows:

[0067] First, addressing the technical problems existing in the prior art and the difficulty in solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0068] The lightweight product identification and tracking system deployed on edge intelligent devices provided by this invention aims to support unmanned vending machines for seamless sales based on dynamic vision. In other words, it is expected that after customers complete identity authentication and open the cabinet door, they will not need to perform additional operations such as scanning tags. After taking the beverage from the cabinet and closing the cabinet door, they can immediately receive the checkout information and complete the automatic deduction, realizing a completely seamless shopping experience of taking and paying immediately.

[0069] This invention improves the following four aspects by designing a lightweight commodity recognition and tracking network based on edge intelligent devices in a vending machine scenario: (1) targeted dataset collection and construction in actual commodity sales scenarios; (2) enhanced recognition of small commodity targets; (3) enhanced target occlusion and multi-target recognition; (4) compression and acceleration of the target detection network; and (5) abnormal behavior detection based on commodity tracking. Specifically, the commodity recognition training dataset is collected and labeled by installing image collection facilities in actual intelligent vending scenarios; a backbone feature extraction network is designed based on Ghost Module to improve network inference speed and reduce network size; a multi-scale commodity detection module is designed based on RFP to improve the feature collection rate of small targets and improve the overall commodity recognition rate; a multi-target detection and target occlusion recognition module is designed based on the DIOU_NMS algorithm to improve the recognition accuracy of retail commodity occlusion and multi-commodity picking detection; a commodity tracking algorithm is added to the lightweight commodity recognition network to analyze user behavior, judge user picking and abnormal operations based on continuous keyframes, and perform order judgment and settlement according to the abnormal behavior processing logic.

[0070] Compared with the prior art, the present invention has the following advantages: In the current image recognition-based product recognition network model, many networks are deployed in the back-end server to decode and recognize the video transmitted back from the front-end device and then return the product recognition information. This makes the model design more complicated, and the time loss in transmission and recognition also greatly reduces the customer experience of similar smart vending machines, which is not conducive to the promotion of this model. To address this, this invention provides a TensorRT-accelerated product recognition and tracking network based on the Jeston Nano edge intelligent device. During the design process, a Ghost module is added for model acceleration and compression, enabling deployment on edge intelligent devices with limited computing and storage resources while maintaining model recognition efficiency. This allows product recognition and tracking to be performed at the front end, reducing response time and improving user experience. Furthermore, a multi-scale detection module is designed based on RFP technology, and a multi-target and product occlusion recognition enhancement module is optimized and designed based on the DIOU_NMS algorithm, improving the accuracy of product detection for small targets, product occlusion, and multi-target detection. In addition, by specifically constructing a product feature dataset, the recognition rate is further improved in specific product recognition scenarios. Finally, a product tracking module is designed to judge user behavior, combining with product recognition to ensure correct order settlement. Through these four innovative design features, this invention significantly improves the network performance of product recognition and tracking on the Jeston Nano intelligent edge device.

[0071] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0072] This invention provides a lightweight product recognition and tracking fusion network deployed on edge intelligent devices. More specifically, it designs a fusion network based on product recognition and tracking, providing suggestions and guidance for deploying the network on edge intelligent devices for embedded applications in corresponding scenarios.

[0073] In addition, this invention designs a multi-scale target monitoring enhancement module for goods based on RFP, thereby optimizing the small target monitoring and recognition performance of the network and improving the overall accuracy of goods recognition.

[0074] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0075] (1) This invention, as the core system, is deployed in an industrial control computer on an unmanned vending machine to identify and track the user's actions of taking goods from the vending machine, completing the user's seamless purchase and payment operation. This invention can be applied to the field of unmanned retail and deployed on various types of unmanned vending machines. Through a product identification and tracking network, it enables users to automatically open and close the vending machine and take goods, completing the transaction and payment. This can significantly reduce the cost of human monitoring and care in the sales field and increase sales profits. At the same time, this invention can provide more contactless shopping application scenarios for seamless vending, which is beneficial to protecting users, reducing unnecessary contact, and bringing huge commercial value to the promotion and application of unmanned vending machines.

[0076] (2) This invention deploys the product identification method on edge intelligent devices and directly networks with intelligent vending machines. It completes the identification and tracking of products taken by users at the front end and transmits the type and quantity of the identified products to the management service back end, connecting to the payment and deduction function. It fills the technical and application gap of product identification network front-end deployment identification in the field of unmanned vending and creates a precedent for the use of intelligent vending in which product identification is combined with front-end identification and the identification results are displayed.

[0077] (3) In view of the technical problem that the current smart vending machine product identification and tracking system cannot balance accuracy and efficiency, the present invention designs a lightweight product identification and tracking system to be deployed on edge smart devices. This system combines the technical advantages of lightweight, multi-scale, multi-target detection and product tracking, and has a high accuracy rate. At the same time, the system’s identification function is front-end, which also saves a lot of transmission time, can accelerate the identification and settlement process, and greatly shorten the shopping process time consumption. Attached Figure Description

[0078] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0079] Figure 1 This is a flowchart of a lightweight product identification and tracking method deployed on an edge intelligent device according to an embodiment of the present invention;

[0080] Figure 2 This is a schematic diagram of a lightweight product identification and tracking method deployed on edge intelligent devices according to an embodiment of the present invention;

[0081] Figure 3 This is a schematic diagram illustrating the operation of the network output feature map based on the Ghost module provided in an embodiment of the present invention;

[0082] Figure 4 This is a schematic diagram of the design of the RFP-based multi-scale target detection enhancement module for goods provided in an embodiment of the present invention. Detailed Implementation

[0083] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0084] I. Explanation of the Implementation Example:

[0085] like Figure 1 As shown, the lightweight product identification and tracking method deployed on edge intelligent devices provided in this embodiment of the invention includes the following steps:

[0086] S101, Collection and construction of targeted datasets for actual product sales scenarios;

[0087] S102, for the application scenario of network model in intelligent edge device embedded application, a new lightweight feature extraction network is designed by adding Ghost Module to the backbone feature extraction network;

[0088] S103, a multi-scale target monitoring enhancement module for commodities based on RFP design;

[0089] S104, Introduce and improve the DIOU_NMS algorithm to design a multi-target detection and occlusion recognition module for goods;

[0090] S105 is designed to integrate behavior tracking algorithms into the product recognition network and establish a product recognition and tracking target detection network at the edge intelligent device.

[0091] S106 transforms the constructed lightweight product identification and tracking network into an ONNX model, converts the ONNX version of the lightweight product identification and tracking network model into a model inference engine for edge intelligent devices, and completes the identification of retail products and user tracking on the Jeston Nano.

[0092] As a preferred embodiment, such as Figure 2 As shown, the lightweight product identification and tracking method deployed on edge intelligent devices provided in this embodiment of the invention specifically includes the following steps:

[0093] Step 1: Targeted Dataset Collection and Construction for Real-World Product Sales Scenarios. The product sales process involves various complex situations, such as motion blur of products, varying degrees of occlusion during product handling, identification of small target products, simultaneous handling of multiple product targets, and the simultaneous appearance of unrelated items with similar shapes in the image. Therefore, during the dataset collection process, the possibility of these different situations must be fully considered.

[0094] During the data collection process, a webcam was installed at the top center of the vending machine to allow testers to simulate the steps of picking up and putting back items. The webcam video stream was acquired via the RTSP protocol, and the video frames were saved. Subsequently, the video frames with 75%, 50%, and 25% occlusion during item handling, small target frames, multi-target frames, and frames affected by irrelevant items were selected proportionally and combined to form a targeted dataset for training and detecting the invented network. Finally, the LabelImg software was used to label the items in the saved video frames.

[0095] Step 2: For embedded applications of intelligent edge devices using the network model, the issue of model lightweighting needs to be addressed. A new lightweight feature extraction network is designed by adding a Ghost Module to the backbone feature extraction network. The speed and training model size of the YOLOv5 object detection and recognition network are insufficient for deployment and detection / recognition on edge intelligent devices. Therefore, a lightweight network design is required. The Ghost method generates feature maps in regular convolutions, then generates Ghost feature maps through simple mapping transformations, and concatenates them to form the output. Its feature generation is as follows: Figure 3 As shown.

[0096] The formula for generating Output is as follows:

[0097] ①

[0098] ②

[0099] Equation ① describes the calculation method for the intrinsic feature map, and equation ② describes the calculation method for the Ghost feature map. Adding a Ghost module to the backbone feature extraction network improves the speed of the resulting model. and model size compression ratio :

[0100] ;

[0101] ;

[0102] in, Input the number of channels. The number of Ghost feature maps generated. This represents the original number of feature maps required. and This represents the length and width of the feature map after convolution. and This represents the kernel size. It can be seen that the speedup of the model obtained by introducing the Ghost module is related to the proportion of Ghost graphs generated in the total output.

[0103] Step 3: Addressing the multi-scale target detection problem arising during actual product sales under camera recognition, the small target detection and recognition capabilities of the target detection network are optimized. A multi-scale target detection enhancement module for products is designed based on RFP (Recursive-Feature Parymid Network). The structure of the multi-scale target detection enhancement module is as follows: Figure 4 As shown, it consists of three process structures:

[0104] 1) Initial features obtained through top-down feature extraction using RFP Top-down upsampling :

[0105] , , ;

[0106] in, This represents the feature data matrix of each feature layer after the convolutional layer. This represents the feature values ​​of each layer obtained by upsampling the input image, calculated by the forward pass of the neural network. get, It includes a feature extraction network consisting of convolution and pooling layers that performs convolution operations on the input image, amplifying the feature parameters of each feature layer of the input image by a factor of 2 for output; It is a convolutional layer obtained by convolving the input image of each layer. The number of channels is modified by convolution, so that the number of channels in each layer is proportional to the channel count. Consistency in Count the features of each feature layer in the neural network;

[0107] 2) To and Perform lateral connections on the corresponding feature layers to obtain features. :

[0108] ;

[0109] in It is and The characteristic matrices are added bit by bit;

[0110] 3) For horizontal links Perform cyclic feature acquisition, repeating steps 1) and 2) to obtain output features. :

[0111] ;

[0112] In the formula, represent By repeating the upsampling, downsampling, and lateral connection operations in steps 1) and 2), the feature pyramid network is reused to obtain more details of small target features.

[0113] Step 4: To address the issues of product occlusion during actual product sales and users taking multiple items at once, the DIOU_NMS algorithm is introduced and improved to design a multi-target product detection and occlusion recognition module. The improved Cluster_IoU loss calculation is then performed. :

[0114] ;

[0115] in, This represents the intersection-union ratio of the bounding boxes, focusing on the overlapping area of ​​bounding boxes in product target detection and recognition; Represents European distance. and This represents the center point of the two bounding boxes, which together measure the center distance of the normalized detection boxes; This represents the length of the diagonal of the two smallest defined bounding boxes. This indicates the similarity ratio between the anchor and the target bounding box (width and height). Based on this module, optimal weights can be selected for the product detection boxes in occluded areas, without missing other products in the image, thus avoiding false positives and false negatives.

[0116] Step 5: Addressing the user behavior detection requirements in product recognition and the embedded mode of edge intelligent devices, a behavior tracking algorithm is designed to integrate into the product recognition network, establishing a product recognition tracking target detection network at the edge intelligent device. The behavior tracking algorithm consists of two parts: tracking and recognition, and behavior logic judgment. Before product sales, the algorithm sets a baseline L, collects edge points of the product sales cabinet using four coordinate points, and constructs the baseline. During product recognition analysis, the continuous keyframes identified by the network are input into a newly created thread, where the distance S between the center point c of the continuous products and L is compared. The comparison results are as follows: 1) When the sign of the distance changes, a picking / placing behavior is determined; 2) When the distance S suddenly exceeds a threshold, abnormal user picking behavior is determined; 3) When S is 0 for a long time, it is determined that the user has not picked up the product or the recognition is incorrect.

[0117] Step 6: Convert the constructed lightweight product identification and tracking network into an ONNX model, and based on the edge intelligent device Jeston Nano, use TensorRT technology to convert the ONNX version of the lightweight product identification and tracking network model into a model inference engine on the edge intelligent device, and complete the identification and tracking of retail products on the Jeston Nano.

[0118] The network model training framework used is PyTorch, which generates binary weight files through model training. PyTorch's ONNX conversion function is used to package the model's weight values, neural network flow information, input and output information of each layer, and other auxiliary information and export them in ONNX format.

[0119] The ONNX file is read, the model is parsed and optimized. The optimization method is inter-layer fusion and tensor fusion, that is, horizontal or vertical merging between model layers, thereby accelerating the model's calculation speed. The model is then saved as a serialized file to the disk on the edge intelligent device Jeston Nano, which TensorRT can directly read and load later. The model inference operation is monitored by connecting Jeston to a smart vending machine, opening the smart vending machine door, triggering user information acquisition, opening the camera to collect video, simulating the taking operation, closing the smart vending machine door, and viewing the product recognition information output. The accuracy is obtained by repeating multiple sets.

[0120] The retail merchandise identification and tracking deployed on the Jeston Nano follows the same process as the previous network identification and tracking, specifically including:

[0121] (1) Recognition: The user simulates taking the product, and the camera transmits the key frame to the lightweight product recognition and tracking network model deployed in Jeston Nano through the intranet for feature analysis and matching. After successful matching, the matching result is displayed on the LED screen to complete the product recognition.

[0122] (2) Tracking: Before the sale of goods, the behavior tracking algorithm sets the baseline L, collects the edge points of the sales cabinet through four point coordinates, and constructs the baseline. When performing product recognition analysis, the continuous key frames identified by the network are input into a new thread, where the distance S between the center point c of the continuous goods and L is compared. When the sign of the distance changes, the picking and putting behavior is determined. When the distance S suddenly exceeds the threshold, the abnormal picking behavior of the user is determined. When S is 0 for a long time, it is determined that the user has not picked up the goods or the recognition is wrong.

[0123] The lightweight product identification and tracking system deployed on edge intelligent devices provided in this embodiment of the invention includes:

[0124] The data acquisition module is used for the collection and construction of targeted datasets in actual product sales scenarios;

[0125] The feature extraction network building module is designed for embedded application scenarios of intelligent edge devices, and adds a Ghost Module to the backbone feature extraction network to design a new lightweight feature extraction network.

[0126] The detection and recognition optimization module is used to optimize the target detection network for small target detection and recognition, and an enhanced module for multi-scale target detection of goods is designed based on RFP.

[0127] The target detection network construction module is used to introduce and improve the DIOU_NMS algorithm to design a multi-target detection and occlusion recognition module for goods; the behavior tracking algorithm is integrated into the goods recognition network to establish a goods recognition and tracking target detection network at the edge intelligent device.

[0128] The product identification and tracking module is used to convert the constructed lightweight product identification and tracking network into an ONNX model, convert the ONNX version of the lightweight product identification and tracking network model into a model inference engine on the edge intelligent device, and complete the identification and tracking of retail products on the Jeston Nano.

[0129] II. Application Examples:

[0130] This invention was applied and deployed on a newly developed dynamic visual smart vending machine by a technology company to realize the purchase process of various beverages in the smart vending machine. The application implementation case of this invention is based on a lightweight backbone feature extraction network designed using GhostModule, a multi-scale product detection enhancement module designed using RFP, a multi-target product detection and occlusion recognition module designed using an improved DIOU_NMS algorithm, and a product tracking module built using a designed behavior tracking algorithm. These four modules are integrated to construct a lightweight product recognition and tracking target detection network. The network is then converted and deployed using the Jeston Nano edge intelligent device, and connected to the smart vending machine to perform the corresponding product detection and recognition tasks.

[0131] III. Evidence of the relevant effects of the embodiments:

[0132] Simulation Experiment: Based on the lightweight product identification and tracking system designed in this invention, a simulation was conducted in a real-world scenario: The lightweight product identification and tracking system was ported and deployed on the edge intelligent device Jeston Nano, and the Jeston Nano was networked with the smart vending machine's intranet, connecting a display screen and a camera to form a complete product identification and sales system and put it into operation.

[0133] Regarding product retrieval, the model simulates the actual shopping environment of users using vending machines, and collects data and trains models on seven commonly sold beverages in vending machines. The seven beverages are: Coca-Cola bottled, Coca-Cola canned, Pepsi bottled, Mirinda bottled, Pulse bottled, Vita Lemon Tea boxed, and Fanta canned.

[0134] The recognition rate results obtained through simulation are shown in Table 1.

[0135] Table 1. Success Rate of Different Beverage Recognition

[0136]

[0137] As can be seen from Table 1, the smart vending machines that deploy the lightweight product identification and tracking system designed in this invention can achieve an average product identification rate of over 95%, reaching the commercially viable identification accuracy.

[0138] Meanwhile, the lightweight product identification and tracking system and method designed in this invention, after being deployed and tested in the edge intelligent device JestonNano, showed that its identification speed, shopping time consumption, and user picking behavior judgment all reached a high level (see Table 2).

[0139] Table 2. Deployment test results of the Jeston Nano edge smart device

[0140]

[0141] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0143] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A lightweight product identification and tracking method deployed on edge intelligent devices, characterized in that, The lightweight product identification and tracking method deployed on edge intelligent devices includes: Step 1: Collection and construction of targeted datasets for actual product sales scenarios; Step 2: For the embedded application scenario of intelligent edge devices, a new lightweight feature extraction network is designed by adding a Ghost Module to the backbone feature extraction network. Step 3: Design a multi-scale target monitoring enhancement module for goods based on RFP; Step 4: Introduce and improve the Cluster_IoU algorithm to design a multi-target detection and occlusion recognition module for goods; Step 5: Design a product tracking algorithm that integrates into the product recognition network to analyze user behavior, identify user picking and abnormal operations based on continuous keyframes, and establish a lightweight product recognition and tracking network at the edge intelligent device. Step six involves converting the constructed lightweight product identification and tracking network into an ONNX model, transforming the ONNX version of the lightweight product identification and tracking network model into a model inference engine for edge intelligent devices, and completing the identification of retail products and user tracking on the Jeston Nano.

2. The lightweight product identification and tracking method deployed on edge intelligent devices as described in claim 1, characterized in that, The targeted dataset collection and construction in step one includes: During the data collection process, a network camera was installed at the top center of the vending machine to simulate the steps of picking up and putting back items. The video stream from the network camera was acquired via the RTSP protocol, and the video frames were saved. Frames showing 75%, 50%, and 25% occlusion during item handling, as well as frames showing small items, multiple items, and irrelevant items affecting the product, were selected proportionally and combined to form a targeted dataset for network training and detection. The LabelImg software was used to label the items in the saved video frames. Step two, which involves adding a Ghost Module to the backbone feature extraction network to design a new lightweight feature extraction network, includes: The Ghost method generates a feature map in a regular convolution and then generates a Ghost feature map through a simple mapping transformation, which is then concatenated to form the Output. The Output is generated and calculated as follows: The formula for calculating the intrinsic map is as follows: ； in, This represents the input matrix after image convolution in the previous neural network layer. The convolution operation, representing a neural network, is generally represented by... and The convolution kernel is used to perform convolution operations on the image feature matrix. This represents the image feature matrix of the layer completed in this convolution operation; The formula for calculating the Ghost feature map is as follows: ； in, This represents the feature matrix of the feature map generated by the ghost module operation, which is performed by... To achieve this, the feature matrices of each layer from the previous step... After a simple linear calculation, the ghost feature layer is generated through mapping. Adding a Ghost module to the backbone feature extraction network improves the speed of the resulting model. and model size compression ratio for: ； ； in, Input the number of channels. The number of Ghost feature maps generated. This represents the original number of feature maps required. and This represents the length and width of the feature map after convolution. and The kernel size is denoted by ; the speedup of the model obtained by introducing the Ghost module is related to the proportion of Ghost graphs generated in the total output.

3. The lightweight product identification and tracking method deployed on edge intelligent devices as described in claim 1, characterized in that, The multi-scale target monitoring enhancement module for goods in step three consists of three structures: (1) Initial features obtained by top-down feature extraction through RFP Top-down upsampling : ， ， ； in, This represents the feature data matrix of each feature layer after the convolutional layer. This represents the feature values ​​of each layer obtained by upsampling the input image, calculated by the forward pass of the neural network. get, It includes a feature extraction network consisting of convolution and pooling layers that performs convolution operations on the input image, amplifying the feature parameters of each feature layer of the input image by a factor of 2 for output; It is a convolutional layer obtained by convolving the input image of each layer. The number of channels is modified by convolution, so that the number of channels in each layer is proportional to the channel count. Consistency in Count the features of each feature layer in the neural network; (2) To and Perform lateral connections on the corresponding feature layers to obtain features. : ； in It is and The characteristic matrices are added bit by bit; (3) For horizontal links Perform cyclic feature acquisition, repeating steps (1) and (2) to obtain output features. : ； In the formula, represent By repeating the upsampling, downsampling, and lateral connection operations in steps (1) and (2), the feature pyramid network is reused to obtain more small target feature details. In step four, the loss calculation of the improved Cluster_IoU algorithm : ； in, This represents the intersection-union ratio of the bounding boxes, focusing on the overlapping area of ​​bounding boxes in product target detection and recognition; Represents European distance. and This represents the center point of the two bounding boxes, which together measure the center distance of the normalized detection boxes; This represents the length of the diagonal of the two smallest defined bounding boxes. This indicates the similarity ratio between the anchor and the target bounding box in terms of length and width.

4. The lightweight product identification and tracking method deployed on edge intelligent devices as described in claim 1, characterized in that, The product tracking algorithm in step five consists of two parts: tracking and identification, and behavioral logic judgment. Before the product is sold, the product tracking algorithm sets a baseline L by collecting the edge points of the product sales cabinet using four coordinate points and constructing the baseline. During product identification and analysis, the continuous keyframes identified by the network are input into a newly opened thread, where the distance S between the center point c of the continuous product and L is compared. When a change in the sign of the distance occurs, a picking or placing behavior is determined. When a sudden change in distance S exceeds a threshold, abnormal picking behavior by the user is determined. When S is 0 for an extended period of time, it is determined that the user has not taken any goods or that the identification is incorrect. In step six, the constructed lightweight product identification and tracking network is transformed into an ONNX model. Based on the edge intelligent device Jeston Nano, TensorRT technology is used to transform the ONNX version of the lightweight product identification and tracking network model into a model inference engine on the edge intelligent device, and the identification and tracking of retail products are completed on the Jeston Nano.

5. A lightweight product identification and tracking system deployed on an edge intelligent device, employing the lightweight product identification and tracking method described in any one of claims 1 to 4, characterized in that, The lightweight product identification and tracking system deployed on edge intelligent devices includes: The data acquisition module is used for the collection and construction of targeted datasets in actual product sales scenarios; The feature extraction network building module is designed for embedded application scenarios of intelligent edge devices, and adds a Ghost Module to the backbone feature extraction network to design a new lightweight feature extraction network. The detection and recognition optimization module is used to optimize the target detection network for small target detection and recognition, and an enhanced module for multi-scale target detection of goods is designed based on RFP. The target detection network construction module is used to introduce and improve the Cluster_IoU algorithm to design a multi-target detection and occlusion recognition module for goods; a goods tracking algorithm is designed to be integrated into the goods recognition network to analyze user behavior, determine user picking and abnormal operations based on continuous keyframes, and establish a lightweight goods recognition and tracking network at the edge intelligent device. The product identification and tracking module is used to convert the constructed lightweight product identification and tracking network into an ONNX model, convert the ONNX version of the lightweight product identification and tracking network model into a model inference engine on the edge intelligent device, and complete the identification and tracking of retail products on JestonNano.

6. A user input program storage medium, wherein the stored computer program causes an electronic device to perform the lightweight commodity identification and tracking method deployed on an edge intelligent device as described in any one of claims 1 to 4.

7. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the lightweight product identification and tracking method deployed on an edge intelligent device as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the lightweight product identification and tracking method deployed on an edge intelligent device as described in any one of claims 1 to 4.

9. An information data processing terminal, characterized in that, The information data processing terminal is used to provide a user input interface when executed on an electronic device to implement the lightweight commodity identification and tracking system deployed on an edge intelligent device as described in claim 5.

Images