A dynamic visual identification method for goods in a container yard based on multi-level classification

Abstract

The application provides a kind of container scene based on multi-level classification commodity dynamic visual identification method, comprising: S1, according to the characteristics of goods in container, set multi-level label for each commodity;S2, obtain the tracking video of commodity, pre-process video;S3, use target detection network to carry out object detection to each frame image, locate the position where commodity appears, and carry out first level label classification to commodity;S4, combined with light flow detection and logical reasoning, track trajectory of commodity, obtain the trajectory of commodity;S5, use residual classification network to carry out second level label and the fine-grained classification of following;S6, use voting strategy to obtain the last level label of each commodity trajectory, finally obtain the entry and exit record of commodity in container.The application can more effectively extract visual identification information, so as to realize accurate and efficient identification of container goods.

Description

Technical Field

[0001] The present invention relates to the field of deep learning algorithms, and in particular to a method for dynamic visual recognition of commodities based on multi-level classification in a container scenario. Background Art

[0002] In unmanned retail scenarios, how to better identify product information and thus automate transactions has always been a topic of widespread concern.

[0003] At present, the main commodity identification methods in container scenarios include: RFID identification technology, gravity monitoring sensing algorithm, static commodity identification algorithm and dynamic identification algorithm.

[0004] RFID identification technology specifically involves attaching a tag to each product, transmitting wireless data to hardware, and using the input wireless tag information to identify the product. The gravity monitoring sensing algorithm primarily uses weight changes and product location information as input. By matching this information with recorded gravity information, the product category is inferred. The static product recognition algorithm relies on a neural network, detecting and identifying changes in the number of products within the input image, thereby inferring the user's shopping list.

[0005] However, existing product identification technologies are susceptible to interference from factors such as product category and location, resulting in limitations in real-world scenarios. Specifically, RFID technology is unsuitable for metal-packaged goods, making it ineffective for identifying most mainstream products. Furthermore, adding RFID tags to products requires significant labor and effort, significantly exceeding the effort required for labeling datasets. Furthermore, gravity detection technology is often used, so using only gravity variations as a feature is insufficient to uncover the rich product information found in real-world scenarios.

[0006] Static visual recognition algorithms cannot obtain rich video information and cannot track products efficiently. In actual application scenarios, problems such as false detection and missed detection may occur.

[0007] Existing dynamic visual recognition algorithms generally use tracking-based detection strategies. However, due to the actual situation of massive product categories, these algorithms perform poorly on the core issue of classification, which can easily lead to errors in the final results. Summary of the Invention

[0008] The purpose of the present invention is to address the defects existing in the prior art and provide a dynamic visual recognition method for goods in a container scenario based on multi-level classification, so as to more effectively extract visual recognition information and thus achieve accurate and efficient recognition of container goods.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A method for dynamic visual recognition of goods based on multi-level classification in a container scenario, comprising:

[0011] S1. Based on the characteristics of the goods in the container, each item is assigned a multi-level label. The multi-level label is used to classify the goods from coarse to fine. The multi-level label includes at least three levels, where the first level of label is classified according to the packaging form of the goods, the second level of label is classified according to the size of the goods, and the third level and below are classified according to more fine-grained characteristics, until the last level of label is the specific information of the goods.

[0012] S2. Obtain the product tracking video and pre-process the video: After extracting frames from the video, use a segmentation algorithm to segment each frame and remove invalid areas;

[0013] S3. Use the object detection network to detect objects in each frame, locate the location of the product, and classify the product into the first-level label;

[0014] S4. Combine optical flow detection and logical reasoning to track the product and obtain the product's trajectory;

[0015] S5. Use the residual classification network to perform fine-grained classification of products at the second level and below;

[0016] S6. Use the voting strategy to obtain the final level label of each product trajectory, and finally obtain the entry and exit records of the products in the container.

[0017] Furthermore, the multi-level label includes three levels, wherein the first level label is divided according to the packaging form of the product, the second level label is divided according to the volume size of the product, and the third level label is the specific information of the product.

[0018] Furthermore, in said S3, after the target detection network completes the object detection, a detection frame is marked on each frame of the image;

[0019] The S4 specifically includes: tracking the products appearing in each frame of the image based on logic; traversing each frame of the image, using optical flow information to predict the movement trajectory of the product, and after obtaining the predicted trajectory, matching the candidate detection frame to the corresponding trajectory through IOU calculation.

[0020] Furthermore, in S5, the residual classification network has the same number of classifiers as the first-level labels and corresponding classifiers; the residual classification network uses the corresponding classifiers to perform fine-grained classification of the second-level labels and below according to the first-level labels of the input samples.

[0021] Furthermore, the training strategy of the residual classification network is a shared backbone network strategy: each classifier in the residual classification network shares the backbone network, and training samples with different primary labels are iteratively input into the network for classification.

[0022] Furthermore, the training strategy of the residual classification network is a multi-level classification strategy: the labels of each level are trained together, and then the losses of each level are weighted together.

[0023] Furthermore, the training strategy of the residual classification network is a category imbalance processing strategy.

[0024] The present invention provides a method for dynamic visual recognition of goods in container scenarios based on multi-level classification, which can more effectively extract visual recognition information, thereby achieving accurate and efficient recognition of container goods. The present invention classifies goods at multiple levels and then, during object detection, only performs coarse-grained classification of the first-level labels simultaneously. The remaining fine-grained classification is achieved through a residual classification network. Through this approach, the present invention can more accurately and efficiently complete the detection and counting tasks of goods in video images, thereby realizing transaction automation in unmanned container scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of a method for dynamic visual recognition of commodities based on multi-level classification in a container scenario provided by the present invention.

[0026] Figure 2 It is a schematic diagram of the architecture of the multi-level tag in the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, the present invention provides a method for dynamic visual recognition of commodities based on multi-level classification in a container scenario, comprising:

[0029] S1. Based on the characteristics of the goods in the container, each item is assigned a multi-level label. The multi-level label is used to classify the goods from coarse to fine. The multi-level label includes at least three levels, where the first level of label is classified according to the packaging form of the goods, the second level of label is classified according to the size of the goods, and the third level and below are classified according to more fine-grained characteristics, until the last level of label is the specific information of the goods.

[0030] Specifically, if Figure 2As shown, in this embodiment, the multi-level labeling includes three levels. The first-level labeling is divided according to the packaging form of the product. For example, the first-level labeling can be divided into four categories: bottled products, canned products, boxed products, and irregular snack products. The second-level labeling is divided according to the size of the product. For example, the second-level labeling of a bottled product can be divided into two categories: medium-sized and small-sized. The third-level labeling contains specific product information, such as a specific flavor of a specific brand of beverage. Multi-level labeling is used to classify products from coarse to fine granularity.

[0031] S2. Obtain the product tracking video and preprocess the video: After extracting frames from the video, use a segmentation algorithm to segment each frame and remove invalid areas.

[0032] Since there are a large number of areas unrelated to the target in the process of tracking goods, these areas may cause certain interference to further detection and classification. Therefore, a segmentation algorithm is used to segment each frame and eliminate some invalid areas to ensure the accuracy of further processing.

[0033] S3. Use the target detection network to detect objects in each frame, locate the location of the product, and classify the products into first-level labels.

[0034] In actual scenarios, there are a large number of commodity categories. If traditional target detection strategies are used, feature extraction frames need to be set for objects of each category, which will make learning too difficult and the detection effect drop sharply. In the present invention, only object classification of the first-level labels is performed while object detection is performed. The first-level labels only perform coarse-grained classification at the morphological level, which reduces the detection requirements. In this way, the detailed recognition and classification are decoupled from the detection work, which can efficiently track the object information in the video and reduce the difficulty of classification. Through this design method, the dynamic visual recognition model can handle new products that have not been seen before but have similar morphology. The generalization ability of the model is guaranteed, and the learning difficulty of introducing new categories will be greatly reduced.

[0035] S4. Combine optical flow detection and logical reasoning to track the product and obtain the product's trajectory.

[0036] After the object detection network in S3 completes object detection, it marks a detection frame on each frame. In S4, the product that appears in each frame is tracked based on logic. Each frame is traversed, and the product's movement trajectory is predicted using optical flow information. After the predicted trajectory is obtained, the candidate detection frame is matched to the corresponding trajectory through the IoU calculation.

[0037] This algorithm efficiently captures trajectory information obtained after video detection. This stage does not rely on deep learning algorithms, thereby improving the model's interpretability and controllability. The design of logical rules makes uncontrollable errors that may occur after the previous detection step manageable. Furthermore, optical flow detection technology can filter out misdetected objects and address missed objects, improving detection accuracy.

[0038] S5. Use the residual classification network to perform fine-grained classification of products at the second level and below;

[0039] For the detection frames obtained after trajectory tracking, the present invention uses a residual classification network to perform fine-grained classification of the products within them. Since the first-level labels of the products are already obtained in S3, it is not necessary to identify the products in all categories during inference; only the subcategories under the specified first-level labels need to be classified. Specifically, the residual classification network has the same number of corresponding classifiers as the first-level labels; the residual classification network uses the corresponding classifiers to perform fine-grained classification of the second-level labels and below based on the first-level labels of the input samples.

[0040] In order to improve the classification accuracy, the present invention introduces three training strategies for the residual classification network. The first is the shared backbone network strategy: each classifier in the residual classification network shares the backbone network, and the training samples of different first-level labels are iteratively input into the network for classification, so that the backbone network can use common and effective features. Then there is the multi-level classification strategy: the labels of each level are trained together, and then the losses of each level are weighted together, so that the performance of the trained network can be improved through multi-level annotation information. Finally, there is the category imbalance processing strategy. Through this strategy, the present invention can use class-balance technology and Focal Loss (focal loss function) to solve the problem of difficult samples and few samples in actual scenarios where there is an imbalance in the number of category samples.

[0041] S6. Use the voting strategy to obtain the final level label of each product trajectory (i.e., the final category of the product), and finally obtain the entry and exit records of the product in the container.

[0042] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for dynamic visual recognition of commodities based on multi-level classification in a container scenario, characterized by: include: S1. Set multi-level labels for each item according to the characteristics of the items in the container; The multi-level labels are used to classify products from a coarse to fine level. The multi-level labels include at least three levels, wherein the first level of labels is based on the packaging form of the product, the second level of labels is based on the size of the product, and the third level and below are based on more fine-grained features, until the last level of labels contains the specific information of the product. S2. Obtain the product tracking video and pre-process the video: After extracting frames from the video, use a segmentation algorithm to segment each frame and remove invalid areas; S3. Use the object detection network to detect objects in each frame, locate the location of the product, and classify the product into the first-level label; S4. Combine optical flow detection and logical reasoning to track the product and obtain the product's trajectory; S5. Use the residual classification network to perform fine-grained classification of products at the second level and below; S6. Use the voting strategy to obtain the final level label of each product trajectory, and finally obtain the entry and exit records of the products in the container; The multi-level label includes three levels, wherein the first level label is divided according to the packaging form of the product, the second level label is divided according to the size of the product, and the third level label is the specific information of the product; In S3, after the target detection network completes object detection, it marks the detection frame on each frame of the image; S4 specifically includes: tracking the products appearing in each frame of the image based on logic; traversing each frame of the image, using optical flow information to predict the movement trajectory of the product, and after obtaining the predicted trajectory, matching the candidate detection frame to the corresponding trajectory through IOU calculation; In S5, the residual classification network has the same number of classifiers as the first-level labels and corresponding classifiers; the residual classification network uses the corresponding classifiers to perform fine-grained classification of the second-level labels and below according to the first-level labels of the input samples.

2. The method for dynamic visual identification of commodities according to claim 1, characterized in that: The training strategy of the residual classification network is a shared backbone network strategy: each classifier in the residual classification network shares the backbone network, and training samples with different primary labels are iteratively input into the network for classification.

3. The method for dynamic visual identification of commodities according to claim 1, characterized in that: The training strategy of the residual classification network is a multi-level classification strategy: the labels of each level are trained together, and then the losses of each level are weighted together.

4. The method for dynamic visual identification of commodities according to claim 1, characterized in that: The training strategy of the residual classification network is a category imbalance processing strategy.

Images