Intelligent visual retail cabinet commodity tracking method and related device

Through the multi-lens collaborative acquisition of image and motion trajectory analysis algorithm, the accurate tracking and damage detection of products in unmanned smart retail cabinets is achieved, solving the problem of consumers taking it out and putting it back, and improving the accuracy and reliability of the system enhances user trust.

CN118942079APending Publication Date: 2024-11-12SHENZHEN GUANJIAN ZHILIAN TECH CO LTD

Patent Information

Application Number
CN202410920317.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In unmanned smart retail cabinets, it is common for consumers to take out the goods and tear them open the packaging or try them and then put them back, which affects the quality and safety of the goods, weakens consumers' confidence in unmanned smart retail cabinets and Trust.

Method used

Multi-lens collaboratively acquire images, match and correlate targets based on the embedding distance and detecting frame position, track product movement trajectory with motion trajectory analysis algorithm, and checkout processing and product damage analysis are carried out when the cabinet door is locked. If it is detected and taken out, it is done. If the goods that are re-placed are damaged, checkout will be conducted or an alarm signal will be generated.

Benefits of technology

Improve the accuracy and real-time performance of product detection and tracking, ensure the accuracy and reliability of automatic checkout, protect the integrity and security of products, enhance consumers' trust in unmanned smart retail cabinets, and enhance user experience. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942079A_ABST
    Figure CN118942079A_ABST
Patent Text Reader

Abstract

The invention relates to an intelligent visual retail cabinet commodity tracking method and a related device, and the method comprises the following steps: S1, obtaining an image based on multi-lens cooperation, and carrying out the target matching and association based on an embedding distance and a detection frame position; cameras are respectively mounted in different areas in the retail cabinet; s2, based on a motion trail analysis algorithm, tracking a commodity moving trail; s3, integrating the trajectory data of the multiple cameras, and judging the in-out state of the commodity based on the movement direction and position of the commodity; and S4, when it is detected that the cabinet door enters the locking state, the taken-out commodities are subjected to checkout processing, the commodities which are taken out and then put in are subjected to commodity damage analysis, and if it is detected that the commodities which are taken out and then put in are damaged, checkout is carried out or an alarm signal is generated. According to the invention, the accuracy and the real-time performance of an article detection and tracking technology are further improved, and a related monitoring and management mechanism is perfected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to an intelligent visual retail cabinet commodity tracking method and related devices. Background Art

[0002] With the continuous advancement of artificial intelligence and Internet of Things technologies, unmanned smart retail cabinets have rapidly developed and gradually become popular as a new retail model. This new retail method has been widely recognized by the market for its convenience, efficiency and intelligence. Unmanned smart retail cabinets have realized the functions of self-service shopping and automatic checkout through built-in sensors and intelligent recognition systems, significantly improving consumers' shopping experience.

[0003] Object detection and tracking technology is one of the core technologies of unmanned smart retail cabinets. In recent years, computer vision technology based on deep learning has made significant progress in the field of object detection and tracking. These technologies can accurately identify and track the movement of goods through the analysis of image and video data, thereby ensuring accurate management and settlement of goods. In unmanned smart retail cabinets, object detection and tracking technology is widely used to monitor the status and location changes of goods, providing efficient inventory management and anti-theft functions.

[0004] One of the core principles of unmanned retail cabinets is to use visual technology to achieve intelligent management. Specifically, users can open the cabinet door by themselves and freely select products from the shelves. When the user closes the cabinet door, the camera and sensor inside the cabinet will capture and record the status and position changes of the products in real time. Through computer vision algorithms, the system can identify the products taken by the user and automatically complete the checkout and deduction process. This process does not require human intervention, greatly improving the convenience and efficiency of shopping.

[0005] Although unmanned smart retail cabinets have made many advances in technology and application, there are still some shortcomings. At present, consumers often take out goods, tear open the packaging, or put them back into the cabinet after trying them. This behavior not only affects the quality and safety of the goods, but also weakens consumers' confidence and trust in unmanned smart retail cabinets. Summary of the invention

[0006] In order to further improve the accuracy and real-time performance of item detection and tracking technology, and improve related monitoring and management mechanisms, the present application provides an intelligent visual retail cabinet merchandise tracking method and related devices.

[0007] In the first aspect, the present application provides a method for tracking merchandise in an intelligent visual retail cabinet, which adopts the following technical solution:

[0008] An intelligent visual retail cabinet commodity tracking method comprises the following steps:

[0009] S1. Acquire images based on multi-lens collaboration, and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet;

[0010] S2. Track the movement trajectory of goods based on the motion trajectory analysis algorithm;

[0011] S3. Integrate the trajectory data of multiple cameras and determine the entry and exit status of the goods based on the movement direction and position of the goods;

[0012] S4. When it is detected that the cabinet door has entered the locked state, the taken-out goods are checked out, and the taken-out and put-back goods are analyzed for damage. If it is detected that the taken-out and put-back goods are damaged, the goods are checked out or an alarm signal is generated.

[0013] Optionally, S4 includes the following sub-steps:

[0014] S41. Determine whether the type of the commodity to be determined is a first-category commodity or a second-category commodity; wherein the commodity to be determined is a commodity that is taken out and then put in again, the first-category commodity is a commodity with a screw-on cap, and the second-category commodity is a commodity with a seal having a toothed tangent line;

[0015] S42. For the first category of goods, determine whether the connecting bridge between the cap body and the anti-theft ring in the twistable bottle cap is broken; if broken, checkout or generate an alarm signal;

[0016] S43. For the second category of goods, determine whether the seal with the serrated cut line is torn; if torn, checkout is performed or an alarm signal is generated.

[0017] Optionally, the S42 includes the following sub-steps:

[0018] S421. Preprocess the image and use the instance segmentation algorithm to capture the unscrewable bottle cap area of ​​the product to be determined;

[0019] S422. The area of ​​the bottle cap that can be twisted is intercepted and standardized;

[0020] S423. Input the standardized image of the twistable bottle cap area into the pre-trained first convolutional neural network to obtain a judgment result; wherein, the first convolutional neural network is trained by a data set containing two types of samples of the cap body and the anti-theft ring, broken and non-broken, and marking the samples corresponding to the broken state, and the output result of the model is the anti-theft ring is broken and non-broken.

[0021] Optionally, the S43 includes the following sub-steps:

[0022] S431. Preprocess the image and use the instance segmentation algorithm to capture the sealing area of ​​the product to be determined;

[0023] S432. The sealing area is intercepted and standardized;

[0024] S433. Input the standardized sealing area image into the pre-trained second convolutional neural network to obtain a judgment result; wherein, the second convolutional neural network is trained by a data set containing two types of samples, one with intact tooth tangents and one with torn teeth, and the corresponding fracture states of the samples are marked, and the output results of the model are torn and non-torn seals.

[0025] Optionally, the S41 includes the following steps:

[0026] S411. Preprocess the image and extract the feature vector of the product from the preprocessed image;

[0027] S412. Match the extracted feature vector with the feature vector in the product database to determine the specific type of product;

[0028] S413. According to the recognition result, the type of matching goods is searched in the goods type database, wherein the goods type database includes the goods name, goods ID, goods picture and its corresponding type;

[0029] S414. According to the database query result, confirm whether the type of the commodity to be determined is a first-category commodity or a second-category commodity.

[0030] In the second aspect, the present application provides an intelligent visual retail cabinet commodity tracking device, which adopts the following technical solution:

[0031] An intelligent visual retail cabinet commodity tracking device, comprising:

[0032] The target matching and association module is used to acquire images based on multi-lens collaboration and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet;

[0033] The trajectory analysis module is used to track the movement trajectory of goods based on the motion trajectory analysis algorithm;

[0034] The entry and exit judgment module is used to integrate the trajectory data of multiple cameras and judge the entry and exit status of the goods based on the movement direction and position of the goods;

[0035] The damage judgment module is used to check out the taken out goods when it detects that the cabinet door has entered a locked state, and to analyze the damage of the goods taken out and put back in. If it is detected that the goods taken out and put back in are damaged, check out or generate an alarm signal.

[0036] In a third aspect, the present application provides a computer device, which adopts the following technical solution:

[0037] A computer device comprising:

[0038] one or more processors;

[0039] Memory;

[0040] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the above-mentioned smart visual retail cabinet product tracking method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0042] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above method.

[0043] The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement: the above-mentioned smart visual retail cabinet product tracking method.

[0044] In summary, the present application includes at least one of the following beneficial technical effects:

[0045] 1. By integrating the trajectory data of multiple cameras, the system can accurately determine the entry and exit status of goods and automatically check out when the cabinet door is closed. In particular, when it is detected that the goods are taken out and put back in, the damage of the goods can be analyzed to ensure the accuracy and reliability of automatic checkout.

[0046] 2. This solution solves the problem of consumers taking out products, tearing them open or trying them out, and then putting them back, avoiding the situation where products are damaged or sold again after being tried out. This not only protects the integrity and safety of the products, but also enhances consumers' trust in unmanned smart retail cabinets and improves users' shopping experience.

[0047] 3. Based on the motion trajectory analysis algorithm, the system can accurately track the movement trajectory of each product, and keep continuous tracking records no matter how the product moves in the cabinet. This effectively prevents the problem of products being stolen or misplaced and unable to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of an application environment of a method for tracking merchandise in an intelligent visual retail cabinet according to an embodiment of the present invention is shown.

[0049] Figure 2 A flowchart of a method for tracking merchandise in an intelligent visual retail cabinet according to an embodiment of the present invention is shown.

[0050] Figure 3 A flowchart of sub-step S1 in one embodiment of the present invention is shown.

[0051] Figure 4 A flowchart of sub-step S2 in one embodiment of the present invention is shown.

[0052] Figure 5 A flowchart of sub-step S3 in one embodiment of the present invention is shown.

[0053] Figure 6 A flowchart of sub-step S4 in one embodiment of the present invention is shown.

[0054] Figure 7 A schematic diagram of a computer device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0055] The present application is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this specification, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid making the disclosed principles complicated and difficult to understand. For the sake of clarity, not all features of the actual implementation are necessary to be described. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes, and may not be selected to delineate or limit the subject matter of the present invention, thereby resorting to the necessary claims to determine such inventive subject matter. References to "a specific implementation" or "specific implementations" in this disclosure mean that the specific features, structures or characteristics described in conjunction with the specific implementation are included in at least one specific implementation, and multiple references to "a specific implementation" or "specific implementations" should not be understood as necessarily referring to the same specific implementation.

[0057] Unless expressly limited, the terms "a", "an", and "the" are not intended to refer to a singular entity, but rather to include general categories of which specific examples may be used for illustration. Thus, the use of the terms "a" or "an" may mean any number of at least one, including "one", "one or more", "at least one", and "one or more than one". The term "or" means any of the alternatives and any combination of the alternatives, including all alternatives, unless the alternatives are expressly indicated to be mutually exclusive. The phrase "at least one of" when combined with a list of items refers to a single item in the list or any combination of the items in the list. The phrase does not require all of the listed items unless expressly limited to that.

[0058] The present application embodiment discloses a method for tracking goods in a smart visual retail cabinet. The method for tracking goods in a smart visual retail cabinet provided by the present invention can be applied in the following aspects: Figure 1 In the application environment, the intelligent visual retail cabinet commodity tracking method is applied in the intelligent visual retail cabinet commodity tracking system, and the intelligent visual retail cabinet commodity tracking system includes a client and a server, wherein the client communicates with the server through a network. The client, also known as the user end, refers to a program corresponding to the server that provides local services to the client. Furthermore, the client is a computer-side program, an APP program for a smart device, or a third-party applet embedded in other APPs. The client can be installed on, but not limited to, various computer devices such as personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0059] Reference Figure 2 , an intelligent visual retail cabinet commodity tracking method includes the following steps S1-S4.

[0060] S1. Acquire images based on multi-lens collaboration, and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet.

[0061] Specifically, refer to Figure 3 , S1 includes the following steps S11-S19.

[0062] S11. Collect continuous images and perform image preprocessing.

[0063] Multiple cameras are configured inside and outside the smart retail cabinet to ensure that all product areas on the shelf are covered to avoid visual blind spots. Image data from multiple cameras is collected in real time to ensure that the image quality and frame rate meet the requirements. The collected images are pre-processed, such as denoising, cropping and correction, to ensure image clarity and consistency. In other embodiments, each frame of the image may also be pre-processed, such as grayscale and denoising, to enhance the feature extraction effect.

[0064] S12. Use the object appearance feature recognition module to extract feature vectors from the preprocessed image.

[0065] Use an object appearance feature recognition module, such as a ResNet network, to extract feature vectors from the preprocessed images. Feature vectors capture high-dimensional features such as color, shape, and texture of the product for subsequent matching and association.

[0066] Specifically, when the system detects a product for the first time, it generates an initial feature vector for the product through the ResNet network, and then stores the generated feature vector together with the product's initial position and ID in the system's tracking list. These feature vectors are used to match and associate newly detected products in subsequent frame processing.

[0067] It should be noted that in the subsequent operations, in each frame of the image, the newly detected product feature vector needs to be matched with the feature vector of the known product. By calculating the embedding distance (such as Euclidean distance or cosine distance) between the new feature vector and the known feature vector, their similarity is determined. Feature vectors with smaller embedding distances are paired and considered to be the same product.

[0068] The "known product feature vectors" mentioned above refer to the feature vectors of products that have been recorded and tracked in the system in each frame of the image. These feature vectors are extracted from each frame of the image through a deep learning model (such as the ResNet network) to represent the appearance characteristics of the product. They are continuously updated and used during the tracking process to achieve continuous tracking and association of products.

[0069] S13. Calculate the embedding distance between the newly detected product feature vector and the known product feature vector Among them, f i and g i are the i-th component of the product feature vector and the known product feature vector respectively.

[0070] Embedding distance is a method to measure the similarity or distance between two points (usually feature vectors) in a high-dimensional space. In the field of deep learning and computer vision, feature vectors (embedding) are extracted by neural networks to represent the high-dimensional features of objects. Embedding distance is used to measure the similarity of two feature vectors in a high-dimensional space.

[0071] S14. Calculate the center distance between the newly detected product detection frame and the known product detection frame Among them, (x1, y1) and (x2, y2) are the center coordinates of the two detection boxes respectively.

[0072] The reason for calculating the center distance between the newly detected product detection frame and the known product detection frame is to improve the accuracy of target tracking. The center distance combined with the embedding distance can more comprehensively reflect the location and appearance characteristics of the product.

[0073] Specifically, for each detection box, calculate the coordinates of its center point Among them, (x min ,x max ,y min ,y max ) is the boundary coordinate of the detection frame. Then calculate the center distance between the newly detected product detection frame and the known product detection frame Finally, calculate the embedding distance between the newly detected product feature vector and the known product feature vector Among them, f i and g i are the i-th components of the two eigenvectors respectively.

[0074] S15. Use the weighted method to perform preliminary matching and calculate the weighted distance d combined =αd embedding +βd center , where α and β are weight parameters, and α>β.

[0075] S16. Use the Hungarian algorithm to perform many-to-many matching, find the matching pair with the smallest total distance, and update the matching pair list and the unmatched detection result list.

[0076] In this step, we first define the distance matrix D, where D ij It represents the comprehensive distance between the i-th newly detected product and the j-th known product. The comprehensive distance is obtained by weighted summation of the embedding distance and the center distance: D ij =αd embedding (i,j)+β center(i,j), where α and β are weight parameters.

[0077] The Hungarian algorithm (also known as the Kuhn-Munkres algorithm) is used to find the matching pair with the minimum total distance on the distance matrix D. The Hungarian algorithm finds the optimal matching pair between the new detection result and the known product through iterative optimization.

[0078] According to the output of the Hungarian algorithm, the matching pair list is updated. For each matching pair, the IDs of the newly detected goods and known goods are recorded, and the feature vector and position of the tracking object are updated. For the detection results that are not matched by the Hungarian algorithm, the unmatched detection result list is updated for subsequent processing (such as rematching or initialization as a new tracking object).

[0079] For example, suppose there are 3 newly detected products and 3 known products in the system:

[0080] Newly detected products: A, B, C

[0081] Known products: 1, 2, 3

[0082] The distance matrix D is as follows:

[0083]

[0084] For each pair of newly detected products and known products, the embedding distance and center distance are calculated and weighted summed to obtain the comprehensive distance. The matching pair with the smallest total distance is found through the Hungarian algorithm. For example, suppose the output result of the Hungarian algorithm is the matching pair: A-1, B-3, C-2.

[0085] The list of matching pairs is updated to:

[0086] A-1: Distance 0.5

[0087] B-3: Distance 0.7

[0088] C-2: Distance 0.9

[0089] Assuming there are no unmatched detection results, the unmatched list is empty.

[0090] S17. Re-match the high-score and low-score detection frames of the unmatched detection results to improve the matching rate.

[0091] This step is to deal with the detection results that failed to match successfully in the preliminary matching and Hungarian algorithm, and to minimize missed detection and false detection through further analysis and re-matching.

[0092] Specifically, we first select detection frames with higher scores from the unmatched detection results. High-scoring detection frames refer to those detection results that have higher confidence scores during the detection process but fail to match in the preliminary matching or Hungarian algorithm. We recalculate the feature vectors of these high-scoring detection frames and compare them with the feature vectors of known products. We calculate the embedding distance and center distance to generate a comprehensive distance. Then we use the weighted distance d combined Matching is performed to try to match these high-scoring detection frames with known products.

[0093] Similarly, select low-scoring detection frames from unmatched detection results. Low-scoring detection frames refer to those detection results with low confidence scores during the detection process. Recalculate the feature vectors for these low-scoring detection frames and compare them with the feature vectors of known products. Calculate the embedding distance and center distance to generate a comprehensive distance. Then use weighted distance for matching and try to match these low-scoring detection frames with known products.

[0094] Finally, the matching results of high-score and low-score detection boxes are updated to the matching pair list. The IDs of newly detected items and known items are recorded, and the feature vector and position of the tracked object are updated.

[0095] For example, suppose that in the preliminary matching and Hungarian algorithm matching, there are the following unmatched detection results:

[0096] High-score detection boxes: D (confidence 0.95), E (confidence 0.90)

[0097] Low score detection box: F (confidence 0.60), G (confidence 0.55)

[0098] First, perform high-score detection frame matching:

[0099] Extract detection boxes D and E. Recalculate the feature vectors of detection boxes D and E and compare them with the known product feature vectors.

[0100] Matching: Perform weighted distance calculation on detection boxes D and E to try to match known products.

[0101] Assume the matching result is:

[0102] D matches known product 4, with a comprehensive distance of 0.6

[0103] E matches known product 5, with an overall distance of 0.7.

[0104] Then perform low-score detection frame matching:

[0105] Extract the detection frames F and G. Recalculate the feature vectors of the detection frames F and G and compare them with the feature vectors of known products. Perform weighted distance calculation on the detection frames F and G and try to match them with known products.

[0106] Assume the matching result is:

[0107] F matches known products 6, with a comprehensive distance of 0.8

[0108] G failed to match successfully.

[0109] Then update the matching pair list to:

[0110] D-4: Comprehensive distance 0.6

[0111] E-5: Comprehensive distance 0.7

[0112] F-6: Comprehensive distance 0.8.

[0113] Update the unmatched list. The unmatched list is updated to:

[0114] G(confidence 0.55).

[0115] S18. For the detection results that are not matched in all the association steps, initialize them as new tracking objects and add them to the tracking list.

[0116] The purpose of this step is to ensure that the system can handle new products and avoid missing any.

[0117] Specifically, first extract all the detection results that fail to match known products from the unmatched detection result list. Then generate a new unique ID for each unmatched detection result for tracking. Record the initial position, feature vector, and ID of each new tracked object and store them in the tracking list. Finally, add the new tracked object to the tracking list and prepare to continue tracking in subsequent frames.

[0118] S19. Update the status of the tracked object and the lost object according to the association result; for the tracked object that is not matched, increase its lost time by 1; for the matched object, reset its lost time to 0. If the lost time of the tracked object exceeds the maximum lost time, remove it from the tracking list and add it to the removal list.

[0119] For example, suppose we have the following objects in the tracking list:

[0120] J(ID:101, lost time:0)

[0121] K(ID:102, lost time:2)

[0122] L(ID:103, lost time:1)

[0123] Matching object state update: Assuming J is successfully matched, reset J’s loss time to 0 and update its latest position and feature vector.

[0124] Unmatched object status update: Assuming K and L do not match, increase the loss time of K by 1 (become 3) and the loss time of L by 1 (become 2).

[0125] Check loss time: If the maximum loss time is set to 2, then K exceeds the maximum loss time and should be removed.

[0126] Add to removal list: Remove K from the tracking list and add it to the removal list.

[0127] For another example, suppose that during the matching process of S16 and S17, the following results are obtained:

[0128] Successfully matched objects: A, B, C, D, E, F.

[0129] Unmatched objects: G,H,I.

[0130] In S18: New tracking objects are generated for H and I and added to the tracking list.

[0131] In S19: update the state of the successfully matched object and reset its lost time. Update the lost time of the unmatched object G and add 1. Check the objects in the tracking list and remove the objects whose lost time exceeds the threshold.

[0132] S2. Track the movement trajectory of goods based on the motion trajectory analysis algorithm.

[0133] Specifically, refer to Figure 4 , the steps of S2 include S21-S29.

[0134] S21. Set the tracking threshold and maximum loss time, extract the initial position, ID and feature vector of each product from the matching results of S1, and initialize the tracking object list.

[0135] Here you can set tracking parameters such as tracking threshold and maximum loss time. The tracking threshold is used to control the maximum distance difference accepted during the matching process. If the distance between two feature vectors is less than the threshold, they are considered to be matched. Therefore, by setting the tracking threshold, you can filter out those non-matching objects with too large a distance to ensure the accuracy of the matching. For example, set the tracking threshold to 0.5.

[0136] The maximum loss time is used to control the number of frames after which the system stops tracking a target. If a target is not matched in several consecutive frames, it is considered lost. By setting the maximum loss time, temporary losses caused by short-term occlusion or detection errors can be handled to avoid giving up tracking too early, and long-term lost targets can be removed in time. For example, the maximum loss time is set to 5 frames.

[0137] S22. Use the feature vector extracted in S1 to fuse the features of multiple frames to generate a fused feature vector Among them, f i is the feature vector of the i-th frame, and N is the number of frames.

[0138] For example, first, extract feature vectors of the same product from multiple consecutive frames. For example, for product A, extract feature vectors from frames 1, 2, and 3.

[0139] Then, the feature vectors from multiple frames are fused to generate a robust feature vector Among them, f i is the feature vector of the i-th frame, and N is the number of frames. Of course, the subscript i can be moved to the superscript for easier understanding. For example, suppose the feature vectors of product A in frames 1, 2, and 3 are:

[0140]

[0141] Calculate the fused feature vector:

[0142]

[0143] Update the feature vector of product A to .

[0144] S23. Predict the position and speed of the product in the next frame based on the trajectory prediction model selected according to the current state.

[0145] This step predicts the next location of the product through a trajectory prediction model (such as a Kalman filter or LSTM network), ensuring that the system can continue to track the product in subsequent frames, even if the product is temporarily out of the camera's field of view.

[0146] Specifically, the model is selected first. It is necessary to select a suitable trajectory prediction model according to the motion state of the tracked object. For example, for products with linear motion, a Kalman filter is used; for products with complex motion, an LSTM network is used. Since users do not perform very complex movements when taking out products, the Kalman filter is used as an example here. As for the use of the Kalman filter as a conventional method, only a simple example is given here. First, initialize the state of the Kalman filter, including parameters such as position, speed, and acceleration. Then initialize the LSTM network and load the pre-trained model. Use the selected prediction model to predict the position and speed of the product in the next frame.

[0147] S24. Match positions and features based on weighted distance.

[0148] This step combines the weighted distance of position and features for matching, ensuring that the detection of the product in the next frame can be accurately associated with the previously tracked object.

[0149] Specifically, first calculate the position distance between the detection result of the current frame and the prediction result. Then calculate the feature distance (embedding distance) between the detection result of the current frame and the prediction result, and calculate the weighted comprehensive distance of the position distance and the feature distance. Finally, match the detection result of the current frame with the prediction result according to the weighted comprehensive distance.

[0150] S25. According to the trajectory prediction result, correct and match the new detection result of the current frame.

[0151] This step corrects and matches the new detection results of the current frame based on the trajectory prediction results. Specifically, the trajectory prediction results are first used to correct the position and features of the detection results in the current frame. Then, the corrected detection results are used to re-match to ensure the accuracy of the match.

[0152] S26. For the unmatched detection results, initialize them as new tracking objects, generate new IDs, and record the initial positions and feature vectors.

[0153] This step ensures that the system can process new product detection results and include them in the tracking list for tracking in subsequent frames.

[0154] S27. Record the motion trajectory data of each tracked object, including the detection box position, feature vector and time of each frame.

[0155] S28. Remove the tracking objects that have exceeded the maximum loss time to maintain the validity of the tracking list. Remove the tracking objects that have exceeded the maximum loss time to maintain the validity of the tracking list.

[0156] S29. Analyze the trajectory data of each tracked object, identify its movement pattern and behavior characteristics, and identify abnormal movement trajectories, mark them as abnormal behaviors, and perform further processing.

[0157] For example, for S26-S29, assume that in an intelligent visual retail cabinet system, there are the following tracking objects in the initial state:

[0158] Tracking object A (ID: 1, position: (10, 20), feature vector: [0.1, 0.2, 0.3])

[0159] Tracking object B (ID: 2, position: (30, 40), feature vector: [0.4, 0.5, 0.6]).

[0160] During processing, newly detected items have the following unmatched detection results:

[0161] Detection result C (position: (50, 60), feature vector: [0.7, 0.8, 0.9])

[0162] Detection result D (position: (70,80), feature vector: [1.0,1.1,1.2]).

[0163] 1. Extract unmatched test results: Extract test results C and D.

[0164] 2. Generate new IDs: Generate new ID 3 for C and new ID 4 for D.

[0165] 3. Record the initial state:

[0166] The list of tracked objects is updated as follows:

[0167] Tracking object A: ID 1, location (10, 20), feature vector [0.1, 0.2, 0.3]

[0168] Tracking object B: ID 2, location (30, 40), feature vector [0.4, 0.5, 0.6]

[0169] Tracking object C: ID 3, location (50, 60), feature vector [0.7, 0.8, 0.9]

[0170] Tracked object D: ID 4, position (70,80), feature vector [1.0,1.1,1.2].

[0171] 4. Record the data of each frame:

[0172] In each frame, the positions, feature vectors and timestamps of all tracked objects are recorded.

[0173] Example data:

[0174] Frame t1: object A (position: (11, 21), feature vector: [0.11, 0.21, 0.31]), timestamp t1

[0175] Frame t2: object B (position: (31, 41), feature vector: [0.41, 0.51, 0.61]), timestamp t2

[0176] ….

[0177] 5.Store trajectory data:

[0178] The data recorded for each frame is stored in the trajectory database.

[0179] 6. Traverse the tracking list:

[0180] Check the loss time of all tracked objects.

[0181] Assume that the loss time of the tracked object B is 6 frames, which exceeds the maximum loss time by 5 frames.

[0182] 7. Remove object: Remove tracking object B from the tracking list.

[0183] 8. Update list:

[0184] The updated tracking list is:

[0185] Tracking object A: ID 1, location (11, 21), feature vector [0.11, 0.21, 0.31]

[0186] Tracking object C: ID 3, location (50, 60), feature vector [0.7, 0.8, 0.9]

[0187] Tracked object D: ID 4, position (70,80), feature vector [1.0,1.1,1.2].

[0188] S3. Integrate the trajectory data of multiple cameras and determine the entry and exit status of the goods based on the movement direction and position of the goods.

[0189] Specifically, refer to Figure 5 , the steps of S3 include S31-S38.

[0190] S31. Align image data from different cameras according to timestamps.

[0191] Ensure that the timestamps of all cameras are synchronized and calibrate the clocks of the cameras for subsequent data integration. For example, use a time synchronization protocol (such as NTP) or a physical synchronization signal for time calibration.

[0192] S32. Identify and match the same product trajectories from different cameras, use feature vectors and detection frame positions for matching, and fuse the trajectory data from multiple cameras.

[0193] Specifically, we first extract the feature vectors of the products from the images of different cameras, use the pre-trained deep learning model (such as ResNet) to extract the product features, and calculate the embedding distance between the feature vectors extracted from different cameras. Among them, f i and g j They are the feature vectors from different cameras.

[0194] Then extract the detection frame positions of the product in different cameras and calculate the distance between the center points of the detection frames Among them, (x i ,y i ) and (x j ,y j) are the center point coordinates of the detection frames in different cameras.

[0195] Finally, the weighted comprehensive distance d is calculated by combining the embedding distance and the center distance. combined (i,j)=αd embedding (i,j)+βd center (i,j). Use weighted comprehensive distance for matching to find the same product trajectory from different cameras. Fuse the matched data of the same product trajectory to generate a unified product trajectory, and update the position and feature vector of the product in the unified trajectory.

[0196] S33. For each tracked object, calculate its motion direction vector in consecutive frames, and use the coordinate difference between the current position and the previous position to calculate the direction vector d = (x t -x t-1 ,y t -y t-1 ), where (x t ,y t ) is the current frame position, x t-1 ,y t-1 ) is the previous frame position.

[0197] The purpose of this step is to analyze the movement trajectory of the product by calculating the direction vector. Specifically, firstly, the coordinates of the center point of the detection frame of the product are extracted from each frame image, and the coordinate difference between the current position and the previous position is calculated.

[0198] Δx=x current -x previous

[0199] Δy=y current -y previous

[0200] Finally, the coordinate difference is used to calculate the direction vector d=(Δx, Δy), where d is the direction vector.

[0201] S34. Smooth the direction vectors of consecutive frames to reduce jitter of direction changes.

[0202] First, collect the direction vector data of the product from multiple consecutive frames. For example, for a certain product, the direction vectors extracted in frames t1, t2, and t3 are (d1, d2, d3) respectively. Select a suitable smoothing algorithm for processing. Common smoothing algorithms include moving average filtering, Kalman filtering, etc.

[0203] S35. Different functional areas are defined in the cabinet. For each tracked object, the coordinates of the center point of the detection frame are used to determine the area to which the position belongs, and to determine whether the position of the current frame enters or leaves the functional area.

[0204] For example, different functional areas are defined in a smart retail cabinet, such as shelf areas, import and export areas. For example:

[0205] Shelf area: defined as the rectangular area from (x1, y1) to (x2, y2).

[0206] Import and export area: defined as the rectangular area from (x3, y3) to (x4, y4).

[0207] Then, the coordinates of the center point of the detection frame of each tracked object in the current frame are extracted. According to the coordinates of the center point of the detection frame, the functional area to which the position of the tracked object in the current frame belongs is determined.

[0208] S36. Define the entry and exit status of the tracked object, wherein the entry and exit status includes entry, exit, and unchanged.

[0209] S37. Determine the state transition of the tracked object according to the trajectory data and the movement direction.

[0210] S38. Update the entry and exit status of the tracked object in real time according to the latest trajectory data and state transition rules.

[0211] According to the state transition result of S37, update the entry and exit status of the tracked object. If the state is "enter", update the object position to the new area. If the state is "leave", mark the object as leaving. Record the time and location of the state change, and trigger the corresponding operation (such as checkout, alarm, etc.) in real time according to the state change.

[0212] For example, there are the following functional areas and location information of tracked objects:

[0213] Functional Areas:

[0214] Shelf area: (10,20) to (50,60).

[0215] Import and export area: (0,0) to (10,20).

[0216] Track the position and orientation of object A in three consecutive frames:

[0217] Frame t1: position (15,30) (in the shelf area), orientation (2,5).

[0218] Frame t2: position (5,10) (in the entrance / exit area), direction (-2,-5).

[0219] Frame t3: position (0,0) (leaving the entrance and exit area), direction (-5,-10).

[0220] Then, the trajectory and direction from frame t1 to t2 are: position (15,30) to (5,10), direction (2,5).

[0221] Trajectory and direction from frame t2 to t3: position (5,10) to (0,0), direction (-5,-10).

[0222] Then determine the state transition from frame t1 to t2, and the state transition from frame t2 to t3. According to the state transition results, update the state of the tracked object, record the time and location of the state change, and trigger corresponding operations, such as checkout and record entry.

[0223] S4. When it is detected that the cabinet door has entered the locked state, the taken-out goods are checked out, and the taken-out and put-back goods are analyzed for damage. If it is detected that the taken-out and put-back goods are damaged, the goods are checked out or an alarm signal is generated.

[0224] Specifically, refer to Figure 6 , S4 includes the following sub-steps S41-S43.

[0225] S41. Determine whether the type of the commodity to be determined is a first-category commodity or a second-category commodity; wherein the commodity to be determined is a commodity that is taken out and then put back in, the first-category commodity is a commodity with a screw-on bottle cap, and the second-category commodity is a commodity with a seal having a toothed tangent line.

[0226] S41 is used to determine whether the commodity to be determined is a first-category commodity or a second-category commodity. Specifically, by identifying the appearance characteristics of the commodity, it is determined whether the commodity has a screw-on cap (first category) or a toothed tangent seal (second category). This step is to prepare for the subsequent determination of the damage of the commodity (such as a broken anti-theft ring or a torn seal).

[0227] Specifically, S41 includes the following steps S411-S414.

[0228] S411. Preprocess the image and extract the feature vector of the product from the preprocessed image.

[0229] S412. Match the extracted feature vector with the feature vector in the commodity database to determine the specific type of the commodity.

[0230] The system uses a deep learning model to extract high-dimensional feature vectors of product images. This feature vector represents the appearance characteristics of the product. The extracted feature vector is then matched with a pre-stored product feature vector database. The most matching product type is determined by calculating the similarity between feature vectors (such as Euclidean distance or cosine similarity). Finally, the specific type of the product is confirmed based on the matching results.

[0231] For example, suppose the feature vector of the product to be determined is [0.12, 0.22, 0.33], and the product feature vector database pre-stored in the system is as follows:

[0232] Yibao: feature vector [0.1, 0.2, 0.3], type: first category product (twist-off bottle cap)

[0233] Lays potato chips: feature vector [0.4, 0.5, 0.6], type: second category product (serrated tangent seal)

[0234] 1. Extract feature vectors:

[0235] The feature vector of the product to be determined is [0.12, 0.22, 0.33].

[0236] 2. Database matching:

[0237] Compare [0.12, 0.22, 0.33] with the feature vectors in the database.

[0238] Calculate the distance to Yibao’s eigenvector [0.1, 0.2, 0.3], and the result is closer.

[0239] Calculate the distance to the Lay's potato chips feature vector [0.4, 0.5, 0.6], and the result is far.

[0240] 3. Confirm the product type:

[0241] Confirm that the product to be determined is Yilibao and its type is the first category of products (twist-off bottle cap).

[0242] S413. According to the recognition result, query the type of the matching product in the product type database, wherein the product type database includes the product name, product ID, product image and its corresponding type.

[0243] Through the matching process of S412, the identification result of the commodity is obtained, including the commodity name and feature vector. According to the identification result, the detailed information of the commodity is queried in the commodity type database. The database includes information such as the commodity name, feature vector, type, and picture. According to the query result, the detailed information of the commodity is confirmed, including the specific type of the commodity (first category or second category).

[0244] S414. According to the database query result, confirm whether the type of the commodity to be determined is a first-category commodity or a second-category commodity.

[0245] S42. For the first category of goods, determine whether the connecting bridge between the cap body and the anti-theft ring in the twistable bottle cap is broken; if broken, checkout is performed or an alarm signal is generated.

[0246] Specifically, S42 includes the following sub-steps S421-S423.

[0247] S421. Preprocess the image and use the instance segmentation algorithm to capture the twistable bottle cap area of ​​the product to be determined.

[0248] Convert the original color image into a grayscale image, and perform denoising on the grayscale image to remove the noise in the image, such as using a Gaussian filter or a median filter to smooth the image and reduce noise interference.

[0249] Use an instance segmentation algorithm (such as Mask R-CNN) to segment the preprocessed image and extract the area of ​​the product where the bottle cap can be twisted. For example, input the preprocessed image into the instance segmentation model to obtain the segmentation mask of each part of the product. Using the instance segmentation result, capture the area of ​​the product where the bottle cap can be twisted. Extract the part representing the bottle cap that can be twisted from the segmentation mask and generate a new image area for subsequent analysis.

[0250] S422. The area of ​​the screw-on bottle cap is cut and standardized.

[0251] The purpose of this step is to extract and standardize the area of ​​the bottle cap that can be twisted. The purpose of this step is to convert the extracted area of ​​the bottle cap that can be twisted into a unified format and size.

[0252] For example, by extracting the part of the segmentation mask that represents the screwable bottle cap, it is cut out from the original image to generate a sub-image containing the screwable bottle cap. The cutout screwable bottle cap area is resized to a fixed size, such as resizing the sub-image to 224x224 pixels, to ensure that all input images have the same size.

[0253] S423. Input the standardized image of the twistable bottle cap area into the pre-trained first convolutional neural network to obtain a judgment result; wherein, the first convolutional neural network is trained by a data set containing two types of samples of the cap body and the anti-theft ring, broken and non-broken, and marking the samples corresponding to the broken state, and the output result of the model is the anti-theft ring is broken and non-broken.

[0254] The purpose of this step is to input the standardized image of the twistable bottle cap area into the pre-trained first convolutional neural network to obtain the judgment result. Through the pre-trained convolutional neural network (CNN), the system can automatically identify and judge whether there is damage in the bottle cap area (such as a broken anti-theft ring), thereby realizing automated product damage detection.

[0255] First, use a pre-trained convolutional neural network model that has been trained on a large number of bottle cap images with damaged and non-damaged annotations. For example, load a ResNet or VGG model trained on the bottle cap damage detection task.

[0256] Then, the image of the screwable bottle cap area standardized in step S422 is input into the pre-trained convolutional neural network model. For example, a standardized image with a size of 224x224 is input into the convolutional neural network. The first convolutional neural network processes the input image and outputs a judgment result indicating whether the bottle cap area is damaged.

[0257] Specifically, TensorFlow and Keras are used here to train the first convolutional neural network model for detecting the damage of the twistable bottle cap. The following is an example description of the specific steps:

[0258] 1. Collect and annotate product images with broken and intact twist-off caps.

[0259] 2. Divide the dataset into training set, validation set and test set.

[0260] 3. Perform preprocessing such as resizing and normalization on the image.

[0261] 4. Perform data augmentation such as random cropping, rotation, translation, and flipping.

[0262] 5. Select ResNet architecture as the convolutional neural network model.

[0263] 6. Define a ResNet model using TensorFlow and Keras.

[0264] 7. Set the learning rate to 0.001, the batch size to 32, and use the Adam optimizer.

[0265] 8. Use binary cross entropy loss function.

[0266] 9. Use the training set data to train the model and optimize the model parameters through back propagation.

[0267] 10. Use the validation set data to validate the model and adjust the learning rate and other hyperparameters to prevent overfitting.

[0268] 11. Use the test set data to evaluate the model performance and calculate the accuracy, precision, recall and F1 score.

[0269] 12. Based on the evaluation results, tune the model, such as adjusting the number of network layers and hyperparameters.

[0270] S43. For the second category of goods, determine whether the seal with the serrated cut line is torn; if torn, checkout is performed or an alarm signal is generated.

[0271] Specifically, S43 includes the following sub-steps S431-S433.

[0272] S431. Preprocess the image and use the instance segmentation algorithm to capture the sealing area of ​​the product to be determined.

[0273] S432. The sealing area is intercepted and standardized.

[0274] S433. Input the standardized sealing area image into the pre-trained second convolutional neural network to obtain a judgment result; wherein, the second convolutional neural network is trained by a data set containing two types of samples, one with intact tooth tangents and one with torn teeth, and the corresponding fracture states of the samples are marked, and the output results of the model are torn and non-torn seals.

[0275] Here, the implementation of S43 and its sub-steps is similar to that of S42, and will not be repeated here.

[0276] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0277] In one embodiment, a smart visual retail cabinet commodity tracking device is provided, which corresponds to the smart visual retail cabinet commodity tracking method in the above embodiment. The smart visual retail cabinet commodity tracking device includes a target matching and association module, a trajectory analysis module, an entry and exit judgment module, and a damage judgment module. The functional modules are described in detail as follows:

[0278] The target matching and association module is used to acquire images based on multi-lens collaboration and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet;

[0279] The trajectory analysis module is used to track the movement trajectory of goods based on the motion trajectory analysis algorithm;

[0280] The entry and exit judgment module is used to integrate the trajectory data of multiple cameras and judge the entry and exit status of the goods based on the movement direction and position of the goods;

[0281] The damage judgment module is used to check out the taken out goods when it detects that the cabinet door has entered a locked state, and to analyze the damage of the goods taken out and put back in. If it is detected that the goods taken out and put back in are damaged, check out or generate an alarm signal.

[0282] For the specific definition of the intelligent visual retail cabinet commodity tracking device, please refer to the definition of the intelligent visual retail cabinet commodity tracking method above, which will not be repeated here. Each module in the above-mentioned intelligent visual retail cabinet commodity tracking device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0283] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for data related to the intelligent visual retail cabinet commodity tracking method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an intelligent visual retail cabinet commodity tracking method is implemented.

[0284] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned smart visual retail cabinet commodity tracking method is implemented, for example Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit of the intelligent visual retail cabinet commodity tracking device in the above embodiment are realized. To avoid repetition, they are not described here.

[0285] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned smart visual retail cabinet commodity tracking method is implemented. For example, Figure 2 Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the intelligent visual retail cabinet commodity tracking device in the above device embodiment are realized.

[0286] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0287] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0288] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for tracking merchandise in an intelligent visual retail cabinet, characterized in that: The following steps are involved: S1. Acquire images based on multi-lens collaboration, and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet; S2. Track the movement trajectory of goods based on the motion trajectory analysis algorithm; S3. Integrate the trajectory data of multiple cameras and determine the entry and exit status of the goods based on the movement direction and position of the goods; S4. When it is detected that the cabinet door has entered the locked state, the taken-out goods are checked out, and the taken-out and put-back goods are analyzed for damage. If it is detected that the taken-out and put-back goods are damaged, the goods are checked out or an alarm signal is generated.

2. The intelligent visual retail cabinet commodity tracking method according to claim 1, characterized in that: The S4 comprises the following sub-steps: S41. Determine whether the type of the commodity to be determined is a first-category commodity or a second-category commodity; wherein the commodity to be determined is a commodity that is taken out and then put in again, the first-category commodity is a commodity with a screw-on cap, and the second-category commodity is a commodity with a seal having a toothed tangent line; S42. For the first category of goods, determine whether the connecting bridge between the cap body and the anti-theft ring in the twistable bottle cap is broken; if broken, checkout or generate an alarm signal; S43. For the second category of goods, determine whether the seal with the serrated cut line is torn; if torn, checkout is performed or an alarm signal is generated.

3. The intelligent visual retail cabinet commodity tracking method according to claim 2, characterized in that: The S42 comprises the following sub-steps: S421. Preprocess the image and use the instance segmentation algorithm to capture the unscrewable bottle cap area of ​​the product to be determined; S422. The area of ​​the bottle cap that can be twisted is intercepted and standardized; S423. Input the standardized image of the twistable bottle cap area into the pre-trained first convolutional neural network to obtain a judgment result; wherein, the first convolutional neural network is trained by a data set containing two types of samples of the cap body and the anti-theft ring, broken and non-broken, and marking the samples corresponding to the broken state, and the output result of the model is the anti-theft ring is broken and non-broken.

4. The intelligent visual retail cabinet commodity tracking method according to claim 3 is characterized in that: The S43 comprises the following sub-steps: S431. Preprocess the image and use the instance segmentation algorithm to capture the sealing area of ​​the product to be determined; S432. The sealing area is intercepted and standardized; S433. Input the standardized sealing area image into the pre-trained second convolutional neural network to obtain a judgment result; wherein, the second convolutional neural network is trained by a data set containing two types of samples, one with intact tooth tangents and one with torn teeth, and the corresponding fracture states of the samples are marked, and the output results of the model are torn and non-torn seals.

5. The intelligent visual retail cabinet commodity tracking method according to any one of claims 1 to 4, characterized in that: The S41 comprises the following steps: S411. Preprocess the image and extract the feature vector of the product from the preprocessed image; S412. Match the extracted feature vector with the feature vector in the product database to determine the specific type of product; S413. According to the recognition result, the type of matching goods is searched in the goods type database, wherein the goods type database includes the goods name, goods ID, goods picture and its corresponding type; S414. According to the database query result, confirm whether the type of the commodity to be determined is a first-category commodity or a second-category commodity.

6. An intelligent visual retail cabinet commodity tracking device, characterized in that: include: The target matching and association module is used to acquire images based on multi-lens collaboration and perform target matching and association based on embedding distance and detection frame position; wherein cameras are installed in different areas of the retail cabinet; The trajectory analysis module is used to track the movement trajectory of goods based on the motion trajectory analysis algorithm; The entry and exit judgment module is used to integrate the trajectory data of multiple cameras and judge the entry and exit status of the goods based on the movement direction and position of the goods; The damage judgment module is used to check out the taken out goods when it detects that the cabinet door has entered a locked state, and to analyze the damage of the goods taken out and put back in. If it is detected that the goods taken out and put back in are damaged, check out or generate an alarm signal.

7. A computer device, characterized in that: It includes: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the smart visual retail cabinet product tracking method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement: the smart visual retail cabinet product tracking method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Lightweight commodity identification and tracking system and method, storage medium, equipment and terminal

    CN115311450A

  • Dynamic visual intelligent vending cabinet commodity detection tracking and post-processing method

    CN118298356A

  • Vending machine

    CN209785129U

Cited By

  • Display cabinet control method and device, medium and display cabinet

    CN116934799A

  • Display cabinet control method, apparatus, medium, and display cabinet

    CN116934799B

  • Unmanned retail monitoring method, device and equipment

    CN121170941A

  • An unmanned retail monitoring method, device and equipment

    CN121170941B