Commodity identification method and device based on image real-time processing

Abstract

The invention discloses a commodity identification method and device based on image real-time processing, and belongs to the technical field of intelligent shopping carts. The method comprises the steps that a real-time video stream and weight change data are acquired; converting the video stream into an image, inputting the image into a pre-constructed target detection model, and outputting an image containing object information; inputting an image containing object information into a feature extraction model to generate an object feature vector; performing target tracking based on the intersection-to-union ratio of the adjacent frame detection frames and the re-identification matching result of the object feature vector, and generating an object motion track; determining a time window of the shopping event, and extracting object movement track data in the time window; performing multi-dimensional feature comparison with a pre-constructed local feature library and a pre-constructed standard feature library to obtain a commodity put-in / take-out identification result; according to the invention, limited hardware resources are fully utilized, internal processing algorithm logic is optimized, and rapid identification of putting in / taking out of commodities is realized.

Description

Technical Field

[0001] The present invention relates to a commodity recognition method and device based on real-time image processing, belonging to the technical field of smart shopping carts. Background Art

[0002] Traditional shopping carts primarily serve as a temporary storage system for items, while checkout still requires a manual checkout counter. High customer traffic often leads to queues, resulting in lengthy shopping processes and a poor shopping experience. With consumers' increasing demand for convenience and a better shopping experience, and offline retailers' in-depth exploration of digital transformation, smart shopping carts have emerged as a new type of shopping tool. They combine advanced technologies such as the Internet of Things, artificial intelligence, and mobile payments to provide consumers with a more convenient and intelligent shopping experience.

[0003] Current smart shopping cart technology primarily relies on weight recognition, product trajectory analysis, barcode scanning, and visual behavior recognition. However, these technologies have significant drawbacks in practical applications: Relying on weight sensors to determine the increase or decrease of goods, it is unable to distinguish the weight changes when goods are stacked. It is easy to make misjudgments due to factors such as changes in the placement of goods and external vibrations. The loss prevention capability is weak and the error rate is high.

[0004] Publication number CN118097519B discloses a method and system for analyzing shopping behavior in an intelligent shopping cart based on product trajectory analysis. This method requires backtracking user behavior videos and extracting product features for identification through event video processing. The processing flow is non-real-time and the response speed is slow, which seriously affects the efficiency of the shopping experience.

[0005] Barcode scanning is highly dependent on user operation standardization. When placing or removing products, the barcode must be facing the camera directly. In real-world scenarios, products are often obscured by random angles and barcodes, resulting in a high rate of missed scans. The motion target processing method, system and device based on shopping behavior recognition, published with publication number CN118196390A, have high computing power costs. They require real-time analysis of continuous frame changes, detection of hand postures and the correlation of grasped objects, and generation of pixel-level segmentation masks. These methods place stringent demands on hardware computing power and are difficult to deploy at low cost. Furthermore, this method has low recognition accuracy. Supermarkets have a wide variety of products, and products with similar appearances are easily misidentified as the same instance by the segmentation model, reducing recognition reliability. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a commodity identification method and device based on real-time image processing, which fully utilizes limited hardware resources, optimizes internal processing algorithm logic, and realizes rapid identification of commodity placement / removal.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a commodity identification method based on real-time image processing, comprising: Obtain real-time video streams from the smart shopping cart's dual cameras and weight change data from the weight sensor; The following operations are performed on the real-time video stream by parallel processing threads: Convert the video stream into an image, input it into a pre-built target detection model, and output an image containing object information; Input the image containing object information into the pre-built feature extraction model to extract features and generate object feature vectors; Target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of object feature vectors to generate object motion trajectories; Determine a time window for the shopping event, and extract object motion trajectory data within the time window based on the object motion trajectory; The object motion trajectory data within the extracted time window is compared with the pre-built local feature library and standard feature library in multiple dimensions to obtain the product placement / removal recognition results.

[0008] Furthermore, the target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching result of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; When any of the following conditions are met, the current object is associated with the target of the previous frame as the same object: Condition 1: The intersection-over-union ratio is greater than the first threshold and the feature similarity is greater than the fourth threshold; or Condition 2: The intersection-over-union ratio is greater than the second threshold but less than the first threshold, and the feature similarity is greater than the third threshold; When the association condition is not met, a new tracking identifier is assigned to the current object; The first threshold is greater than the second threshold, and the third threshold is greater than the fourth threshold; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0009] Furthermore, the target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching result of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; For each pair of object matching combinations, the weighted score is calculated according to the following formula: Weighted score = w1×intersection-over-union + w2×feature similarity; Among them, w1 represents the weight of the intersection-over-union ratio, w2 represents the weight of the feature similarity, w1+w2=1; When the weighted score is greater than the association threshold, the current object and the corresponding object in the previous frame are associated as the same target; When the weighted score is less than or equal to the association threshold, a new tracking identifier is assigned to the current object; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0010] Furthermore, the target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching result of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, calculate the intersection-union ratio of its detection frame and all object detection frames in the previous frame; When the maximum intersection-over-union ratio exceeds a first threshold, the object is associated with the object in the previous frame with the maximum intersection-over-union ratio as the same target; When the maximum intersection-over-union ratio is lower than the first threshold but higher than the second threshold: Extract the feature vector of the object; Calculate the similarity between the feature vector and the feature vectors of all objects in the previous frame; When the maximum similarity exceeds a third threshold, the object is associated with the object in the previous frame with the maximum similarity; When the above association conditions are not met, a new tracking identifier is assigned to the object; The object motion trajectory is updated based on the association result, where the object motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0011] Furthermore, the extracted object motion trajectory data within the time window is compared with the local feature library and the standard feature library in multiple dimensions to obtain the product insertion / removal recognition result, including: When placing products: If there is scanned barcode data, the object's motion trajectory data is matched 1:1 with the features of the corresponding barcode in the standard feature library; If the code is not scanned, the object's motion trajectory data is compared with all product features in the standard feature library for a 1:N similarity calculation, and the result with the highest similarity is output, where N is a natural number greater than 1; When taking out the product: Perform 1:N matching between the object motion trajectory data and N historical event features in the local feature library; When the application provides an add-to-cart list, the object's motion trajectory data is additionally compared with the standard features of M items in the add-to-cart list, where M is a natural number greater than 1. Output the two sets of comparison results to the application side for comprehensive decision-making.

[0012] Furthermore, determining the time window of the shopping event includes at least one of the following methods: Method 1: Determine the time window for shopping events based on the time threshold before and after the weight sensor stabilizes; Method 2: By monitoring images in real time, the system identifies the states of objects entering the shopping cart, objects remaining stationary in the shopping cart, and objects leaving the shopping cart. The time of state changes is recorded to determine the time window of the shopping event. Method 3: Combining methods 1 and 2, the state change time of the object in the image is obtained under the condition of stable weight, so as to determine the time window of the shopping event.

[0013] Furthermore, the construction of the local feature library and the standard feature library includes: Dynamically store the feature vector of each item added during the current shopping cycle to obtain a local feature library; The mapping relationship between the product barcode and the multi-dimensional feature vector is pre-stored to obtain the standard feature library.

[0014] Furthermore, the target detection model is a single-stage target detection model, and the feature extraction model is a residual neural network model.

[0015] Furthermore, the method further includes: directly reusing the calculation result of the previous frame for the feature vector of the stationary object.

[0016] Furthermore, the object information includes object detection frame, category and confidence level.

[0017] In a second aspect, the present invention provides a commodity identification device based on real-time image processing, comprising: Multi-source acquisition module, used to synchronously acquire dual-camera video streams and weight sensor data; The parallel processing module deploys a multi-threaded architecture and is used to execute any of the aforementioned commodity recognition methods based on real-time image processing.

[0018] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the aforementioned methods when executed by a processor.

[0019] In a fourth aspect, the present invention provides a computer device, comprising: Memory, used to store computer programs / instructions; A processor is configured to execute the computer program / instructions to implement the steps of any of the aforementioned methods.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the aforementioned methods when executed by a processor.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a commodity recognition method and device based on real-time image processing. By combining the dual cameras and weight sensor on the smart shopping cart, the object motion trajectory data within the time window is extracted, and multi-dimensional feature comparison is performed with the pre-built local feature library and standard feature library to obtain the commodity insertion / removal recognition results. This method fully utilizes limited hardware resources, optimizes the internal processing algorithm logic, realizes the rapid recognition of commodity insertion / removal, and provides rapid feedback on user operations, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of a commodity identification method based on real-time image processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] Example 1: This example introduces a commodity identification method based on real-time image processing, including: Obtain real-time video streams from the smart shopping cart's dual cameras and weight change data from the weight sensor; The following operations are performed on the real-time video stream by parallel processing threads: Convert the video stream into an image, input it into a pre-built target detection model, and output an image containing object information; Input the image containing object information into the pre-built feature extraction model to extract features and generate object feature vectors; Target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of object feature vectors to generate object motion trajectories; Determine a time window for the shopping event, and extract object motion trajectory data within the time window based on the object motion trajectory; The object motion trajectory data within the extracted time window is compared with the pre-built local feature library and standard feature library in multiple dimensions to obtain the product placement / removal recognition results.

[0025] The smart shopping cart integrates a powerful deep learning processor. This embodiment fully utilizes computing and storage resources such as the deep learning processor, CPU, and memory to achieve real-time processing of multi-camera data, real-time tracking of objects in the smart shopping cart, and quickly realize product identification feedback to the APP decision-making based on APP needs.

[0026] The application process of the commodity recognition method based on real-time image processing provided in this embodiment specifically involves the following steps: 1. Real-time shopping event tracking: By fully leveraging the model's computing acceleration hardware platform, the video data on the smart shopping cart is processed in real time. The target detection model detects all objects in each frame, obtaining the corresponding target's rect (detection box), category, and confidence level. Feature extraction is then performed on multiple detected objects to obtain object features. The inter-frame object rect intersection-over-union (IOU) (a measure of the degree of overlap between two bounding boxes (or regions) calculated by calculating the ratio of the intersection area to the union area of the two boxes) and ReID technology (focusing on the target's appearance features and determining whether they belong to the same target by extracting and comparing the target's feature vectors) are used to achieve real-time target tracking. Specifically, the following steps are involved: Method 1: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; When any of the following conditions are met, the current object is associated with the target of the previous frame as the same object: Condition 1: The intersection-over-union ratio is greater than the first threshold and the feature similarity is greater than the fourth threshold; or Condition 2: The intersection-over-union ratio is greater than the second threshold but less than the first threshold, and the feature similarity is greater than the third threshold; When the association condition is not met, a new tracking identifier is assigned to the current object; The first threshold is greater than the second threshold, and the third threshold is greater than the fourth threshold; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0027] Method 2: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; For each pair of object matching combinations, the weighted score is calculated according to the following formula: Weighted score = w1×intersection-over-union + w2×feature similarity; Among them, w1 represents the weight of the intersection-over-union ratio, w2 represents the weight of the feature similarity, w1+w2=1; When the weighted score is greater than the association threshold, the current object and the corresponding object in the previous frame are associated as the same target; When the weighted score is less than or equal to the association threshold, a new tracking identifier is assigned to the current object; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0028] Method 3: For each detected object in the current frame, calculate the intersection-union ratio of its detection frame and all object detection frames in the previous frame; When the maximum intersection-over-union ratio exceeds a first threshold, the object is associated with the object in the previous frame with the maximum intersection-over-union ratio as the same target; When the maximum intersection-over-union ratio is lower than the first threshold but higher than the second threshold: Extract the feature vector of the object; Calculate the similarity between the feature vector and the feature vectors of all objects in the previous frame; When the maximum similarity exceeds a third threshold, the object is associated with the object in the previous frame with the maximum similarity; When the above association conditions are not met, a new tracking identifier is assigned to the object; The object motion trajectory is updated based on the association result, where the object motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

[0029] In real time, the object's movement and stillness are determined based on the various tracking results within a certain period of time, and the time of movement of the object is calculated. By integrating the intrusion analysis of the moving object in the images of all real-time video sources, the movement status and time of the moving object in the smart shopping cart can be obtained. The basic information (rect, category, confidence, and features, etc.) of each tracked target in a fixed window for a long time can be cached in real time. 2. Real-time shopping event time determination: 1) Confirm the movement time of moving objects in the smart shopping cart through images; 2) The time of the shopping event can also be obtained based on the stable weight trigger time of the weight sensor and the movement time of the moving object in the smart shopping cart; The image moving object determination is based on the multi-frame tracking information within the time window to judge the movement of each tracked object. For example, if the change of its rect position meets a certain direction and is greater than a preset threshold, it is judged to be moving, otherwise it is stationary. Combine the motion continuity of multiple cameras and merge the events to get the time of the final event when the difference between event segments is small and meets the motion trajectory of the shopping event; 3. Product identification for this shopping event: 1) Perform trajectory analysis on the tracking results during this period to obtain complete moving and static trajectories. Compare the target object in all frames of the moving trajectory with the existing local feature library (the local feature library formed by each additional trajectory) to obtain the corresponding comparison results to see if they are consistent; 2) Based on the weight change and the code scan, the trajectory of the moving object can be obtained and then compared with the characteristics of the specified product. This includes a pre-established standard feature library and a local feature library corresponding to the trajectory of each event during the user's shopping process. The comparison results combined with the weight can be used to determine whether the product in the current event is correct, whether it is a previously placed product, and which type of product is being removed.

[0030] The following describes the contents involved in the above embodiment in conjunction with a preferred embodiment.

[0031] like Figure 1 As shown, the smart shopping cart tablet integrates the Rockchip RK3588 high-performance processor, including a 6TOPS NPU (Neural Network Processing Unit), 3 cores, 8GB of memory, and two cameras, Camera0 and Camera1 (frame rate 25 frames / second, resolution 1024*1280); 1. Launch the smart shopping cart app; 2. Start the weight sensor; 3. The real-time image processing module initializes the main thread: it reads the object detection model (such as the YOLO object detection model) into NPU Core0 (core numbered 0) and Core1 (core numbered 1), and reads the feature extraction model (such as the ResNet residual network model) into NPU Core2 (core numbered 2). It also reads the pre-collected and generated standard library (which can be downloaded locally from the server or pre-prepared locally) into memory and the two cameras. 4. Object detection; The real-time image data format is converted and scaled to the model input size and input into the corresponding NPU Core, for example: Thread 2: Camera0 data is preprocessed (converting YUV to RGB) and resized (a term used in computer programming to adjust the size of a window, image, or page element, which can be calculated on the CPU, RGA, or GPU). Thread 3 passes the data to NPU Core0, and the data from Camera 1 is processed and passed to NPU Core1, where the target detection model is calculated and post-processed to obtain the corresponding object information (rect, category, confidence) in the image. 5. Feature calculation; Thread 4: Combines the result of step 3 with its image to obtain the object's rect image, which is then preprocessed (including resizing and format conversion). All processed rect image data of the object is sequentially passed to NPU Core 2 for feature calculation. To reduce computation, the object information of the current frame is compared with the object information of the previous frame (data frame from the same camera) before being passed in. For objects that are identical (i.e., stationary or nearly stationary), the features calculated in the previous frame are directly reused. 6. Real-time tracking; Thread 5: Use the tracking algorithm to obtain the tracking results of the object information of Camera0 including the features calculated in 4, and cache them in the corresponding data buffer buffer0; Thread 6: The object information of Camera 1 including the features calculated in step 4 is used to obtain the tracking results through the tracking algorithm and cached in the corresponding data cache buffer 1; 7. Event time statistics (thread 7); Method 1: Determine the stable weight spanning a period of time forward and backward based on the weight sensor or in combination with the inertial measurement unit sensor. For example, if the weight increases, use the data from x seconds before the weight stabilizes; if the weight decreases, use the data from y seconds before the weight stabilizes and z seconds after the weight stabilizes, totaling y + z seconds, to obtain the time span (t0-t1). There are two methods to determine the stable weight. Method 1: Determine the stable weight time based solely on the weight stability provided by the weight sensor. Method 2: When the weight sensor provides weight stability, it is also necessary to use the inertial measurement unit sensor to determine whether the current vehicle body state is stationary or in uniform motion (for example, the total acceleration is the acceleration due to gravity). The time at this time is the stable weight time. Method 2: Count data from buffer0 and buffer1 for a period of time (12 frames) in real time, analyze whether an object moves in the smart shopping cart, remove the bottom fluctuations in the frame, and if there is movement, record the time of the moving point and the time of rest. Combine the results of 0 and 1 to accurately obtain the time span of the object's movement (t0-t1). Method 3: Combining Methods 1 and 2, obtaining the movement time (t0-t1) of the object confirmed in the image under the condition of stable weight, and eliminating false recognition; 8. Trajectory analysis; According to the time obtained in step 7, the data in buffer0 and buffer1 are analyzed separately to obtain the user product running trajectory of different cameras; The features of all object information in the trajectory each time the weight increases are recorded to form a local feature library of the shopping user.

[0032] 9. Compare; The features of the object information in the trajectories of Camera0 and Camera1 are compared with the feature library in different ways, and the results are fed back to the APP side for use. For example, when adding a product to a cart, if there is a product barcode, a 1:1 feature similarity statistics can be directly performed with the corresponding standard library data. If there is no barcode, the trajectory feature 1 of the behavior can be compared with the feature N of all product libraries, or the APP side can specify it, such as the scanned product. Similarly, when taking out a product, the trajectory features of the behavior can be compared with all data in the local feature library, and the APP side can pass in a list of purchased product information codes. The trajectory features of the behavior can be compared with the features of the purchased product information codes in the standard feature library one by one. The two comparison results are reported and transmitted to the APP side for decision-making to observe whether there is the same product behavior or the same placing and taking out behavior.

[0033] Among them: Steps 8 and 9 can be performed in separate threads, merged threads, or in the main thread, as long as the decision-making time requirement on the app side is met.

[0034] If a high-performance CPU or deep learning processor is used, the multi-threaded processing mentioned above can be omitted and a small number of threads can be used. Video frame rate and resolution can be adjusted to accommodate less powerful processors.

[0035] Example 2: This embodiment provides a commodity identification device based on real-time image processing, including: Multi-source acquisition module, used to synchronously acquire dual-camera video streams and weight sensor data; The parallel processing module deploys a multi-threaded architecture and is used to execute the commodity recognition method based on real-time image processing described in any one of the embodiments 1.

[0036] The parallel processing module includes a neural network processor core 0, a neural network processor core 1 and a neural network processor core 2, wherein the target detection model is deployed in the neural network processor core 0 and the neural network processor core 1, respectively used to perform target detection on images captured and converted by dual cameras, and the feature extraction model is deployed in the neural network processor core 2, used to perform feature extraction on images output by the target detection model.

[0037] The specific functional implementation of each of the above modules can be found in the relevant content of the method in Example 1 and will not be elaborated on here.

[0038] Example 3: This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods described in Example 1 are implemented.

[0039] Embodiment 4: This embodiment provides a computer device, including: Memory, used to store computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of any one of the methods described in Example 1.

[0040] Example 5: This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of any method described in Example 1 when executed by a processor.

[0041] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0042] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0043] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0044] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and are not intended to limit its scope of protection. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the disclosed claims to be approved.

Claims

1. A commodity identification method based on real-time image processing, characterized in that: include: Obtain real-time video streams from the smart shopping cart's dual cameras and weight change data from the weight sensor; The following operations are performed on the real-time video stream by parallel processing threads: Convert the video stream into an image, input it into a pre-built target detection model, and output an image containing object information; Input the image containing object information into the pre-built feature extraction model to extract features and generate object feature vectors; Target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of object feature vectors to generate object motion trajectories; Determine a time window for the shopping event, and extract object motion trajectory data within the time window based on the object motion trajectory; The object motion trajectory data within the extracted time window is compared with the pre-built local feature library and standard feature library in multiple dimensions to obtain the product placement / removal recognition results.

2. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; When any of the following conditions are met, the current object is associated with the target of the previous frame as the same object: Condition 1: The intersection-over-union ratio is greater than the first threshold and the feature similarity is greater than the fourth threshold; or Condition 2: The intersection-over-union ratio is greater than the second threshold but less than the first threshold, and the feature similarity is greater than the third threshold; When the association condition is not met, a new tracking identifier is assigned to the current object; The first threshold is greater than the second threshold, and the third threshold is greater than the fourth threshold; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

3. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, perform the following operations: Calculate the intersection-and-union ratio of its detection frame and the detection frames of each object in the previous frame; Calculate the similarity between its feature vector and the feature vector of each object in the previous frame; For each pair of object matching combinations, the weighted score is calculated according to the following formula: Weighted score = w1×intersection-over-union + w2×feature similarity; Among them, w1 represents the weight of the intersection-over-union ratio, w2 represents the weight of the feature similarity, w1+w2=1; When the weighted score is greater than the association threshold, the current object and the corresponding object in the previous frame are associated as the same target; When the weighted score is less than or equal to the association threshold, a new tracking identifier is assigned to the current object; The object's motion trajectory is updated based on the association result, where the motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

4. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The target tracking is performed based on the intersection-over-union ratio of adjacent frame detection frames and the re-identification matching results of the object feature vector to generate the object motion trajectory, including: For each detected object in the current frame, calculate the intersection-union ratio of its detection frame and all object detection frames in the previous frame; When the maximum intersection-over-union ratio exceeds a first threshold, the object is associated with the object in the previous frame with the maximum intersection-over-union ratio as the same target; When the maximum intersection-over-union ratio is lower than the first threshold but higher than the second threshold: Extract the feature vector of the object; Calculate the similarity between the feature vector and the feature vectors of all objects in the previous frame; When the maximum similarity exceeds a third threshold, the object is associated with the object in the previous frame with the maximum similarity; When the above association conditions are not met, a new tracking identifier is assigned to the object; The object motion trajectory is updated based on the association result, where the object motion trajectory includes a position sequence, a feature vector sequence, and a corresponding tracking identifier of the object in consecutive frames.

5. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The extracted object motion trajectory data within the time window is compared with the local feature library and the standard feature library in multiple dimensions to obtain the product insertion / removal recognition result, including: When placing products: If there is scanned barcode data, the object's motion trajectory data is matched 1:1 with the features of the corresponding barcode in the standard feature library; If the code is not scanned, the object's motion trajectory data is compared with all product features in the standard feature library for a 1:N similarity calculation, and the result with the highest similarity is output, where N is a natural number greater than 1; When taking out the product: Perform 1:N matching between the object motion trajectory data and N historical event features in the local feature library; When the application provides an add-to-cart list, the object's motion trajectory data is additionally compared with the standard features of M items in the add-to-cart list, where M is a natural number greater than 1. Output the two sets of comparison results to the application side for comprehensive decision-making.

6. The commodity identification method based on real-time image processing according to claim 1, characterized in that: Determining the time window of the shopping event includes at least one of the following methods: Method 1: Determine the time window for shopping events based on the time threshold before and after the weight sensor stabilizes; Method 2: By monitoring images in real time, the system identifies the states of objects entering the shopping cart, objects remaining stationary in the shopping cart, and objects leaving the shopping cart. The time of state changes is recorded to determine the time window of the shopping event. Method 3: Combining methods 1 and 2, the state change time of the object in the image is obtained under the condition of stable weight, so as to determine the time window of the shopping event.

7. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The construction of the local feature library and the standard feature library includes: Dynamically store the feature vector of each item added during the current shopping cycle to obtain a local feature library; The mapping relationship between the product barcode and the multi-dimensional feature vector is pre-stored to obtain the standard feature library.

8. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The method further includes: directly reusing a calculation result of a previous frame for the feature vector of the stationary object.

9. The commodity identification method based on real-time image processing according to claim 1, characterized in that: The object information includes object detection frame, category and confidence.

10. A commodity identification device based on real-time image processing, characterized in that: include: Multi-source acquisition module, used to synchronously acquire dual-camera video streams and weight sensor data; The parallel processing module deploys a multi-threaded architecture and is used to execute the commodity recognition method based on real-time image processing according to any one of claims 1 to 9.

Images