A commodity identification method and device based on image real-time processing

Abstract

The application discloses a kind of based on image real-time processing's commodity identification method and device, belongs to intelligent shopping cart technical field method includes: obtaining real-time video stream, and weight change data;Video stream is converted into image, input to pre-constructed target detection model, and the image containing object information is output;The image containing object information is input to feature extraction model, and object feature vector is generated;Based on the intersection-over-union of adjacent frame detection frame and the re-identification matching result of object feature vector carries out target tracking, and generates object motion trajectory;Determine the time window of shopping event, extract object motion trajectory data in time window;And with pre-constructed local feature library, standard feature library carries out multidimensional feature comparison, obtains commodity put in / take out identification result;The application makes full use of limited hardware resources, optimizes internal processing algorithm logic, realizes the rapid identification of commodity put in / take out.

Description

TECHNICAL FIELD

[0001] The application relates to a commodity identification method and device based on image real-time processing, and belongs to the technical field of intelligent shopping carts. BACKGROUND

[0002] Traditional shopping carts mainly realize the function of temporary storage of commodities, and settlement still needs to go to the manual cash desk. When the passenger flow is large, queuing often occurs, resulting in a long shopping process and poor experience. With the continuous pursuit of consumers for shopping convenience and experience, and the in-depth exploration of offline retailers in the aspect of digital transformation, intelligent shopping carts as a new type of shopping tool have emerged as the times require. It combines advanced technologies such as the Internet of Things, artificial intelligence, mobile payment, etc., and provides consumers with a more convenient and intelligent shopping experience.

[0003] Current intelligent shopping cart technology mainly relies on weight recognition, commodity trajectory analysis, barcode scanning and visual behavior recognition, but there are significant defects in actual application:

[0004] The weight sensor is used to determine the increase and decrease of commodities, which cannot distinguish the weight change when the commodities are stacked, and is prone to misjudgment due to factors such as change of commodity placement position, external vibration, etc. The anti-theft ability is weak and the error rate is high.

[0005] The application number CN118097519B discloses an intelligent shopping cart shopping behavior analysis method and system based on commodity trajectory analysis. This method needs to backtrack user behavior video, extracts commodity features for recognition through event video processing, and the processing flow is not real-time, the response speed is slow, and the shopping experience efficiency is seriously affected.

[0006] Barcode scanning highly depends on the normativeness of user operation. When the commodity is put in / taken out, the barcode must face the camera. In actual scenarios, the angle of the commodity is random, the barcode is frequently blocked, and the missing scanning rate is high.

[0007] The application number CN118196390A discloses a moving target processing method, system and device based on shopping behavior recognition. The method has a high cost of computing power, needs to analyze continuous frame changes in real time, detect hand gestures and the relevance of grasped objects, and generate pixel-level segmentation masks. The method has high requirements for hardware computing power and is difficult to deploy at low cost. Moreover, the method has low recognition accuracy. The types of commodities in supermarkets are diverse, and similar commodities are easily misjudged as the same instance by the segmentation model, reducing the recognition reliability. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art, provide a commodity identification method and device based on image real-time processing, make full use of limited hardware resources, optimize the internal processing algorithm logic, and realize fast identification of commodity putting in / taking out.

[0009] To achieve the above object, the present application is implemented by using the following technical scheme:

[0010] In a first aspect, the present application provides a commodity identification method based on image real-time processing, comprising:

[0011] Obtaining real-time video stream collected by double cameras of an intelligent shopping cart and weight change data of a weight sensor;

[0012] Performing the following operations on the real-time video stream by parallel processing threads:

[0013] Converting the video stream into an image, inputting the image into a pre-constructed target detection model, and outputting an image containing object information;

[0014] Inputting the image containing object information into a pre-constructed feature extraction model to perform feature extraction and generate an object feature vector;

[0015] Performing target tracking based on the intersection-over-union of adjacent frame detection boxes and the re-identification matching result of the object feature vector to generate an object motion trajectory;

[0016] Determining a time window of a shopping event, combining the object motion trajectory, and extracting object motion trajectory data within the time window;

[0017] Comparing the extracted object motion trajectory data within the time window with a pre-constructed local feature library and a standard feature library in multiple dimensions to obtain commodity put-in / take-out identification results.

[0018] Further, the target tracking based on the intersection-over-union of adjacent frame detection boxes and the re-identification matching result of the object feature vector to generate an object motion trajectory comprises:

[0019] For each detected object of the current frame, performing the following operations:

[0020] Calculating the intersection-over-union value of its detection box and each object detection box of the previous frame;

[0021] Calculating the similarity of its feature vector and each object feature vector of the previous frame;

[0022] When any of the following conditions is met, the current object is associated with the target of the previous frame as the same object:

[0023] Condition one: the intersection-over-union value is greater than a first threshold value and the feature similarity is greater than a fourth threshold value; or

[0024] Condition two: the intersection-over-union value is greater than a second threshold value but less than the first threshold value, and the feature similarity is greater than a third threshold value;

[0025] When the association condition is not met, a new tracking identifier is assigned to the current object;

[0026] The first threshold is greater than the second threshold, and the third threshold is greater than the fourth threshold.

[0027] updating the object motion trajectory based on the association result, the motion trajectory including a sequence of positions of the object in consecutive frames, a sequence of feature vectors, and corresponding tracking identifiers.

[0028] Further, the target tracking is performed based on the intersection-over-union of the bounding boxes of adjacent frames and the re-identification matching result of the object feature vectors to generate the object motion trajectory, including:

[0029] For each detected object of the current frame, the following operations are performed:

[0030] calculating the intersection-over-union of the bounding box of the object and the bounding boxes of objects of the previous frame;

[0031] calculating the similarity of the feature vector of the object and the feature vectors of objects of the previous frame;

[0032] For each pair of object matching combination, a weighted score is calculated according to the following formula:

[0033] Weighted score = w1×intersection-over-union + w2×feature similarity;

[0034] wherein w1 represents the weight of the intersection-over-union, w2 represents the weight of the feature similarity, and w1+w2=1.

[0035] When the weighted score is greater than the association threshold, the current object and the corresponding object of the previous frame are associated as the same target;

[0036] When the weighted score is less than or equal to the association threshold, a new tracking identifier is assigned to the current object;

[0037] updating the object motion trajectory based on the association result, the motion trajectory including a sequence of positions of the object in consecutive frames, a sequence of feature vectors, and corresponding tracking identifiers.

[0038] Further, the target tracking is performed based on the intersection-over-union of the bounding boxes of adjacent frames and the re-identification matching result of the object feature vectors to generate the object motion trajectory, including:

[0039] For each detected object of the current frame, the intersection-over-union of the bounding box of the object and the bounding boxes of all objects of the previous frame is calculated;

[0040] When the maximum intersection-over-union value exceeds the first threshold, the object and the object of the previous frame with the maximum intersection-over-union are associated as the same target;

[0041] When the maximum intersection-over-union value is lower than the first threshold but higher than the second threshold:

[0042] extracting the feature vector of the object;

[0043] calculating the similarity of the feature vector with the feature vectors of all objects in the previous frame;

[0044] when the maximum similarity exceeds a third threshold, associating the object with the object in the previous frame with the maximum similarity;

[0045] when the association condition is not met, assigning a new tracking identifier to the object;

[0046] updating the object motion trajectory based on the association result, the object motion trajectory including a sequence of positions of the object in consecutive frames, a sequence of feature vectors, and a corresponding tracking identifier.

[0047] Further, the extracted object motion trajectory data in the time window is compared with the local feature library and the standard feature library in multiple dimensions to obtain a goods put-in / taken-out recognition result, including:

[0048] When putting in goods:

[0049] If there is scan code data, the object motion trajectory data is matched with the features of the corresponding bar code in the standard feature library in 1:1;

[0050] If there is no scan code, the object motion trajectory data is matched with all the features of the goods in the standard feature library in 1:N similarity calculation, and the result with the highest similarity is output, where N is a natural number greater than 1;

[0051] When taking out goods:

[0052] The object motion trajectory data is matched with N historical event features in the local feature library in 1:N;

[0053] When the application side provides a wish list, the object motion trajectory data is additionally compared with the standard features of M goods in the wish list, where M is a natural number greater than 1;

[0054] The two sets of comparison results are output to the application side for comprehensive decision-making.

[0055] Further, the time window of the shopping event includes at least one of the following ways:

[0056] Method one: determining the time window of the shopping event based on the time threshold before and after the weight sensor stabilizes;

[0057] Method two: by monitoring the image in real time, recognizing the states of the object entering the shopping cart, the object in the shopping cart being stationary, and the object leaving the shopping cart, recording the state change time to determine the time window of the shopping event;

[0058] The third mode: the state change time of the object in the image is obtained under the condition of weight stability, so as to determine the time window of the shopping event.

[0059] Further, the construction of the local feature library and the standard feature library comprises:

[0060] The feature vector of each put-in commodity in the current shopping period is dynamically stored to obtain the local feature library.

[0061] The mapping relationship between the pre-stored commodity bar code and the multi-dimensional feature vector is obtained to obtain the standard feature library.

[0062] Further, the target detection model is a single-stage target detection model, and the feature extraction model is a residual neural network model.

[0063] Further, the method further comprises: directly reusing the calculation result of the previous frame for the feature vector of the stationary object.

[0064] Further, the object information comprises an object detection frame, a category and a confidence.

[0065] In a second aspect, the application provides a commodity identification device based on image real-time processing, comprising:

[0066] A multi-source acquisition module is configured to synchronously acquire double-camera video streams and weight sensor data.

[0067] A parallel processing module is configured to deploy a multi-thread architecture and is configured to execute the commodity identification method based on image real-time processing according to any one of the preceding aspects.

[0068] In a third aspect, the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the method according to any one of the preceding aspects.

[0069] In a fourth aspect, the application provides a computer device, comprising:

[0070] A memory is configured to store a computer program / instruction.

[0071] A processor is configured to execute the computer program / instruction to implement the steps of the method according to any one of the preceding aspects.

[0072] In a fifth aspect, the application provides a computer program product, comprising a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the method according to any one of the preceding aspects.

[0073] Compared with the prior art, the application has the following beneficial effects:

[0074] The application provides a commodity recognition method and device based on image real-time processing, which combines a double camera and a weight sensor on an intelligent shopping cart, extracts object motion trajectory data in a time window, performs multi-dimensional feature comparison with a pre-constructed local feature library and a standard feature library, obtains commodity putting-in / taking-out recognition results, fully utilizes limited hardware resources, optimizes internal processing algorithm logic, realizes rapid commodity putting-in / taking-out recognition, realizes rapid feedback of user operation, and improves user experience. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is a flowchart of a commodity recognition method based on image real-time processing provided by an embodiment of the application. DETAILED DESCRIPTION

[0076] The application will be further described below in conjunction with the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0077] Embodiment 1: This embodiment introduces a commodity recognition method based on image real-time processing, which comprises the following steps:

[0078] Obtaining real-time video stream collected by a double camera of an intelligent shopping cart and weight change data of a weight sensor;

[0079] Performing the following operations on the real-time video stream through a parallel processing thread:

[0080] Converting the video stream into an image, inputting the image into a pre-constructed target detection model, and outputting an image containing object information;

[0081] Inputting the image containing object information into a pre-constructed feature extraction model to perform feature extraction, and generating an object feature vector;

[0082] Performing target tracking based on an intersection-over-union ratio of adjacent frames and a re-identification matching result of the object feature vector, and generating an object motion trajectory;

[0083] Determining a time window of a shopping event, combining the object motion trajectory, and extracting object motion trajectory data in the time window;

[0084] Performing multi-dimensional feature comparison between the extracted object motion trajectory data in the time window and a pre-constructed local feature library and a standard feature library, and obtaining commodity putting-in / taking-out recognition results.

[0085] The intelligent shopping cart is integrated with a strong deep learning processor, and the embodiment fully utilizes the deep learning processor, CPU, memory and other computing and storage resources to realize real-time processing of multi-camera data, real-time tracking of objects in the intelligent shopping cart, and rapid identification of goods combined with APP requirements and feedback to APP decision-making.

[0086] The application process of the method for identifying goods based on image real-time processing provided in the embodiment specifically involves the following steps:

[0087] 1. Real-time shopping event tracking

[0088] By fully utilizing the computing acceleration hardware platform of the model, real-time processing of video data on the intelligent shopping cart is realized, target detection of all objects in each frame of image is realized through a target detection model, the rect (detection box), category and confidence of the corresponding target are obtained, the features of the objects are obtained through feature extraction of the multiple detected objects, the real-time tracking of the target is realized by using the inter-frame object rect intersection-over-union (IOU, an index for measuring the degree of overlap between two bounding boxes (or regions). It is obtained by calculating the ratio of the intersection area to the union area of the two boxes) and reid technology (focusing on the appearance features of the target, the feature vectors of the targets are extracted and compared to determine whether they belong to the same target), and the real-time tracking of the target is realized, specifically including:

[0089] Method one

[0090] For each detected object of the current frame, the following operations are performed:

[0091] Calculate the intersection-over-union of its detection box and the detection boxes of the objects in the previous frame;

[0092] Calculate the similarity of its feature vector and the feature vectors of the objects in the previous frame;

[0093] When any of the following conditions is met, the current object is associated with the target in the previous frame as the same object:

[0094] Condition one: the intersection-over-union is greater than a first threshold value and the feature similarity is greater than a fourth threshold value; or

[0095] Condition two: the intersection-over-union is greater than a second threshold value but less than the first threshold value, and the feature similarity is greater than a third threshold value;

[0096] When the association condition is not met, a new tracking identifier is assigned to the current object;

[0097] The first threshold value is greater than the second threshold value, and the third threshold value is greater than the fourth threshold value;

[0098] Update the object motion trajectory based on the association result, and the motion trajectory includes a sequence of positions, a sequence of feature vectors and a corresponding tracking identifier of the object in consecutive frames.

[0099] Method two:

[0100] For each detected object of the current frame, the following operations are performed:

[0101] Calculate the intersection-over-union value of its detection box and each object detection box of the previous frame;

[0102] Calculate the similarity of its feature vector and each object feature vector of the previous frame;

[0103] For each pair of object matching combination, calculate the weighted score according to the following formula:

[0104] Weighted score = w1 x intersection-over-union + w2 x feature similarity;

[0105] Wherein, w1 represents the weight of intersection-over-union, w2 represents the weight of feature similarity, w1 + w2 = 1;

[0106] When the weighted score is greater than the association threshold, the current object and the corresponding object of the previous frame are associated as the same target;

[0107] When the weighted score is less than or equal to the association threshold, a new tracking identifier is assigned to the current object;

[0108] Update the object motion trajectory based on the association result, the motion trajectory including the sequence of positions, the sequence of feature vectors and the corresponding tracking identifier of the object in consecutive frames.

[0109] Method three:

[0110] For each detected object of the current frame, calculate the intersection-over-union value of its detection box and all object detection boxes of the previous frame;

[0111] When the maximum intersection-over-union value exceeds the first threshold, the object and the object of the previous frame with the maximum intersection-over-union are associated as the same target;

[0112] When the maximum intersection-over-union value is lower than the first threshold but higher than the second threshold:

[0113] Extract the feature vector of the object;

[0114] Calculate the similarity of the feature vector and the feature vector of all objects of the previous frame;

[0115] When the maximum similarity exceeds the third threshold, the object and the object of the previous frame with the maximum similarity are associated;

[0116] When the above association conditions are not met, a new tracking identifier is assigned to the object;

[0117] Update the object motion trajectory based on the association result, the object motion trajectory including the sequence of positions, the sequence of feature vectors and the corresponding tracking identifier of the object in consecutive frames.

[0118] Real-time moving and static judgment according to each tracking result of the object in a certain time, calculation of the time of the moving object, comprehensive analysis of the intrusion of the moving object in the image of all real-time video sources, the movement of the moving object in the intelligent shopping cart and the time can be obtained, and the basic information (rect, category, confidence and features, etc.) of each tracked target is cached for a fixed window time;

[0119] 2, Real-time shopping event time determination:

[0120] 1) Confirm the movement time of the image moving object in the intelligent shopping cart;

[0121] 2) The time of the shopping event can also be obtained according to the stable weight triggering time of the weight sensor combined with the movement time of the image moving object in the intelligent shopping cart;

[0122] Among them, the image moving object determination is to determine the movement of each tracked object according to the multi-frame tracking information in the time window, for example, if the rect position change meets a certain direction and is greater than a predetermined threshold, it is determined to be moving, otherwise it is static;

[0123] The final event time is obtained by combining the movement continuity of multiple cameras and according to the difference value of the event segment and meeting the shopping event motion trajectory;

[0124] 3, This shopping event commodity identification:

[0125] 1) Track the results of this period of time to obtain complete moving and static trajectories, and compare all frames of the target object in the moving trajectory with the existing local feature library (local feature library formed by each added trajectory) to obtain the corresponding comparison result and whether it is consistent;

[0126] 2) According to the change of weight and scanning code, the moving object trajectory can be compared with the specified commodity, including the standard feature library established in advance and the local feature library corresponding to each event trajectory in the shopping process, and the comparison result combined with the weight can determine whether the current event commodity is correct or not or whether it is the previously placed commodity or the placed commodity.

[0127] The contents involved in the above embodiments will be described below in conjunction with a preferred embodiment.

[0128] For example, Figure 1As shown, the smart shopping cart tablet integrates the RK3588 high-performance processor of Ruichai Micro, which contains a 6TOPS NPU (Neural Network Processing Unit), contains three cores, 8G memory and two cameras Camera0 and Camera1 (frame rate 25 frames / second, resolution 1024*1280);

[0129] 1. Start the smart shopping cart app;

[0130] 2. Start the weight sensor;

[0131] 3. Image real-time processing module initializes the main thread: reads the target detection model (such as the yolo target detection model) into NPU Core0 (core number 0) and Core1 (core number 1) respectively, reads the feature extraction model (such as the Resnet residual network model) into NPU Core2 (core number 2), and reads the pre-acquired standard library (the standard library can be downloaded from the server side to the local or pre-prepared to the local) into the memory, two cameras;

[0132] 4. Target detection;

[0133] Real-time image data format conversion and scaling to the size of the model input are input into the corresponding NPU Core, for example:

[0134] Thread 2: Camera0 data is preprocessed (format conversion YUV to RGB) and resized (a computer programming term for adjusting the size of a window, image, or page element, which can be calculated in CPU, RGA, or GPU);

[0135] Thread 3 is transmitted to NPU Core0, and Camera1 data is processed and transmitted to NPU Core1 after processing, and the target detection model is calculated respectively, and the corresponding post-processing is performed to obtain the object information (rect, category, confidence) in the image;

[0136] 5. Feature calculation;

[0137] Thread 4: combines the results of 3 with its image to obtain the rect image of the object, and performs preprocessing (including resize, format conversion) on it, and all processed rect image data of the object is transmitted to NPU Core2 for feature calculation. In order to reduce calculation, the object information of the current frame and the object information of the previous frame (data frame of the same camera) of the frame are compared before being transmitted, and the object features of the same object (i.e. static or nearly static) are directly reused the features calculated in the previous frame;

[0138] 6. Real-time tracking;

[0139] Thread 5: The object information of Camera0 includes the features calculated in 4 through the tracking algorithm to obtain the tracking result, and is cached to the corresponding data cache buffer0;

[0140] Thread 6: The object information of Camera1 includes the features calculated in 4 through the tracking algorithm to obtain the tracking result, and is cached to the corresponding data cache buffer1;

[0141] 7. Event time statistics (thread 7);

[0142] Method one: according to the weight sensor or combined with the inertial measurement unit sensor to determine the stable weight across a period of time, for example, weight increase, x seconds of data before weight stabilization is used; weight decrease, y+z seconds of data before and after weight stabilization, total y+z seconds, to obtain the time span (t0~t1), there are two ways to determine the stable weight, method one: only through the weight sensor to determine the stable weight time, method two: the weight sensor gives the weight stabilization and also needs to judge the current vehicle body state through the inertial measurement unit sensor whether it is static or uniform motion (for example, the combined acceleration is the gravitational acceleration), at this time, the time is the stable weight time;

[0143] Method two: from buffer0 and buffer1 respectively, real-time per frame statistics a period of time (12 frames) of data, analyze whether there is an object moving in the intelligent shopping cart, remove the bottom fluctuation in the box, if there is movement, record the moving point time and the static time, merge the results of 0 and 1, so as to accurately obtain the object moving time span (t0~t1);

[0144] Method three: comprehensive method one and method two, in the case of weight stabilization, the object moving time (t0~t1) confirmed in the image is obtained, and the misrecognition is removed;

[0145] 8. Trajectory analysis;

[0146] According to the time in step 7, the data in buffer0 and buffer1 are analyzed respectively to obtain the user's commodity running trajectory of different cameras;

[0147] Record the features of all object information in the trajectory at each weight increase to form a local feature library of the shopping user.

[0148] 9. Comparison;

[0149] The features in the object information in the trajectories of Camera0 and Camera1 are compared in different ways with the feature library, and the results are fed back to the APP side for use. For example, when adding to cart, if there is a product barcode, it can be directly compared with the corresponding standard library data for 1:1 feature similarity statistics; if there is no barcode, the trajectory features of this behavior can be compared with all the features N of the product library, or the APP side can be used to specify, for example, the scanned product; similarly, when taking out the product, the trajectory features of this behavior can be compared with all the data in the local feature library, and the APP side can transmit the purchased product information code list, and the trajectory features of this behavior can be compared with the features of the purchased product information code in the standard feature library one by one, and the two comparison results are reported to the APP side for decision-making to observe whether there is the same product behavior or the put-in and take-out behavior.

[0150] Steps 8 and 9 can be in a separate thread, or combined thread, or in the main thread, as long as the app side decision-making time requirement is met.

[0151] If a high-performance CPU and a deep learning processor are used, the above multi-thread processing can not be used, and a few threads can be used for processing.

[0152] The video frame rate and resolution can be adjusted to meet the performance of the slightly poor computing processor.

[0153] Embodiment 2 provides a kind of based on image real-time processing's product identification device, comprising:

[0154] Multi-source acquisition module, for synchronously acquiring double camera video stream and weight sensor data;

[0155] Parallel processing module, deploy multi-thread architecture, for executing the product identification method based on image real-time processing of any one of embodiment 1.

[0156] The parallel processing module includes a neural network processor core 0, a neural network processor core 1 and a neural network processor core 2, wherein the target detection model is deployed on the neural network processor core 0 and the neural network processor core 1, and is used for target detection on the image converted by the double camera, and the feature extraction model is deployed on the neural network processor core 2, and is used for feature extraction on the image output by the target detection model.

[0157] The specific function implementation of each module is referred to the related content in the method of embodiment 1, and is not described here.

[0158] Embodiment 3 provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method of any one of embodiment 1.

[0159] Embodiment 4 provides a computer device comprising:

[0160] a memory for storing computer programs / instructions;

[0161] a processor for executing the computer programs / instructions to implement the steps of the method of any one of Embodiment 1.

[0162] Embodiment 5 provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of any one of Embodiment 1.

[0163] The above merely provides the preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

[0164] Those skilled in the art will understand that the embodiments of the present disclosure can be provided as methods, systems or computer program products. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage etc.) containing computer-usable program code.

[0165] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flow Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks. Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0166] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the flow Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks. Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0167] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processes, and the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0168] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present disclosure, but not to limit the scope of protection, although the present disclosure has been described in detail with reference to the above examples, those skilled in the art should understand: after reading the present disclosure, the skilled in the art can make various changes, modifications or equivalent replacements to the specific embodiments of the present disclosure, but these changes, modifications or equivalent replacements are all within the protection scope of the disclosed claims.

Claims

1. A commodity recognition method based on image real-time processing, characterized in that, The method comprises the following steps: acquiring real-time video stream collected by a dual camera of an intelligent shopping cart and weight change data of a weight sensor; performing the following operations on the real-time video stream through parallel processing threads: converting the video stream into an image, inputting the image into a pre-constructed target detection model, and outputting an image containing object information; inputting the image containing object information into a pre-constructed feature extraction model to perform feature extraction and generate an object feature vector; performing target tracking based on an intersection-over-union of adjacent frame detection boxes and a re-identification matching result of the object feature vector to generate an object motion trajectory; including: for each detection object of the current frame, calculating the intersection-over-union of the detection box and each object detection box of the previous frame; calculating the similarity of the feature vector and each object feature vector of the previous frame; when the intersection-over-union is greater than a first threshold value and the feature similarity is greater than a fourth threshold value; or the intersection-over-union is greater than a second threshold value but less than the first threshold value, and the feature similarity is greater than a third threshold value; associating the current object with the target of the previous frame as the same object; wherein the first threshold value is greater than the second threshold value, and the third threshold value is greater than the fourth threshold value; or, for each pair of object matching combination, calculating a weighted score, and when the weighted score is greater than an association threshold value, associating the current object with the corresponding object of the previous frame as the same target; or, when the maximum intersection-over-union exceeds the first threshold value, associating the object with the object of the previous frame with the maximum intersection-over-union as the same target; when the maximum intersection-over-union is lower than the first threshold value but higher than the second threshold value, and the maximum similarity exceeds the third threshold value, associating the object with the object of the previous frame with the maximum similarity; when the association condition is not met, assigning a new tracking identifier to the current object; updating the object motion trajectory based on the association result; determining a time window of a shopping event, combining the object motion trajectory, and extracting object motion trajectory data within the time window; the determination of the time window of the shopping event comprises at least one of the following modes: mode one: determining the time window of the shopping event based on the time threshold value before and after the weight sensor is stable; mode two: monitoring the image in real time to identify the states of the object entering the shopping cart, the object in the shopping cart being stationary, and the object leaving the shopping cart, recording the state change time to determine the time window of the shopping event; mode three: comprehensively considering mode one and mode two, and acquiring the state change time of the object in the image in the case of weight stability to determine the time window of the shopping event; performing multi-dimensional feature comparison between the extracted object motion trajectory data within the time window and a pre-constructed local feature library and a standard feature library to obtain a product putting / taking recognition result, including: when a product is put in: if there is scan code data, performing 1:1 matching between the object motion trajectory data and the features of the corresponding bar code in the standard feature library; if there is no scan code, performing 1:N similarity calculation between the object motion trajectory data and all product features in the standard feature library, and outputting the result with the highest similarity, wherein N is a natural number greater than 1; when a product is taken out: performing 1:N matching between the object motion trajectory data and N historical event features in the local feature library; When the application end provides the add-to-list, additionally, the object motion trajectory data is compared with the standard features of M commodities in the add-to-list, wherein M is a natural number greater than 1; Two sets of comparison results are output to the application end for comprehensive decision-making.

2. The image-based real-time processing method for product identification according to claim 1, wherein, The motion trajectory includes a position sequence, a feature vector sequence and a corresponding tracking identifier of the object in consecutive frames. 3.The image-based real-time processing method for product recognition according to claim 1, wherein, The weighted score is calculated, and the formula is as follows: Weighted score = w1 x intersection over union + w2 x feature similarity; Wherein, w1 represents the weight of the intersection over union, w2 represents the weight of the feature similarity, and w1 + w2 = 1.

4. The image-based real-time processing method for product identification according to claim 1, wherein, The construction of the local feature library and the standard feature library includes: The feature vector of each put-in commodity in the current shopping period is dynamically stored to obtain the local feature library; The mapping relationship between the commodity bar code and the multi-dimensional feature vector is pre-stored to obtain the standard feature library. 5.The image-based real-time processing method for product identification according to claim 1, wherein, The method further includes directly reusing the feature vector of the stationary object and the calculation result of the previous frame. 6.The image-based real-time processing method for product identification according to claim 1, wherein, The object information includes an object detection frame, a category and a confidence.

7. A commodity recognition apparatus based on real-time image processing, characterized by, It includes: A multi-source acquisition module is configured to synchronously acquire double-camera video streams and weight sensor data; A parallel processing module is configured to deploy a multi-thread architecture and execute the commodity identification method based on image real-time processing according to any one of claims 1-6.

Images