Commodity track association method and device based on multi-modal information, equipment and medium

Abstract

The invention relates to the technical field of commodity trajectory tracking and management. The invention relates to a commodity track association method and device based on multi-modal information, equipment and a medium, and the method comprises the steps: obtaining the multi-modal information related to a single order, the multi-modal information comprising video information and code scanning information, and automatically tracking the position of a commodity in a video through a multi-target tracking model based on the video information, the method comprises the steps of obtaining commodity track information, automatically associating the commodity track information with commodity information in code scanning information according to the commodity track information in combination with the code scanning information to form a commodity track information set, inputting the commodity track information set into a deep neural network classifier, and automatically outputting label information of commodity tracks. And according to the label information of the commodity track, the abnormal commodity type is automatically identified, and an abnormal report is generated, and the abnormal report comprises a scanning missing abnormal report, a scale missing abnormal report and a wrong scanning abnormal report. According to the invention, the effect of accurate tracking of commodities is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of commodity trajectory tracking and management, and in particular to a commodity trajectory association method, device, equipment and medium based on multi-modal information. Background Art

[0002] At present, with the rapid development of the retail industry and e-commerce, in the process of commodity sales, management and settlement, the accurate acquisition and processing of commodity information have become increasingly complex. In traditional commodity management methods, mainly rely on single information acquisition methods such as barcode scanning and two-dimensional code scanning. However, these traditional methods often have problems such as information isolation and low accuracy in the process of commodity processing. Especially when dealing with a large number of commodities, abnormal situations such as missed scanning and wrong scanning are likely to occur, resulting in the loss or error of commodity information.

[0003] Most of the existing commodity tracking technologies rely on a single information source, such as relying only on a video surveillance system or a barcode scanner for commodity tracking. For the processing of video information, the existing technology usually uses a multi-object tracking model to identify the position of commodities. However, in a high-dynamic environment, the accuracy of video tracking is often low. Especially in situations such as commodity occlusion and rapid movement, position tracking errors are likely to occur, thus affecting subsequent commodity information identification and anomaly detection. At the same time, although barcode scanning can quickly identify commodity information, the single scanned code information cannot provide the spatio-temporal dynamic information of commodities, resulting in difficulty in accurately capturing the movement trajectory and abnormal state of commodities.

[0004] Therefore, how to effectively integrate multiple information sources, and improve the accuracy and efficiency of commodity information processing through high-precision commodity trajectory tracking and anomaly identification has become a technical problem to be solved urgently. Summary of the Invention

[0005] In order to achieve precise tracking of commodities, the present application provides a commodity trajectory association method, device, equipment and medium based on multi-modal information.

[0006] The first above-mentioned invention object of the present application is achieved through the following technical solutions: A commodity trajectory association method based on multi-modal information, the commodity trajectory association method based on multi-modal information includes: acquiring multi-modal information related to a single order, the multi-modal information including video information and scanned code information; Based on the video information, automatically track the position of commodities in the video through a multi-object tracking model to obtain the trajectory information of the commodities; According to the commodity trajectory information, in combination with the scanned code information, automatically associate the commodity trajectory information with the commodity information in the scanned code information to form a commodity trajectory information set; Input the set of the commodity trajectory information into a deep neural network classifier to automatically output the label information of the commodity trajectory; Automatically identify the abnormal commodity types according to the label information of the commodity trajectory, and generate an exception report, where the exception report includes a missed scanning exception report, a missed weighing exception report, and a mis-scanning exception report.

[0007] By adopting the above technical solution, by constructing a multi-object tracking model in the commodity trajectory association method based on multi-modal information, the position information of commodities in the video can be extracted in real time and accurately tracked, providing information support for the efficient construction of the commodity trajectory; by combining the scanning information and the video tracking information, based on the time window matching method, the commodity information in different information sources can be accurately associated to ensure the integrity of the trajectory information set and provide a reliable basis for subsequent anomaly detection; by analyzing the spatio-temporal features of the commodity trajectory through a deep neural network classifier, the normal and abnormal commodity trajectories can be intelligently identified, and the label information can be output to provide decision support for the automatic identification and report generation of abnormal commodity types; by automatically generating exception reports such as missed scanning, missed weighing, and mis-scanning according to the trajectory label information, the problems occurring in the logistics can be discovered and processed in a timely manner, thereby effectively improving the accuracy of commodity processing, reducing human errors, and improving the operation efficiency.

[0008] In a preferred example of the present application, it can be further configured that: based on the video information, through a multi-object tracking model, automatically perform position tracking on the commodities in the video to obtain the trajectory information of the commodities, including: Perform frame segmentation on the video information through the multi-object tracking model, and extract the commodity image regions in each frame; The multi-object tracking model identifies the commodity positions at each moment in the commodity image region to obtain the preliminary positioning information of the commodities; The multi-object tracking model identifies the position changes in the preliminary commodity positioning information based on the feature matching algorithm to obtain the position change information, and constructs the motion trajectory of the commodity based on the position change information to generate the commodity trajectory information.

[0009] By adopting the above technical solution, the commodities in the video can be tracked in real time through the multi-object tracking model, the image information of the commodities can be accurately extracted and their position information can be obtained, laying a foundation for the subsequent analysis of the motion trajectory of the commodities; by identifying the specific positions of the commodities at each moment and accurately identifying the position changes of the commodities in consecutive frames in combination with the feature matching algorithm, the motion trajectory of the commodities can be effectively constructed, thereby ensuring the high accuracy of the commodity trajectory information; by continuously analyzing the motion trajectory of the commodities, reliable information support can be provided for subsequent anomaly detection, commodity identification, and other related tasks, improving the automation and intelligent level of the entire commodity processing process.

[0010] In a preferred example, the present application can be further configured as follows: The multi-object tracking model is based on a feature matching algorithm to identify the position changes in the preliminary product positioning information, and obtain position change information, including: The multi-object tracking model extracts the feature point information of the product according to the position of the product at each moment in the preliminary product positioning information; According to the feature point information, the multi-object tracking model matches the feature points of the same product in consecutive frames through a feature matching algorithm, and obtains the position change information of the product in consecutive frames.

[0011] By adopting the above technical solution, it is possible to accurately capture the movement trajectory of the product in the video through feature point extraction and matching, realize the efficient tracking of the product position change. By extracting feature points from the product positioning information at each moment, the same product can be accurately matched between different frames, avoiding mis-matching caused by image noise or complex background, thus ensuring the reliability of the position change information. Using the feature matching algorithm, the position change of the product in consecutive frames can be dynamically obtained, further improving the accuracy and stability of the multi-object tracking model, and providing accurate information support for the subsequent analysis of the product movement trajectory and anomaly detection.

[0012] In a preferred example, the present application can be further configured as follows: Based on the position change information, construct the movement trajectory of the product and generate the product trajectory information, including: According to the position change information, the multi-object tracking model calculates the moving displacement of the product at each moment; Based on the moving displacement, the multi-object tracking model associates the position of the product at each moment with the position of the product at the previous moment, tracks the movement path of the product, and connects the positions of the product at each moment to construct the continuous movement trajectory of the product; The multi-object tracking model generates the product trajectory information according to the continuous movement trajectory of the product.

[0013] By adopting the above technical solution, it is possible to accurately quantify the movement process of the product by calculating the moving displacement of the product at each moment, avoiding the tracking error caused by the lack of accurate displacement calculation in the traditional method; based on the moving displacement, further combined with the positions of the product at consecutive moments, effectively construct the continuous movement trajectory of the product, realizing the accurate tracking of the multiple movement processes of the product in the video; by tracking the movement path of the product, the position information of the product at different moments can be connected to generate complete product trajectory information, providing a reliable information basis for subsequent anomaly detection, product identification and trajectory analysis, thus enhancing the application ability of the multi-object tracking model in complex scenarios.

[0014] In a preferred example, the present application can be further configured as follows: according to the commodity trajectory information, in combination with the scanning code information, automatically associate the commodity trajectory information with the commodity information in the scanning code information to form a commodity trajectory information set, including: Based on the timestamp in the commodity trajectory information and the scanning timestamp in the scanning code information, through the time window matching method, screen out the commodity information within the same time period; According to the commodity information within the same time period, judge whether the commodity positions in the commodity trajectory information and the scanning code information match to obtain a commodity position matching result; If the commodity position matching result is successful, confirm that there is an association between the commodity trajectory information and the commodity information in the scanning code information, and generate a complete commodity trajectory information set.

[0015] By adopting the above technical solution, it is possible to accurately screen out the commodity information within the same time period through the time window matching method, so as to ensure the matching of commodity positions in the same time dimension, avoid the problem of incorrect matching of commodity information in different time periods, and further improve the accuracy of commodity trajectory data by judging whether the commodity positions in the commodity trajectory information and the scanning code information match, providing accurate data support for subsequent analysis. When the commodity position matching is successful, it is possible to confirm the association between the commodity trajectory information and the scanning code information, generate a complete commodity trajectory information set, provide a reliable basis for the further processing of multi-modal information and anomaly detection, and thus greatly improve the efficiency and accuracy of commodity information association.

[0016] In a preferred example, the present application can be further configured as follows: input the commodity trajectory information set into a deep neural network classifier to automatically output the label information of the commodity trajectory, including: Based on the spatio-temporal feature information in the commodity trajectory information set, construct a spatio-temporal feature vector of the commodity; Through the deep neural network classifier, analyze the spatio-temporal feature vector of the commodity, identify the normal or abnormal trajectory of the commodity, and output the label information of the commodity trajectory.

[0017] By adopting the above technical solution, it is possible to organically combine the time and space features in the commodity trajectory information set by constructing the spatio-temporal feature vector of the commodity, further improving the expression ability of the trajectory information and the comprehensive analysis accuracy of the data. Through the analysis of the spatio-temporal feature vector by the deep neural network classifier, it is possible to automatically perform pattern recognition on the commodity trajectory, accurately distinguish the normal trajectory and the abnormal trajectory of the commodity, and effectively reduce the error of manual discrimination.

[0018] In a preferred example, the present application can be further configured as follows: according to the label information of the commodity trajectory, automatically identify the abnormal commodity type and generate an abnormal report, and the abnormal report includes a missed scanning abnormal report, a missed weighing abnormal report, and a mis-scanning abnormal report, including: According to the label information of the commodity trajectory, identify the missed scanning abnormality. If it is determined to be a missed scanning abnormality, it is determined that the commodity has not been scanned and the missed scanning abnormal report is generated; According to the label information of the commodity trajectory, identify the missed weighing abnormality. If it is determined to be a missed weighing abnormality, it is confirmed that the weighing information of the commodity is missing or there is an error, and the missed weighing abnormal report is generated; According to the label information of the commodity trajectory, identify the mis-scanning abnormality. If it is determined to be a mis-scanning abnormality, it is determined that the commodity does not match the commodity detail information in the commodity database and the mis-scanning abnormal report is generated.

[0019] By adopting the above technical solutions, it is possible to automatically identify and classify abnormal types through the label information of the commodity trajectory, improving the accuracy and efficiency of abnormal detection; through the identification of missed scanning abnormalities, it is possible to timely discover un-scanned commodities and generate detailed missed scanning abnormal reports, thus avoiding the situation of missing commodities; through the identification of missed weighing abnormalities, it is possible to accurately judge the absence or error of the commodity weighing information and generate a missed weighing abnormal report to ensure the accuracy of the weighing result; through the identification of mis-scanning abnormalities, it is possible to accurately compare the commodity information with the commodity detail information in the database, timely discover the mismatch situation, generate a mis-scanning abnormal report, and effectively improve the accuracy and reliability of commodity scanning and transactions.

[0020] The above-mentioned second invention object of the present application is achieved through the following technical solutions: A commodity trajectory association device based on multi-modal information, the commodity trajectory association device based on multi-modal information includes: a multi-modal information acquisition module for acquiring multi-modal information related to a single order, and the multi-modal information includes video information and scanning code information; A commodity trajectory tracking module for automatically tracking the position of the commodity in the video through a multi-object tracking model based on the video information to obtain the trajectory information of the commodity; A commodity trajectory information association module for automatically associating the commodity trajectory information with the commodity information in the scanning code information according to the commodity trajectory information to form a commodity trajectory information set; A deep neural network classification module for inputting the commodity trajectory information set into a deep neural network classifier to automatically output the label information of the commodity trajectory; An abnormal commodity identification and report generation module, which is used to automatically identify the types of abnormal commodities according to the label information of the commodity trajectory and generate an abnormal report, where the abnormal report includes a missed scanning abnormal report, a missed weighing abnormal report, and a mis-scanning abnormal report.

[0021] By adopting the above technical solution, by constructing a multi-object tracking model in the commodity trajectory association method based on multi-modal information, the position information of commodities in the video can be extracted in real time and accurately tracked, providing information support for the efficient construction of the commodity trajectory; by combining the scanning information and the video tracking information, based on the time window matching method, the commodity information in different information sources can be accurately associated to ensure the integrity of the trajectory information set and provide a reliable basis for subsequent abnormal detection; by analyzing the spatio-temporal features of the commodity trajectory through a deep neural network classifier, the normal and abnormal commodity trajectories can be intelligently identified and label information can be output, providing decision support for the automatic identification of abnormal commodity types and report generation; by automatically generating abnormal reports such as missed scanning, missed weighing, and mis-scanning according to the trajectory label information, the problems occurring in the logistics can be timely discovered and processed, thereby effectively improving the accuracy of commodity processing, reducing human errors, and improving the operation efficiency.

[0022] The above object three of the present application is achieved by the following technical solution: A computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where when the processor executes the computer program, the steps of the above commodity trajectory association method based on multi-modal information are implemented.

[0023] The above object four of the present application is achieved by the following technical solution: A computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above commodity trajectory association method based on multi-modal information are implemented.

[0024] In summary, the present application includes at least one of the following beneficial technical effects: 1. By constructing a multi-object tracking model in the commodity trajectory association method based on multi-modal information, it is possible to extract the commodity position information in the video in real time and perform accurate tracking, providing information support for the efficient construction of the commodity trajectory; by combining the barcode scanning information and the video tracking information, based on the time window matching method, it is possible to accurately associate the commodity information in different information sources, ensuring the integrity of the trajectory information set and providing a reliable basis for subsequent anomaly detection; by analyzing the spatio-temporal features of the commodity trajectory through a deep neural network classifier, it is possible to intelligently identify normal and abnormal commodity trajectories and output label information, providing decision support for the automatic identification and report generation of abnormal commodity types; by automatically generating abnormal reports such as missed scans, missed weighings, and mis-scans according to the trajectory label information, it is possible to timely discover and handle problems occurring in the logistics, thereby effectively improving the accuracy of commodity processing, reducing human errors, and improving operational efficiency. 2. It is possible to accurately quantify the movement process of the commodity by calculating the movement displacement of the commodity at each moment, avoiding the tracking errors caused by the lack of accurate displacement calculation in the traditional method; based on the movement displacement, further combining the commodity positions at consecutive moments, it is possible to effectively construct the continuous movement trajectory of the commodity and achieve the accurate tracking of the multiple movement processes of the commodity in the video; by tracking the movement path of the commodity, it is possible to connect the position information of the commodity at different moments and generate complete commodity trajectory information, providing a reliable information basis for subsequent anomaly detection, commodity identification, and trajectory analysis, thereby enhancing the application ability of the multi-object tracking model in complex scenarios. 3. By constructing a multi-object tracking model in the commodity trajectory association method based on multi-modal information, it is possible to extract the commodity position information in the video in real time and perform accurate tracking, providing information support for the efficient construction of the commodity trajectory; by combining the barcode scanning information and the video tracking information, based on the time window matching method, it is possible to accurately associate the commodity information in different information sources, ensuring the integrity of the trajectory information set and providing a reliable basis for subsequent anomaly detection; by analyzing the spatio-temporal features of the commodity trajectory through a deep neural network classifier, it is possible to intelligently identify normal and abnormal commodity trajectories and output label information, providing decision support for the automatic identification and report generation of abnormal commodity types; by automatically generating abnormal reports such as missed scans, missed weighings, and mis-scans according to the trajectory label information, it is possible to timely discover and handle problems occurring in the logistics, thereby effectively improving the accuracy of commodity processing, reducing human errors, and improving operational efficiency. Description of the Drawings

[0025] Figure 1 It is a schematic structural diagram of the commodity trajectory association method based on multi-modal information in an embodiment of the application; Figure 2 It is a flowchart of the implementation of step S20 in the commodity trajectory association method based on multi-modal information in an embodiment of the application; Figure 3It is a flowchart of the implementation of step S203 in the commodity trajectory association method based on multi-modal information in an embodiment of the present application; Figure 4 It is another flowchart of the implementation of step S203 in the commodity trajectory association method based on multi-modal information in an embodiment of the present application; Figure 5 It is a flowchart of the implementation of step S30 in the commodity trajectory association method based on multi-modal information in an embodiment of the present application; Figure 6 It is a flowchart of the implementation of step S40 in the commodity trajectory association method based on multi-modal information in an embodiment of the present application; Figure 7 It is a flowchart of the implementation of step S50 in the commodity trajectory association method based on multi-modal information in an embodiment of the present application; Figure 8 It is a principle block diagram of a commodity trajectory association device based on multi-modal information in an embodiment of the present application; Figure 9 It is a schematic diagram of a device in an embodiment of the present application. Detailed implementation manners

[0026] The present application will be further described in detail below with reference to the accompanying drawings.

[0027] In one embodiment, as Figure 1 shown, the present application discloses a commodity trajectory association method based on multi-modal information, which specifically includes the following steps: S10: Obtain multi-modal information related to a single order, where the multi-modal information includes video information and scanning code information.

[0028] Specifically, first, the system obtains the detailed information of the current order through the order management system or the point of sale terminal, including the order number, commodity information, purchase quantity, etc. By connecting to the sales terminal, multi-modal data related to the order commodity is obtained. The video information is used to collect the image data of the commodity during the transaction process in real time through a camera. These video frames reflect the dynamic information of the commodity, including the movement trajectory and position transformation of the commodity; the scanning code information is used to read the barcode or two-dimensional code of the commodity through a scanning device (such as a barcode scanner, mobile phone, etc.) to obtain information such as the unique identifier, commodity name, price, and quantity of the commodity.

[0029] S20: Based on the video information, automatically track the position of the commodity in the video through a multi-object tracking model to obtain the trajectory information of the commodity.

[0030] Specifically, each frame of the video information is extracted and converted into a data format suitable for the input of the multi-object tracking model. Usually, through scaling and normalization processing, the images are made to have a unified size and data range. The multi-object tracking model automatically detects the products in each frame of the image and locates their initial positions. The multi-object tracking model continuously monitors the movement trajectories of the products by tracking the positions of the detected product bounding boxes in each frame of the image. During the tracking process, the multi-object tracking model not only uses the appearance features of the products for re-identifying the objects, but also combines the movement information of the products in consecutive frames to predict their position changes, so as to ensure that the products can be accurately identified and tracked after occlusion, overlap or leaving the field of view. Specifically, the model predicts and updates the movement trajectories of the products through a Kalman filter, calculates the movement features such as the position, speed, and acceleration of the products, and uses the appearance information for matching to reduce the tracking error. In addition, multi-object tracking performs separate ID tracking on each product in the video to ensure that the trajectory information of each product can be consistent over different time periods. Finally, after being processed by the multi-object tracking model, the movement trajectory information of the products is obtained. S30: According to the product trajectory information and in combination with the scanning code information, automatically associate the product trajectory information with the product information in the scanning code information to form a product trajectory information set.

[0031] Specifically, based on a preset time window (for example, ±1 second), the scanning code records that overlap with the time period of the product trajectory information are filtered out from the scanning code information. Next, for the product information within the same time period that is filtered out, in combination with the features such as the position, speed, and size of the product, it is judged whether the product position in the product trajectory information is consistent with the product position in the scanning code information. To improve the matching accuracy, a feature matching algorithm is adopted, and the physical features in the product trajectory (such as the appearance, shape, or color of the product) are compared with the product information recorded in the scanning code information to confirm whether the product is the same object. In the case of successful position matching, further in combination with the unique identifier of the product (such as the barcode or QR code of the product), the product trajectory information is associated with the product information in the scanning code information. If the product positions match and the identifiers are the same, it is confirmed that this product trajectory and the product information in the scanning code information belong to the same product, and a complete trajectory information set of this product is generated.

[0032] S40: Input the product trajectory information set into a deep neural network classifier to automatically output the label information of the product trajectory.

[0033] Specifically, first, each commodity trajectory in the commodity trajectory information set is converted into a spatio-temporal feature vector, which includes data such as timestamps, position changes, speeds, and accelerations. Specifically, the commodity positions, speeds, and accelerations at each time point are extracted from the commodity trajectory information, and by calculating the change rates of positions and accelerations, the motion patterns of the commodity in each time period are obtained. Then, these spatio-temporal feature vectors are used as inputs and passed into a deep neural network classifier for analysis. The deep neural network learns based on the existing training data through multiple neuron layers, and identifies the differences between normal and abnormal trajectories. During the classification process, the deep neural network continuously adjusts the weights through an optimization algorithm, and can automatically determine whether the commodity trajectory is normal or abnormal based on the input feature vector, and output corresponding label information, such as "normal trajectory" or "abnormal trajectory".

[0034] S50: According to the label information of the commodity trajectory, automatically identify the abnormal commodity type and generate an abnormal report, which includes a missed scanning abnormal report, a missed weighing abnormal report, and a mis-scanning abnormal report.

[0035] Specifically, according to the label information of the commodity trajectory, each commodity trajectory is classified. If the label information indicates that the commodity trajectory is an abnormal trajectory, the abnormal type is further identified. If the label information shows that there is a missed scanning abnormality in the commodity trajectory, by analyzing the missing or incorrect scanning records in the scanning information of the commodity, it is judged whether there is a commodity that has not been correctly scanned. If it is judged as a missed scanning abnormality, a missed scanning abnormal report is generated, and the report contains detailed information about the un-scanned commodity, including the commodity number, the trajectory time point, the reason for the missed scanning, etc. If the label information shows that there is a missed weighing abnormality in the commodity trajectory, according to the weight information of the commodity, it is judged whether there is a missing or error in the weighing process of the commodity. If there is an error or missing, a missed weighing abnormal report is generated, and the report contains the weighing data of the commodity, the weighing error range, and related influencing factors. If the label information shows that there is a mis-scanning abnormality in the commodity trajectory, by comparing the scanning information of the commodity with the commodity information in the commodity database, it is judged whether there is a situation where the scanned commodity does not match the commodity information in the database. If a mismatch is found, a mis-scanning abnormal report is generated, and the report will list the detailed situation of the mismatch of the commodity information, such as the differences between the commodity name, the commodity barcode, and the relevant information in the commodity database, etc.

[0036] In one embodiment, as Figure 2 shown, in step S20, that is, based on the video information, through a multi-object tracking model, the positions of the commodities in the video are automatically tracked to obtain the trajectory information of the commodities, including: S201: Perform frame segmentation on the video information through a multi-object tracking model, and extract the commodity image regions in each frame.

[0037] Specifically, the video information is preprocessed and converted into a series of consecutive image frames, each frame representing a specific time point. Then, frame segmentation technology is used to extract each frame in the video stream. During the frame segmentation process, image denoising algorithms (such as median filtering or Gaussian blur) can be used to remove the noise in the frame and improve the image quality. Next, based on the background modeling and foreground detection methods, the commodity regions contained in each frame are identified. For the multi-commodity scenario, the multi-object tracking model identifies the commodities in each frame and frames their image regions. The image region of each commodity is identified by a bounding box, and a unique identifier is assigned to each commodity to ensure that different commodities can be accurately distinguished in subsequent tracking and analysis.

[0038] S202: The multi-object tracking model identifies the positions of the commodities at each moment in the commodity image region to obtain the preliminary positioning information of the commodities.

[0039] Specifically, through the multi-object tracking model, the commodity image region in each frame is compared with the commodity image region in the previous frame, and the position of the commodity is estimated and corrected based on the kinematic model (such as Kalman filtering). First, the multi-object tracking model identifies the commodities in each frame and assigns a unique ID to each commodity. Then, the multi-object tracking model matches the commodity image region in the current frame with the commodity image region in the previous frame to calculate the movement trajectory of the commodity. Through this matching method, the position of each commodity can be updated in real time to obtain the preliminary positioning information of the commodities. For commodities with large movements or frequent occlusions, data association algorithms can be used to solve the data association problem in multi-object tracking to ensure that the position of each commodity is accurately identified at each moment. When a commodity is occluded or disappears for a short time, the multi-object tracking model will maintain the tracking information of the commodity through prediction and interpolation techniques (such as Kalman filtering or particle filtering) and restore its position when the commodity reappears.

[0040] Furthermore, the data association algorithm refers to the algorithm used to match and associate different positions of the same commodity in multiple frames of video. The purpose of this algorithm is to ensure the consistency of the commodity positions in the video at different time points. By analyzing information such as the appearance features, movement trajectories, and spatial relationships of the commodities, the data association algorithm can determine whether a commodity is the same object between consecutive frames.

[0041] S203: The multi-object tracking model identifies the position changes in the preliminary positioning information of the commodities based on the feature matching algorithm to obtain the position change information. Based on the position change information, the movement trajectory of the commodity is constructed to generate the commodity trajectory information.

[0042] Specifically, in each frame, first, feature points of the commodity are extracted from the preliminary positioning information of the commodity, such as corner points, edges, texture or color features, etc. These feature points can uniquely identify the commodity and be matched in subsequent frames. Using feature matching algorithms, such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), the feature points in adjacent frames are matched, and the similarity and matching degree between them are calculated. For the matched feature points, by calculating the change in position, the movement information of the commodity at different time points can be obtained, and these changes are the displacement information of the commodity. Based on this displacement information, the movement speed and direction of the commodity at each moment are further calculated, and combined with the timestamp information, the position change of the commodity is tracked frame by frame, so as to construct the complete movement trajectory of the commodity.

[0043] In one embodiment, as Figure 3 shown, in step S203, that is, the multi-target tracking model based on the feature matching algorithm identifies the position change in the preliminary positioning information of the commodity and obtains the position change information, including: S2031: The multi-target tracking model extracts the feature point information of the commodity according to the position of the commodity at each moment in the preliminary positioning information of the commodity.

[0044] Specifically, first, through the preliminary positioning information (for example, the bounding box coordinates or key point positions of the commodity in each frame), the position range of the commodity in the current frame is determined. Then, based on this position area, an image feature extraction algorithm is used to extract the feature points of the commodity in this area. Feature points are positions in the image with high stability and significance. They can be corners, edges or texture features, and these points can maintain a certain degree of stability under different perspectives, illuminations and scales.

[0045] S2032: According to the feature point information, the multi-target tracking model matches the feature points of the same commodity in consecutive frames through the feature matching algorithm, and obtains the position change information of the commodity in consecutive frames.

[0046] Specifically, the feature point information extracted from the previous frame is used as the matching benchmark in the current frame. In the current frame, a feature matching algorithm is used to match the extracted feature points. This matching process calculates the similarity between feature points and selects the feature point pairs with the highest matching degree. During the matching process, the feature matching algorithm determines which feature points belong to the same commodity and pairs them by calculating the Euclidean distance or other similarity measurement methods between feature points. Through these matching results, the position change of the commodity between the current frame and the previous frame can be determined. Specifically, the feature matching algorithm calculates the displacement amount and movement direction of the commodity based on the displacement of feature points in consecutive frames, and further infers the movement trajectory of the commodity in the video. If the number of matched feature points reaches a preset threshold, the matching result is considered valid, and position change information is generated accordingly.

[0047] In one embodiment, as Figure 4 shown, in step S203, that is, based on the position change information, a movement trajectory of the commodity is constructed to generate commodity trajectory information, including: S2033: According to the position change information, the multi-object tracking model calculates the moving displacement amount of the commodity at each moment.

[0048] Specifically, in each frame, first, the feature point information of the commodity in this frame is extracted and matched with the feature points in the previous frame. To calculate the displacement amount between consecutive frames of the commodity, an innovative displacement calculation method is adopted: assuming that the position of the feature point of the commodity in the previous frame is P1(x1, y1), and the position of the feature point in the current frame is P2(x2, y2), then the moving displacement amount ΔP is: where, w x and w y are the weight coefficients on the x-axis and y-axis respectively. These weight coefficients are dynamically adjusted according to the feature distribution of the image, giving different priorities to movements in different directions. For example, if the commodity moves more on the horizontal axis, w x can be set to a larger value. In this way, the displacement amount not only considers the position change of the commodity but also weights the result according to the actual movement direction, thus more accurately reflecting the displacement information of the commodity at each moment.

[0049] S2034: Based on the moving displacement amount, the multi-object tracking model tracks the movement path of the commodity by associating the position of the commodity at each moment with the position of the commodity at the previous moment, and connects the positions of the commodity at each moment to construct a continuous movement trajectory of the commodity.

[0050] Specifically, first, the commodity position P0(x0, y0) at the initial moment is set as the starting point. In each subsequent frame, the position of the commodity is updated through the commodity position at the previous moment and the displacement ΔP at the current moment. Specifically, the calculated position P n (x n , y n ) of the commodity in the current frame can be determined as follows: P n (x n , y n ) = P n-1 (x n-1 , y n-1 ) + ΔPn, where ΔPn is the displacement calculated from the previous frame. After dynamically adjusting the weight, it is accurate to the specific position change of the commodity at each moment. To ensure accuracy and stability, the direction and speed of the commodity's movement are also considered during the displacement calculation. If the movement speed of the commodity is relatively fast at a certain moment, the displacement at that moment is correspondingly increased, and vice versa. In this way, the movement trajectories of the commodity in consecutive frames are more precisely connected. In practical applications, considering that the commodity may be briefly occluded or have a large position change, a correction step based on interpolation is adopted: if the displacement of the commodity between two frames is too large, the model will perform interpolation processing to generate synthetic intermediate frames between these frames, thereby filling the missing points in the trajectory. Finally, the commodity position points P0, P1, …, P n at each moment are connected into a smooth trajectory curve to construct the continuous movement trajectory of the commodity.

[0051] S2035: The multi-object tracking model generates commodity trajectory information based on the continuous movement trajectory of the commodity.

[0052] Specifically, after the continuous movement trajectory of the commodity is established, first, the model marks the time stamps for the movement trajectories of each commodity. Each frame position data P n (x n , y n ) of each commodity will correspond to an accurate time label t n, indicating the actual position of the product at this moment. This time information ensures that the product's trajectory can be dynamically updated over time, forming a complete time series dataset. Then, based on the position information of each product at different time points, the model smooths the product trajectory to remove possible noise points and discontinuous trajectories. For example, if there are sudden changes or abnormal variations (such as sudden jumps or position losses) in the product's trajectory within a short period, through smoothing algorithms (such as Kalman filtering or weighted average method), these abnormal points will be identified and corrected by appropriate interpolation or prediction methods to ensure that the generated trajectory is smoother and conforms to the normal movement law of the product. Subsequently, the model extracts key information of each product trajectory, such as movement speed, acceleration, moving direction, etc. By analyzing the displacement ΔPn at each moment and combining the speed changes between adjacent frames, the movement characteristics of the product can be obtained. In particular, by calculating the difference in displacement between consecutive frames, it is possible to identify whether the product has a trend of acceleration or deceleration and generate a trajectory dataset containing this information. Finally, all trajectory information is classified and organized according to the number or identifier of the product in the video to form complete product trajectory information.

[0053] In one embodiment, as Figure 5 shown, in step S30, that is, according to the product trajectory information and in combination with the scan code information, the product trajectory information is automatically associated with the product information in the scan code information to form a product trajectory information set, including: S301: Based on the time stamp in the product trajectory information and the scan time stamp in the scan code information, through the time window matching method, the product information within the same time period is screened out.

[0054] Specifically, take the time stamp in the product trajectory information and the scan time stamp in the scan code information. Next, set a suitable time window, and the length of this time window can be adjusted according to actual needs, usually several seconds or several minutes, to limit the matching time range. For each piece of product trajectory information, the algorithm checks whether its time stamp falls within this time window. If it does, it is considered that this trajectory information may correspond to a piece of scan code information. At this time, through the comparison of time stamps, the product trajectory that meets the time conditions is preliminarily matched with the product information in the scan code information. To ensure the accuracy of the matching, a tolerance (such as ±0.5 seconds or ±1 second) can also be set to handle the small time differences caused by factors such as camera delay and scanning device asynchronization. If the time stamp in the product trajectory information coincides with the scan time stamp in the scan code information within the set tolerance range, it is considered that this product trajectory corresponds to the scan code information, and then the product information within the same time period is screened out.

[0055] S302: Determine whether the product location in the product trajectory information matches the product location in the scan code information based on the product information within the same time period, and obtain the product location matching result.

[0056] Specifically, extract the product location information in the product information filtered within the same time period. This location information may include the coordinates of the product in the video, the shelf location where the product is located, the identification of the product, etc. At the same time, extract the trajectory points related to the product trajectory information, which contain the location information of the product during this time period. Next, determine the match by calculating the similarity between the trajectory points and the product location in the scan code information. The specific operation may include a judgment based on the spatial distance. For example, calculate the Euclidean distance between the product trajectory points and the product location in the scan code information. If the distance is less than the set threshold, it is considered that the product location match is successful; if the distance is greater than this threshold, it is considered that the match fails. In addition, comprehensive judgment can also be combined with the size, shape information of the product, etc. If the product location has a large deviation, it may be caused by changes in the camera angle or scanning errors, and the matching sensitivity can also be improved by appropriately adjusting the tolerance. Through these judgment methods, the result of the product location match is obtained. If the match is successful, it can be confirmed that there is an association between the product trajectory information and the product information in the scan code information.

[0057] S303: If the product location matching result is successful, confirm that there is an association between the product trajectory information and the product information in the scan code information, and generate a complete product trajectory information set.

[0058] Specifically, after confirming that the product location match is successful, extract the matching product trajectory information, including all location information, timestamps, and related feature data of the product in the video. Subsequently, integrate the product barcode, product number, price, product name, etc. information in the matching scan code information with the product trajectory information to form a complete product trajectory information set. To ensure the accuracy of the information, the system can perform data verification on the trajectory information to check whether the changes in the product location in consecutive frames match the product information in the scan code information, such as the appearance frequency of the product in the image, the residence time of the product in a specific area, etc. If the information is consistent and matches, a complete product trajectory information set is generated, containing all dynamic data of the product, including time, location, size change, item recognition result, etc.

[0059] In one embodiment, as Figure 6 shown, in step S40, the product trajectory information set is input into a deep neural network classifier to automatically output the label information of the product trajectory, including: S401: Based on the spatio-temporal feature information in the product trajectory information set, construct a spatio-temporal feature vector of the product.

[0060] Specifically, first, extract the position information and motion state data of each commodity at different time points from the commodity trajectory information set, including the spatial coordinates of the commodity (such as pixel positions in an image or physical positions in an actual coordinate system), and the time stamps at each moment. For each commodity, the spatio-temporal feature vector will include the following features at different time points: position features, such as the X and Y coordinates of the commodity at a specific moment; velocity features, calculated as the velocity of the commodity between two consecutive time points, usually the ratio of the change in position to the time difference; direction features, indicating the moving direction of the commodity, usually obtained by calculating the angle between the positions of the commodity at two adjacent moments; acceleration features, calculating the acceleration value through the change in the position of the commodity between multiple time points; and time features, such as the appearance time and departure time of the commodity. Then, to ensure the unity and comparability of the feature vectors, all the extracted features will be normalized, for example, by standardizing the values of position, velocity, acceleration, etc., so that the dimensions of these data are consistent and avoid a certain feature having too much influence on the model. Finally, combine the changes of each commodity in the time dimension to construct a vector containing spatio-temporal features.

[0061] S402: Analyze the spatio-temporal feature vector of the commodity through a deep neural network classifier to identify the normal or abnormal trajectory of the commodity and output the label information of the commodity trajectory.

[0062] Specifically, after receiving the spatio-temporal feature vector of the commodity, the deep neural network classifier first preprocesses the input data, including standardization and normalization, so that the model can better process input data of different scales. Next, the deep neural network (DNN) processes and learns these spatio-temporal features layer by layer through multiple layers of neurons. The neurons in each layer transform the output of the previous layer into a new feature representation through activation functions (such as ReLU, Sigmoid, or Tanh, etc.) to capture the complex non-linear relationships in the data. During the training phase of the neural network, a labeled data set is used to supervise the learning process of the model, and the model is trained to be able to identify abnormal patterns in the commodity trajectory, such as missed scans, missed weighings, or mis-scans, etc. The deep neural network processes the input spatio-temporal feature vector layer by layer, extracts features in the hidden layer, and learns the patterns of spatio-temporal trajectories. These patterns can include the normal movement trajectory of the commodity, pauses, sudden accelerations, or other abnormal behaviors. The network adjusts the network weights according to the error between the actual output and the target label through the backpropagation algorithm, so that the network gradually learns the correct classification boundary. After sufficient training, the classifier can judge whether the trajectory of the commodity is abnormal based on the input spatio-temporal feature vector and classify the commodity trajectory as "normal" or "abnormal", for example, missed scan anomaly, missed weighing anomaly, or mis-scan anomaly. Finally, the label information output by the model is the final classification result of the commodity trajectory, indicating whether there is abnormal behavior in the trajectory of the commodity.

[0063] In one embodiment, as Figure 7 shown, in step S50, based on the label information of the product trajectory, the abnormal product type is automatically identified and an abnormal report is generated. The abnormal report includes a missing scan abnormal report, a missing weighing abnormal report, and a mis-scan abnormal report, including: S501: Based on the label information of the product trajectory, identify the missing scan abnormality. If it is determined to be a missing scan abnormality, it is determined that the product has not been scanned and a missing scan abnormal report is generated.

[0064] Specifically, when identifying the missing scan abnormality, first analyze the label information of the product trajectory to check whether there is a scan record of the product within the time period when it should be scanned. If there is no corresponding scanning event or the scanning information is empty for the product within a certain time period, it can be determined as a missing scan abnormality. To further confirm the missing scan abnormality, it is necessary to compare the time stamps in the product trajectory information and the scanning information. If it is found that the product trajectory fails to match the scanning information within the corresponding time range, it is determined that the product has not been scanned. Finally, a missing scan abnormal report is generated by the system, which will list in detail the identification information of the product, the trajectory path of the product, and the missing scanning information, ensuring that relevant personnel can quickly identify and handle the missing scan problem.

[0065] S502: Based on the label information of the product trajectory, identify the missing weighing abnormality. If it is determined to be a missing weighing abnormality, it is confirmed that the weighing information of the product is missing or there is an error, and a missing weighing abnormal report is generated.

[0066] Specifically, when identifying the missing weighing abnormality, first analyze the label information of the product trajectory to check whether a product has completed the weighing operation within the specified time range. If there is a weighing action for the product, but no valid weighing data can be obtained (such as the weighing sensor does not record data or the data is lost), it can be determined as a missing weighing abnormality. If there is an error in the weighing data, for example, the deviation from the expected weight is too large, it can also be regarded as a missing weighing abnormality. At this time, the system compares the time stamps in the product trajectory with the time stamps of the weighing records to verify whether there is an error in the weighing process of the product and generates a missing weighing abnormal report. The report contains the missing or abnormal weighing information to help the staff timely discover and correct the problems in the weighing process.

[0067] S503: Based on the label information of the product trajectory, identify the mis-scan abnormality. If it is determined to be a mis-scan abnormality, it is determined that the product does not match the product details information in the product database and a mis-scan abnormal report is generated.

[0068] Specifically, when identifying mis-scanning anomalies, by analyzing the label information of the product trajectory, it is checked whether the product matches the product information in the database during scanning. If the scanning record of the product is inconsistent with the product name, barcode or other identification information in the database, and the product information displayed in the trajectory does not match the scanning record, it can be determined as a mis-scanning anomaly. To further confirm, the system will compare the information such as the timestamp, product type, weight, etc. in the product trajectory information with the product details in the product database. If a mismatch is found, it is recorded as a mis-scanning anomaly. Finally, the system generates a mis-scanning anomaly report, which lists the mismatched product information, the incorrect scanning record, and the correct product information in the database to help the staff identify and correct the scanning error.

[0069] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0070] In one embodiment, a product trajectory association device based on multi-modal information is provided. The product trajectory association device based on multi-modal information corresponds one-to-one with the product trajectory association method based on multi-modal information in the above embodiment. As Figure 8 shown, the product trajectory association device based on multi-modal information includes a multi-modal information acquisition module, a product trajectory tracking module, a product trajectory information association module, a deep neural network classification module, and an abnormal product identification and report generation module. The detailed description of each functional module is as follows: The multi-modal information acquisition module is used to acquire multi-modal information related to a single order. The multi-modal information includes video information and scanning information; The product trajectory tracking module is used to automatically track the position of the product in the video based on the video information through a multi-object tracking model to obtain the product trajectory information; The product trajectory information association module is used to automatically associate the product trajectory information with the product information in the scanning information according to the product trajectory information and the scanning information to form a product trajectory information set; The deep neural network classification module is used to input the product trajectory information set into a deep neural network classifier to automatically output the label information of the product trajectory; The abnormal product identification and report generation module is used to automatically identify the type of abnormal product according to the label information of the product trajectory and generate an abnormal report. The abnormal report includes a missed scanning anomaly report, a missed weighing anomaly report, and a mis-scanning anomaly report.

[0071] Optionally, the product trajectory tracking module includes: A product image region extraction sub-module, which is used to perform frame segmentation on video information through a multi-object tracking model and extract the product image region in each frame; A product position recognition sub-module, which is used to recognize the product position at each moment in the product image region through a multi-object tracking model to obtain the preliminary product positioning information; A product motion trajectory construction sub-module, which is used to recognize the position change in the preliminary product positioning information through a multi-object tracking model based on a feature matching algorithm to obtain the position change information, and construct the product motion trajectory based on the position change information to generate the product trajectory information.

[0072] Optionally, the product position recognition sub-module includes: A product feature point extraction unit, which is used to extract the feature point information of the product through a multi-object tracking model according to the position of the product at each moment in the preliminary product positioning information; A feature point matching and position change recognition unit, which is used to match the feature points of the same product in consecutive frames through a feature matching algorithm according to the feature point information to obtain the position change information of the product in consecutive frames.

[0073] Optionally, the product motion trajectory construction sub-module includes: A moving displacement calculation unit, which is used to calculate the moving displacement of the product at each moment through a multi-object tracking model according to the position change information; A product motion path tracking unit, which is used to track the product motion path by associating the product position at each moment with the product position at the previous moment based on the moving displacement, and connect the product positions at each moment to construct the continuous motion trajectory of the product; A product trajectory generation unit, which is used to generate the product trajectory information through a multi-object tracking model according to the continuous motion trajectory of the product.

[0074] Optionally, the product trajectory information association module includes: A time window matching sub-module, which is used to filter out the product information within the same time period through a time window matching method based on the time stamp in the product trajectory information and the scanning time stamp in the scanning code information; A product position matching judgment sub-module, which is used to judge whether the product position in the product trajectory information matches the product position in the scanning code information according to the product information within the same time period to obtain the product position matching result; A product trajectory association sub-module, which is used to confirm the association between the product trajectory information and the product information in the scanning code information and generate a complete product trajectory information set if the product position matching result is successful.

[0075] Optionally, the deep neural network classification module includes: A spatio-temporal feature vector construction sub-module, configured to construct a spatio-temporal feature vector of a commodity based on spatio-temporal feature information in a commodity trajectory information set; A trajectory classification and label output sub-module, configured to analyze the spatio-temporal feature vector of the commodity through a deep neural network classifier, identify normal or abnormal trajectories of the commodity, and output label information of the commodity trajectory.

[0076] Optionally, the abnormal commodity identification and report generation module includes: A missed scan anomaly identification and report generation sub-module, configured to identify missed scan anomalies according to the label information of the commodity trajectory, and if it is determined that there is a missed scan anomaly, determine that the commodity has not been scanned and generate a missed scan anomaly report; A missing weighing anomaly identification and report generation sub-module, configured to identify missing weighing anomalies according to the label information of the commodity trajectory, and if it is determined that there is a missing weighing anomaly, confirm that the commodity weighing information is missing or there is an error, and generate a missing weighing anomaly report; A mis-scan anomaly identification and report generation sub-module, configured to identify mis-scan anomalies according to the label information of the commodity trajectory, and if it is determined that there is a mis-scan anomaly, determine that the commodity does not match the commodity detail information in the commodity database and generate a mis-scan anomaly report.

[0077] For the specific limitations of the commodity trajectory association device based on multi-modal information, reference can be made to the limitations of the commodity trajectory association method based on multi-modal information in the above text, which will not be elaborated here. Each module in the above commodity trajectory association device based on multi-modal information can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0078] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for the commodity database. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a commodity trajectory association method based on multi-modal information.

[0079] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Obtain multi-modal information related to a single order, where the multi-modal information includes video information and scan code information; Based on the video information, automatically track the positions of the commodities in the video through a multi-object tracking model to obtain the trajectory information of the commodities; According to the commodity trajectory information and in combination with the scan code information, automatically associate the commodity trajectory information with the commodity information in the scan code information to form a commodity trajectory information set; Input the commodity trajectory information set into a deep neural network classifier to automatically output the label information of the commodity trajectory; According to the label information of the commodity trajectory, automatically identify the types of abnormal commodities and generate an abnormal report, where the abnormal report includes a missed scan abnormal report, a missed weighing abnormal report, and a wrong scan abnormal report.

[0080] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Obtain multi-modal information related to a single order, where the multi-modal information includes video information and scan code information; Based on the video information, automatically track the positions of the commodities in the video through a multi-object tracking model to obtain the trajectory information of the commodities; According to the commodity trajectory information and in combination with the scan code information, automatically associate the commodity trajectory information with the commodity information in the scan code information to form a commodity trajectory information set; Input the commodity trajectory information set into a deep neural network classifier to automatically output the label information of the commodity trajectory; According to the label information of the commodity trajectory, automatically identify the types of abnormal commodities and generate an abnormal report, where the abnormal report includes a missed scan abnormal report, a missed weighing abnormal report, and a wrong scan abnormal report.

[0081] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0082] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0083] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for associating commodity trajectories based on multi-modal information, characterized in that, The commodity trajectory association method based on multi-modal information includes: Obtain multi-modal information related to a single order, where the multi-modal information includes video information and scanning code information; Based on the video information, automatically track the position of the commodity in the video through a multi-object tracking model to obtain the trajectory information of the commodity; According to the commodity trajectory information, in combination with the scanning code information, automatically associate the commodity trajectory information with the commodity information in the scanning code information to form a commodity trajectory information set; Input the commodity trajectory information set into a deep neural network classifier to automatically output the label information of the commodity trajectory; According to the label information of the commodity trajectory, automatically identify abnormal commodity types and generate an abnormal report, where the abnormal report includes a missed scanning abnormal report, a missed weighing abnormal report, and a mis-scanning abnormal report.

2. The method for associating commodity trajectories based on multi-modal information according to claim 1, wherein The step of automatically tracking the position of the commodity in the video through a multi-object tracking model based on the video information to obtain the trajectory information of the commodity includes: Perform frame segmentation on the video information through the multi-object tracking model to extract the commodity image area in each frame; The multi-object tracking model identifies the position of the commodity at each moment in the commodity image area to obtain the preliminary positioning information of the commodity; The multi-object tracking model identifies the position change in the preliminary positioning information of the commodity based on a feature matching algorithm to obtain position change information, and based on the position change information, constructs the movement trajectory of the commodity and generates the commodity trajectory information.

3. The method for associating commodity trajectories based on multimodal information according to claim 2, wherein The step that the multi-object tracking model identifies the position change in the preliminary positioning information of the commodity based on a feature matching algorithm to obtain position change information includes: The multi-object tracking model extracts the feature point information of the commodity according to the position of the commodity at each moment in the preliminary positioning information of the commodity; According to the feature point information, the multi-object tracking model matches the feature points of the same commodity in consecutive frames through a feature matching algorithm to obtain the position change information of the commodity in consecutive frames.

4. The method for associating commodity trajectories based on multimodal information according to claim 2, wherein The step of constructing the movement trajectory of the commodity based on the position change information and generating the commodity trajectory information includes: According to the position change information, the multi-object tracking model calculates the moving displacement of the commodity at each moment; Based on the moving displacement, the multi-object tracking model associates the position of the commodity at each moment with the position of the commodity at the previous moment to track the movement path of the commodity and connect the positions of the commodity at each moment to construct the continuous movement trajectory of the commodity; The multi-object tracking model generates the commodity trajectory information according to the continuous movement trajectory of the commodity.

5. The method for associating commodity trajectories based on multi-modal information according to claim 1, wherein The step of automatically associating the commodity trajectory information with the commodity information in the scanning code information according to the commodity trajectory information, in combination with the scanning code information, to form a commodity trajectory information set includes: Based on the time stamp in the commodity trajectory information and the scanning time stamp in the scanning code information, through a time window matching method, screen out the commodity information within the same time period; According to the commodity information within the same time period, judge whether the commodity trajectory information matches the commodity position in the scanning code information to obtain a commodity position matching result; If the result of the product location matching is successful, confirm that there is an association between the product trajectory information and the product information in the scanned code information, and generate a complete set of product trajectory information.

6. The method for associating commodity trajectories based on multimodal information according to claim 1, wherein Input the set of product trajectory information into a deep neural network classifier, and automatically output the label information of the product trajectory, including: Based on the spatio-temporal feature information in the set of product trajectory information, construct a spatio-temporal feature vector of the product; Analyze the spatio-temporal feature vector of the product through the deep neural network classifier, identify the normal or abnormal trajectory of the product, and output the label information of the product trajectory.

7. The method for associating commodity trajectories based on multi-modal information according to claim 1, wherein According to the label information of the product trajectory, automatically identify the type of abnormal product and generate an abnormal report. The abnormal report includes a missed scan abnormal report, a missed weighing abnormal report, and a mis-scan abnormal report, including: According to the label information of the product trajectory, identify the missed scan abnormality. If it is determined to be a missed scan abnormality, it is determined that the product has not been scanned and the missed scan abnormal report is generated; According to the label information of the product trajectory, identify the missed weighing abnormality. If it is determined to be a missed weighing abnormality, it is confirmed that the product weighing information is missing or there is an error, and the missed weighing abnormal report is generated; According to the label information of the product trajectory, identify the mis-scan abnormality. If it is determined to be a mis-scan abnormality, it is determined that the product does not match the product details information in the product database and the mis-scan abnormal report is generated.

8. An apparatus for associating product trajectories based on multi-modal information, characterized in that, The product trajectory association device based on multi-modal information includes: A multi-modal information acquisition module for acquiring multi-modal information related to a single order. The multi-modal information includes video information and scanned code information; A product trajectory tracking module for automatically tracking the position of the product in the video through a multi-object tracking model based on the video information to obtain the product trajectory information; A product trajectory information association module for automatically associating the product trajectory information with the product information in the scanned code information according to the product trajectory information and in combination with the scanned code information to form a set of product trajectory information; A deep neural network classification module for inputting the set of product trajectory information into a deep neural network classifier and automatically outputting the label information of the product trajectory; An abnormal product identification and report generation module for automatically identifying the type of abnormal product according to the label information of the product trajectory and generating an abnormal report. The abnormal report includes a missed scan abnormal report, a missed weighing abnormal report, and a mis-scan abnormal report.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the product trajectory association method based on multi-modal information according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the product trajectory association method based on multi-modal information according to any one of claims 1 to 7 are implemented.

Images