Method and device for monitoring shopping process

Abstract

The invention discloses a shopping process monitoring method and device. The method comprises the following steps: acquiring accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor and video frame image data detected by a visual sensor; determining the advancing state of the shopping cart according to the accelerometer data and the gyroscope data; according to the advancing state of the shopping cart and the change of the weight data, determining a weight data identification result and the confidence of the weight data identification result when the commodity is put into or taken out of the shopping cart; according to the video frame image data, determining a video frame image data identification result and the confidence of the video frame image data identification result when the commodity is put into or taken out of the shopping cart; and according to the weight data identification result and the confidence thereof, the video frame image data identification result and the confidence thereof, and the dynamically determined weight data influence weight and the video frame image data influence weight, fusing and determining a final result that the commodity is put into or taken out of the shopping cart and the confidence thereof. The shopping process can be accurately monitored.

Description

Technical Field

[0001] The present invention relates to the technical field of shopping carts, and in particular to a method and device for monitoring a shopping process. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] When smart shopping carts are deployed in supermarkets, there's a need to prevent irregular shopping behavior. Because they're self-service, consumers inevitably try to circumvent the normal shopping process. Smart shopping carts need to detect and alert or prevent these behaviors. Accurately determining when items are added or removed from the cart is a crucial step in preventing irregular shopping behavior. Many shopping carts on the market rely on a single sensor to detect item placement, such as weight or vision. There are also methods that combine weight and vision sensors, but these existing methods have limitations and can't accurately detect when items are added or removed from the cart. Summary of the Invention

[0004] An embodiment of the present invention provides a method for monitoring a shopping process, for accurately monitoring the shopping process. The method includes:

[0005] Acquire accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor;

[0006] determining a moving state of the shopping cart based on the accelerometer data and the gyroscope data;

[0007] Determining, based on the travel status of the shopping cart and the change in the weight data, a weight data recognition result and a confidence level of whether the product is placed in or taken out of the shopping cart;

[0008] Determining, based on the video frame image data, a video frame image data recognition result indicating whether a product is placed in or taken out of a shopping cart and its confidence level;

[0009] Based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight, the final result of whether the product is placed in or taken out of the shopping cart and its confidence level are determined by fusion.

[0010] An embodiment of the present invention further provides a shopping process monitoring device for accurately monitoring the shopping process, the device comprising:

[0011] an acquisition unit, configured to acquire accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor;

[0012] a state determination unit, configured to determine a travel state of the shopping cart based on the accelerometer data and the gyroscope data;

[0013] a weight data recognition unit, configured to determine a weight data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the travel state of the shopping cart and changes in the weight data;

[0014] a video frame data recognition unit, configured to determine, based on the video frame image data, a video frame image data recognition result indicating whether a product is placed in or taken out of a shopping cart and its confidence level;

[0015] A final detection unit is used to determine the final result of whether the product is placed in or taken out of the shopping cart and its confidence level based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight.

[0016] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned shopping process monitoring method when executing the computer program.

[0017] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned shopping process monitoring method.

[0018] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned shopping process monitoring method.

[0019] In an embodiment of the present invention, a shopping process monitoring solution comprises: obtaining accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor; determining a shopping cart movement state based on the accelerometer data and gyroscope data; determining a weight data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the shopping cart movement state and changes in weight data; determining a video frame image data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the video frame image data; and integrating a final result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, a dynamically determined weight data influence weight, and a video frame image data influence weight. This allows for accurate monitoring of the shopping process and improves the accuracy of monitoring the shopping process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 Schematic diagram of the flow of a method for monitoring a shopping process according to an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the system architecture for monitoring the shopping process in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a process for establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data in an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the structure of a shopping process monitoring device according to an embodiment of the present invention;

[0025] Figure 5 FIG. 1 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0027] The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant laws and regulations.

[0028] This embodiment of the present invention proposes a method for analyzing and determining the shopping process based on intelligent hardware, including a weight sensor mounted on the bottom of the wheel and a visual sensor mounted on the device. Furthermore, a unique and novel fusion algorithm is designed to effectively calculate and determine the data from the weight and visual sensors, more accurately determining whether items are placed in or removed from the shopping cart. This provides a monitoring solution for the shopping process, which is described in detail below.

[0029] Figure 1 FIG. 1 is a flow chart of a method for monitoring a shopping process according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0030] Step 101: Acquire accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor;

[0031] Step 102: Determine the travel state of the shopping cart based on the accelerometer data and the gyroscope data;

[0032] Step 103: Determine the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data;

[0033] Step 104: Determine, based on the video frame image data, a recognition result of the video frame image data indicating whether a product is placed in or taken out of the shopping cart and its confidence level;

[0034] Step 105: Based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight, a fusion is performed to determine the final result of whether the product is placed in or taken out of the shopping cart and its confidence level.

[0035] By process Figure 1It can be seen that the shopping process monitoring method provided by the embodiment of the present invention works as follows: acquiring accelerometer data and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the visual sensor; determining the travel state of the shopping cart based on the accelerometer data and gyroscope data; determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel state of the shopping cart and the change in the weight data; determining the video frame image data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the video frame image data; based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight level, the final result of determining whether the product is placed in or taken out of the shopping cart and its confidence level are integrated to accurately monitor the shopping process and improve the accuracy of monitoring the shopping process. The following is a detailed introduction to the shopping process monitoring method.

[0036] The shopping process monitoring method provided by the present invention monitors the movement state, weight changes, and visual video frame data of a shopping cart during use, capturing the shopping process. It also proposes a fusion judgment algorithm to effectively determine the information determined by the two sensor data and output the final information on whether an item has been placed or removed. This is described in detail below.

[0037] In order to facilitate understanding of the shopping process monitoring method provided by the embodiment of the present invention, the system architecture for monitoring the shopping process is first introduced.

[0038] The embodiment of the present invention can add several weight sensors and several visual cameras on the basis of the necessary hardware of the smart shopping cart to capture the shopping behavior of shoppers and accurately determine the process of placing or taking out goods.

[0039] Figure 2 FIG. 1 is a schematic diagram of a system architecture for monitoring the shopping process in an embodiment of the present invention. Figure 2 As shown, embodiments of the present invention may include:

[0040] Barcode scanner 1: A common barcode scanner gun or barcode scanner that can scan and identify the barcode on the product, connect to the computing and interactive device 5, and send the identification result to the computing and interactive device 5.

[0041] Weight sensor 2: Designed to be placed on the wheel of a shopping cart, it can be flexibly installed and removed from the support frame of a common shopping cart. The weight sensor has the ability to sense weight, that is, it can sense weight changes and specific values; the weight sensor has communication capabilities and can connect to computing and interactive devices and transmit data.

[0042] Visual sensor 3: installed above and around the shopping cart basket, there are more than one visual sensor 3; connected to the computing and interactive device 5 and transmits data.

[0043] The inertial measurement unit 4 acquires accelerometer data and gyroscope data in real time.

[0044] Computing and interactive device 5: An electronic device with a touchscreen and a processing unit with a certain computing power. Installed on the handle of a shopping cart, it connects to a barcode scanner, weight sensor, and visual camera (sensor). It can run an operating system and software, presenting data on an interactive screen, and process data transmitted by the barcode scanner, weight sensor, and visual camera. The shopping process monitoring method provided in this embodiment of the present invention is a method applied to a computing and interactive device.

[0045] The shopping process monitoring method provided by the embodiment of the present invention may include the following steps.

[0046] 1. Real-time data acquisition: Acquire real-time data from IMU (Inertial Measurement Unit) sensors, real-time data from weight sensors, and video frame images;

[0047] 2. Determine whether the shopping cart is in motion based on the non-zero acceleration detected by the accelerometer;

[0048] 3. Detect whether goods are put in or taken out based on weight changes;

[0049] 4. Detect whether a product is put in or taken out based on the video frame data;

[0050] 5. Integrate the two types of judgment information to comprehensively judge the situation of goods entering and leaving.

[0051] The following is a detailed introduction to the monitoring method of the shopping process.

[0052] 1. Acquire data in real time, i.e., step 101 above.

[0053] Acquire real-time data (accelerometer data and gyroscope data) from the IMU (Inertial Measurement Unit) sensor, real-time data (weight data) from the weight sensor, and video frame image data, and transmit the real-time data to the computing and interactive devices.

[0054] 2. Based on the accelerometer data and the gyroscope data, determine whether the shopping cart is in motion, i.e., step 102 above.

[0055] Based on the accelerometer and gyroscope data obtained by the IMU sensor, data preprocessing is performed first, including denoising and calibration. The following steps are performed on the preprocessed data:

[0056] 1) Calculate the modulus a of the acceleration on the three axes (X, Y, Z) of the accelerometer total :

[0057]

[0058] 2) According to the gyroscope value, calculate the sum of the absolute values ​​of the angular velocity of each axis to obtain r total .

[0059] 3) If the modulus of acceleration a total Equal to or close to the preset modulus threshold (e.g. 9.8 m / s 2 ), and r total If the total angular velocity is less than a preset threshold (e.g., 0.2 rad / s) and does not change much within a certain period of time (within a preset time period) (the modulus change of acceleration is less than or equal to the preset modulus change threshold, and the total angular velocity absolute value change is less than or equal to the preset angular velocity absolute value change threshold), the shopping cart is considered to be stationary.

[0060] 4) If the modulus of acceleration a total and r total If both acceleration modulus and angular velocity absolute values ​​are greater than a threshold (the modulus of acceleration is greater than a preset modulus threshold, and the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value threshold), or if there is a significant change within a short period of time (within a preset period of time, such as 0.5 seconds) (the change in modulus is greater than a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value change threshold), the shopping cart is considered to be in motion.

[0061] As can be seen from the above, in one embodiment, determining the moving state of the shopping cart based on the accelerometer data and the gyroscope data may include:

[0062] Determining the modulus of acceleration on three axes in the accelerometer data;

[0063] According to the gyroscope data, determine the sum of the absolute values ​​of the angular velocity of each axis;

[0064] The shopping cart is determined to be in a stationary state when the modulus of acceleration is less than or equal to a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is less than or equal to a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value change threshold.

[0065] As can be seen from the above, in one embodiment, determining the moving state of the shopping cart based on the accelerometer data and the gyroscope data may further include:

[0066] The shopping cart is determined to be in motion when the modulus of acceleration is greater than a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is greater than a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value change threshold.

[0067] In specific implementation, the above specific implementation method of determining the travel state (moving state or stationary state) of the shopping cart based on the accelerometer data and the gyroscope data can improve the monitoring accuracy of the subsequent shopping process.

[0068] 3. Detect whether there is any goods entering or exiting based on the weight change, i.e., step 103 above.

[0069] If the shopping cart is stationary:

[0070] Determine the weight sensor change data (weight data change) within a time interval (preset time interval, such as 1 second):

[0071] If the weight value increases from a stable state (for example, the weight waveform is a stable waveform) (for example, the weight waveform is a waveform with an increasing trend), and finally remains stable (maintained within the preset weight threshold range), and the difference is greater than a certain threshold (preset weight threshold, such as 10g), it is determined that the product is put in; if the weight value decreases from a stable state, and finally remains stable (maintained within the preset weight threshold range), and the difference is greater than a certain threshold (preset weight threshold, such as 10g), it is determined that the product is taken out; and, at this time, the confidence value Conf_a (confidence of the weight data recognition result) is set to 1.

[0072] As can be seen from the above, in one embodiment, based on the travel state of the shopping cart and the change in the weight data, determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart may include:

[0073] When it is detected that the moving state of the shopping cart is stationary, the change of the weight data within the preset time interval is detected;

[0074] If it is determined according to the test results that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being placed in the shopping cart are obtained.

[0075] As can be seen from the above, in one embodiment, determining the weight data recognition result and its confidence level of whether an item is placed in or taken out of the shopping cart based on the travel state of the shopping cart and the change in the weight data may further include:

[0076] If it is determined according to the detection results that the weight value changes from a stable state to a decreasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being taken out of the shopping cart are obtained.

[0077] In specific implementation, the above specific implementation method of obtaining the weight data recognition result and the confidence level of the product being taken out of the shopping cart when the moving state of the shopping cart is detected to be stationary can further improve the accuracy of shopping process monitoring.

[0078] If the shopping cart is in motion:

[0079] Determine the weight sensor change data (weight data change) within a time interval (preset time interval, such as 1 second):

[0080] First, the variance of the values ​​within the time series interval is calculated. If the variance of the weight change data is greater than a certain threshold (preset variance threshold), then the weight sensor change data within the time series interval is determined to be of the type of violent fluctuation (determining that the weight data change within the preset time series interval is of the type of violent fluctuation), and the use of the weight sensor data in the time series interval to determine the entry and exit of goods is abandoned, that is, the confidence value Conf_a is set to 0. If the weight change data is not of the type of violent fluctuation (if the variance value is less than or equal to the preset variance threshold, it is determined that the weight data change within the preset time series interval is not of the type of violent fluctuation), then further judgment is made on the weight change data within a time series interval:

[0081] If the weight value increases from a stable state and finally remains stable, and the difference is greater than a certain threshold (preset weight threshold), it is determined that the product has been placed (the weight data recognition result of the product being placed in the shopping cart is obtained); if the weight value decreases from a stable state and finally remains stable, and the difference is greater than a certain threshold (preset weight threshold), it is determined that the product has been taken out (the weight data recognition result of the product being taken out of the shopping cart is obtained);

[0082] The weight value data for determining "put in" and "take out" (determining whether the product is the weight value corresponding to when it is placed in the shopping cart) is sent to the "reliability scoring model" (i.e., confidence recognition model) for scoring, and the score is used as the confidence value Conf_a (confidence of the weight data recognition result) that can be used for judgment, and the value is between [0,1].

[0083] The "reliability scoring model" is a discriminant classification model trained based on machine learning methods. It is trained using historically collected weight sensor change data sets and corresponding labels. It can ultimately score the weight sensor change data within an input time series interval and output a score value between [0, 1].

[0084] As can be seen from the above, in one embodiment, based on the travel state of the shopping cart and the change in the weight data, determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart may include:

[0085] When it is detected that the shopping cart is in a moving state, the weight data change within a preset time interval is detected;

[0086] Determine a variance value of the weight data within a preset time series interval, and if the variance value is less than or equal to a preset variance threshold, determine that a change in the weight data within the preset time series interval does not belong to a type of drastic fluctuation;

[0087] When it is determined that the weight data change within the preset time interval does not belong to the type of drastic fluctuation, if it is determined according to the detection result that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value in the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been added to the shopping cart;

[0088] The confidence level of the weight data recognition result of the product placed in the shopping cart is determined based on the corresponding weight value when the product is placed in the shopping cart.

[0089] As can be seen from the above, in one embodiment, determining the weight data recognition result and its confidence level of whether an item is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data further includes:

[0090] If the weight value changes from a stable state to a decreasing state according to the detection result and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been taken out of the shopping cart;

[0091] The confidence level of the weight data recognition result of the product taken out of the shopping cart is determined based on the corresponding weight value when the product is taken out of the shopping cart.

[0092] As can be seen from the above, in one embodiment, determining the weight data recognition result and its confidence level of whether an item is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data further includes:

[0093] If the variance value is greater than the preset variance threshold, it is determined that the weight data change in the preset time series interval is a type of drastic fluctuation, and the use of the weight data in the time series interval to judge the entry and exit of goods is abandoned, and the confidence level is determined to be 0.

[0094] In specific implementation, the above specific implementation method of obtaining the weight data recognition result and the confidence level of the product being taken out of the shopping cart when the shopping cart is detected to be in motion can further improve the accuracy of shopping process monitoring.

[0095] As can be seen from the above, in one embodiment, determining the confidence level of the weight data recognition result of the product taken out of the shopping cart based on the corresponding weight value of the product when it is determined to be taken out of the shopping cart may include:

[0096] The weight value corresponding to when the product is taken out of the shopping cart is input into the confidence recognition model to obtain the confidence of the weight data recognition result of the product taken out of the shopping cart. The confidence value range is within the interval [0,1]. The confidence recognition model is pre-trained based on sample data of the relationship between historical weight values ​​and confidence levels. The input of the model is the weight value, and the output is the confidence level.

[0097] In specific implementation, the above-mentioned method of determining the confidence level of the weight data recognition result of the removed shopping cart can improve the efficiency and accuracy of determining the confidence level.

[0098] 4. Detect whether there are any goods entering or exiting based on the video frame data, i.e., step 104 above.

[0099] Judge the video frame data within a time interval (such as 1 second):

[0100] The time-series video frame data is fed into the video understanding model, which is built and trained based on deep learning methods.

[0101] The video understanding model receives the input time-series video segment data, outputs whether the time-series video segment belongs to one of the categories "put in", "take out", or "put in and take out at the same time", and outputs the confidence of the judgment as the confidence Conf_b of the put in or take out action.

[0102] The details of the video understanding model are as follows:

[0103] 1) Model Architecture: Using a transformer-based architecture, this patent prioritizes the Swin Transformer3D model. Based on this architecture, 1. A dynamic attention mechanism, AdaFocus, is introduced to enable the model to adaptively focus on key frames and key areas in the video, reducing computational redundancy while improving the ability to recognize complex behaviors. 2. The hierarchical structure of the model is improved, integrating features from different layers and scales in the second half of the model to further improve the recognition accuracy of product behaviors at different scales.

[0104] 2) Model Pre-training: Use large-scale video data, including the Kinetics-700 dataset, UCF101, HMDB51, and NTU RGB+D datasets, to pre-train the improved Swin Transformer 3D model built in step 1).

[0105] 3) Data Collection and Labeling: Video data of items being placed in and removed from shopping carts is collected by visual sensors installed on the shopping carts. This data can be from laboratory scenarios or scenes of shopping carts being used in stores. The collected video data is manually labeled, marking each video segment of approximately one second in length. Each segment is annotated with attributes or labels related to the placement and removal of items in and out of the shopping cart. This forms a standard dataset for subsequent model fine-tuning, training, and validation.

[0106] 4) Data Preprocessing and Augmentation: The labeled video dataset undergoes further frame rate adjustment, resolution scaling, and normalization to meet the input requirements of the improved Swin Transformer 3D model. Video data is augmented using techniques such as random cropping, flipping, illumination transformation, and random acceleration to improve the model's robustness to varying lighting conditions, viewpoint changes, and motion speeds.

[0107] 5) Model fine-tuning training: Use the labeled video data to train the improved SwinTransformer 3D model built in step 1. Use the basic cross-entropy loss function, AdamW as the optimizer, and cosine annealing learning rate scheduler to ensure that the model converges stably during training.

[0108] As can be seen from the above, in one embodiment, determining, based on the video frame image data, a recognition result of the video frame image data indicating that a product is placed in or taken out of a shopping cart and its confidence level includes: inputting the video frame image data into a video understanding model pre-trained according to the following method to obtain a recognition result of the video frame image data indicating that a product is placed in or taken out of a shopping cart and its confidence level:

[0109] The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model to enable the model to adaptively focus on key frames and key areas in video frame image data. The features of different layers and scales in the second half of the model are fused to obtain the improved Swin Transformer 3D model.

[0110] Pre-train the improved Swin Transformer 3D model using video data larger than the preset scale to obtain a pre-trained video understanding model;

[0111] The pre-trained video understanding model is fine-tuned and verified using historical video frame image data of labeled items being placed in or taken out of the shopping cart to obtain an optimized video understanding model.

[0112] As can be seen from the above, in one embodiment, the shopping process monitoring method may further include:

[0113] The historical video frame image data of marked items being placed in or taken out of the shopping cart is preprocessed by frame rate adjustment, resolution scaling, and normalization to obtain preprocessed video frame image data to adapt to the input requirements of the improved SwinTransformer 3D model;

[0114] The preprocessed video frame image data is enhanced by using random cropping, flipping, illumination transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, perspective changes and motion speeds, and obtain enhanced video frame image data.

[0115] 5. Integrate the two types of judgment information to comprehensively judge the entry and exit of goods, i.e., the above-mentioned step 105.

[0116] Configure two influence weights for the decision-making information based on weight sensor data and visual video frame data. Set the default influence weight based on weight sensor data to α = 0.5, and the default influence weight based on visual video frame data to β = 0.5. In practice, the values ​​of α and β can be calculated based on experience or using machine learning methods and ultimately adjusted to other more optimal values, such as α = 0.65 and β = 0.35.

[0117] However, the present invention further uses a method for dynamically calculating the optimal values ​​of α and β based on historical data, that is, it can dynamically calculate according to the values ​​of Conf_a and Conf_b at that time. The specific description is:

[0118] Collect a large amount of historical data, such as 100,000 video examples. When an insertion or removal action occurs, record Conf_a and Conf_b at that time. Also record Conf_a and Conf_b that are not 0 when no action occurs. For Conf_a and Conf_b, when their values ​​are greater than 0.8, check whether the action actually occurred. If the action did occur, it is recorded as correct, otherwise it is recorded as an error. Finally, the cumulative calculation is performed to calculate the accuracy of the weight sensor judgment and the visual sensor judgment, Acc. a and Acc b .

[0119] With Acc calculated based on history a and Acc b , for each action case, different α and β thresholds can be dynamically set according to the different values ​​of Conf_a and Conf_b. The formula (model for dynamically determining the weight of weight data influence) is:

[0120]

[0121] The model for dynamically determining the influence weight of video frame image data is: β = 1-α;

[0122] Among them, α is the weight data influence weight, β is the video frame image data influence weight, Acc a Acc is the weight data detection accuracy determined based on historical data. b is the detection accuracy of the video frame image data determined based on historical data, Conf_a is the confidence of the current weight data recognition result, and Conf_b is the confidence of the current video frame image data recognition result.

[0123] Specifically, the timeline is used to check the entry and exit information of goods. For two types of judgment information, after one judgment is completed, wait for the other judgment signal for a maximum of n seconds (e.g., 1 second). If no judgment signal is received after n seconds, the corresponding influence weight is set to 0. For judgment signals that have been received, the influence weight is set to 1. For example, if the visual sensor does not receive a judgment signal within 1 second after the weight sensor receives a signal, set Conf_a to 1 and Conf_b to 0.

[0124] In one embodiment, the shopping process monitoring method may further include adjusting the video frame image data recognition result or the confidence level of the video frame image data recognition result based on the weight data recognition result and the determination time of the video frame image data recognition result as follows:

[0125] After one of the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds;

[0126] If the waiting time exceeds the preset number of seconds and no other recognition result is received, the influence weight value corresponding to the other recognition result is adjusted to 0;

[0127] If the waiting time is less than or equal to the preset number of seconds, another recognition result has been received, and the influence weight corresponding to the other recognition result is 1.

[0128] During specific implementation, the embodiment of the present invention can effectively and accurately analyze and judge the process of putting in or taking out goods during the shopping process by adjusting different confidence values ​​and influence weights.

[0129] Finally, the final decision confidence Conf is calculated, that is, based on the weight data recognition result and its confidence, the video frame image data recognition result and its confidence, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight, the final result of determining whether the product is placed in or taken out of the shopping cart and its confidence, including: determining the final result of whether the product is placed in or taken out of the shopping cart according to the following fusion model:

[0130] Conf=λ(α×Conf_a+β×Conf_b);

[0131] Among them, Conf is the final result, α is the influence weight of weight data, β is the influence weight of video frame image data, Conf_a is the confidence of the current weight data recognition result, Conf_b is the confidence of the current video frame image data recognition result, λ is the degree of confidence influence, λ=1 / n, n is the difference between the time when weight data recognition result is determined and the time when video frame image data is determined.

[0132] Furthermore, for the confidence level λ, the shorter the time difference between the two judgment signals received within n seconds, the larger the λ value, set to λ = 1 / n. The implication here is that the more synchronized the two judgment signals are, the higher the probability that the action will occur.

[0133] If Conf is greater than the set threshold, it is determined that a product has been put in or taken out.

[0134] From the above, it can be seen that in one embodiment, Figure 3 As shown, the above-mentioned shopping process monitoring method may further include establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data according to the following method:

[0135] Step 201: Acquire multiple historical video frame image data;

[0136] Step 202: Based on the plurality of historical video frame image data, the confidence level of the historical weight data recognition result when a product is placed in or taken out, the confidence level of the historical video frame image data recognition result, and the confidence level of the historical weight data recognition result that is not zero when no product is placed in or taken out, as well as the confidence level of the historical video frame image data recognition result;

[0137] Step 203: When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, checking whether a product placement or removal action actually occurred at this time to obtain a verification result;

[0138] Step 204: Determine the weight data detection accuracy and the video frame image data detection accuracy based on the inspection results;

[0139] Step 205: Based on the weight data detection accuracy and the video frame image data detection accuracy, a model for dynamically determining the weight of the weight data and a model for dynamically determining the weight of the video frame image data are established.

[0140] In specific implementation, the above-mentioned specific implementation methods of establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data can further improve the accuracy of shopping process monitoring.

[0141] In summary, the shopping process monitoring method provided by the embodiment of the present invention effectively combines the data of IMU sensors, weight sensors, and visual sensors, and adjusts different confidence values ​​and influence weights through fusion algorithm logic. It can effectively and accurately analyze and judge the process of placing or taking out goods during the shopping process, and can more accurately judge whether goods are placed in or taken out of the shopping cart.

[0142] The present invention also provides a shopping process monitoring device, as described in the following embodiments. Since the principle of the device to solve the problem is similar to that of the shopping process monitoring method, the implementation of the device can refer to the implementation of the shopping process monitoring method, and the repeated parts will not be repeated.

[0143] Figure 4 FIG. 1 is a schematic diagram of the structure of a monitoring device for a shopping process according to an embodiment of the present invention. Figure 4 As shown, the device includes:

[0144] Acquisition unit 01, used to acquire accelerometer data and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the visual sensor;

[0145] a state determination unit 02, configured to determine the travel state of the shopping cart based on the accelerometer data and the gyroscope data;

[0146] The weight data recognition unit 03 is used to determine the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change of the weight data;

[0147] The video frame data recognition unit 04 is configured to determine, based on the video frame image data, a video frame image data recognition result indicating whether a product is placed in or taken out of a shopping cart and its confidence level;

[0148] The final detection unit 05 is used to determine the final result of whether the product is placed in or taken out of the shopping cart and its confidence level based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight.

[0149] In one embodiment, the state determination unit is specifically configured to:

[0150] Determining the modulus of acceleration on three axes in the accelerometer data;

[0151] According to the gyroscope data, determine the sum of the absolute values ​​of the angular velocity of each axis;

[0152] The shopping cart is determined to be in a stationary state when the modulus of acceleration is less than or equal to a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is less than or equal to a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value change threshold.

[0153] In one embodiment, the state determination unit is further configured to:

[0154] The shopping cart is determined to be in motion when the modulus of acceleration is greater than a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is greater than a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value change threshold.

[0155] In one embodiment, the weight data identification unit is specifically configured to:

[0156] When it is detected that the moving state of the shopping cart is stationary, the change of the weight data within the preset time interval is detected;

[0157] If it is determined according to the test results that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being placed in the shopping cart are obtained.

[0158] In one embodiment, the weight data identification unit is further configured to:

[0159] If it is determined according to the detection results that the weight value changes from a stable state to a decreasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being taken out of the shopping cart are obtained.

[0160] In one embodiment, the weight data identification unit is specifically configured to:

[0161] When it is detected that the shopping cart is in a moving state, the weight data change within a preset time interval is detected;

[0162] Determine a variance value of the weight data within a preset time series interval, and if the variance value is less than or equal to a preset variance threshold, determine that a change in the weight data within the preset time series interval does not belong to a type of drastic fluctuation;

[0163] When it is determined that the weight data change within the preset time interval does not belong to the type of drastic fluctuation, if it is determined according to the detection result that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value in the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been added to the shopping cart;

[0164] The confidence level of the weight data recognition result of the product placed in the shopping cart is determined based on the corresponding weight value when the product is placed in the shopping cart.

[0165] In one embodiment, the weight data identification unit is further configured to:

[0166] If the weight value changes from a stable state to a decreasing state according to the detection result and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been taken out of the shopping cart;

[0167] The confidence level of the weight data recognition result of the product taken out of the shopping cart is determined based on the corresponding weight value when the product is taken out of the shopping cart.

[0168] In one embodiment, the weight data identification unit is further configured to:

[0169] If the variance value is greater than the preset variance threshold, it is determined that the weight data change in the preset time series interval is a type of drastic fluctuation, and the use of the weight data in the time series interval to judge the entry and exit of goods is abandoned, and the confidence level is determined to be 0.

[0170] In one embodiment, the video frame data recognition unit is specifically configured to input the video frame image data into a video understanding model pre-trained according to the following method to obtain a video frame image data recognition result and a confidence level of whether a product is placed in or taken out of a shopping cart:

[0171] The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model to enable the model to adaptively focus on key frames and key areas in video frame image data. The features of different layers and scales in the second half of the model are fused to obtain the improved Swin Transformer 3D model.

[0172] Pre-train the improved Swin Transformer 3D model using video data larger than the preset scale to obtain a pre-trained video understanding model;

[0173] The pre-trained video understanding model is fine-tuned and verified using historical video frame image data of labeled items being placed in or taken out of the shopping cart to obtain an optimized video understanding model.

[0174] In one embodiment, the video frame data identification unit is further configured to:

[0175] The historical video frame image data of marked items being placed in or taken out of the shopping cart is preprocessed by frame rate adjustment, resolution scaling, and normalization to obtain preprocessed video frame image data to adapt to the input requirements of the improved SwinTransformer 3D model;

[0176] The preprocessed video frame image data is enhanced by using random cropping, flipping, illumination transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, perspective changes and motion speeds, and obtain enhanced video frame image data.

[0177] In one embodiment, the shopping process monitoring device may further include: an establishment unit configured to establish a model for dynamically determining the weight influence of weight data and a model for dynamically determining the weight influence of video frame image data according to the following method:

[0178] Acquire multiple historical video frame image data;

[0179] Based on the plurality of historical video frame image data, the confidence level of the historical weight data recognition result when a product is placed in or taken out, as well as the confidence level of the historical video frame image data recognition result, and the confidence level of the historical weight data recognition result that is not zero when no product is placed in or taken out, as well as the confidence level of the historical video frame image data recognition result;

[0180] When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, checking whether a commodity insertion or removal action actually occurs at this time to obtain a verification result;

[0181] Determine the weight data detection accuracy and the video frame image data detection accuracy based on the inspection results;

[0182] According to the weight data detection accuracy and the video frame image data detection accuracy, a model for dynamically determining the weight of the weight data influence and a model for dynamically determining the weight of the video frame image data influence are established.

[0183] In one embodiment, the shopping process monitoring device may further include an adjustment unit configured to adjust the video frame image data recognition result or the confidence level of the video frame image data recognition result according to the following determination time based on the weight data recognition result and the video frame image data recognition result:

[0184] After one of the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds;

[0185] If the waiting time exceeds the preset number of seconds and no other recognition result is received, the influence weight value corresponding to the other recognition result is adjusted to 0;

[0186] If the waiting time is less than or equal to the preset number of seconds, another recognition result has been received, and the influence weight corresponding to the other recognition result is 1.

[0187] In one embodiment, the model for dynamically determining the weight of weight data influence is:

[0188]

[0189] The model for dynamically determining the influence weight of video frame image data is:

[0190] β=1-α;

[0191] Among them, α is the weight data influence weight, β is the video frame image data influence weight, Acc aAcc is the weight data detection accuracy determined based on historical data. b is the detection accuracy of the video frame image data determined based on historical data, Conf_a is the confidence of the current weight data recognition result, and Conf_b is the confidence of the current video frame image data recognition result.

[0192] In one embodiment, the final detection unit is specifically configured to determine the final result of whether the product is placed in or taken out of the shopping cart according to the following fusion model:

[0193] Conf=λ(α×Conf_a+β×Conf_b);

[0194] Among them, Conf is the final result, α is the influence weight of weight data, β is the influence weight of video frame image data, Conf_a is the confidence of the current weight data recognition result, Conf_b is the confidence of the current video frame image data recognition result, λ is the degree of confidence influence, λ=1 / n, n is the difference between the time when weight data recognition result is determined and the time when video frame image data is determined.

[0195] Based on the above invention concept, Figure 5 As shown, the present invention also proposes a computer device 500, including a memory 510, a processor 520 and a computer program 530 stored in the memory 510 and executable on the processor 520, wherein the processor 520 implements the aforementioned shopping process monitoring method when executing the computer program 530.

[0196] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned shopping process monitoring method.

[0197] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned shopping process monitoring method.

[0198] In an embodiment of the present invention, a shopping process monitoring solution comprises: obtaining accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor; determining a shopping cart movement state based on the accelerometer data and gyroscope data; determining a weight data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the shopping cart movement state and changes in weight data; determining a video frame image data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the video frame image data; and integrating a final result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, a dynamically determined weight data influence weight, and a video frame image data influence weight. This allows for accurate monitoring of the shopping process and improves the accuracy of monitoring the shopping process.

[0199] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0203] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for monitoring a shopping process, characterized in that: include: Acquire accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor; determining a moving state of the shopping cart based on the accelerometer data and the gyroscope data; Determining, based on the travel status of the shopping cart and the change in the weight data, a weight data recognition result and a confidence level of whether the product is placed in or taken out of the shopping cart; Determining, based on the video frame image data, a video frame image data recognition result indicating whether a product is placed in or taken out of a shopping cart and its confidence level; Based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight, the final result of whether the product is placed in or taken out of the shopping cart and its confidence level are determined by fusion.

2. The method according to claim 1, wherein Determining the travel state of the shopping cart based on the accelerometer data and the gyroscope data, including: Determining the modulus of acceleration on three axes in the accelerometer data; According to the gyroscope data, determine the sum of the absolute values ​​of the angular velocity of each axis; The shopping cart is determined to be in a stationary state when the modulus of acceleration is less than or equal to a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is less than or equal to a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is less than or equal to a preset angular velocity absolute value change threshold.

3. The method according to claim 2, wherein Determining the moving state of the shopping cart based on the accelerometer data and the gyroscope data further includes: The shopping cart is determined to be in motion when the modulus of acceleration is greater than a preset modulus threshold, the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value threshold, and the change in the modulus of acceleration within a preset time period is greater than a preset modulus change threshold, and the change in the sum of the absolute values ​​of the angular velocities is greater than a preset angular velocity absolute value change threshold.

4. The method according to claim 1, wherein Determining the weight data recognition result and its confidence level of whether an item is placed in or taken out of the shopping cart based on the movement status of the shopping cart and the change in the weight data includes: When it is detected that the moving state of the shopping cart is stationary, the change of the weight data within the preset time interval is detected; If it is determined according to the test results that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being placed in the shopping cart are obtained.

5. The method according to claim 4, wherein Determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data, further comprising: If it is determined according to the detection results that the weight value changes from a stable state to a decreasing state and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value maintained stable is greater than the preset weight threshold, the weight data recognition result and confidence level of the product being taken out of the shopping cart are obtained.

6. The method according to claim 1, wherein Determining the weight data recognition result and its confidence level of whether an item is placed in or taken out of the shopping cart based on the movement status of the shopping cart and the change in the weight data includes: When it is detected that the shopping cart is in a moving state, the weight data change within a preset time interval is detected; Determine a variance value of the weight data within a preset time series interval, and if the variance value is less than or equal to a preset variance threshold, determine that a change in the weight data within the preset time series interval does not belong to a type of drastic fluctuation; When it is determined that the weight data change within the preset time interval does not belong to the type of drastic fluctuation, if it is determined according to the detection result that the weight value changes from a stable state to an increasing state and remains stable after a preset period of time, and the weight difference between the initial weight value in the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been added to the shopping cart; The confidence level of the weight data recognition result of the product placed in the shopping cart is determined based on the corresponding weight value when the product is placed in the shopping cart.

7. The method according to claim 6, wherein Determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data, further comprising: If the weight value changes from a stable state to a decreasing state according to the detection result and remains stable after a preset period of time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, a weight data recognition result is obtained indicating that the product has been taken out of the shopping cart; The confidence level of the weight data recognition result of the product taken out of the shopping cart is determined based on the corresponding weight value when the product is taken out of the shopping cart.

8. The method according to claim 6, wherein Determining the weight data recognition result and its confidence level of whether the product is placed in or taken out of the shopping cart based on the travel status of the shopping cart and the change in the weight data, further comprising: If the variance value is greater than the preset variance threshold, it is determined that the weight data change in the preset time series interval is a type of drastic fluctuation, and the use of the weight data in the time series interval to judge the entry and exit of goods is abandoned, and the confidence level is determined to be 0.

9. The method according to claim 6, wherein Determining the confidence level of the weight data recognition result of the product taken out of the shopping cart based on the corresponding weight value of the product when it is determined that the product is taken out of the shopping cart includes: The weight value corresponding to when the product is taken out of the shopping cart is input into the confidence recognition model to obtain the confidence of the weight data recognition result of the product taken out of the shopping cart. The confidence value range is within the interval [0,1]. The confidence recognition model is pre-trained and generated based on sample data of the relationship between historical weight values ​​and confidence levels.

10. The method according to claim 1, wherein Determining, based on the video frame image data, a recognition result of the video frame image data indicating that a product is placed in or taken out of a shopping cart and its confidence level, includes: inputting the video frame image data into a video understanding model pre-trained according to the following method to obtain a recognition result of the video frame image data indicating that a product is placed in or taken out of a shopping cart and its confidence level: The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model to enable the model to adaptively focus on key frames and key areas in video frame image data. The features of different layers and scales in the second half of the model are fused to obtain the improved Swin Transformer 3D model. Pre-train the improved Swin Transformer 3D model using video data larger than the preset scale to obtain a pre-trained video understanding model; The pre-trained video understanding model is fine-tuned and verified using historical video frame image data of labeled items being placed in or taken out of the shopping cart to obtain an optimized video understanding model.

11. The method according to claim 10, wherein Also includes: The historical video frame image data of marked items being placed in or taken out of the shopping cart is preprocessed by frame rate adjustment, resolution scaling, and normalization to obtain preprocessed video frame image data to adapt to the input requirements of the improved Swin Transformer 3D model; The preprocessed video frame image data is enhanced by using random cropping, flipping, illumination transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, perspective changes and motion speeds, and obtain enhanced video frame image data.

12. The method according to claim 1, wherein The method also includes establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data according to the following method: Acquire multiple historical video frame image data; Based on the plurality of historical video frame image data, the confidence level of the historical weight data recognition result when a product is placed in or taken out, as well as the confidence level of the historical video frame image data recognition result, and the confidence level of the historical weight data recognition result that is not zero when no product is placed in or taken out, as well as the confidence level of the historical video frame image data recognition result; When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, checking whether a commodity insertion or removal action actually occurs at this time to obtain a verification result; Determine the weight data detection accuracy and the video frame image data detection accuracy based on the inspection results; According to the weight data detection accuracy and the video frame image data detection accuracy, a model for dynamically determining the weight of the weight data influence and a model for dynamically determining the weight of the video frame image data influence are established.

13. The method according to claim 1, wherein The method further includes adjusting the video frame image data recognition result or the confidence level of the video frame image data recognition result according to the determination time of the weight data recognition result and the video frame image data recognition result: After one of the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds; If the waiting time exceeds the preset number of seconds and no other recognition result is received, the influence weight value corresponding to the other recognition result is adjusted to 0; If the waiting time is less than or equal to the preset number of seconds, another recognition result has been received, and the influence weight corresponding to the other recognition result is 1.

14. The method according to claim 1, wherein The model for dynamically determining the impact weight of weight data is: The model for dynamically determining the influence weight of video frame image data is: β＝1-α； Among them, α is the weight data influence weight, β is the video frame image data influence weight, Acc a Acc is the weight data detection accuracy determined based on historical data. b is the detection accuracy of the video frame image data determined based on historical data, Conf_a is the confidence of the current weight data recognition result, and Conf_b is the confidence of the current video frame image data recognition result.

15. The method according to claim 1, wherein Determining the final result of whether the product is placed in or taken out of the shopping cart and its confidence level based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight, including: determining the final result of whether the product is placed in or taken out of the shopping cart according to the following fusion model: Conf=λ(α×Conf_a+β×Conf_b); Among them, Conf is the final result, α is the influence weight of weight data, β is the influence weight of video frame image data, Conf_a is the confidence of the current weight data recognition result, Conf_b is the confidence of the current video frame image data recognition result, λ is the degree of confidence influence, λ=1 / n, n is the difference between the time when weight data recognition result is determined and the time when video frame image data is determined.

16. A shopping process monitoring device, characterized in that: include: an acquisition unit, configured to acquire accelerometer data and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a visual sensor; a state determination unit, configured to determine a travel state of the shopping cart based on the accelerometer data and the gyroscope data; a weight data recognition unit, configured to determine a weight data recognition result and a confidence level of whether a product is placed in or taken out of the shopping cart based on the travel state of the shopping cart and changes in the weight data; a video frame data recognition unit, configured to determine, based on the video frame image data, a video frame image data recognition result indicating whether a product is placed in or taken out of a shopping cart and its confidence level; A final detection unit is used to determine the final result of whether the product is placed in or taken out of the shopping cart and its confidence level based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data impact weight, and the dynamically determined video frame image data impact weight.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 15 is implemented.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

19. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images