Intelligent shopping cart user behavior decision method and device integrating multi-type sensor information

Abstract

The present invention discloses a method and device for making user behavior decisions on a smart shopping cart by integrating multi-type sensor information. The method comprises: obtaining weight information, image information and motion trajectory characteristics of commodities in a time period from a start time t0 to an end time t1; obtaining the credibility of the weight perceived by the vehicle state by analyzing the data of each sensor in the event time period, further ensuring the credibility of the weight and the confidence level of the weight feedback to the image through the image, thereby obtaining a more accurate result and enabling a more accurate evaluation of the event.

Description

Technical Field

[0001] The present invention relates to a method and a device for making user behavior decisions on an intelligent shopping cart by integrating information from multiple types of sensors, and belongs to the field of intelligent shopping carts. Background Art

[0002] With the development of intelligent technology, smart shopping carts have become an important development direction of the retail industry. Traditional shopping carts are usually only used to place goods, while smart shopping carts help consumers complete the shopping process more efficiently through intelligent and automated methods, improve shopping efficiency, and optimize shopping experience.

[0003] There are three types of smart shopping carts on the market, one is a smart shopping cart based on weighing, one is a smart shopping cart based on image method, and the other is a smart shopping cart that combines the two methods. The smart shopping cart that includes weighing is the same as the existing basic scale. The reliability of the weight obtained requires the scale to be horizontal and still. The image sensor recognizes the items put in or taken out through image recognition, which has a certain recognition error and does not perceive the weight of the goods. The combination of the two methods is based on the first match of weight. If the match is unsuccessful, the image matching is used to pass. Otherwise, it will fail. It is a serial method.

[0004] In a pure weight smart shopping cart, in order to distinguish the weight difference between different types of goods, the weight sensor used in the smart shopping cart is usually of high precision and high sensitivity. In the process of shopping in a supermarket, the shopping behavior of users is random and complex, and the purchased goods are diverse. For example, the user's touching and holding the shopping box will cause the weight value to shift, and the large-sized goods leaning against the frame will also cause the actual weight to be less than the standard weight of the goods. Changes in the packaging of the goods will also cause the actual weight of the goods to be inconsistent with the weight of the goods marked in the database. In addition, there are many goods with different prices but the same quality in the supermarket. These situations often cause consumers to be unable to settle their purchases according to the normal operating procedures.

[0005] In a pure image smart shopping cart, the camera installed on the shopping cart body is used to accurately capture the user's shopping process, and the product target detection, tracking, comparison and recognition algorithm is used to identify the category and quantity of the products that the user puts in or takes out of the shopping cart. Due to the randomness of the user's shopping behavior, the product image imaging angle is uncertain. The occlusion of hands and products during the shopping process will cause the loss of product appearance information. The strong light characteristics of some product packaging will cause the product image to be blurred. These factors will reduce the accuracy of the product image detection and recognition algorithm. In addition, for products with highly similar packaging and bulk products, it is impossible to use product image information to accurately identify the product type.

[0006] At present, the user behavior decision-making method of the intelligent shopping cart that integrates image information and gravity information uses the gravity value of the gravity signal in a stable state as a decision basis, and then integrates the trajectory characteristics of the moving goods to finally make a decision on the user's shopping behavior. It belongs to a serial decision-making process. The reduction of the confidence of one kind of information will cause the confidence of the final decision result to be reduced. The movement, collision, handrail, and goods pressing the frame of the shopping cart will cause the gravity information to be inaccurate or unable to reach a stable state; fast purchase, goods being blocked during shopping, and missed detection of the goods detection algorithm will cause the trajectory of the moving goods to be too short and the image information to be inaccurate. These situations will cause the confidence of the final decision result to be reduced. The reliability of the decision requires that both the image information and the gravity information have a high confidence, which reduces the universality of this serial decision algorithm. Summary of the invention

[0007] The present invention provides a method and device for making user behavior decisions on an intelligent shopping cart by integrating information from multiple types of sensors, thereby solving the problems disclosed in the background technology.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] Smart shopping cart user behavior decision-making method integrating multi-type sensor information:

[0010] Determine the time period from the start time t0 to the end time t1 of the user's shopping event;

[0011] Obtain the weight information, image information and motion trajectory characteristics of the product in the time period from the start time t0 to the end time t1;

[0012] Calculate the confidence of the user's shopping event based on the weight information and image information of the product in the time period from the start time t0 to the end time t1;

[0013] If the confidence level of the user's shopping event is higher than a threshold, the corresponding shopping event type is matched according to the movement trajectory characteristics of the product and the corresponding weight information changes.

[0014] Furthermore, the method for determining the time period from the start time t0 to the end time t1 of the user shopping event includes:

[0015] If the user scans the code to purchase items, the scanning time is taken as the starting time t0; if the user does not scan the code, the image intrusion detection algorithm detects that an item has invaded the smart shopping cart, and the starting time t0 is a certain time before the invasion of the smart shopping cart. If there is no intrusion information, the weight change is detected and stabilizes after the change, the starting time t0 is a certain time before the weight change before the stable weight, and the ending time t1 is a certain time after the weight stabilizes.

[0016] Furthermore, it also includes obtaining angle information and acceleration information of the smart shopping cart;

[0017] Determine the state of the smart shopping cart according to the angle information and acceleration information of the smart shopping cart;

[0018] If the smart shopping cart is stationary and level, the weight information is highly reliable;

[0019] If the smart shopping cart is stationary on a ramp and weight data needs to be compensated, the weight information is highly reliable;

[0020] If there are multiple consecutive stable weight values ​​during the horizontal movement of the vehicle, and the weight value fluctuation is less than the set threshold, the weight information of the average weight value is more reliable;

[0021] If the weight value fluctuation is greater than the set threshold, the weight information is not reliable.

[0022] Furthermore, the method for calculating the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1 is:

[0023] According to the normalized formula calculating the confidence level;

[0024] λ w W is the credibility weight of the weight information; conf is the weight confidence; i is the credibility weight of the image information; I conf is the image confidence.

[0025] Furthermore, the weight of the credibility of the weight information λ w , weight confidence W conf , image information credibility weight λ i and image confidence I conf The method to obtain is:

[0026] The image intrusion detection algorithm detects that an item has invaded the smart shopping cart;

[0027] Obtaining the object's motion trajectory feature information;

[0028] Compare the item image with the feature information in the item's motion trajectory with the preset product image database to obtain the topN and its confidence level, and search the database based on the barcode corresponding to the topN products to obtain the corresponding product weight and upper and lower limits;

[0029] Obtain the weight λ of the credibility of the weight information according to the product weight w ; The weight confidence W is obtained by the difference between the weight and the standard weight conf; Obtain the image information credibility weight λ according to the image comparison result i and image confidence I conf .

[0030] Furthermore, the shopping event types include:

[0031] Place items into the shopping cart, take items out of the shopping cart, and organize items in the shopping cart.

[0032] Furthermore, if the direction of the product's movement trajectory is: moving from top to bottom, from the left and right sides of the imaging plane to the longitudinal center line, or the trajectory state changes from moving to stationary, and any of the above three conditions is met, combined with the change in gravity information, if the gravity value increases significantly, then the shopping event type is determined to be the product being placed in the shopping cart;

[0033] If the direction of the product's movement trajectory is: from bottom to top, from the longitudinal center line of the imaging plane to the left or right sides, or the trajectory state changes from static to moving, and any of the above three conditions are met, combined with the change in gravity information, if the gravity value decreases significantly, the shopping event type is determined to be the product being taken out of the shopping cart;

[0034] If the product's movement trajectory has no obvious direction, and the track state has no obvious regularity, and the difference in gravity value from the start time t0 to the end time t1 is within the error range, then the shopping event type is determined to be sorting the products in the shopping cart.

[0035] Accordingly, the present invention also provides an intelligent shopping cart user behavior decision-making device integrating multi-type sensor information, comprising:

[0036] The time period interception module is used to determine the time period from the start time t0 to the end time t1 of the user's shopping event;

[0037] A gravity sensor is used to obtain the weight information of the goods in the time period from the start time t0 to the end time t1, and a camera is used to obtain the image information of the goods in the time period from the start time t0 to the end time t1;

[0038] The movement trajectory module is used to obtain the movement trajectory of the product in the time period from the start time t0 to the end time t1;

[0039] A confidence calculation module, used to calculate the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1;

[0040] The shopping event type matching module is used to match the corresponding shopping event type according to the movement trajectory characteristics of the product and the corresponding weight information changes when the confidence level of the user's shopping event is higher than a threshold.

[0041] Furthermore, it also includes a code scanning module and an image intrusion detection module. The code scanning module is used to scan the code to purchase items, and the time period interception module uses the code scanning time as the starting time t0; if the user has no code scanning event, the image intrusion detection module detects that an item has invaded the smart shopping cart, and the starting time t0 is a certain time before the invasion of the smart shopping cart. If there is no intrusion information, the gravity sensor detects a weight change, and it is stable after the change, and the starting time t0 is a certain time before the weight change before the stable weight, and the end time t1 is a certain time after the weight stabilizes.

[0042] Furthermore, it also includes a weight information credibility assessment module, wherein the weight information credibility assessment module is provided with a gyroscope magnetometer and an acceleration sensor;

[0043] The gyro magnetometer is used to obtain the angle information of the smart shopping cart in the time period from the start time t0 to the end time t1;

[0044] The acceleration sensor is used to obtain acceleration information of the smart shopping cart during a time period from a start time t0 to an end time t1;

[0045] The weight information credibility evaluation module is used to determine the state of the smart shopping cart according to the angle information and acceleration information of the smart shopping cart;

[0046] If the smart shopping cart is stationary and level, the weight information is highly reliable;

[0047] If the smart shopping cart is stationary on a ramp and weight data needs to be compensated, the weight information is highly reliable;

[0048] If there are multiple consecutive stable weight values ​​during the horizontal movement of the vehicle, and the weight value fluctuation is less than the set threshold, the weight information of the average weight value is more reliable;

[0049] If the weight value fluctuation is greater than the set threshold, the weight information is not reliable.

[0050] The beneficial effects achieved by the present invention are:

[0051] By analyzing the data from each sensor during the event period, we can obtain the credibility of the vehicle's state-perceived weight. Through images, we can further ensure the credibility of the weight and the confidence level of the weight feedback to the image, thereby obtaining more accurate results and enabling a more accurate assessment of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the overall process of the intelligent shopping cart user behavior decision method integrating multi-type sensor information of the present invention;

[0053] Figure 2A schematic diagram of a weighted decision process of a user behavior decision method for an intelligent shopping cart that integrates information from multiple types of sensors in the present invention;

[0054] Figure 3 This is a schematic diagram of the neural network decision process of the intelligent shopping cart user behavior decision method that integrates multi-type sensor information in the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0056] like Figure 1 As shown, the present invention provides a user behavior decision method for a smart shopping cart that integrates multi-type sensor information. By collecting multi-sensor data segments within the time period when a single shopping event occurs when the smart shopping cart is used, the sensors include weight sensors, image sensors, and motion sensing sensors such as gyroscopes, magnetometers, and accelerometers. According to the reliability of weight information and image information, corresponding weight information is allocated, and the judgment result and corresponding confidence level of the smart shopping cart on the event are calculated in combination with a custom formula or a neural network model.

[0057] The present invention adopts a formulaic intelligent comprehensive decision-making method, which includes the following steps:

[0058] 1. Smart shopping app is turned on;

[0059] 2. Normal business logic execution, real-time collection of sensor data within the time period of a shopping event, including motion sensing sensors, weight sensors, and image sensors;

[0060] 3. If the user scans the code to purchase items, the sensor data with this as the starting time t0 is used for intelligent analysis and decision-making; if the code is not scanned, the image intrusion detection algorithm detects that an object has invaded the smart shopping cart, and the time before is used as the starting time t0 of the decision algorithm data. If there is no intrusion information, the weight sensor detects a weight change, and it is stable after the change. The time before the weight change before the stable weight is used as the starting time t0 of the decision algorithm data, and the time after the weight is stable is used as the end time t1, and the sensor data of the time period t0-t1 is collected; if there is no time period, no intelligent decision is made;

[0061] 4. If Figure 2 As shown, intelligent decision-making adopts weighted decision-making, analyzes the data of each sensor, calculates the weight of the stable change value of the weight in the current event and the weight of the image result, and uses the normalized formula Calculate the confidence result of the current event and the event type obtained through data analysis. Here λ w: Weight information credibility weight; W conf : weight confidence; λ i : Image information credibility weight; I conf : Image confidence. The specific analysis is as follows:

[0062] 1) Obtain motion sensor data within a time period, including acceleration, attitude angle information, weight data, and video stream. If the user scans the code, query the database to obtain the product's weight, upper and lower limits, and standard image samples of the product;

[0063] 2) Obtain the posture and movement of the vehicle at each moment in the event through gyroscopes, magnetometers and accelerometers to evaluate the reliability of weight data;

[0064] 3) Analyze the weight data and divide the weight data into stable section, variable section, stable section, etc. Then analyze the vehicle's motion state and attitude angle in each period of time to obtain the reliability of the weight data in that period of time. For example, in the stable section, the vehicle is stationary and horizontal, so the stable weight value is highly reliable. If the vehicle is stationary on a slope, the weight data needs to be compensated, and the stable weight value is also highly reliable. If there are multiple continuous stable values ​​and the fluctuation is relatively small during the horizontal movement of the vehicle, the average weight value is more reliable. If the weight value fluctuates greatly, the weight value is basically unreliable.

[0065] 4) Image algorithm analyzes the video stream frame by frame:

[0066] (1) Intrusion detected

[0067] (2) Is there any frame interference, such as a pressing frame? If there is a hand-held frame, the weight during this time period is completely unreliable. If the object is pressing the frame throughout the process and the vehicle is stationary, the stable weight change difference during this time period is credible;

[0068] (3) Analyze the tracked moving object trajectory and obtain trajectory characteristics;

[0069] (4) Compare the product images in the moving object trajectory with the product image database to obtain the topN and their confidence levels, and search the database based on the barcodes corresponding to the topN products to obtain the corresponding product weights and upper and lower limits;

[0070] 5) If there is weight information corresponding to the scan, the weight λ is obtained according to the credibility of the weight w , and then the weight is obtained by the difference between the weight and the standard weight to obtain the weight confidence W conf ; Then obtain the confidence and weight of the image according to the image comparison results;

[0071] 6) If there is no weight information corresponding to the scan, then according to step (4) in step 4), obtain the topN product list, calculate the weight confidence one by one, and adjust the topN confidence weight according to the weight confidence, that is, the more consistent the weight is, the greater the weight is. The topN recalculates the image weight and sorts it, obtains the topN confidence result, and returns it to the business layer;

[0072] 7) If there is a barcode scanning event, the product image in the moving object trajectory is compared with the image sample of the barcode in the product image database for similarity, and the similarity is used as the confidence I of the image information conf ;

[0073] 8) If there is no code scanning event, the product image in the moving object trajectory is compared with all the image samples under the barcode in the product image database for similarity, and the maximum similarity value is selected as the confidence I of the image information conf ;

[0074] 9) Determine the image information weight λ i It is necessary to comprehensively consider factors such as the trajectory length, the number of image samples in the trajectory, the direction of the trajectory relative to the shopping cart frame, and the state contained in the trajectory. If a trajectory is short in the video and contains fewer target samples, the weight λ is assigned. i A smaller value, otherwise λ i A larger value. If the direction of a trajectory relative to the shopping cart frame is single and clear, the weight λ is given. i A larger value, if the trajectory motion is complex, then the weight λ is given i A smaller value. If a trajectory contains two states, motion and stillness, there are two cases: ① The state changes from stillness to motion, which means that the algorithm has captured a complete process of taking the goods out of the shopping cart; if the state changes from stillness to motion, ② The state changes from motion to stillness, which means that the algorithm has captured a complete process of putting the goods into the shopping cart; in this case, λ is assigned i A larger value;

[0075] 10) Determination of shopping event types. Shopping events are divided into three categories: (I) placing items into the shopping cart; (II) taking items out of the shopping cart; and (III) sorting items in the shopping cart.

[0076] (1) If the direction of the product's motion trajectory is (a) from top to bottom, (b) from the left and right sides of the imaging plane to the longitudinal center line; (c) the trajectory state changes from motion to stillness, and any of the above three conditions is met, combined with the change in the weight of the gravity scale, if the weight increases significantly, the event is determined to be type (I) product placement in the shopping cart.

[0077] (2) If the direction of the product's trajectory is (a) from bottom to top, (b) moving from the longitudinal center line of the imaging plane to the left or right, or (c) the trajectory state changes from static to moving, and any of the above three conditions is met, combined with the change in the weight of the gravity scale, if the weight decreases significantly, the event is determined to be type (II) product removal from the shopping cart.

[0078] (3) If the product's trajectory has no obvious direction and the track state has no obvious regularity, and the difference between the gravity value at the event occurrence time t0 and the event end time t1 is within the error range, the event type is determined to be (III) sorting the items in the shopping cart.

[0079] like Figure 3 As shown, as an alternative to the above weighted decision, a neural network algorithm is used for decision making, which is specifically implemented as follows:

[0080] Train the model:

[0081] 1) Extract the data of each sensor within the event time period;

[0082] 2) Calculate the data features of each sensor, such as combined acceleration, gyroscope data, angle information, etc., as well as the trajectory features of the moving goods in the video, the border interference features, and the product comparison features, and annotate the event;

[0083] 3) Use neural network algorithm to train and obtain the model;

[0084] Compute decisions in real time:

[0085] 1) Load the model

[0086] 2) Obtain the data of each sensor in the time period t0-t1;

[0087] 3) Calculate each feature according to the method in step 5.2) and input it into the model to obtain the decision result.

[0088] The business layer updates the previous decision results based on the confidence level.

[0089] Based on the same concept of the above method, the present invention also provides an intelligent shopping cart user behavior decision-making device integrating multi-type sensor information, including:

[0090] The time period interception module is used to determine the time period from the start time t0 to the end time t1 of the user's shopping event;

[0091] A gravity sensor is used to obtain the weight information of the goods in the time period from the start time t0 to the end time t1, and a camera is used to obtain the image information of the goods in the time period from the start time t0 to the end time t1;

[0092] The movement trajectory module is used to obtain the movement trajectory of the product in the time period from the start time t0 to the end time t1;

[0093] A confidence calculation module, used to calculate the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1;

[0094] The shopping event type matching module is used to match the corresponding shopping event type according to the movement trajectory characteristics of the product and the corresponding weight information changes when the confidence level of the user's shopping event is higher than a threshold.

[0095] It also includes a code scanning module and an image intrusion detection module. The code scanning module is used to scan the code to purchase items. The time period interception module uses the code scanning time as the starting time t0. If the user does not have a code scanning event, the image intrusion detection module detects that an item has invaded the smart shopping cart, and the starting time t0 is a certain time before the invasion of the smart shopping cart. If there is no intrusion information, the gravity sensor detects a weight change, and it is stable after the change. The starting time t0 is a certain time before the weight change before the stable weight starts, and the end time t1 is a certain time after the weight stabilizes.

[0096] It also includes a weight information credibility assessment module, wherein the weight information credibility assessment module is provided with a gyroscope magnetometer and an acceleration sensor;

[0097] The gyro magnetometer is used to obtain the angle information of the smart shopping cart in the time period from the start time t0 to the end time t1;

[0098] The acceleration sensor is used to obtain acceleration information of the smart shopping cart during a time period from a start time t0 to an end time t1;

[0099] The weight information credibility evaluation module is used to determine the state of the smart shopping cart according to the angle information and acceleration information of the smart shopping cart;

[0100] If the smart shopping cart is stationary and level, the weight information is highly reliable;

[0101] If the smart shopping cart is stationary on a ramp and weight data needs to be compensated, the weight information is highly reliable;

[0102] If there are multiple consecutive stable weight values ​​during the horizontal movement of the vehicle, and the weight value fluctuation is less than the set threshold, the weight information of the average weight value is more reliable;

[0103] If the weight value fluctuation is greater than the set threshold, the weight information is not reliable.

[0104] The weight information credibility evaluation module, the confidence calculation module and the shopping event type matching module constitute the decision module of the device.

[0105] Example:

[0106] This embodiment provides a preferred solution for a smart shopping cart user behavior decision method that integrates multi-type sensor information. The specific steps are as follows:

[0107] Formulated intelligent decision making

[0108] 1. Open the smart shopping app;

[0109] 2. Normal business logic execution: the decision module collects data from various sensors in real time during the time period of a shopping event. The sensors include weight sensors, image sensors, and motion sensing sensors such as gyroscopes, magnetometers, and accelerometers.

[0110] 3. When a user scans a code to purchase an item, the scanning time is recorded as the starting time t0;

[0111] 4. If the code is not scanned, the image intrusion detection algorithm detects that an object has invaded the smart shopping cart, and the previous 1 second is used as the decision algorithm data starting point t0;

[0112] 5. If there is no intrusion information, the weight sensor detects a weight change, and the weight is stable after the change, and a certain time before the weight change before the stable weight begins, such as 3 seconds, is used as the decision algorithm data start time t0;

[0113] 6. A certain time after the weight is stable, such as 500ms, is the termination time (it can also be the moment when the weight is stable) t1;

[0114] 7. Collect the data of each sensor in the time period t0-t1; if there is no time period, no intelligent decision is made; Analyze the data of each sensor, calculate the weight of the stable change value of the weight in the current event and the weight of the image result, and use the normalized formula Calculate the confidence result of the current event and the event type obtained through data analysis. Here λw: weight information credibility weight; Wconf: weight confidence; λi: image information credibility weight; Iconf: image confidence. The specific analysis is as follows:

[0115] 1) Obtain motion sensor data within a time period, including acceleration, attitude angle information, weight data, and video stream. If the user scans the code, query the database to obtain the product's weight, upper and lower limits, and standard image samples of the product;

[0116] 2) The posture and movement of the vehicle at each moment in the event are obtained through motion sensors and attitude angle information to evaluate the reliability of weight data; Analyze weight data and divide the weight data into stable segment, variable segment, stable segment, etc. Then analyze the vehicle's motion state and attitude angle in each time period to obtain the reliability of the weight data in that time period. For example, in the stable segment, the vehicle is stationary and horizontal, so the stable weight value of 1000g is highly credible with a weight of 1. If the vehicle is stationary on a slope, the weight data needs to be compensated, and the stable weight value is highly credible with a weight of 0.9. If there are multiple continuous stable values ​​and relatively small fluctuations during the horizontal movement of the vehicle, the average weight value is more credible with a weight of 0.8. If the weight value fluctuates greatly, the weight value is basically unreliable with a weight of 0.2.

[0117] 3) Image algorithm analyzes the video stream frame by frame:

[0118] (1) Intrusion detected

[0119] (2) Whether there is any border interference, such as pressing the frame. If there is a hand-held frame, the weight during this time period is completely unreliable and the updated weight is 0; if the object is pressing the frame throughout the process and the vehicle is stationary, the stable weight change difference during this time period is credible and the weight remains unchanged;

[0120] (3) Analyze the trajectory of the tracked moving object and obtain trajectory characteristics, such as the spatial trend of change;

[0121] (4) Compare the product images in the moving object trajectory with the product image database to obtain the topN and their confidence, for example, N=10, and search the database according to the barcode corresponding to the top10 products to obtain the corresponding product weight and upper and lower limits; If there is weight information of the corresponding barcode scanned, obtain the weight λw according to the credibility of the weight, and then obtain the weight confidence Wconf by the difference between the weight and the weight, for example, the weight obtained by scanning the barcode is 995g, Wconf=1-(1000-995) / 995; then obtain the image confidence and weight according to the image comparison result; If there is no scan, then according to ④ in 4), obtain the list of topN products, calculate the weight confidence one by one, for example, top10, according to these 10 comparison results, obtain their corresponding barcode and the weight information of the product, calculate the weight confidence of the 10 products through the weight confidence calculation method similar to scanning, and then readjust the confidence weight of top10 according to the weight confidence, that is, the more consistent the weight is, the greater the weight is, the top10 recalculates the image weight and sorts it, obtains the confidence result of top10, and returns it to the business layer;

[0122] The detailed calculation of image weight is as follows: if there is a code scanning event, the product image in the moving object trajectory is compared with the image sample of the barcode in the product image database for similarity, and the similarity is used as the confidence Iconf of the image information; if there is no code scanning event, the product image in the moving object trajectory is compared with the image samples under all barcodes in the product image database for similarity, and the maximum similarity value is selected as the confidence Iconf of the image information; to determine the image information weight λi, it is necessary to comprehensively consider factors such as the trajectory length x0, the number of image samples in the trajectory x1, the direction of movement of the trajectory relative to the shopping cart frame x2, and the state x3 contained in the trajectory, for example, the weight λi=w0*x0+w1*x1+w2*x2+w3*x3 is used. If a trajectory is short in the video and contains fewer target samples, a smaller value is assigned to the weight λi, otherwise a larger value is assigned to λi. If the direction of movement of a trajectory is single and clear relative to the shopping cart frame, a larger value is assigned to the weight λi, and if the trajectory movement form is complex, a smaller value is assigned to the weight λi. If a trajectory contains two states, motion and stillness, there are two cases: ① If the state changes from stillness to motion, it means that the algorithm has captured a complete process of taking out the goods from the shopping cart; if the state changes from stillness to motion, ② If the state changes from motion to stillness, it means that the algorithm has captured a complete process of putting the goods into the shopping cart; in this case, a larger value is assigned to λi;

[0123] Shopping event type determination: Shopping events are divided into three categories: (I) placing goods into the shopping cart; (II) taking goods out of the shopping cart; (III) arranging goods in the shopping cart. If the direction of the goods' motion trajectory is (a) from top to bottom, (b) moving from the left and right sides of the imaging plane to the longitudinal center line; (c) the trajectory state changes from motion to stillness, and any one of the above three conditions is met, combined with the change in the weighing value of the gravity scale, if the weighing value has increased significantly, the event is determined to be type (I) goods placed in the shopping cart. If the direction of the goods' motion trajectory is (a) from bottom to top, (b) moving from the longitudinal center line of the imaging plane to the left and right sides; (c) the trajectory state changes from stillness to motion, and any one of the above three conditions is met, combined with the change in the weighing value of the gravity scale, if the weighing value has decreased significantly, the event is determined to be type (II) taking goods out of the shopping cart. If the goods' motion trajectory has no obvious direction, and the track state has no obvious regularity, and the difference in gravity value at the event occurrence time t0 and the event end time t1 is within the error range, the event type is determined to be (III) arranging goods in the shopping cart.

[0124] Similarly, neural network weighted decision-making is used to replace the above weighted decision-making. The specific steps are as follows:

[0125] AI Intelligent Decision-making

[0126] The neural network method calculates the features of each sensor in the event (t0~t1) through a large amount of data in advance, and then trains the model through a neural network, such as a GRU / LSTM model, to obtain a model. The required features are as follows:

[0127] 1. Motion sensing sensor: mean and variance of the combined acceleration, mean and variance of the pitch angle in the stable state within the event

[0128] 2. Weight sensor: stable change of scale value, that is, the weight change value in the shopping cart, the amplitude and duration of the scale value change when it is unstable

[0129] 3. Image sensor: trajectory features (trajectory length, movement direction, trajectory status, number of samples in the trajectory), comparison of topN results.

[0130] Then in actual use:

[0131] 1. Open the smart shopping app and load the model;

[0132] 2. Normal business logic execution: the decision module collects data from various sensors (including motion sensors, weight sensors, and image sensors) in real time during the time period when a shopping event occurs.

[0133] 3. When a user scans a code to purchase an item, the scanning time is recorded as the starting time t0;

[0134] 4. If the code is not scanned, the image intrusion detection algorithm detects that an object has invaded the smart shopping cart, and the previous 2 seconds is used as the decision algorithm data start time t0;

[0135] 5. If there is no intrusion information, the weight sensor detects a weight change, and the weight is stable after the change, and a certain time before the weight change before the stable weight begins, such as 5 seconds, is used as the decision algorithm data start time t0;

[0136] 6. The end time is a certain time after the weight stabilizes (it can also be the moment when the weight stabilizes) t1;

[0137] Collect the data of each sensor in the time period t0-t1; if there is no time period, no intelligent decision is made;

[0138] 7. Calculate the features according to the training method and input them into the model to obtain the model results, determine the results of the event and report them.

[0139] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0140] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to execute a smart shopping cart user behavior decision method that fuses multi-type sensor information.

[0141] A computing device includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing a smart shopping cart user behavior decision method that fuses multi-type sensor information.

[0142] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A smart shopping cart user behavior decision method integrating multi-type sensor information, characterized in that: include: Determine the time period from the start time t0 to the end time t1 of the user's shopping event; Obtain the weight information, image information and motion trajectory characteristics of the product in the time period from the start time t0 to the end time t1; Calculate the confidence of the user's shopping event based on the weight information and image information of the product in the time period from the start time t0 to the end time t1; If the confidence level of the user's shopping event is higher than the threshold, the corresponding shopping event type is matched according to the movement trajectory characteristics of the product and the corresponding weight information changes; The method for calculating the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1 is: According to the normalized formula calculating the confidence level; λ w is the weight of the credibility of the weight information; W conf is the weight confidence; λ i is the credibility weight of the image information; I conf is the image confidence; Weight Information Credibility Weight λ w , weight confidence W conf , Image information credibility weight λ i and image confidence I conf The methods for obtaining include: Obtain motion sensor data within a time period, including acceleration, attitude angle information, weight data, and video stream. If the user scans the code, query the database to obtain the product's weight, upper and lower limits, and standard image samples of the product. The posture and movement of the vehicle at each moment in the event are obtained through gyroscopes, magnetometers and accelerometers to evaluate the reliability of weight data; Analyze the weight data and divide it into stable section, variable section and stable section. Then analyze the vehicle's motion state and attitude angle in each section to obtain the reliability of the weight data in that section. In the stable section, the vehicle is stationary and horizontal, so the stable weight value is highly reliable. If the vehicle is stationary on a slope, the weight data needs to be compensated, and the stable weight value is highly reliable. If there are multiple continuous stable values ​​and the fluctuation is relatively small during the horizontal movement of the vehicle, the average weight value is more reliable. If the weight value fluctuates greatly, the weight value is unreliable. The image algorithm analyzes the video stream frame by frame: An intrusion event is detected; Is there any frame interference? If there is a handrail, the weight in this time period is completely unreliable. If the object is pressed against the frame and the vehicle is stationary, the stable weight change difference in this time period is credible. Analyze the tracked moving object trajectory and obtain trajectory characteristics; Compare the product images in the moving object trajectory with the product image database to obtain the topN and their confidence levels, and search the database based on the barcodes corresponding to the topN products to obtain the corresponding product weights and upper and lower limits; If there is weight information corresponding to the scan, the weight is obtained according to the credibility of the weight. λ w , and then the weight is obtained by comparing the weight with the standard weight. W conf ; Then obtain the confidence and weight of the image according to the image comparison results; If there is no weight information corresponding to the scan, obtain the topN product list, calculate the weight confidence one by one, and adjust the topN confidence weight according to the weight confidence. That is, the more consistent the weight is, the greater the weight is. TopN recalculates the image weight and sorts it, obtains the topN confidence result, and returns it to the business layer. If there is a barcode scanning event, the product image in the moving object trajectory is compared with the image sample of the barcode in the product image database for similarity, and the similarity is used as the confidence of the image information I conf ; If there is no barcode scanning event, the product image in the moving object trajectory is compared with the image samples under all barcodes in the product image database for similarity, and the maximum similarity value is selected as the confidence of the image information. I conf ; Determine image information weight λ i It is necessary to comprehensively consider the trajectory length, the number of image samples in the trajectory, the direction of movement of the trajectory relative to the shopping cart frame, and the state contained in the trajectory; if a trajectory is shorter in the video and contains fewer target samples, a weight is assigned. λ i A smaller value, otherwise λ i A larger value; if the direction of a trajectory is single and clear relative to the shopping cart frame, then the weight is given λ i A larger value, if the trajectory motion is complex, then give weight λ i A smaller value; if a trajectory contains two states, motion and stillness, there are two situations: if the state changes from stillness to motion, it means that the algorithm has captured a complete process of the goods being taken out of the shopping cart; if the state changes from stillness to motion, and from motion to stillness, it means that the algorithm has captured a complete process of the goods being put into the shopping cart; in this case, the value is given λ i A larger value.

2. The method for intelligent shopping cart user behavior decision-making based on integration of multi-type sensor information according to claim 1 is characterized by: The method for determining the time period from the start time t0 to the end time t1 of the user shopping event includes: If the user scans the code to purchase items, the scanning time is taken as the starting time t0; if the user does not scan the code, the image intrusion detection algorithm detects that an item has invaded the smart shopping cart, and the starting time t0 is a certain time before the invasion of the smart shopping cart. If there is no intrusion information, the weight change is detected and stabilizes after the change, the starting time t0 is a certain time before the weight change before the stable weight, and the ending time t1 is a certain time after the weight stabilizes.

3. The method for intelligent shopping cart user behavior decision-making based on integration of multi-type sensor information according to claim 1 is characterized by: It also includes obtaining angle information and acceleration information of the smart shopping cart; Determine the state of the smart shopping cart according to the angle information and acceleration information of the smart shopping cart; If the smart shopping cart is stationary and level, the weight information is highly reliable; If the smart shopping cart is stationary on a ramp and weight data needs to be compensated, the weight information is highly reliable; If there are multiple continuous stable weight values ​​during the horizontal movement of the vehicle, and the weight value fluctuation is less than the set threshold, the weight information of the average weight value is more reliable; If the weight value fluctuation is greater than the set threshold, the weight information is not reliable.

4. The method for intelligent shopping cart user behavior decision-making based on integration of multi-type sensor information according to claim 1 is characterized by: The shopping event types include: Place items into the shopping cart, take items out of the shopping cart, and organize items in the shopping cart.

5. The method for intelligent shopping cart user behavior decision-making based on integration of multi-type sensor information according to claim 4 is characterized by: If the direction of the product's movement trajectory is: moving from top to bottom, from the left and right sides of the imaging plane to the longitudinal center line, or the trajectory state changes from moving to stationary, and any of the above three conditions are met, combined with the change in gravity information, if the gravity value has increased significantly, then the shopping event type is determined to be the product being placed in the shopping cart; If the direction of the product's movement trajectory is: from bottom to top, from the longitudinal center line of the imaging plane to the left or right sides, or the trajectory state changes from static to moving, and any of the above three conditions are met, combined with the change in gravity information, if the gravity value decreases significantly, the shopping event type is determined to be the product being taken out of the shopping cart; If the product's movement trajectory has no obvious direction, and the track state has no obvious regularity, and the difference in gravity value from the start time t0 to the end time t1 is within the error range, then the shopping event type is determined to be sorting the products in the shopping cart.

6. An intelligent shopping cart user behavior decision-making device integrating multi-type sensor information, characterized in that: include: The time period interception module is used to determine the time period from the start time t0 to the end time t1 of the user's shopping event; A gravity sensor is used to obtain the weight information of the goods in the time period from the start time t0 to the end time t1, and a camera is used to obtain the image information of the goods in the time period from the start time t0 to the end time t1; The movement trajectory module is used to obtain the movement trajectory of the product in the time period from the start time t0 to the end time t1; A confidence calculation module, used to calculate the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1; A shopping event type matching module is used to match the corresponding shopping event type according to the movement trajectory characteristics of the product and the corresponding weight information change when the confidence of the user's shopping event is higher than a threshold; It also includes a weight information credibility assessment module, which is provided with a gyroscope magnetometer and an acceleration sensor; The gyro magnetometer is used to obtain the angle information of the smart shopping cart in the time period from the start time t0 to the end time t1; The acceleration sensor is used to obtain acceleration information of the smart shopping cart during a time period from a start time t0 to an end time t1; The weight information credibility evaluation module is used to determine the state of the smart shopping cart according to the angle information and acceleration information of the smart shopping cart; If the smart shopping cart is stationary and level, the weight information is highly reliable; If the smart shopping cart is stationary on a ramp and weight data needs to be compensated, the weight information is highly reliable; If there are multiple continuous stable weight values ​​during the horizontal movement of the vehicle, and the weight value fluctuation is less than the set threshold, the weight information of the average weight value is more reliable; If the weight value fluctuation is greater than the set threshold, the weight information is not reliable; The method for calculating the confidence of the user's shopping event based on the weight information and image information of the goods in the time period from the start time t0 to the end time t1 is: According to the normalized formula calculating the confidence level; λ w is the weight of the credibility of the weight information; W conf is the weight confidence; λ i is the credibility weight of the image information; I conf is the image confidence; Weight Information Credibility Weight λ w , weight confidence W conf , Image information credibility weight λ i and image confidence I conf The methods for obtaining include: Obtain motion sensor data within a time period, including acceleration, attitude angle information, weight data, and video stream. If the user scans the code, query the database to obtain the product's weight, upper and lower limits, and standard image samples of the product. The posture and movement of the vehicle at each moment in the event are obtained through gyroscopes, magnetometers and accelerometers to evaluate the reliability of weight data; Analyze the weight data and divide it into stable section, variable section and stable section. Then analyze the vehicle's motion state and attitude angle in each section to obtain the reliability of the weight data in that section. In the stable section, the vehicle is stationary and horizontal, so the stable weight value is highly reliable. If the vehicle is stationary on a slope, the weight data needs to be compensated, and the stable weight value is highly reliable. If there are multiple continuous stable values ​​and the fluctuation is relatively small during the horizontal movement of the vehicle, the average weight value is more reliable. If the weight value fluctuates greatly, the weight value is unreliable. The image algorithm analyzes the video stream frame by frame: An intrusion event is detected; Is there any frame interference? If there is a handrail, the weight in this time period is completely unreliable. If the object is pressed against the frame and the vehicle is stationary, the stable weight change difference in this time period is credible. Analyze the tracked moving object trajectory and obtain trajectory characteristics; Compare the product images in the moving object trajectory with the product image database to obtain the topN and their confidence levels, and search the database based on the barcodes corresponding to the topN products to obtain the corresponding product weights and upper and lower limits; If there is weight information corresponding to the scan, the weight is obtained according to the credibility of the weight. λ w , and then the weight is obtained by comparing the weight with the standard weight. W conf ; Then obtain the confidence and weight of the image according to the image comparison results; If there is no weight information corresponding to the scan, obtain the topN product list, calculate the weight confidence one by one, and adjust the topN confidence weight according to the weight confidence. That is, the more consistent the weight is, the greater the weight is. TopN recalculates the image weight and sorts it, obtains the topN confidence result, and returns it to the business layer. If there is a barcode scanning event, the product image in the moving object trajectory is compared with the image sample of the barcode in the product image database for similarity, and the similarity is used as the confidence of the image information I conf ; If there is no barcode scanning event, the product image in the moving object trajectory is compared with the image samples under all barcodes in the product image database for similarity, and the maximum similarity value is selected as the confidence of the image information. I conf ; Determine image information weight λ i It is necessary to comprehensively consider the trajectory length, the number of image samples in the trajectory, the direction of movement of the trajectory relative to the shopping cart frame, and the state contained in the trajectory; if a trajectory is shorter in the video and contains fewer target samples, a weight is assigned. λ i A smaller value, otherwise λ i A larger value; if the direction of a trajectory is single and clear relative to the shopping cart frame, then the weight is given λ i A larger value, if the trajectory motion is complex, then give weight λ i A smaller value; if a trajectory contains two states, motion and stillness, there are two situations: if the state changes from stillness to motion, it means that the algorithm has captured a complete process of the goods being taken out of the shopping cart; if the state changes from stillness to motion, and from motion to stillness, it means that the algorithm has captured a complete process of the goods being put into the shopping cart; in this case, the value is given λ i A larger value.

7. The intelligent shopping cart user behavior decision-making device integrating multi-type sensor information according to claim 6 is characterized by: It also includes a code scanning module and an image intrusion detection module. The code scanning module is used to scan the code to purchase items. The time period interception module uses the code scanning time as the starting time t0. If the user does not have a code scanning event, the image intrusion detection module detects that an item has invaded the smart shopping cart, and the starting time t0 is a certain time before the invasion of the smart shopping cart. If there is no intrusion information, the gravity sensor detects a weight change, and it is stable after the change. The starting time t0 is a certain time before the weight change before the stable weight starts, and the end time t1 is a certain time after the weight stabilizes.

Images