Method and apparatus for detecting status of product entering or exiting shopping cart, and shopping cart

Abstract

The present application discloses a method and apparatus for detecting the status of a product entering or exiting a shopping cart, and a shopping cart. The method comprises: acquiring sensed weight values at a plurality of moments within a current time period which are acquired by a weight sensor at a preset time interval; inputting the weight values at the plurality of moments within the current time period into a weight waveform category identifier to obtain a weight waveform category of the weight sensor in the current time period, the identifier being generated by pre-training using a data sample set of the relationships between sensed weight values at a plurality of historical moments and weight waveform categories; and, when the weight waveform category of the weight sensor in the current time period is a stable status category, on the basis of the difference between an average weight value at all moments corresponding to the current time period of the weight sensor and an average weight value at all moments corresponding to a previous stable status category period, detecting the status of a product entering or exiting a shopping cart.

Description

Method and device for detecting commodity in-out state of shopping cart and shopping cart

[0001] Related Applications

[0002] The present application claims priority to the Chinese Patent Application No. 202410841121.2 filed on June 26, 2024, and incorporates by reference the entire disclosure of the aforementioned patent application as part of the present application. TECHNICAL FIELD

[0003] The present application relates to the technical field of self-service shopping, and in particular to a method and device for detecting commodity in-out state of shopping cart and shopping cart. BACKGROUND

[0004] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior publication, square, or subject matter disclosed previously and addresses exists or is "prior art".

[0005] At present, in order to better meet the needs of customers, improve customer shopping experience and optimize business operation, the supermarket industry is also constantly transforming towards intelligence and digitization. Shopping settlement is an important part of intelligent and digital transformation. Intelligent shopping cart appears in the supermarket environment in this background, the main purpose is to provide self-service shopping options, simplify the steps necessary for traditional shopping and payment, so that customers do not need to interact with cashiers at checkout counters, and improve the shopping experience. At the same time, for the business, it can reduce the configuration of cashiers to reduce operating costs.

[0006] Generally speaking, intelligent shopping cart has a series of advanced functions, including automatic scanning and settlement, navigation and commodity positioning, item identification and weighing, personalized recommendation and advertising, data analysis and inventory management, etc. Self-service code scanning and settlement is the core function of intelligent shopping cart, which is generally based on hardware such as code scanner, tablet computer device and visual camera to complete intelligent detection and identification of commodities or barcodes, update the shopping list on the tablet computer in real time, and finally the shopper checks the list for settlement.

[0007] In the shopping process, detection and identification of commodities are the most core part, and accurate detection of put-in or taken-out commodities is a basic requirement. Only good detection of put-in or taken-out commodities can effectively improve the efficiency of shopping settlement, and plays an important role in preventing abnormal shopping behavior. The existing method for detecting commodity in-out state of shopping cart has the problem of low precision. SUMMARY

[0008] The present application provides a method for detecting commodity in-out state of shopping cart to improve the precision of detecting commodity in-out state of shopping cart, which comprises:

[0009] acquire the sensed weight values at multiple time points in the current period collected by the weight sensor at preset time intervals;

[0010] input the sensed weight values at multiple time points in the current period into a weight waveform category identifier generated by pre-training, to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category identifier is generated by pre-training using a relationship data sample set between the sensed weight values at multiple historical time points and weight waveform categories;

[0011] when the weight waveform category of the current period of the weight sensor is a stable state category, detect the state of the goods entering or leaving the shopping cart according to a weight difference between the average weight value at all time points corresponding to the current period of the weight sensor and the average weight value at all time points corresponding to the previous stable state category period.

[0012] The embodiment of the application further provides a device for detecting the state of goods entering or leaving a shopping cart, to improve the accuracy of detecting the state of goods entering or leaving the shopping cart, which comprises:

[0013] an acquisition unit, configured to acquire the sensed weight values at multiple time points in the current period collected by the weight sensor at preset time intervals;

[0014] an identification unit, configured to input the sensed weight values at multiple time points in the current period into a weight waveform category identifier generated by pre-training, to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category identifier is generated by pre-training using a relationship data sample set between the sensed weight values at multiple historical time points and weight waveform categories;

[0015] a detection unit, configured to, when the weight waveform category of the current period of the weight sensor is a stable state category, detect the state of the goods entering or leaving the shopping cart according to a weight difference between the average weight value at all time points corresponding to the current period of the weight sensor and the average weight value at all time points corresponding to the previous stable state category period.

[0016] The embodiment of the application further provides a shopping cart, to improve the accuracy of detecting the state of goods entering or leaving the shopping cart, which comprises:

[0017] a weight sensor, configured to collect the sensed weight values at multiple time points in each period at preset time intervals, and send the sensed weight values at multiple time points in each period to the device for detecting the state of goods entering or leaving the shopping cart;

[0018] the device for detecting the state of goods entering or leaving the shopping cart as described above.

[0019] The embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for detecting the state of the goods entering or leaving the shopping cart when executing the computer program.

[0020] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method for detecting the state of the goods entering or leaving the shopping cart.

[0021] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the method for detecting the state of the goods entering or leaving the shopping cart.

[0022] In the embodiment of the present application, the method for detecting the state of the goods entering or leaving the shopping cart comprises the following steps: acquiring the sensing weight values of the weight sensor at multiple time points in a current period collected at a preset time interval; inputting the sensing weight values of the multiple time points in the current period into a weight waveform category identifier generated by pre-training to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category identifier is generated by pre-training using a relationship data sample set between the sensing weight values of the multiple historical time points and the weight waveform category; when the weight waveform category of the current period of the weight sensor is a stable state category, the state of the goods entering or leaving the shopping cart is detected according to the weight difference between the average weight values of all time points corresponding to the current period of the weight sensor and the average weight values of all time points corresponding to the last stable state category period, so that the accuracy of detecting the state of the goods entering or leaving the shopping cart can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort. In the drawings:

[0024] Fig. 1 is a flowchart of the method for detecting the state of the goods entering or leaving the shopping cart in the embodiment of the present application;

[0025] Fig. 2A and Fig. 2B are schematic diagrams of weight waveforms of multiple periods in the embodiment of the present application;

[0026] Fig. 3 is a schematic diagram of the principle of detecting the state of the goods entering or leaving the shopping cart in the embodiment of the present application;

[0027] Fig. 4 is a schematic diagram of the structure of the device for detecting the state of the goods entering or leaving the shopping cart in the embodiment of the present application;

[0028] Fig. 5 is a schematic diagram of the circuit structure of the shopping cart in an embodiment of the present application;

[0029] Fig. 6 is a schematic diagram of the circuit structure of the shopping cart in another embodiment of the present application;

[0030] Fig. 7 is a schematic diagram of the computer device structure in an embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present application and their descriptions are used to explain the present application but not to limit the present application.

[0032] Fig. 1 is a schematic diagram of the flow of the method for detecting the state of the goods entering and leaving the shopping cart in an embodiment of the present application. As shown in Fig. 1, the method includes the following steps:

[0033] Step 101: acquiring the sensing weight values at multiple time points in the current period collected by the weight sensor at a preset time interval;

[0034] Step 102: inputting the sensing weight values at multiple time points in the current period into the weight waveform category recognizer generated in advance to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category recognizer is generated in advance by using the relationship data sample set between the sensing weight values at multiple historical time points and the weight waveform category;

[0035] Step 103: when the weight waveform category of the current period of the weight sensor is the stable state category, detecting the state of the goods entering and leaving the shopping cart according to the weight difference between the average weight values at all time points corresponding to the current period of the weight sensor and the average weight values at all time points corresponding to the previous stable state category period.

[0036] In the embodiments of the present application, the method for detecting the state of the goods entering and leaving the shopping cart works as follows: acquiring the sensing weight values at multiple time points in the current period collected by the weight sensor at a preset time interval; inputting the sensing weight values at multiple time points in the current period into the weight waveform category recognizer generated in advance to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category recognizer is generated in advance by using the relationship data sample set between the sensing weight values at multiple historical time points and the weight waveform category; when the weight waveform category of the current period of the weight sensor is the stable state category, detecting and judging the state of the goods entering and leaving the shopping cart according to the weight difference between the average weight values at all time points corresponding to the current period of the weight sensor and the average weight values at all time points corresponding to the previous stable state category period, which can improve the accuracy of detecting and judging the state of the goods entering and leaving the shopping cart. The method for detecting the state of the goods entering and leaving the shopping cart is described in detail below.

[0037] The embodiment of the present application provides a method for detecting and judging the state of goods entering and leaving a shopping cart based on a weight sensor and a visual sensor. The method is used for calculating and processing the acquired weight sensor data and image data by using a method combining machine learning and deep learning, and accurately judging the state of goods being put in or taken out. Details are introduced below.

[0038] The method for detecting the state of goods entering and leaving a shopping cart provided by the embodiment of the present application can be used in the following components.

[0039] 1. A weight sensor is installed at the bottom of a shopping cart basket or the bottom of a shopping cart.

[0040] 2. At least one visual sensor (acquisition device, for example, a camera) is installed on the shopping cart, and the vision can cover the shopping cart basket (basket) area.

[0041] 3. A computing device (the method for detecting the state of goods entering and leaving a shopping cart provided by the embodiment of the present application is applied to the computing device, and the computing device is the device for detecting the state of goods entering and leaving a shopping cart provided by the embodiment of the present application).

[0042] The method for detecting the state of goods entering and leaving a shopping cart provided by the embodiment of the present application is as follows.

[0043] The embodiment of the present application samples, processes and judges the weight fluctuation change mode (weight waveform category) of the weight sensor, combines the visual information collected by the visual sensor, checks and determines the effective "steady" state, and then calculates the weight difference between two stable states to judge whether the goods are put in or taken out. The method for detecting the state of goods entering and leaving a shopping cart is described in detail below in combination with FIG. 2A, FIG. 2B and FIG. 3 (the rhombus judgment box in FIG. 3 corresponds to the downward arrow representing "yes" and the right arrow representing "no").

[0044] 1. Sampling weight value. The weight sensor samples and acquires the sensed gravity value at a fixed time interval (preset time interval, for example, 20 milliseconds), that is, samples and calculates the physical gravity value, for example, samples and calculates once every 20 milliseconds, and sends to the computing device (device for detecting and judging the state of goods entering and leaving a shopping cart).

[0045] 2. Kalman filter correction. To further eliminate errors, the computing device performs Kalman filter denoising correction on the data sent by the weight sensor; that is, in an embodiment, the above-mentioned method of detecting the state of the goods entering and leaving the shopping cart further comprises: performing Kalman filter denoising correction processing on the sensed weight value in each preset time interval period to obtain the sensed weight value in each preset time interval period after correction processing, and in subsequent steps, sampling the sensed weight value at multiple time points in the current period from the sensed weight value in each preset time interval period after correction processing to input the weight waveform category recognizer generated by pre-training to obtain the weight waveform category of the weight sensor in the current period, so that a more accurate weight waveform category can be obtained, and the accuracy of detecting the state of the goods entering and leaving the shopping cart is further improved.

[0046] 3. Sampling time sequence interval weight value. According to a preset fixed time interval value m, n weight values on the time sequence are sampled. Specifically, for example, if n is set to 5, for the current time t, a total of 5 corresponding weight values at time t, t-1, t-2, t-3, and t-4 can be sampled.

[0047] In specific implementation, n at this time can be set to a value that is more practical, such as a sampling frequency of 20 milliseconds, and n can be set to a value in the interval [25, 50]. Assuming that n = 50, the time sequence weight value in a 1000 millisecond interval is viewed. For example, the width of one box in FIGS. 2A and 2B (the interval is 1000 milliseconds, and 50 values are sampled and sent to the subsequent classifier, that is, the weight waveform category recognizer).

[0048] As described above, in an embodiment, the sensed weight value at multiple time points in the current period collected by the weight sensor at a preset time interval includes:

[0049] The sensed weight value in the current time interval period collected by the weight sensor at a preset time interval is obtained.

[0050] The sensed weight value at multiple time points in the current period is sampled from the sensed weight value in the current time interval period.

[0051] In specific implementation, in the above-mentioned first step, the number of sensed weight values collected by the weight sensor at a fixed time interval is large, and the embodiment of the present application samples the sensed weight value at multiple time points that can represent the characteristics of the current period to input the weight waveform category recognizer, which can improve the accuracy and efficiency of subsequent weight waveform category recognition.

[0052] 4. Determine the weight waveform type. After normalization, the n weight values sampled at the current time are input into the "weight change pattern" classifier (weight waveform category identifier) to obtain the current weight fluctuation category. In one embodiment, the inductive weight values at multiple time points in the current period are input into the pre-trained weight waveform category identifier to obtain the weight waveform category of the weight sensor in the current period. After normalization, the inductive weight values at multiple time points in the current period are input into the pre-trained weight waveform category identifier to obtain the weight waveform category of the weight sensor in the current period, which can obtain a more accurate weight waveform category, thereby improving the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0053] In a specific implementation, the normalization process here can use the "min-max" normalization method, that is, for each value of the n weight values in time sequence:

[0054] Finally, the normalized value is between 0 and 1.

[0055] The classifier (weight waveform category identifier) in the embodiments of the present application uses a machine learning algorithm model, including but not limited to decision tree, KNN (k-nearest neighbor classification), support vector machine, random forest, and recurrent neural network RNN (Recurrent Neural Network), etc. The embodiments of the present application preferably use the LSTM (Long Short-Term Memory) algorithm model, which is used to capture time sequence change information. In one embodiment, the weight waveform category identifier is a long short-term memory network LSTM identifier, which can obtain a more accurate weight waveform category and further improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0056] In a specific implementation, the categories of the classifier (weight waveform category identifier) in the embodiments of the present application can be divided into the following 7 categories, as shown in Table 1 below:

[0057] Table 1

[0058] In practice, the waveform changes shown in FIGS. 2A and 2B in the embodiments of the present application are only for illustration, and there should be a certain difference from the actual waveform of each type, so before deploying the classifier model on the shopping cart for use, the actual weight fluctuation data in the actual situation is collected according to the above-mentioned category setting, and after collecting a certain amount of weight fluctuation data, the above-mentioned classifier, i.e. the weight waveform category identifier, is trained using the weight fluctuation data. The input of the weight waveform category identifier is the inductive weight value at multiple time points in the current period, and the output of the identifier is the weight waveform category as shown in Table 1 above. As shown in FIGS. 2A and 2B, a plurality of periods (the width of each rectangular box corresponds to a period) can be included, each period corresponds to a weight waveform category, each period includes a plurality of time points, and each time point corresponds to a weight value.

[0059] In practice, both the "0-value smooth" and the "fluctuation smooth" in the embodiments of the present application belong to the "smooth state".

[0060] 5. As shown in FIG. 3, it is determined whether it is "0-value smooth". The weight change pattern category (weight waveform category) output by the classifier (weight waveform category identifier). If it is 0-value smooth, the weight difference between the weight value of the last "smooth state" before the time t and the weight value at this time (for example, the weight difference between the weight value corresponding to the dark green box (box 1) and the weight value corresponding to the same dark green box (box 1) or the light green box (box 2) corresponding to the real fluctuation state in FIGS. 2A and 2B) is calculated and reported. That is, in an embodiment, when the weight waveform category of the current period of the weight sensor is the smooth state category, the state of the goods entering and leaving the shopping cart is detected according to the weight difference between the average weight value of all time points corresponding to the current period of the weight sensor and the average weight value of all time points corresponding to the last smooth state category period, including (or also including): when the weight waveform category of the current period of the weight sensor is the 0-value smooth state category, determining the third weight difference between the average weight value of the last real fluctuation smooth state category or 0-value smooth state category period before the current period and the average weight value of the current period; and detecting the state of the goods entering and leaving the shopping cart according to the third weight difference.

[0061] 6. As shown in FIG. 3, it is determined whether it is "fluctuation smooth". The weight change pattern category output by the classifier. If it is determined that the state at the current time point is "fluctuation smooth", the next step 7 is performed.

[0062] 7. As shown in FIG. 3, a hand is detected. The n RGB images sampled at the current time are input into the detection model, i.e., the hand detection model, to determine whether a hand appears on the shopping cart basket area (e.g., the edge). The input of the hand detection model can be multiple shopping cart basket area images in each time period, and the output can be the detection result of whether a hand appears on the shopping cart basket area (e.g., the edge) or whether there is a pressing situation. In an embodiment, the hand detection model is a yolov5 target detection model trained using a deep learning method, which can improve the accuracy of detecting the hand result and thus improve the accuracy of detecting the state of the goods entering and exiting the shopping cart. Before deployment, a certain number of image datasets of hands pressing on the shopping cart basket edge or goods are used to label the hand region, and after labeling is completed, the target detection model is trained using the dataset.

[0063] 8. If the output result in step 7 is that there is no hand or pressing situation, it is determined that the "real fluctuation is stable", at this time, the average weight value in the "real fluctuation is stable" state is calculated, then the weight difference between the average weight value at this time and the last "0 value is stable" or "real fluctuation is stable" before the time t is calculated, and the state of the goods entering and exiting the shopping cart is reported and judged.

[0064] As described above, in an embodiment, when the weight waveform category of the weight sensor in the current time period is the stable state category, the state of the goods entering and exiting the shopping cart is detected according to the weight difference between the average weight value of all times corresponding to the current time period of the weight sensor and the average weight value of all times corresponding to the last stable state category period.

[0065] When the weight waveform category of the weight sensor in the current time period is not the 0 value stable state category, it is determined whether the weight fluctuation category of the weight sensor in the current time period is the fluctuation stable state category.

[0066] When the weight waveform category of the weight sensor in the current time period is the fluctuation stable state category, the multiple shopping cart basket area images collected by the collection device in the current time period are input into the hand detection model to detect whether a hand appears on the shopping cart basket area. The hand detection model is generated by pre-training using the image dataset of the hand pressing on the shopping cart basket area (e.g., the edge), and the collection device is arranged on the shopping cart in a position visually covering the shopping cart basket area.

[0067] When it is detected that a hand is present on the shopping cart basket area, and when it is detected that the hand is pressed down, the following operation of continuing secondary inspection and confirmation is performed:

[0068] In a specific implementation, the embodiments of the present application can accurately determine the state of the goods being put into or taken out of the shopping cart under the condition of using a weight sensor as the main and a visual camera as the auxiliary, and the accuracy is higher and more reliable.

[0069] 9. As shown in FIG. 3, secondary inspection. If the output result in step 7 is that the hand is pressed down, the secondary inspection and confirmation is continued. The confirmation method is as follows: checking whether the waveform of “artificially pressed up” or “abnormal fluctuation up” exists in the waveform category of the previous waveform category before the “fluctuation stable” state at this moment, and calculating the variance of n weight values in the time interval in the “fluctuation stable” state at this moment. If it is judged that the waveform is “artificially pressed up” or “abnormal fluctuation up”, and the weight fluctuation variance value is greater than a certain threshold value, it is proved that the “fluctuation stable” is caused by artificial pressing down, and it is judged as “abnormal fluctuation stable”, and the abnormality is reported. If it is judged that the waveform is not “artificially pressed up” or “abnormal fluctuation up”, it is judged as “real fluctuation stable”, at this moment, the average weight value in the “real fluctuation stable” state is calculated, then the weight difference between the average weight value at this moment and the average weight value of the last “0 value stable” or “real fluctuation stable” before the moment t (current moment) is calculated, and the state of the goods being put into or taken out of the shopping cart is determined.

[0070] As known from the above, in an embodiment, the method for detecting the state of the goods being put into or taken out of the shopping cart further includes: when it is detected that a hand is present on the shopping cart basket area, and when it is detected that the hand is pressed down, the following operation of continuing secondary inspection and confirmation is performed:

[0071] checking whether the waveform of “artificially pressed up” or “abnormal fluctuation up” exists in the waveform category of the previous waveform category before the “fluctuation stable” state at this moment;

[0072] when it is checked that the waveform of “artificially pressed up” or “abnormal fluctuation up” exists, determining the variance value of the inductive weight values at all moments in the current period in the “fluctuation stable” state of the current period;

[0073] when the variance value of the inductive weight values is greater than a preset variance threshold value, determining that the “fluctuation stable” state of the current period is caused by artificial pressing down, judging that the waveform category of the current period is the “abnormal fluctuation stable” state category, and issuing an abnormal alarm.

[0074] In the implementation, the embodiment of the present application proposes a method of using the combination of vision and weight fluctuation change mode (weight waveform category) for secondary inspection, which can improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0075] From the above, in one embodiment, the operation of the secondary inspection confirmation further includes:

[0076] When the weight waveform category of the artificial downward pressure and upward or abnormal fluctuation upward is not found in the inspection, it is determined that the current period weight waveform category is a real fluctuation smooth category, the current period average weight value in the real fluctuation smooth state is calculated, the second weight difference value between the average weight value of the period before the last real fluctuation smooth state category or the 0 value smooth state category and the current period average weight value is determined, and the state of goods entering and leaving the shopping cart is detected according to the second weight difference value.

[0077] In the implementation, the embodiment of the present application proposes a method of using the combination of vision and weight waveform category for secondary inspection, which can improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0078] 10. Determine the entry and exit of goods. According to the weight difference value between two “smooth states”, the entry and exit of goods are determined. That is, if the difference value is positive and the value is greater than a certain threshold (preset weight difference threshold), it is determined that the goods are put in; if the difference value is negative and greater than a certain threshold (preset weight difference threshold), it is determined that the goods are taken out. For example, after determining that the goods are put in, the put-in goods can be identified and the first goods identifier obtained by identification can be added to the settlement list for goods settlement; after determining that the goods are taken out, the second goods identifier of the taken-out goods can be determined and removed from the settlement list for goods settlement.

[0079] From the above, in one embodiment, according to the weight difference value between the average weight value of all time points corresponding to the current period of the weight sensor and the average weight value of all time points corresponding to the period of the last smooth state category, the state of goods entering and leaving the shopping cart is detected, including: if the weight difference value is positive and the weight difference value is greater than the preset weight difference threshold, it is detected that the goods are put into the shopping cart; if the weight difference value is negative and the weight difference value is greater than the preset weight difference threshold, it is detected that the goods are taken out of the shopping cart.

[0080] In the implementation, the embodiment of the present application proposes a method of using the combination of vision and weight waveform category for secondary inspection, which can improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0081] In summary, the method for detecting the state of goods entering and leaving the shopping cart provided by the embodiments of the present application has the following advantages:

[0082] 1) The embodiments of the present application propose a method of using an LSTM classifier, defining, sampling, labeling, and training the classifier model, and finally determining the pattern category of the time series weight fluctuation change, which can improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0083] 2) The embodiments of the present application propose a method of using visual and weight fluctuation change pattern (weight waveform category) combination secondary check, which can improve the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0084] 3) The embodiments of the present application perform denoising and normalization processing on the weight change value, which achieves more accurate judgment and avoids errors, and further improves the accuracy of detecting the state of goods entering and leaving the shopping cart.

[0085] 4) The embodiments of the present application can accurately determine the state of goods being placed into or taken out of the shopping cart under the condition of using a weight sensor as the main and a visual camera as the auxiliary, which is more accurate and reliable.

[0086] The embodiments of the present application also provide a device for detecting the state of goods entering and leaving the shopping cart, as described in the following embodiments. Since the principle of solving the problem of the device is similar to that of the method for detecting the state of goods entering and leaving the shopping cart, the implementation of the device can be referred to the implementation of the method for detecting the state of goods entering and leaving the shopping cart, and the repeated parts will not be described again.

[0087] FIG. 4 is a structural schematic diagram of the device for detecting the state of goods entering and leaving the shopping cart (i.e., the computing device mentioned above) in the embodiments of the present application, as shown in FIG. 4, the device comprises:

[0088] The acquisition unit 21 is configured to acquire the inductive weight values at multiple time points in the current period collected by the weight sensor at a preset time interval;

[0089] The recognition unit 22 is configured to input the inductive weight values at multiple time points in the current period into the weight waveform category recognizer generated by pre-training, to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category recognizer is generated by pre-training using a relationship data sample set between the inductive weight values at multiple historical time points and the weight waveform category;

[0090] The detection unit 23 is configured to, when the weight waveform category of the current period of the weight sensor is a steady state category, detect the state of goods entering and leaving the shopping cart according to the weight difference between the average weight values of all time points corresponding to the current period of the weight sensor and the average weight values of all time points corresponding to the last steady state category period.

[0091] In one embodiment, the detection unit is specifically used for:

[0092] When the weight waveform category of the current period of the weight sensor is not the 0-value stable state category, it is determined whether the weight fluctuation category of the current period of the weight sensor is the fluctuation stable state category.

[0093] When the weight waveform category of the current period of the weight sensor is the fluctuation stable state category, the current period of the plurality of shopping cart basket region images collected by the collection device is input into the hand detection model to detect whether a hand appears on the shopping cart basket region; the hand detection model is generated by pre-training using a hand image dataset in which a hand is pressed on a shopping cart basket region, and the collection device is arranged on the shopping cart in a position visually covering the shopping cart basket region.

[0094] When no hand is detected to appear on the shopping cart basket region, it is determined that the weight waveform category of the current period is a real fluctuation stable category, the average weight value of the current period in the real fluctuation stable state category is determined, and the first weight difference value between the average weight value of the current period and the average weight value of the period before the last real fluctuation stable state category or the 0-value stable state category is determined; according to the first weight difference value, the state of the goods entering and exiting the shopping cart is detected.

[0095] In one embodiment, the detection unit is further used for: when it is detected that a hand appears on the shopping cart basket region and it is detected that a hand is pressed, the following operations of continuing the secondary inspection and confirmation are performed:

[0096] Inspecting whether the last weight waveform category before the fluctuation stable state category of the current period exists a manually pressed upward or an abnormally fluctuated upward weight waveform category;

[0097] When it is inspected that the manually pressed upward or the abnormally fluctuated upward weight waveform category exists, the variance value of the inductive weight value at all times in the current period in the fluctuation stable state category of the current period is determined.

[0098] When the variance value of the inductive weight value is greater than a preset variance threshold value, it is determined that the fluctuation stable state of the current period is caused by manual pressing, it is determined that the weight waveform category of the current period is an abnormal fluctuation stable state category, and an abnormal alarm is issued.

[0099] In one embodiment, the secondary inspection and confirmation operation further includes:

[0100] When it is found that there is no weight waveform category of human down pressure or abnormal fluctuation up pressure, the weight waveform category of the current period is determined as a real fluctuation stable state category, a second weight difference value between the average weight value of the period before the last real fluctuation stable state category or the 0 value stable state category and the average weight value of the current period is determined, and the state of the commodity entering or leaving the shopping cart is detected according to the second weight difference value.

[0101] In one embodiment, the detection unit is further configured to:

[0102] When the weight waveform category of the current period of the weight sensor is the 0 value stable state category, a third weight difference value between the average weight value of the period before the last real fluctuation stable state category or the 0 value stable state category and the average weight value of the current period is determined, and the state of the commodity entering or leaving the shopping cart is detected according to the third weight difference value.

[0103] In one embodiment, the acquisition unit is specifically configured to:

[0104] The acquisition unit is configured to acquire the sensing weight value in the current time interval period collected by the weight sensor at a preset time interval;

[0105] The acquisition unit is configured to sample the sensing weight value at a plurality of time points in the current period from the sensing weight value in the current time interval period.

[0106] In one embodiment, the hand detection model is a yolov5 target detection model trained using a deep learning method.

[0107] In one embodiment, the device for detecting the state of the commodity entering or leaving the shopping cart further comprises a correction processing unit configured to perform Kalman filtering denoising correction processing on the sensing weight value at a plurality of time points in each period to obtain the sensing weight value at a plurality of time points in each period after correction processing.

[0108] In one embodiment, the recognition unit is specifically configured to input the sensing weight value at a plurality of time points in the current period into a pre-trained weight waveform category recognizer after normalization processing to obtain the weight waveform category of the current period of the weight sensor.

[0109] In one embodiment, the weight waveform category recognizer is a long short-term memory network (LSTM) recognizer.

[0110] In one embodiment, the detection unit is specifically configured to:

[0111] If the weight difference value is positive and greater than a preset weight difference value threshold, it is detected that the commodity is in a state of being put into the shopping cart; if the weight difference value is negative and greater than the preset weight difference value threshold, it is detected that the commodity is in a state of being taken out of the shopping cart.

[0112] The application also provides a shopping cart for detecting the state of goods entering and leaving the shopping cart, as described in the following embodiment. Since the shopping cart solves the problem by the same principle as the method for detecting the state of goods entering and leaving the shopping cart, the implementation of the shopping cart can refer to the implementation of the method for detecting the state of goods entering and leaving the shopping cart, and the repeated parts will not be described again.

[0113] FIG. 5 is a structural schematic diagram of a shopping cart circuit according to an embodiment of the application. As shown in FIG. 5, the shopping cart includes:

[0114] a weight sensor 1 configured to collect the sensed weight values at multiple time points in each time period at preset time intervals, and send the sensed weight values at the multiple time points in each time period to the device for detecting the state of goods entering and leaving the shopping cart.

[0115] a device 2 for detecting the state of goods entering and leaving the shopping cart.

[0116] As shown in FIG. 6, in one embodiment, the shopping cart described above can further include a collection device 3 arranged at a position on the shopping cart that visually covers the basket area of the shopping cart, and configured to collect multiple images of the basket area of the shopping cart in the current time period and input the hand detection model.

[0117] Based on the foregoing inventive concept, as shown in FIG. 7, the application further provides a computer device 500 including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520, wherein the processor 520 implements the method for detecting the state of goods entering and leaving the shopping cart when executing the computer program 530.

[0118] The application also provides a computer readable storage medium storing a computer program, wherein the computer program is executable by a processor to implement the method for detecting the state of goods entering and leaving the shopping cart.

[0119] The application also provides a computer program product including a computer program, wherein the computer program is executable by a processor to implement the method for detecting the state of goods entering and leaving the shopping cart.

[0120] In the embodiments of the present application, the scheme for detecting the state of the goods entering and leaving the shopping cart comprises the following steps: acquiring the sensing weight values of the weight sensor at multiple time points in a current period at a preset time interval; inputting the sensing weight values of the multiple time points in the current period into a weight waveform category identifier generated by pre-training, to obtain the weight waveform category of the current period of the weight sensor; the weight waveform category identifier is generated by pre-training using a relationship data sample set between the sensing weight values of the multiple historical time points and the weight waveform category; when the weight waveform category of the current period of the weight sensor is a stable state category, the state of the goods entering and leaving the shopping cart is detected according to the weight difference between the average weight values of all time points corresponding to the current period of the weight sensor and the average weight values of all time points corresponding to the last stable state category period, so that the accuracy of detecting the state of the goods entering and leaving the shopping cart can be improved.

[0121] The acquisition, storage, use, processing and the like of data in the technical scheme of the present application comply with relevant provisions of laws and regulations.

[0122] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0123] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0124] These computer program instructions can also be stored in a computer readable storage medium which can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product comprising instruction devices, which implement the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0125] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0126] The above specific embodiments are described for the purpose of further explaining the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting the status of goods entering and leaving a shopping cart, characterized in that, include: Acquire the sensed weight values ​​at multiple moments within the current time period, collected by the weight sensor at preset time intervals; Input the sensed weight values ​​from multiple moments within the current time period into a pre-trained weight waveform category recognizer to determine the weight waveform category of the weight sensor in the current time period. The weight waveform category recognizer is pre-trained using a data sample set showing the relationship between the sensed weight values ​​and weight waveform categories at multiple historical moments. as well as When the weight waveform of the weight sensor is in a stable state category during the current period, the status of the goods entering or leaving the shopping cart is detected based on the weight difference between the average weight value of all times corresponding to the current period and the average weight value of all times corresponding to the previous stable state category period.

2. The method as described in claim 1, characterized in that, When the weight waveform of the weight sensor in the current time period is in a stable state, the status of the goods entering or leaving the shopping cart is detected based on the weight difference between the average weight value of all times in the current time period and the average weight value of all times in the previous stable state period, including: When the weight waveform category of the weight sensor in the current time period is not a zero value and is in a stable state category, determine whether the weight fluctuation category of the weight sensor in the current time period is a fluctuation and is in a stable state category. When the weight waveform of the weight sensor in the current time period is classified as a fluctuating and stable state, multiple images of the shopping cart basket area acquired by the acquisition device in the current time period are input into a hand detection model to detect whether a hand appears on the shopping cart basket area. The hand detection model is pre-trained using an image dataset of hands pressing down on the shopping cart basket area, and the acquisition device is positioned on the shopping cart to visually cover the shopping cart basket area. When no hand is detected in the shopping cart basket area, the weight waveform category of the current time period is determined to be a true stable fluctuation category. The average weight value of the current time period under the true stable fluctuation state category is determined. The first weight difference between the average weight value of the previous true stable fluctuation state category or 0-value stable state category and the average weight value of the current time period is determined. Based on the first weight difference, the status of goods entering and leaving the shopping cart is detected.

3. The method as described in claim 2, characterized in that, Also includes: If a hand is detected in the shopping cart basket area, and if a hand is detected pressing down, the following secondary verification procedure is performed: Check whether there is a weight waveform category that is artificially suppressed or abnormally fluctuating upward before the current period's stable fluctuation state category. When a weight waveform category with artificial downward pressure or abnormal fluctuation is detected, the variance of the sensed weight value at all times within the current time period is determined under the category of stable fluctuation in the current time period. as well as When the variance of the sensed weight value is greater than the preset variance threshold, it is determined that the current period's stable fluctuation state is caused by human pressure, the weight waveform category of the current period is judged to be an abnormal stable fluctuation state category, and an abnormal alarm is issued.

4. The method as described in claim 3, characterized in that, The secondary verification and confirmation process also includes: When no artificially inflated or abnormally fluctuating weight waveform category is found, the weight waveform category of the current period is determined to be a true stable fluctuation state category. The second weight difference is determined between the average weight value of the previous true stable fluctuation state category or 0-value stable state category and the average weight value of the current period. Based on the second weight difference, the status of goods entering and leaving the shopping cart is detected.

5. The method as described in claim 1, characterized in that, When the weight waveform of the weight sensor in the current time period is in a stable state, the status of the goods entering or leaving the shopping cart is detected based on the weight difference between the average weight value of all times in the current time period and the average weight value of all times in the previous stable state period, including: When the weight waveform category of the weight sensor in the current time period is a 0-value stable state category, a third weight difference is determined between the average weight value of the previous real fluctuation stable state category or 0-value stable state category and the average weight value of the current time period; based on the third weight difference, the status of goods entering and leaving the shopping cart is detected.

6. The method as described in claim 2, characterized in that, The hand detection model is a YOLOv5 object detection model trained using deep learning methods.

7. The method as described in claim 1, characterized in that, Acquire the sensed weight values ​​from the weight sensor at preset time intervals within the current time period, including: Acquire the sensed weight value within the current time interval period collected by the weight sensor at preset time intervals; and Sample the sensed weight values ​​at multiple moments within the current time interval from the sensed weight values ​​within the current time interval.

8. The method as described in claim 1, characterized in that, Also includes: Kalman filtering is applied to the sensible weight values ​​at multiple times within each time period to denoise and correct them, resulting in the corrected sensible weight values ​​at multiple times within each time period.

9. The method as described in claim 1, characterized in that, The sensor inputs the sensed weight values ​​from multiple moments within the current time period into a pre-trained weight waveform category recognizer to determine the weight waveform category of the weight sensor for the current time period. This process includes: normalizing the sensed weight values ​​from multiple moments within the current time period and then inputting them into the pre-trained weight waveform category recognizer to determine the weight waveform category of the weight sensor for the current time period.

10. The method as described in claim 1, characterized in that, The weight waveform category recognizer is a Long Short Time Memory (LSTM) network recognizer.

11. The method as described in claim 1, characterized in that, The status of items entering or leaving the shopping cart is detected based on the weight difference between the average weight value of all times in the current time period and the average weight value of all times in the previous stable state category time period, including: If the weight difference is positive and greater than the preset weight difference threshold, the item is detected as having been added to the shopping cart; if the weight difference is negative and greater than the preset weight difference threshold, the item is detected as having been removed from the shopping cart.

12. A device for detecting the status of goods entering and leaving a shopping cart, characterized in that, include: The acquisition unit is used to acquire the sensed weight values ​​at multiple moments within the current time period collected by the weight sensor at preset time intervals; The identification unit is used to input the sensed weight values ​​at multiple moments within the current time period into a pre-trained weight waveform category recognizer to determine the weight waveform category of the weight sensor in the current time period. The weight waveform category recognizer is generated by pre-training using a data sample set of relationships between sensed weight values ​​and weight waveform categories at multiple historical moments. as well as The detection unit is used to detect the status of goods entering or leaving the shopping cart when the weight waveform category of the weight sensor in the current period is a stable state category, based on the weight difference between the average weight value of all times in the current period and the average weight value of all times in the previous stable state category period.

13. The apparatus as claimed in claim 12, characterized in that, The detection unit is specifically used for: When the weight waveform category of the weight sensor in the current time period is not a zero value and is in a stable state category, determine whether the weight fluctuation category of the weight sensor in the current time period is a fluctuation and is in a stable state category. When the weight waveform of the weight sensor in the current time period is in the fluctuating and stable state category, multiple images of the shopping cart basket area collected by the acquisition device in the current time period are input into the hand detection model to detect whether a hand appears on the shopping cart basket area; the hand detection model is pre-trained using an image dataset of hands pressing down on the shopping cart basket area, and the acquisition device is set on the shopping cart at a position that visually covers the shopping cart basket area; as well as When no hand is detected in the shopping cart basket area, the weight waveform category of the current time period is determined to be a true fluctuation and stability category. The average weight value of the goods in the current time period under the true fluctuation and stability category is determined. The first weight difference between the average weight value of the previous true fluctuation and stability category or 0-value stability category and the average weight value of the current time period is determined. Based on the first weight difference, the status of the goods entering or leaving the shopping cart is detected.

14. A shopping cart, characterized in that, include: A weight sensor is used to collect the sensed weight values ​​at multiple moments within each time period at preset time intervals, and send the sensed weight values ​​at multiple moments within each time period to a device that detects the status of goods entering and leaving the shopping cart. as well as The apparatus for detecting the status of goods entering and leaving a shopping cart as described in any one of claims 12 to 13.

15. The shopping cart as described in claim 14, characterized in that, Also includes: The data acquisition device is positioned on the shopping cart, visually covering the area of ​​the shopping cart basket, to acquire multiple images of the shopping cart basket area during the current time period and input them into the hand detection model.

16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

Images