Transaction processing method and system for goods based on weighing sensor and video recognition

Abstract

The application discloses a kind of based on weighing sensor and video identification commodity transaction processing method and system.Method includes hardware trigger synchronous start weighing sensing unit and video acquisition unit, make weight data and image data frame alignment;After enhancing image, the type and quantity of the article that user takes out are extracted, and compared with pre-stored data, and if qualified, output unlocking signal, otherwise output penalty signal.After unlocking, read the weight signal of weighing sensor, compare with pre-stored single product unit price weight, consistent then output commodity type and quantity signal;For multiple commodity situation, the standard weighing value of each single product is subtracted from the distribution of weighing information and the error covariance is calculated, the covariance is used to detect malicious tampering weighing behavior, and output risk or pass signal.The scheme combines weighing data and video recognition data to improve the accuracy and reliability of commodity identification.

Description

Technical Field

[0001] This application relates to the field of smart vending machine technology, and in particular to a commodity transaction processing method and system based on weighing sensors and video recognition. Background Technology

[0002] In the field of smart vending machines, the development of product recognition and transaction systems is receiving increasing attention.

[0003] In existing technologies, single weighing systems are easily affected by environmental interference and have low recognition accuracy, while pure vision recognition systems are prone to failure in low light or when the goods are obscured.

[0004] For example, CN113175986A discloses a weighing management system that improves recognition efficiency and accuracy by combining a visual recognition positioning detection system, an intelligent weighing storage system, and an information input and access control system. It solves the problems of identifying misplaced or missing goods and providing early warnings of erroneous operations, and achieves full-process monitoring and control.

[0005] However, the system has difficulty accurately separating and identifying multiple items when they are taken out at the same time, and its ability to detect malicious behavior (such as item substitution or weight fraud) is limited.

[0006] In addition, CN117523538A proposed an AI visual recognition scale based on deep integration of software and hardware, which improves the system's tracking, inventory and intelligent replenishment capabilities by setting up user units and merchant units.

[0007] However, the accuracy and reliability of this identification scale in complex environments still need to be improved. Summary of the Invention

[0008] Based on this, embodiments of this application provide a commodity transaction processing method and system based on weighing sensors and video recognition, combining weighing data and video recognition data to improve the accuracy and reliability of commodity recognition. Through comprehensive analysis of multimodal data, this algorithm can accurately identify the type and quantity of commodities in various complex environments.

[0009] Firstly, a commodity transaction processing method based on weighing sensors and video recognition is provided, the method comprising:

[0010] The weighing sensing unit and the video acquisition unit are started synchronously through a hardware trigger signal, so that the acquired weight data and image data are aligned frame by frame in time sequence.

[0011] Image enhancement is performed on the acquired image data; wherein, the image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering;

[0012] The system uses video recognition technology to extract the type and quantity of items taken out by the user from the enhanced image and compares them with pre-stored data. If the items belong to the pre-stored product type and the quantity does not exceed the limit, an unlock signal is output; otherwise, a penalty signal is output.

[0013] Upon receiving the unlock signal, the system reads the weight signal sensed by the weighing sensor and compares it with the weight of the item's unit price in the pre-stored product information. If the weights match, the system directly outputs the product type and quantity signal and processes the transaction based on the output product type and quantity signal.

[0014] Optionally, the method further includes:

[0015] If the weight-to-price ratio exceeds the allowable error, a multi-product signal is output. Based on the multi-product signal, the standard weight value of a single product is directly subtracted from the weight information distribution of multiple products to calculate the error covariance. The error covariance is used to detect cheating behavior such as malicious tampering with the weighing. If cheating is determined, a risk signal is output; otherwise, a pass signal is output.

[0016] Optionally, the method further includes:

[0017] When a risk signal is received, the electronic lock time of the item is increased and the calculation process is restarted;

[0018] When a pass signal is received, a standardized bill is generated based on the identified product type, quantity, and pre-stored price information.

[0019] Optionally, the brightness mapping specifically includes:

[0020] If the image brightness is below 0.03, it will be mapped to 12% of the value range of 0 to 255.

[0021] If the brightness is greater than or equal to 0.03 and less than or equal to 0.5, it is mapped to 0.5 times the value range of 0 to 255.

[0022] If the brightness is greater than or equal to 0.5 and less than or equal to 0.97, it is mapped to 0.75 times the value range of 0 to 255.

[0023] If the brightness is greater than or equal to 0.97, it is mapped to 1.1 times the value range of 0 to 255.

[0024] Optionally, the contrast scaling specifically includes:

[0025] If the image contrast is less than 1, the pixel value is multiplied by 1.2;

[0026] If the contrast ratio is greater than or equal to 1 and less than or equal to 10, the pixel value remains unchanged; if the contrast ratio is greater than 10, the pixel value is reduced to one-tenth.

[0027] Optionally, the saturation adjustment specifically includes:

[0028] When the image saturation is below 1, the value of each pixel increases by 5;

[0029] When the saturation is between 1 and 20, the original value remains unchanged; when the saturation exceeds 20, the value of each pixel is reduced by 5.

[0030] Optionally, before comparing the weight signal sensed by the weighing sensor with the weight of the item's unit price in the pre-stored product information, the method further includes:

[0031] The three-dimensional spatial features and the largest divisible region of the object are extracted by the convolutional neural network, and the volume and weight information of each individual item are calculated and stored.

[0032] For packaged goods, the category is determined based on their external outline; for unpackaged goods, the category is determined based on their lower weight limit.

[0033] The type and quantity of goods are determined by combining the weight signal, and then a comparison is performed with the unit price and weight of each item.

[0034] Secondly, a commodity transaction processing system based on weighing sensors and video recognition is provided, the system comprising:

[0035] The synchronous start-up module is used to synchronously start the weighing sensing unit and the video acquisition unit through a hardware trigger signal, so that the acquired weight data and image data are aligned frame by frame in time sequence.

[0036] An image enhancement module is used to enhance the acquired image data; wherein, the image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering;

[0037] The behavior recognition module is used to extract the type and quantity of items taken by the user from the enhanced image using video recognition technology and compare them with pre-stored data. If the items belong to the pre-stored product type and the quantity does not exceed the limit, an unlock signal is output; otherwise, a penalty signal is output.

[0038] The single-item recognition module is used to read the weight signal sensed by the weighing sensor after receiving the unlocking signal, and compare it with the weight of the unit price of the item in the pre-stored product information. If the weights match, the module directly outputs the product type and quantity signal, and performs product transaction processing based on the output product type and quantity signal.

[0039] Optionally, the system further includes:

[0040] The malicious detection module is used to output a multi-product signal if the weight-to-price ratio of the weight exceeds the allowable error. Based on the multi-product signal, the standard weight value of a single product is directly subtracted from the weight information distribution of multiple products to calculate the error covariance. The error covariance is used to detect malicious tampering with the weighing. If it is determined to be cheating, a risk signal is output; otherwise, a pass signal is output.

[0041] Optionally, the system further includes:

[0042] The output module is used to increase the electronic lock time of the item and restart the calculation program when a risk signal is received; when a pass signal is received, it generates a uniform format bill based on the identified product type, quantity and pre-stored price information.

[0043] In this application, the weighing sensing unit and the video acquisition unit are synchronously activated via a hardware trigger signal, aligning the acquired weight data and image data frame by frame in time sequence. Image enhancement is performed on the acquired image data, and video recognition technology is used to extract the type and quantity of items taken by the user from the enhanced image, comparing it with pre-stored data. Upon receiving an unlocking signal, the weight signal sensed by the weighing sensor is read and compared with the weight of the item's unit price in the pre-stored product information. If the weights match, the product type and quantity signal are directly output, and the transaction is processed based on the output product type and quantity signal. In addition, the standard weighing value of a single product is directly subtracted from the weighing information distribution of multiple products to calculate the error covariance. The error covariance is used to detect cheating behavior such as malicious tampering with the weighing. It can be seen that the beneficial effects of the technical solution provided by the embodiments of this application include at least the following:

[0044] (1) It effectively solves the problem that smart vending machines have difficulty accurately identifying the type and quantity of goods under special circumstances such as being upside down, tilted, or loosely packaged.

[0045] (2) A human behavior recognition algorithm was designed to improve the detection capability of malicious behavior. This improved the fault tolerance and robustness of the system and facilitated the development of backend operation and statistics modules. Attached Figure Description

[0046] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart illustrating the steps of a commodity transaction processing method based on weighing sensors and video recognition, provided in an embodiment of this application;

[0048] Figure 2 This is a system block diagram of a commodity transaction processing method based on weighing sensors and video recognition, provided for an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0051] This application proposes an intelligent commodity recognition algorithm based on multimodal data fusion. This algorithm combines weighing data and video recognition data to improve the accuracy and reliability of commodity recognition. Through comprehensive analysis of multimodal data, the algorithm can accurately identify the type and quantity of commodities in various complex environments.

[0052] The core of the technical solution in this application lies in achieving high-precision commodity recognition through the deep integration of weighing sensors and video recognition technology, thereby effectively solving the shortcomings of existing technologies.

[0053] The core objective of this invention is to provide an innovative fusion algorithm based on weighing sensors and video recognition technology to achieve high-precision and high-reliability identification of product type and quantity, thereby improving the credibility of smart vending machines.

[0054] Please refer to Figure 1 The document illustrates a flowchart of a commodity transaction processing method based on weighing sensors and video recognition, according to an embodiment of this application. This method may include the following steps:

[0055] Step 1: Synchronously start the weighing sensing unit and the video acquisition unit through a hardware trigger signal to align the acquired weight data and image data frame by frame in time sequence;

[0056] Step 2: Perform image enhancement on the acquired image data.

[0057] Image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering.

[0058] Step 3: Use video recognition technology to extract the type and quantity of items taken out by the user from the enhanced image and compare it with the pre-stored data.

[0059] If the product is a pre-stored item and the quantity is within the limit, an unlock signal is output; otherwise, a penalty signal is output.

[0060] Step 4: After receiving the unlock signal, read the weight signal sensed by the weighing sensor and compare it with the weight of the item's unit price in the pre-stored product information. If the weights match, directly output the product type and quantity signal, and process the transaction based on the output product type and quantity signal.

[0061] Step 5: If the weight-to-price ratio exceeds the allowable error, output a multi-product signal and process the multi-product signal.

[0062] Specifically, based on multi-commodity signals, the standard weight value of a single commodity is directly subtracted from the weight information distribution of multiple commodities to calculate the error covariance. The error covariance is used to detect cheating behavior such as malicious tampering with the weighing. If cheating is determined, a risk signal is output; otherwise, a pass signal is output.

[0063] Step Six: When a risk signal is received, increase the electronic lock time of the item and restart the calculation process; when a pass signal is received, generate a standardized bill based on the identified product type, quantity, and pre-stored price information.

[0064] In summary, this invention proposes an innovative fusion algorithm that combines weighing sensors and video recognition technology. Its features include the following steps: S1. Integrating the weighing module and video acquisition module; S2. Applying video preprocessing technology to enhance image quality; S3. Utilizing a human behavior recognition algorithm combining deep neural networks and support vector machines; S4. Achieving analysis of commodity weighing information, as well as identification of commodity type and quantity; S5. Performing multi-commodity weighing information analysis and detecting malicious behavior; S6. Outputting calculation results in a unified format. Steps S2 to S5 are all implemented on ARM and Nvidia Jetson platforms using the YOLOv3 algorithm.

[0065] Furthermore, S1 includes a weighing module and a video acquisition module, wherein: the weighing module uses an HX711 chip and a load sensor to jointly build a precise weighing sensing system; the video acquisition module preferably uses a depth camera as the acquisition device, but can also use an ordinary high-definition camera as a backup.

[0066] Conversion formula: Actual weight = (ADC_Value - Zero_Offset) / Scale_Factor

[0067] ADC_Value: The 24-bit digital value of the ADC output by the HX711;

[0068] Zero_Offset: Zero-point offset calibration value;

[0069] Scale_Factor: Scale factor (obtained through calibration with standard weights);

[0070] Formula for judging the stability of weighing:

[0071] Weight_Stability = Threshold_stability

[0072] Weight_Stability: Weight stability index value;

[0073] Wi: The weight value of the i-th sample;

[0074] The average weight taken N times;

[0075] n: Number of samples;

[0076] Threshold_stability: Stability threshold.

[0077] Additionally, an optional embodiment of this application is provided:

[0078] The method includes the following steps: S100: The user retrieves items from the locker using a dedicated withdrawal device; S102: Video recognition technology is used to extract the type and quantity of items retrieved by the user from the video, and compared with pre-stored data to determine whether the items retrieved by the user belong to the pre-stored product type and whether the quantity exceeds the preset quantity; if the items retrieved by the user belong to the pre-stored product type and the quantity does not exceed the preset quantity, then proceed to S104; otherwise, proceed to S105; S104: Based on the video recognition result, the electronic lock of the corresponding product is opened, and the user retrieves the items from the locker; S106: A weighing sensor is used to sense the user's purchase... The weight of the goods is fed back to the central processing unit; S108: The central processing unit calculates the remaining weight based on the weight signal sensed by the weighing sensor and the pre-stored goods information to determine whether the remaining weight is greater than the weight of the unit price of the item; if the remaining weight is greater than the weight of the unit price of the item, then S112 is executed; otherwise, S110 is executed; S110: If the number of items that cannot be weighed exceeds the preset quantity, the locking time of the items is increased and the calculation program is restarted; S112: The central processing unit calculates the settlement based on the quantity and price information of the goods identified by the video recognition system, and the user pays the corresponding amount.

[0079] The working principle of the above technical solution is as follows: When a user takes an item from the locker using a dedicated withdrawal device, the video recognition system begins to track and record the user's behavior and extracts the type and quantity of the item taken out by the user. It then compares this information with the pre-stored data to determine whether the item taken out by the user belongs to the pre-stored product type and whether the quantity exceeds the preset quantity.

[0080] If the item the user wants to retrieve is a pre-stored item and the quantity does not exceed the preset quantity, the user can retrieve the item from the locker normally.

[0081] If a video recognition process fails to identify an item, or if the number of unidentifiable items exceeds a preset limit, a penalty mechanism will be activated to automatically extend the time the electronic lock on that item is locked, and the calculation process will restart.

[0082] When the user picks up an item and places it within the counting area, the weighing sensor begins to sense the weight of the item and transmits the signal to the central processing unit.

[0083] The central processing unit calculates whether the remaining weight is greater than the weight of the item's unit price based on the weight signal sensed by the weighing sensor and the pre-stored product information.

[0084] If the remaining weight is greater than the unit price of the item, it means that the item taken by the user did not exceed the unit of measurement of the product.

[0085] The central processing unit calculates the amount based on the quantity and price information of the goods identified by the video recognition system, and the user pays the corresponding amount.

[0086] Beneficial Effects: This invention solves the problem of traditional vending machines' difficulty in monitoring user theft by fusing weighing data with video data.

[0087] Traditional vending machines mainly rely on weighing sensors to monitor theft of goods, while users' theft can only be monitored afterward through cameras.

[0088] This invention addresses this deficiency by enabling real-time tracking and capture of users, and providing immediate warnings of potential theft.

[0089] Furthermore, this invention uses a binocular camera for tracking. Compared to a traditional monocular camera, a binocular camera has more accurate stereo imaging capabilities and can more accurately measure the size, distance, and other attributes of objects. This makes the object positioning more accurate and facilitates subsequent video analysis.

[0090] Furthermore, since the present invention uses a deep learning network to train the model, it is unaffected by changes in lighting conditions, and can therefore function normally even under varying lighting conditions.

[0091] like Figure 2 This application also provides a commodity transaction processing system based on weighing sensors and video recognition, which may include:

[0092] The synchronous start-up module is used to synchronously start the weighing sensing unit and the video acquisition unit through a hardware trigger signal, so that the acquired weight data and image data are aligned frame by frame in time sequence.

[0093] The image enhancement module is used to enhance the acquired image data; the image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering.

[0094] The behavior recognition module is used to extract the type and quantity of items taken by the user from the enhanced image using video recognition technology and compare them with pre-stored data. If the items belong to the pre-stored product type and the quantity does not exceed the limit, an unlock signal is output; otherwise, a penalty signal is output.

[0095] The single-item recognition module is used to read the weight signal sensed by the weighing sensor after receiving the unlocking signal, and compare it with the weight of the unit price of the item in the pre-stored product information. If the weights match, the module directly outputs the product type and quantity signal, and performs product transaction processing based on the output product type and quantity signal.

[0096] The system also includes:

[0097] The malicious detection module is used to output a multi-product signal if the weight-to-price ratio of the weight exceeds the allowable error. Based on the multi-product signal, the standard weight value of a single product is directly subtracted from the weight information distribution of multiple products to calculate the error covariance. The error covariance is used to detect malicious tampering with the weighing. If it is determined to be cheating, a risk signal is output; otherwise, a pass signal is output.

[0098] The system also includes:

[0099] The output module is used to increase the electronic lock time of the item and restart the calculation program when a risk signal is received; when a pass signal is received, it generates a uniform format bill based on the identified product type, quantity and pre-stored price information.

[0100] Specific limitations regarding the commodity transaction processing system based on weighing sensors and video recognition can be found in the limitations of the commodity transaction processing method based on weighing sensors and video recognition described above, and will not be repeated here. Each module in the aforementioned commodity transaction processing system based on weighing sensors and video recognition can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0101] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A commodity transaction processing method based on weighing sensors and video recognition, characterized in that, The method includes: The weighing sensing unit and the video acquisition unit are started synchronously through a hardware trigger signal, so that the acquired weight data and image data are aligned frame by frame in time sequence. Image enhancement is performed on the acquired image data; wherein, the image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering; The system uses video recognition technology to extract the type and quantity of items taken out by the user from the enhanced image and compares them with pre-stored data. If the items belong to the pre-stored product type and the quantity does not exceed the limit, an unlock signal is output; otherwise, a penalty signal is output. Upon receiving the unlock signal, the system reads the weight signal sensed by the weighing sensor and compares it with the weight of the item's unit price in the pre-stored product information. If the weights match, the system directly outputs the product type and quantity signal and processes the transaction based on the output product type and quantity signal.

2. The commodity transaction processing method according to claim 1, characterized in that, The method further includes: If the weight-to-price ratio exceeds the allowable error, a multi-product signal is output. Based on the multi-product signal, the standard weight value of a single product is directly subtracted from the weight information distribution of multiple products to calculate the error covariance. The error covariance is used to detect cheating behavior such as malicious tampering with the weighing. If cheating is determined, a risk signal is output; otherwise, a pass signal is output.

3. The commodity transaction processing method according to claim 2, characterized in that, The method further includes: When a risk signal is received, the electronic lock time of the item is increased and the calculation process is restarted; When a pass signal is received, a standardized bill is generated based on the identified product type, quantity, and pre-stored price information.

4. The commodity transaction processing method according to claim 1, characterized in that, The brightness mapping specifically includes: If the image brightness is below 0.03, it will be mapped to 12% of the value range of 0 to 255. If the brightness is greater than or equal to 0.03 and less than or equal to 0.5, it is mapped to 0.5 times the value range of 0 to 255. If the brightness is greater than or equal to 0.5 and less than or equal to 0.97, it is mapped to 0.75 times the value range of 0 to 255. If the brightness is greater than or equal to 0.97, it is mapped to 1.1 times the value range of 0 to 255.

5. The commodity transaction processing method according to claim 1, characterized in that, The contrast scaling specifically includes: If the image contrast is less than 1, the pixel value is multiplied by 1.2; If the contrast ratio is greater than or equal to 1 and less than or equal to 10, the pixel value remains unchanged; if the contrast ratio is greater than 10, the pixel value is reduced to one-tenth.

6. The commodity transaction processing method according to claim 1, characterized in that, The saturation adjustment specifically includes: When the image saturation is below 1, the value of each pixel increases by 5; When the saturation is between 1 and 20, the original value remains unchanged; when the saturation exceeds 20, the value of each pixel is reduced by 5.

7. The commodity transaction processing method according to claim 1, characterized in that, Before comparing the weight signal sensed by the weighing sensor with the weight of the item's unit price in the pre-stored product information, the method further includes: The three-dimensional spatial features and the largest divisible region of the object are extracted by the convolutional neural network, and the volume and weight information of each individual item are calculated and stored. For packaged goods, the category is determined based on their external outline; for unpackaged goods, the category is determined based on their lower weight limit. The type and quantity of goods are determined by combining the weight signal, and then a comparison is performed with the unit price and weight of each item.

8. A commodity transaction processing system based on weighing sensors and video recognition, characterized in that, The system includes: The synchronous start-up module is used to synchronously start the weighing sensing unit and the video acquisition unit through a hardware trigger signal, so that the acquired weight data and image data are aligned frame by frame in time sequence. An image enhancement module is used to enhance the acquired image data; wherein, the image enhancement includes at least brightness mapping, contrast scaling, saturation adjustment, and bilateral filtering; The behavior recognition module is used to extract the type and quantity of items taken by the user from the enhanced image using video recognition technology and compare them with pre-stored data. If the items belong to the pre-stored product type and the quantity does not exceed the limit, an unlock signal is output; otherwise, a penalty signal is output. The single-item recognition module is used to read the weight signal sensed by the weighing sensor after receiving the unlocking signal, and compare it with the weight of the unit price of the item in the pre-stored product information. If the weights match, the module directly outputs the product type and quantity signal, and performs product transaction processing based on the output product type and quantity signal.

9. The commodity transaction processing system according to claim 8, characterized in that, The system also includes: The malicious detection module is used to output a multi-product signal if the weight-to-price ratio of the weight exceeds the allowable error. Based on the multi-product signal, the standard weight value of a single product is directly subtracted from the weight information distribution of multiple products to calculate the error covariance. The error covariance is used to detect cheating behavior such as malicious tampering with the weighing. If cheating is determined, a risk signal is output; otherwise, a pass signal is output.

10. The commodity transaction processing system according to claim 9, characterized in that, The system also includes: The output module is used to increase the electronic lock time of the item and restart the calculation program when a risk signal is received; when a pass signal is received, it generates a uniform format bill based on the identified product type, quantity and pre-stored price information.

Images