Intelligent shopping cart self-service payment system based on multi-sensor fusion and implementation method thereof

Abstract

The invention discloses an intelligent shopping cart self-service payment system based on multi-sensor fusion and an implementation method thereof, and the system comprises a self-service payment system which comprises a commodity recognition module, a data processing module, a settlement and payment module, a safety management module, a user interaction module, and a communication and network module. According to the invention, through integration of the RFID reader, the high-definition camera, the weight sensor and the ultrasonic sensor, high-precision identification of commodities in the shopping cart is realized by adopting a multi-modal data fusion algorithm, the system supports multiple payment modes, and the shopping cart is convenient to use. The block chain technology and the multi-layer encryption algorithm are utilized to guarantee the security of the payment process, the response speed and the data processing capability of the system are improved through the combination of edge computing and cloud computing, and the system improves the shopping experience, is suitable for various retail scenes, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of smart shopping cart self-service payment systems, and in particular to a smart shopping cart self-service payment system based on multi-sensor fusion and an implementation method thereof. Background Art

[0002] Smart shopping carts support consumers to shop while browsing and scan codes by themselves. After consumers scan the barcode of the product, the smart shopping cart will automatically pop up the price information of the product for consumers to verify. After shopping, consumers can complete the settlement through the shopping cart at any location in the supermarket, avoiding the trouble of waiting in line and improving the shopping experience. The smart shopping cart can quickly and accurately identify the product information and automatically calculate the total price through the built-in high-definition camera and image recognition technology, eliminating the tedious queuing and settlement process. Consumers only need to put the product into the shopping cart, and the system will automatically complete the settlement, greatly saving shopping time. The smart shopping cart can provide consumers with personalized product recommendations through deep learning of user purchasing habits, which means that consumers can obtain instant discount product information and can instantly redeem electronic coupons and enjoy shopping at preferential prices. The smart shopping cart supports multiple payment methods, such as WeChat, Alipay, etc. Consumers can complete the settlement through these payment methods after shopping without contacting the cashier, further improving the convenience of payment.

[0003] However, the traditional smart shopping cart self-service payment system still has the following disadvantages:

[0004] Existing smart shopping carts and self-service payment systems still have many shortcomings in terms of product identification, payment security, user experience, etc. For example, product identification relies on a single RFID or visual recognition technology, which is easily subject to environmental and technical limitations; the payment method is single and user acceptance is low; the system security protection measures are not perfect and are vulnerable to network attacks. Summary of the invention

[0005] The purpose of the present invention is to provide a smart shopping cart self-service payment system based on multi-sensor fusion and its implementation method, so as to solve the problems that the existing smart shopping carts and self-service payment systems proposed in the above background technology still have many deficiencies in product identification, payment security, user experience, etc. For example, product identification relies on a single RFID or visual recognition technology, which is easily subject to environmental and technical limitations; the payment method is single and user acceptance is low; the system security protection measures are imperfect and vulnerable to network attacks.

[0006] To achieve the above-mentioned object, the present invention provides the following technical solutions: a smart shopping cart self-service payment system based on multi-sensor fusion, comprising a self-service payment system, wherein the self-service payment system comprises a commodity identification module, a data processing module, a settlement and payment module, a security management module, a user interaction module and a communication and network module;

[0007] The commodity identification module identifies the shape and weight of shopping commodities through sensors, and is used to collect commodity information in the shopping cart;

[0008] The data processing module processes the data information of commodity identification, and is used to pre-process, extract features and fuse data of the collected multi-modal data to identify commodities in the shopping cart;

[0009] The settlement and payment module settles the commodity price and optimizes the payment calculation based on the commodity identification information, and is used to generate a shopping list based on the identified commodities, calculate the total price, and complete the payment process;

[0010] The security management module records the payment transaction information in the alliance chain. Each node is composed of merchants, banks and payment institutions and securely encrypts the payment information to ensure the data security of the system and the security of the payment process, including data encryption, identity authentication and blockchain transaction records;

[0011] The user interaction module performs personalized recommendations and friendly interactions with users, and is used to interact with users and display shopping lists, recommended products, and payment information;

[0012] The communication and network module completes data interaction through 5G communication technology and Internet of Things protocol, and is used to realize the communication connection between the shopping cart and the cloud server and payment gateway.

[0013] As a preferred technical solution of the present invention, the commodity identification module includes a multi-sensor fusion unit and a data fusion unit. The multi-sensor fusion unit collects the appearance and weight of commodity data through an RFID reader, a high-definition camera, a weight sensor, and an ultrasonic sensor, and improves the accuracy of commodity identification through a multimodal data fusion algorithm. The specific process of the data fusion unit is as follows: S1, data acquisition: RFID data acquisition reads the RFID tag information in the shopping cart to obtain the commodity ID; visual data acquisition: the camera captures the real-time image in the shopping cart; weight data acquisition: the weight sensor obtains the current total weight of the shopping cart; ultrasonic data acquisition: detects the position and movement of items in the shopping cart; S2, data preprocessing: RFID data preprocessing removes repeated and erroneously read tag information; image preprocessing: image denoising, grayscale, and size normalization; weight data preprocessing: filter out noise and calculate the weight change ΔW; ultrasonic data preprocessing; filtering processing to locate the position of the item; S3, feature extraction: visual feature extraction, using a convolutional neural network CNN to extract image features, and the model structure is specifically as follows:

[0014] Input layer: the image in the shopping cart (size is WXIIX3),

[0015] Convolution + pooling layer xN: extract high-level extracts,

[0016] Fully connected continuous layer: classify features,

[0017] Output layer: Sofmax activation, output product rate,

[0018] Loss function:

[0019]

[0020] Among them, y i is the true label, is the predicted probability, C is the total number of commodity categories,

[0021] RFID feature extraction: directly obtain product ID and match product database; weight feature extraction: calculate weight change:

[0022] ΔW=W current -W previous

[0023] Ultrasonic feature extraction: calculate the object location coordinates (x, y, z); S4, data fusion and decision-making: use weighted Bayesian estimation to fuse multi-source data, Bayesian fusion formula:

[0024]

[0025] Where: P(C kD): Given data D, the posterior probability that the item belongs to category CK;

[0026] P(C k ): Prior probability, set to uniform distribution or adjusted according to historical data;

[0027] P(D i |C k ): likelihood function of the data in the i-th category Ck;

[0028] wi: weight of the i-th data, set according to the reliability of the sensor;

[0029] Z: normalization constant; decision rule, choose P(C k D) Maximum Category C k As the recognition result; S5, algorithm flow:

[0030] As a preferred technical solution of the present invention, the data processing module is provided with edge computing combined with cloud computing and distributed data processing. The edge computing combined with cloud computing is specifically edge computing and cloud computing. The edge computing is deployed on an embedded processor of a shopping cart terminal, responsible for real-time data collection, preprocessing and partial feature extraction, reducing the amount of data transmission and improving real-time performance. The cloud computing is deployed on a high-performance server in the cloud, responsible for complex model training, data storage and global optimization, providing an API interface for terminal devices to call. The distributed data processing is specifically as follows: S1. Data transmission protocol: using the message queue telemetry transmission (MQTT) protocol, which is lightweight and efficient and suitable for IoT devices; S2. Real-time data processing framework: using Apache Flink to realize real-time streaming data processing; S3. Data storage: short-term data and long-term data. The short-term data uses the Redis memory database to store real-time status and cache. The long-term data uses the Hadoop distributed file system (HDFS) or NoSQL database to store historical data and logs.

[0031] As a preferred technical solution of the present invention, the settlement and payment module is provided with payment process optimization and payment algorithm. The payment process optimization is specifically as follows: the dynamic payment strategy recommends payment methods based on user portraits using machine learning models (such as KNN, decision trees); the one-click payment process user confirms the shopping list, the system calls the bound payment account, performs fast identity authentication such as fingerprint, face recognition, automatically completes payment, and updates the payment status. The payment algorithm is specifically as follows: S1, payment request generation: payment amount calculation:

[0032]

[0033] Where: Pi: the price of the i-th product; Qi: the quantity of the i-th product; D: the discount amount (full discount, discount), generate a payment request object; S2, payment process: identity authentication: use multi-factor authentication MFA, such as password + biometrics, biometric matching algorithm: face recognition: use a deep learning model to enter FaceNet, calculate the face feature vector f, and perform similarity calculation. If the similarity is higher than the threshold, the verification is passed. Encrypt payment request: use the RSA encryption algorithm to encrypt the payment request, public key encryption:

[0034] C=E pub (M),

[0035] Among them, M is the plaintext payment request, C is the ciphertext, and sending the payment request: the encrypted payment request is sent to the payment gateway via the HTTPS protocol.

[0036] The security management module includes blockchain technology applications and multi-layer encryption and privacy protection. The blockchain technology applications are as follows: transaction records are put on the chain, and payment transaction information is recorded in the alliance chain. Each node is composed of merchants, banks and payment institutions. The block structure is:

[0037]

[0038] The consensus mechanism adopts the Practical Byzantine Fault Tolerance (PBFT) algorithm to improve the transaction confirmation speed. The multi-layer encryption and privacy protection are specifically data encryption: symmetric encryption, using AES-256 to encrypt user sensitive data, encryption:

[0039] C=AES key (M);

[0040] Decryption:

[0041]

[0042] Key management, use the key management service KMS, regularly change keys, communication encryption: TLS handshake process: the client supports the protocol version and encryption algorithm, the server selects the protocol and algorithm, sends the digital certificate, the client verifies the certificate, generates the session key, and the session key is used for encrypted communication; privacy protection, differential privacy, when analyzing data, add noise to protect user privacy, Laplace mechanism:

[0043]

[0044] Among them, ε is the privacy budget parameter.

[0045] As a preferred technical solution of the present invention, the user interaction module is provided with personalized recommendation and friendly interaction interface. The personalized recommendation is based on collaborative filtering of users through collaborative filtering algorithm to calculate user similarity and recommendation score prediction. The deep learning recommendation model uses neural networks DNN and RNN to model the interaction relationship between users and commodities. The friendly interaction interface is specifically S1. Interface design: adopting high-resolution touch screen, providing intuitive operation interface, supporting multi-language display, and convenient for users with different backgrounds to use; S2. Voice interaction: integrating voice recognition and synthesis technology, supporting voice commands and feedback.

[0046] As a preferred technical solution of the present invention, the communication and network module is provided with 5G communication technology and Internet of Things protocol. The 5G communication technology improves data transmission rate, reduces latency, and supports the connection of large-scale devices. The Internet of Things protocol uses MQTT or CoAP to achieve efficient communication between devices.

[0047] The present invention provides a method for realizing a smart shopping cart self-service payment system based on multi-sensor fusion, comprising the following steps:

[0048] Step 1: Product identification: The product identification module collects multimodal data in the shopping cart, including RFID tag information, product images, weight data, and ultrasonic ranging data;

[0049] Step 2, data processing: In the data processing module, the collected data is preprocessed and feature extracted, including data cleaning, filtering, visual feature extraction and weight change calculation;

[0050] Step 3: Data calculation: Use a multimodal data fusion algorithm to fuse the features of different sensors, identify the items in the shopping cart, and update the shopping list;

[0051] Step 4: List payment: In the settlement and payment module, calculate the total price based on the shopping list, generate a payment request, and complete the payment through identity authentication;

[0052] Step 5: Security management: In the security management module, payment transactions are encrypted and recorded to ensure data security and the security of the payment process;

[0053] Step 6: User interaction: Through the user interaction module, information is exchanged with users to provide a friendly operation interface and personalized services.

[0054] As a preferred technical solution of the present invention, a weighted Bayesian estimation algorithm is used in step three to calculate the posterior probability that the product belongs to each category to improve the accuracy of product identification. The identity authentication in step four includes multi-factor authentication, which combines passwords and biometrics to improve the security of the payment process. Blockchain technology is used to record payment transactions in step five, and smart contracts are used to automatically execute transactions to improve the credibility and transparency of data. In step six, collaborative filtering and deep learning algorithms are used to provide users with personalized product recommendations and promotional information. The entire system adopts a distributed data processing method that combines edge computing with cloud computing, performs real-time data processing on the edge side, and performs model training and global optimization on the cloud.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: by integrating multiple devices such as RFID readers, high-definition cameras, weight sensors and ultrasonic sensors, and adopting a multimodal data fusion algorithm, high-precision identification of items in the shopping cart can be achieved. The system supports multiple payment methods, and uses blockchain technology and multi-layer encryption algorithms to ensure the security of the payment process. Through the combination of edge computing and cloud computing, the response speed and data processing capabilities of the system are improved. The system improves the shopping experience, is suitable for various retail scenarios, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the architecture of the self-service payment system of the present invention;

[0057] Figure 2 This is a schematic diagram of the architecture of the commodity identification module of the present invention;

[0058] Figure 3 It is a schematic diagram of the architecture of the data processing module of the present invention;

[0059] Figure 4 This is a schematic diagram of the architecture of the settlement and payment module of the present invention;

[0060] Figure 5 is a schematic diagram of the architecture of the security management module of the present invention;

[0061] Figure 6 A schematic diagram of the architecture of the user interaction module of the present invention;

[0062] Figure 7 It is a schematic diagram of the architecture of the communication and network module of the present invention;

[0063] Figure 8 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] See also Figure 1-8 ,The present invention provides a smart shopping cart self-service payment system based on multi-sensor fusion, including a self-service payment system, the self-service payment system including a commodity identification module, a data processing module, a settlement and payment module, a security management module, a user interaction module and a communication and network module;

[0066] The commodity recognition module uses sensors to identify the shape and weight of shopping commodities and collects commodity information in the shopping cart;

[0067] The data processing module processes the data information of commodity identification and is used to pre-process, extract features and fuse data of the collected multi-modal data to identify the commodities in the shopping cart;

[0068] The settlement and payment module settles the commodity price and optimizes the payment calculation based on the commodity identification information, and is used to generate a shopping list based on the identified commodities, calculate the total price, and complete the payment process;

[0069] The security management module records payment transaction information in the alliance chain. Each node is composed of merchants, banks and payment institutions and encrypts the payment information securely to ensure the data security of the system and the security of the payment process, including data encryption, identity authentication and blockchain transaction records;

[0070] The user interaction module makes personalized recommendations and interacts with users in a friendly manner, and is used to interact with users and display shopping lists, recommended products, and payment information;

[0071] The communication and network module completes data interaction through 5G communication technology and Internet of Things protocol, and is used to realize the communication connection between the shopping cart and the cloud server and payment gateway.

[0072] The commodity identification module includes a multi-sensor fusion unit and a data fusion unit. The multi-sensor fusion unit collects commodity data shape and weight through RFID readers, high-definition cameras, weight sensors, and ultrasonic sensors, and uses multimodal data fusion algorithms to improve the accuracy of commodity identification. The specific process of the data fusion unit is as follows: S1, data collection: RFID data collection reads the RFID tag information in the shopping cart and obtains the commodity ID; visual data collection: the camera captures the real-time image in the shopping cart; weight data collection: the weight sensor obtains the current total weight of the shopping cart; ultrasonic data collection: detects the position and movement of items in the shopping cart; S2, data preprocessing: RFID data preprocessing removes repeated and incorrectly read label information; image preprocessing: image denoising, grayscale, and size normalization; weight data preprocessing: filter out noise and calculate weight change ΔW; ultrasonic data preprocessing; filtering processing, locate the position of the item; S3, feature extraction: visual feature extraction, using convolutional neural network CNN to extract image features, the model structure is as follows:

[0073] Input layer: the image in the shopping cart (size is WXIIX3),

[0074] Convolution + pooling layer xN: extract high-level extracts,

[0075] Fully connected continuous layer: classify features,

[0076] Output layer: Sofmax activation, output product rate,

[0077] Loss function:

[0078]

[0079] Among them, y i is the true label, is the predicted probability, C is the total number of commodity categories,

[0080] RFID feature extraction: directly obtain product ID and match product database; weight feature extraction: calculate weight change:

[0081] ΔW=W current -W previous

[0082] Ultrasonic feature extraction: calculate the object location coordinates (x, y, z); S4, data fusion and decision-making: use weighted Bayesian estimation to fuse multi-source data, Bayesian fusion formula:

[0083]

[0084] Where: P(C k D): Given data D, the posterior probability that the item belongs to category CK;

[0085] P(C k ): Prior probability, set to uniform distribution or adjusted according to historical data;

[0086] P(D i |C k ): likelihood function of the data in the i-th category Ck;

[0087] wi: weight of the i-th data, set according to the reliability of the sensor;

[0088] Z: normalization constant; decision rule, choose P(C k D) Maximum Category C k As the recognition result; S5, algorithm flow: RFID readers are installed inside and on the edge of the shopping cart to read the RFID tag information on the goods. High-definition cameras are installed above and inside the shopping cart. The camera's field of view covers the entire internal space of the shopping cart and is used to capture images of the goods placed in the shopping cart. Weight sensors are installed at multiple locations at the bottom of the shopping cart to measure the total weight and weight change of the items in the shopping cart. Ultrasonic sensors are installed inside the shopping cart to detect the location and movement trajectory of the items.

[0089] The data processing module is equipped with edge computing combined with cloud computing and distributed data processing. The combination of edge computing and cloud computing is specifically edge computing and cloud computing. Edge computing is deployed on the embedded processor of the shopping cart terminal, which is responsible for real-time data collection, preprocessing and partial feature extraction, reducing the amount of data transmission and improving real-time performance. Cloud computing is deployed on high-performance servers in the cloud, which is responsible for complex model training, data storage and global optimization, and provides API interfaces for terminal devices to call. Distributed data processing is specifically as follows: S1. Data transmission protocol: using the message queue telemetry transmission (MQTT) protocol, which is lightweight and efficient and suitable for IoT devices; S2. Real-time data processing framework: using Apache Flink to realize real-time streaming data processing; S3. Data storage: short-term data and long-term data. Short-term data uses the Redis memory database to store real-time status and cache. Long-term data uses the Hadoop distributed file system (HDFS) or NoSQL database to store historical data and logs. The edge computing unit embedded processor is located inside the shopping cart and is responsible for real-time data collection, preprocessing and partial feature extraction. The cloud computing server is located in a remote data center and is responsible for complex model training, global data analysis and optimization.

[0090] The settlement and payment module is equipped with payment process optimization and payment algorithm. The payment process optimization is as follows: dynamic payment strategy recommends payment method based on user portrait using machine learning model (such as KNN, decision tree); one-click payment process: user confirms shopping list, system calls bound payment account, performs fast identity authentication such as fingerprint, face recognition, automatically completes payment, and updates payment status. The payment algorithm is as follows: S1, payment request generation: payment amount calculation:

[0091]

[0092] Where: Pi: the price of the i-th product; Qi: the quantity of the i-th product; D: the discount amount (full discount, discount), generate a payment request object; S2, payment process: identity authentication: use multi-factor authentication MFA, such as password + biometrics, biometric matching algorithm: face recognition: use a deep learning model to enter FaceNet, calculate the face feature vector f, and perform similarity calculation. If the similarity is higher than the threshold, the verification is passed. Encrypt payment request: use the RSA encryption algorithm to encrypt the payment request, public key encryption:

[0093] C=E pub (M),

[0094] Among them, M is a plaintext payment request, C is a ciphertext, and the payment request is sent: the encrypted payment request is sent to the payment gateway through the HTTPS protocol. The payment request generation unit calculates the total amount and generates a payment request based on the product list and discount policy. The identity authentication unit supports multiple identity authentication methods, including passwords and biometrics (such as face and fingerprint). The payment result processing unit receives feedback from the payment gateway, updates the payment status, and notifies the user of the payment result.

[0095] The security management module includes blockchain technology applications and multi-layer encryption and privacy protection. The specific application of blockchain technology is: transaction records are put on the chain, and payment transaction information is recorded in the alliance chain. Each node is composed of merchants, banks and payment institutions. The block structure is:

[0096] ```Plain Text

[0097] Block{

[0098] Index,

[0099] Timestamp,

[0100] Transactions:

[0101] {

[0102] TransactionID,

[0103] UserID,

[0104] OrderID,

[0105] Amount,

[0106] PaymentMethod,

[0107] Status

[0108] }, ...

[0110] ],

[0111] PreviousHash,

[0112] Hash,

[0113] Nonce

[0114] }

[0115] ```

[0116] The consensus mechanism uses the Practical Byzantine Fault Tolerance (PBFT) algorithm to improve the transaction confirmation speed. Multi-layer encryption and privacy protection are specifically for data encryption: symmetric encryption, using AES-256 to encrypt user sensitive data, encryption:

[0117] C=AES key (M);

[0118] Decryption:

[0119]

[0120] Key management, use the key management service KMS, regularly change keys, communication encryption: TLS handshake process: the client supports the protocol version and encryption algorithm, the server selects the protocol and algorithm, sends the digital certificate, the client verifies the certificate, generates the session key, and the session key is used for encrypted communication; privacy protection, differential privacy, when analyzing data, add noise to protect user privacy, Laplace mechanism:

[0121]

[0122] Among them, ε is the privacy budget parameter. The blockchain network nodes participate in the nodes of the alliance chain, record transaction information, and ensure that the data cannot be tampered with. The data encryption unit is responsible for encrypting, storing and transmitting sensitive data, and the key management unit manages the generation, storage and rotation of encryption keys.

[0123] The user interaction module is equipped with personalized recommendations and a friendly interactive interface. Personalized recommendations are made through collaborative filtering algorithms, based on user collaborative filtering, to calculate user similarity and recommendation score predictions. The deep learning recommendation model uses neural networks such as DNN and RNN to model the interactive relationship between users and products. The friendly interactive interface is specifically S1. Interface design: It uses a high-resolution touch screen to provide an intuitive operating interface and supports multi-language display, which is convenient for users from different backgrounds to use; S2. Voice interaction: It integrates voice recognition and synthesis technology, supports voice commands and feedback, and the touch screen interface is installed on the handle of the shopping cart, which is convenient for users to view shopping lists, choose payment methods, and view recommended products. The voice interaction system has a built-in microphone and speaker to support voice commands and feedback.

[0124] The communication and network module is equipped with 5G communication technology and Internet of Things protocol. 5G communication technology: improves data transmission rate, reduces latency, and supports the connection of large-scale devices. The Internet of Things protocol uses MQTT or CoAP to achieve efficient communication between devices. The 5G communication module provides high-speed data transmission and supports real-time communication between shopping carts and cloud servers. The MQTT protocol interface is used for efficient communication between Internet of Things devices.

[0125] The present invention provides a method for realizing a smart shopping cart self-service payment system based on multi-sensor fusion, comprising the following steps:

[0126] Step 1: Product identification: The product identification module collects multimodal data in the shopping cart, including RFID tag information, product images, weight data, and ultrasonic ranging data;

[0127] Step 2, data processing: In the data processing module, the collected data is preprocessed and feature extracted, including data cleaning, filtering, visual feature extraction and weight change calculation;

[0128] Step 3: Data calculation: Use a multimodal data fusion algorithm to fuse the features of different sensors, identify the items in the shopping cart, and update the shopping list;

[0129] Step 4: List payment: In the settlement and payment module, calculate the total price based on the shopping list, generate a payment request, and complete the payment through identity authentication;

[0130] Step 5: Security management: In the security management module, payment transactions are encrypted and recorded to ensure data security and the security of the payment process;

[0131] Step 6: User interaction: Through the user interaction module, information is exchanged with users to provide a friendly operation interface and personalized services.

[0132] In step three, a weighted Bayesian estimation algorithm is used to calculate the posterior probability that the product belongs to each category to improve the accuracy of product identification. The identity authentication in step four includes multi-factor authentication, which combines passwords and biometrics to improve the security of the payment process. In step five, blockchain technology is used to record payment transactions, and smart contracts are used to automatically execute transactions to improve the credibility and transparency of data. In step six, collaborative filtering and deep learning algorithms are used to provide users with personalized product recommendations and promotional information. The entire system adopts a distributed data processing method that combines edge computing with cloud computing, performs real-time data processing on the edge side, and performs model training and global optimization on the cloud.

[0133] In the present invention, the RFID reader is used to read the RFID tag information of the commodity; the high-definition camera is used to capture the commodity image in the shopping cart; the weight sensor is used to detect the weight change in the shopping cart; the ultrasonic sensor is used to detect the position and movement of the items in the shopping cart. The data processing module adopts a multimodal data fusion algorithm to fuse the RFID data, visual features, weight features and ultrasonic features to improve the accuracy and robustness of commodity recognition. The multimodal data fusion algorithm is based on weighted Bayesian estimation, fuses the data of different sensors, calculates the posterior probability that the commodity belongs to each category, and selects the category with the maximum probability as the recognition result. The data processing module uses a deep learning model to extract features from the visual data, and uses a pre-trained convolutional neural network (CNN) model to extract high-level visual features of the commodity. The settlement and payment module supports multiple Payment methods include mobile payment, biometric payment and contactless payment, and multi-factor authentication is used for identity authentication. Biometric payment includes face recognition payment and fingerprint recognition payment. Deep learning models are used to extract biometric features, and identity authentication is performed through similarity calculation. The security management module uses blockchain technology to record payment transactions, and uses smart contracts and consensus mechanisms to ensure the immutability and transparency of transaction data. The blockchain network uses the Practical Byzantine Fault Tolerance algorithm (PBFT) as a consensus mechanism. Each node is composed of merchants, banks and payment institutions. The communication and network module uses 5G communication technology to achieve high-speed data transmission between the shopping cart and the cloud server and payment gateway. The user interaction module includes a high-resolution touch screen and a voice interaction system, providing shopping list display, payment method selection, personalized recommendations and voice command functions.

[0134] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart shopping cart self-service payment system based on multi-sensor fusion, including a self-service payment system, characterized in that: The self-service payment system includes a commodity identification module, a data processing module, a settlement and payment module, a security management module, a user interaction module and a communication and network module; The commodity identification module identifies the shape and weight of shopping commodities through sensors, and is used to collect commodity information in the shopping cart; The data processing module processes the data information of commodity identification, and is used to pre-process, extract features and fuse data of the collected multi-modal data to identify commodities in the shopping cart; The settlement and payment module settles the commodity price and optimizes the payment calculation based on the commodity identification information, and is used to generate a shopping list based on the identified commodities, calculate the total price, and complete the payment process; The security management module records the payment transaction information in the alliance chain. Each node is composed of merchants, banks and payment institutions and securely encrypts the payment information to ensure the data security of the system and the security of the payment process, including data encryption, identity authentication and blockchain transaction records; The user interaction module performs personalized recommendations and friendly interactions with users, and is used to interact with users and display shopping lists, recommended products, and payment information; The communication and network module completes data interaction through 5G communication technology and Internet of Things protocol, and is used to realize the communication connection between the shopping cart and the cloud server and payment gateway.

2. According to claim 1, a smart shopping cart self-service payment system based on multi-sensor fusion is characterized by: The commodity identification module includes a multi-sensor fusion unit and a data fusion unit. The multi-sensor fusion unit collects commodity data shape and weight through RFID readers, high-definition cameras, weight sensors, and ultrasonic sensors, and improves the accuracy of commodity identification through multi-modal data fusion algorithms. The specific process of the data fusion unit is as follows: S1, data collection: RFID data collection reads RFID tag information in the shopping cart to obtain commodity ID; visual data collection: the camera captures real-time images in the shopping cart; weight data collection: the weight sensor obtains the current total weight of the shopping cart; ultrasonic data collection: detects the position and movement of items in the shopping cart; S2, data preprocessing: RFID data preprocessing removes repeated and incorrectly read tag information; image preprocessing: image denoising, grayscale, and size normalization; weight data preprocessing: filter out noise and calculate weight change ΔW; ultrasonic data preprocessing; filtering processing to locate the position of the item; S3, feature extraction: visual feature extraction, using convolutional neural network CNN to extract image features, the model structure is specifically as follows: Input layer: the image in the shopping cart (size is WXIIX3), Convolution + pooling layer xN: extract high-level extracts, Fully connected continuous layer: classify features, Output layer: Sofmax activation, output product rate, Loss function: Among them, y i is the true label, is the predicted probability, C is the total number of commodity categories, RFID feature extraction: directly obtain product ID and match product database; weight feature extraction: calculate weight change: ΔW=W current -IN previous Ultrasonic feature extraction: calculate the object location coordinates (x, y, z); S4, data fusion and decision-making: use weighted Bayesian estimation to fuse multi-source data, Bayesian fusion formula: Where: P(C k D): Given data D, the posterior probability that the item belongs to category CK; P(C k ): Prior probability, set to uniform distribution or adjusted according to historical data; P(D i |C k ): likelihood function of the data in the i-th category Ck; wi: weight of the i-th data, set according to the reliability of the sensor; Z: normalization constant; decision rule, choose P(C k D) Maximum Category C k As the recognition result; S5, algorithm flow:

3. According to claim 1, a smart shopping cart self-service payment system based on multi-sensor fusion is characterized by: The data processing module is provided with a combination of edge computing and cloud computing and distributed data processing. The combination of edge computing and cloud computing is specifically edge computing and cloud computing. The edge computing is deployed on an embedded processor of the shopping cart terminal, which is responsible for real-time data collection, preprocessing and partial feature extraction, reducing the amount of data transmission and improving real-time performance. The cloud computing is deployed on a high-performance server in the cloud, which is responsible for complex model training, data storage and global optimization, and provides an API interface for terminal devices to call. The distributed data processing is specifically as follows: S1. Data transmission protocol: using the message queue telemetry transmission (MQTT) protocol, which is lightweight and efficient and suitable for IoT devices; S2. Real-time data processing framework: using Apache Flink to realize real-time streaming data processing; S3. Data storage: short-term data and long-term data. The short-term data uses the Redis memory database to store real-time status and cache. The long-term data uses the Hadoop distributed file system (HDFS) or NoSQL database to store historical data and logs.

4. According to claim 1, a smart shopping cart self-service payment system based on multi-sensor fusion is characterized by: The settlement and payment module is provided with payment process optimization and payment algorithm. The payment process optimization is specifically as follows: dynamic payment strategy recommends payment method based on user portrait using machine learning model (such as KNN, decision tree); one-click payment process user confirms shopping list, system calls bound payment account, performs fast identity authentication such as fingerprint, face recognition, automatically completes payment, and updates payment status. The payment algorithm is specifically as follows: S1, payment request generation: payment amount calculation: Where: Pi: the price of the i-th product; Qi: the quantity of the i-th product; D: the discount amount (full discount, discount), generate a payment request object; S2, payment process: identity authentication: use multi-factor authentication MFA, such as password + biometrics, biometric matching algorithm: face recognition: use a deep learning model to enter FaceNet, calculate the face feature vector f, and perform similarity calculation. If the similarity is higher than the threshold, the verification is passed. Encrypt payment request: use the RSA encryption algorithm to encrypt the payment request, public key encryption: C=E pub (M), Among them, M is the plaintext payment request, C is the ciphertext, and sending the payment request: the encrypted payment request is sent to the payment gateway via the HTTPS protocol.

5. According to claim 1, a smart shopping cart self-service payment system based on multi-sensor fusion is characterized in that: The security management module includes blockchain technology applications and multi-layer encryption and privacy protection. The blockchain technology applications are as follows: transaction records are put on the chain, and payment transaction information is recorded in the alliance chain. Each node is composed of merchants, banks and payment institutions. The block structure is: The consensus mechanism adopts the Practical Byzantine Fault Tolerance (PBFT) algorithm to improve the transaction confirmation speed. The multi-layer encryption and privacy protection are specifically data encryption: symmetric encryption, using AES-256 to encrypt user sensitive data, encryption: C=AES key (M); Decryption: Key management, use the key management service KMS, regularly change keys, communication encryption: TLS handshake process: the client supports the protocol version and encryption algorithm, the server selects the protocol and algorithm, sends the digital certificate, the client verifies the certificate, generates the session key, and the session key is used for encrypted communication; privacy protection, differential privacy, when analyzing data, add noise to protect user privacy, Laplace mechanism: Among them, ε is the privacy budget parameter.

6. The intelligent shopping cart self-service payment system based on multi-sensor fusion according to claim 1, characterized in that: The user interaction module is provided with personalized recommendation and friendly interaction interface. The personalized recommendation is based on collaborative filtering of users through collaborative filtering algorithm, calculates user similarity and recommendation score prediction. The deep learning recommendation model uses neural network DNN and RNN to model the interaction relationship between users and commodities. The friendly interaction interface is specifically S1. Interface design: adopts high-resolution touch screen, provides intuitive operation interface, supports multi-language display, and is convenient for users with different backgrounds to use; S2. Voice interaction: integrates voice recognition and synthesis technology, and supports voice commands and feedback.

7. The intelligent shopping cart self-service payment system based on multi-sensor fusion according to claim 1, characterized in that: The communication and network module is equipped with 5G communication technology and Internet of Things protocol. The 5G communication technology improves data transmission rate, reduces latency, and supports the connection of large-scale devices. The Internet of Things protocol uses MQTT or CoAP to achieve efficient communication between devices.

8. A method for implementing a smart shopping cart self-service payment system based on multi-sensor fusion according to any one of claims 1 to 7, characterized in that: The following steps are involved: Step 1: Product identification: The product identification module collects multimodal data in the shopping cart, including RFID tag information, product images, weight data, and ultrasonic ranging data; Step 2, data processing: In the data processing module, the collected data is preprocessed and feature extracted, including data cleaning, filtering, visual feature extraction and weight change calculation; Step 3: Data calculation: Use a multimodal data fusion algorithm to fuse the features of different sensors, identify the items in the shopping cart, and update the shopping list; Step 4: List payment: In the settlement and payment module, calculate the total price based on the shopping list, generate a payment request, and complete the payment through identity authentication; Step 5: Security management: In the security management module, payment transactions are encrypted and recorded to ensure data security and the security of the payment process; Step 6: User interaction: Through the user interaction module, information is exchanged with users to provide a friendly operation interface and personalized services.

9. The method for implementing a smart shopping cart self-service payment system based on multi-sensor fusion according to claim 8, characterized in that: In the step three, a weighted Bayesian estimation algorithm is used to calculate the posterior probability that the product belongs to each category to improve the accuracy of product identification. The identity authentication in the step four includes multi-factor authentication, which combines passwords and biometrics to improve the security of the payment process. In the step five, blockchain technology is used to record payment transactions, and smart contracts are used to automatically execute transactions to improve the credibility and transparency of data. In the step six, collaborative filtering and deep learning algorithms are used to provide users with personalized product recommendations and promotional information. The entire system adopts a distributed data processing method that combines edge computing with cloud computing, performs real-time data processing on the edge side, and performs model training and global optimization on the cloud.

Images