Commodity identification and settlement method and system based on multi-modal feature fusion

Abstract

The invention discloses a commodity identification and settlement method and system based on multi-modal feature fusion, and the method comprises the steps: obtaining a customer ID, and building a customer-sensor mapping table; multi-modal data acquisition is carried out through a visual sensor, a weight sensor, a UWB base station and a radio frequency sensor, and the acquired data is extracted and processed; constructing a commodity feature vector, calculating a commodity existence probability P (A | F) by adopting a weighted Bayesian network, and dynamically adjusting a modal weight: judging the size of P (A | F), and when P (A | F) is greater than or equal to 0.9, judging that a customer takes the commodity by a system, and adding the commodity into a shopping cart; and a client reads the commodity label through an exit channel type RFID reader-writer and compares an identification result with a result obtained in the above step, if the identification result is consistent with the result obtained in the above step, money is automatically deducted, and if the identification result is not consistent with the result obtained in the above step, a manual channel rechecking is triggered. According to the method, the recognition accuracy is effectively improved, the system robustness is enhanced, the settlement process is simplified, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of smart retail and computer vision technology, and in particular to a product recognition and settlement method and system based on multimodal feature fusion. Background Technology

[0002] With the rapid development of technology, smart retail is gradually becoming a new trend in the retail industry. Traditional product identification and checkout methods mainly rely on manual operation, which suffers from problems such as low efficiency, high labor costs, and susceptibility to errors. In recent years, product identification technology based on computer vision has made some progress, such as using convolutional neural networks (CNNs) to extract features and classify product images. However, single image features have limitations when facing complex scenarios, such as diverse product placement angles, changing lighting conditions, and similar product appearances, which can easily lead to a decrease in recognition accuracy.

[0003] Furthermore, existing product recognition systems often only consider image information, neglecting other modalities such as product weight, shape, and barcodes. This multimodal information can provide richer features for product recognition, helping to improve accuracy and robustness. Summary of the Invention

[0004] To overcome the aforementioned problems in the existing technology, this invention proposes a product identification and settlement method and system based on multimodal feature fusion.

[0005] The technical solution adopted by this invention to solve its technical problem is: a commodity identification and settlement method based on multimodal feature fusion, comprising the following steps: Step 1: Obtain customer ID, establish customer-sensor mapping table, track customer location via UWB base station, and trigger activation of corresponding shelf sensors based on the location of the customer in the shelf area. Step 2: Multimodal data is acquired using visual sensors, weight sensors, UWB base stations, and radio frequency sensors, and the acquired data is extracted and processed. Step 3, construct the product feature vector F: F=[V,W,P,D,R,T] Where V represents the confidence level of the product category extracted from the data collected by the vision sensor, W represents the matching degree extracted from the data collected by the weight sensor, P represents the time the customer spends in front of the shelf, D represents the distance between the customer and the shelf, R represents the matching degree of the RFID tag, and T represents the timestamp. Step 4: Calculate the probability of item presence P(A|F) using a weighted Bayesian network, and dynamically adjust the modal weights: P(A|F)= ; Where P(Fi|A) represents the conditional probability, and P(A) represents the initial probability of the existence of item A. Represents conditional probability. This represents the initial probability that product B exists. Let F represent the set of all goods, where i represents the feature index in the product feature vector F, i=1,2,……6; and n represents the number of features in the product feature vector F, n=6. Step 5: Determine the value of P(A|F). When P(A|F)≥0.9, the system determines that the customer has taken the product and adds the product to the shopping cart. Step 6: The customer reads the product tag through the exit channel using an RFID reader and compares the identification result with the result obtained in steps 1-5. If they match, the payment is automatically deducted; otherwise, a manual review channel is triggered.

[0006] In the above-mentioned product recognition and settlement method based on multimodal feature fusion, in step 2, the visual sensor acquires product images and inputs the acquired images into the YOLOv8 model. The model outputs the category confidence and three-dimensional bounding box coordinates of each detected product. The weight sensor records the initial weight of the shelf in real time. When a customer picks up or puts down a product, the weight sensor calculates the weight change ΔW in real time. The UWB base station calculates the distance D between the customer and the shelf and the dwell time P, and uses a clustering algorithm to divide the customer activity area and associate it with the shelf number; The radio frequency sensor reads the product label in real time and outputs the label ID and signal strength.

[0007] The above-mentioned product recognition and settlement method based on multimodal feature fusion uses ResNet-50 to extract product color and texture features from images acquired by the visual sensor.

[0008] In the above-mentioned product recognition and settlement method based on multimodal feature fusion, the matching degree W extracted from the data collected by the weight sensor in step 3 is calculated using the following formula: W=ΔW 理论 / ΔW 实际 ΔW 理论 ΔW is calculated based on the known weight of the goods and the possible quantity to be taken. 实际 This refers to the weight change measured in real time.

[0009] In the above-mentioned product recognition and settlement method based on multimodal feature fusion, in step 2, the visual sensor, weight sensor, UWB base station, and radio frequency sensor periodically send heartbeat signals to the edge computing layer. If the edge computing layer does not receive a heartbeat signal from a certain sensor within a certain period of time, it determines that the sensor is faulty. The sensor weights are dynamically adjusted during the calculation in steps 3 and 4.

[0010] A product recognition and settlement system based on multimodal feature fusion employs the product recognition and settlement method based on multimodal feature fusion as described above. It includes a multimodal sensor layer, an edge computing layer, and a cloud management layer. The multimodal sensor layer is used for data acquisition, and the edge computing layer is used for extracting and processing the data acquired by the multimodal sensor layer. The cloud management layer is communicatively connected to the edge computing layer, and the data from the edge computing layer is uploaded to the cloud management layer in real time. An alarm is triggered in case of an anomaly. A database and parameters are configured on the cloud management layer.

[0011] The aforementioned product identification and settlement system based on multimodal feature fusion includes a multimodal sensor layer comprising a visual sensor installed on the top of the shelf, a weight sensor independently configured for each shelf unit, UWB base stations deployed at the four corners of the store, and a radio frequency sensor integrated into the exit aisle. The data collected by the visual sensor is used to extract product category confidence scores, the data collected by the weight sensor is used to calculate matching scores, the UWB base stations are used to calculate the distance between the customer and the shelf and the dwell time, and to perform motion trajectory analysis; the radio frequency sensor outputs the tag ID and signal strength.

[0012] The aforementioned commodity recognition and settlement system based on multimodal feature fusion also includes a status detection module, which detects the sensor status in real time and dynamically adjusts the weight of features extracted from the data collected by the sensor when the sensor fails.

[0013] The beneficial effects of this invention are: 1. Improved recognition accuracy: By fusing multimodal features such as image, weight, and space, the complementarity between different modal information is fully utilized, which effectively improves the accuracy of product recognition. Especially when facing complex scenarios, it can significantly reduce the false recognition rate and the missed recognition rate.

[0014] 2. Enhanced system robustness: Multimodal feature fusion makes the system more adaptable to factors such as product placement angle, lighting conditions, and similar product appearance, enabling it to operate stably in various environments and improving the system's reliability and stability.

[0015] 3. Simplified checkout process: Users no longer need to scan product barcodes one by one or manually enter product information. They only need to place the product in the corresponding collection area, and the system can automatically complete product identification and checkout, which greatly simplifies the checkout process and improves the user experience.

[0016] 4. Reduced Costs: The system reduces the need for manual operations, thus lowering labor costs. Simultaneously, an efficient product identification and checkout system improves operational efficiency in retail operations, reduces inventory backlog and losses, and further reduces operating costs. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a flowchart of the invention; Figure 3 This is a flowchart of the multimodal feature fusion process of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] This embodiment discloses a product recognition and automated settlement method and system based on multimodal feature fusion (vision, weight, spatial positioning, radio frequency signal), which is applicable to scenarios such as unmanned supermarkets, smart vending machines, and self-checkout. It aims to solve problems such as low efficiency of manual settlement, high product misidentification rate, and weak anti-theft and loss prevention capabilities in traditional retail.

[0020] The architecture diagram of the product recognition and settlement system based on multimodal feature fusion in this embodiment is as follows: Figure 1 As shown, the system includes a multimodal sensor layer, an edge computing layer, a cloud management layer, and a status detection module. The multimodal sensor layer is used for data acquisition, and the edge computing layer is used for extracting and processing the data acquired by the multimodal sensor layer. The cloud management layer is communicatively connected to the edge computing layer, and the edge computing layer uploads data to the cloud management layer in real time, triggering alarms in case of abnormalities. A database and parameters are configured on the cloud management layer. The status detection module monitors the sensor status in real time, and dynamically adjusts the weights of the extracted features from the sensor data when the sensor fails.

[0021] Edge computing layer: Deploy NVIDIA Jetson AGX Orin edge computing nodes (computing power ≥ 275 TOPS) to run multimodal feature fusion algorithms and behavior analysis models; Cloud-based management layer: Enables device management, data storage, and model iteration based on the Alibaba Cloud IoT platform.

[0022] In this embodiment, the multimodal sensor layer includes a vision sensor, a weight sensor, a UWB base station, and a radio frequency sensor. The data collected by the vision sensor is used to extract the confidence level of the product category, the data collected by the weight sensor is used to calculate the matching degree, the UWB base station is used to calculate the distance between the customer and the shelf and the dwell time, and to perform motion trajectory analysis; the radio frequency sensor outputs the tag ID and signal strength.

[0023] Visual sensor: Deployed on the top of the shelf, using an RGB-D camera (resolution ≥4K, depth accuracy ±2mm) to output the 3D point cloud and color features of the product in real time; Weight sensor: Each shelf unit is independently configured with a range of 0-10kg, an accuracy of ±1g, and a sampling frequency of ≥200Hz; UWB base stations: Deployed at the four corners of the store, with a coverage radius of ≥20m and a positioning accuracy of ±5cm, supporting spatial location tracking of customers and goods; Radio frequency sensor (RFID reader / writer in this embodiment): integrated into the exit channel, supports EPCC1G2 protocol, reading rate ≥80 tags / second, false reading rate <0.05%.

[0024] The system setup process in this embodiment includes the following steps: (I) Hardware Selection 1. Edge Computing Devices: Based on the system's requirements for computing performance and real-time operation, NVIDIA Jetson AGX Orin was selected as the edge computing node to meet the operational needs of feature extraction and fusion algorithms. It also possesses heat dissipation and protection capabilities to adapt to the complex environmental conditions of supermarkets.

[0025] 2. Sensor equipment Visual sensor: A high-resolution, low-latency RGB-D camera, such as the Intel RealSense D455, is selected. This camera can simultaneously acquire RGB images and depth maps, and has good anti-interference capabilities, making it suitable for use in shopping mall environments.

[0026] Weight sensor: Select a high-precision weight sensor with an appropriate measuring range, such as a Mettler Toledo weighing sensor. Determine the sensor's range based on the weight range of the goods in the store to ensure accurate weight measurement.

[0027] Positioning sensor: Employs a UWB positioning module, such as Decawave's DW1000 chip. UWB positioning technology offers advantages such as high precision and low power consumption, enabling real-time and accurate acquisition of customer location information.

[0028] Radio Frequency Sensor: A high-performance RFID reader, such as Impinj's Speedway R420, is selected. This reader features fast reading speed and high recognition accuracy, meeting the anti-theft requirements of shopping malls.

[0029] 3. Network Equipment: To ensure stable and reliable data transmission between edge computing nodes and the cloud management platform, high-performance routers and switches will be selected. Simultaneously, wireless communication technologies such as Wi-Fi 6 or 5G will be considered to meet the communication needs between devices.

[0030] (ii) Equipment installation and wiring 1. Edge Computing Node Installation: Install the NVIDIA Jetson AGX Orin edge computing nodes in the mall's server room or equipment room, ensuring good ventilation and proper heat dissipation. Simultaneously, provide a stable power supply to the devices and connect them to network equipment to enable communication with the cloud management platform.

[0031] 2. Sensor Installation Visual sensor installation: Mount the RGB-D camera above or to the side of the shelf, ensuring the camera covers the product area. During installation, adjust the camera's angle and height so that the acquired images and depth maps accurately reflect the location and characteristics of the products. Also, avoid exposing the camera to direct sunlight or obstructions.

[0032] Weight sensor installation: Install the weight sensor at the bottom of the shelf or on the product placement platform, ensuring a tight connection between the sensor and the shelf or platform for accurate weight measurement. During installation, pay attention to sensor calibration and zeroing to guarantee accurate measurement results.

[0033] Location sensor installation: Install UWB positioning base stations on the ceiling or walls of the shopping mall, ensuring coverage of the entire mall area. The installation height and spacing of the base stations should be rationally planned according to the mall's area and layout to guarantee positioning accuracy and stability. Simultaneously, have customers wear or carry UWB positioning tags to obtain their real-time location information.

[0034] RFID sensor installation: Install RFID readers near mall exits, entrances, or shelves, ensuring coverage of areas where merchandise may pass. During installation, adjust the reader's antenna angle and power to improve reading efficiency and identification accuracy.

[0035] Light intensity sensor installation: Install light intensity sensors in different areas of the shopping mall, such as next to shelves and in aisles, to obtain comprehensive light intensity data. The sensor installation location should avoid direct light or obstruction to ensure the accuracy of the measurement results.

[0036] 3. Cabling Requirements: During equipment installation, plan the cabling carefully to ensure neatness and safety. Power cables should meet safety standards and be properly grounded. Data cables should be shielded to reduce electromagnetic interference. Label the cables clearly for easy maintenance and management later.

[0037] (III) Software Deployment and Configuration Operating System Installation: Install a suitable operating system on the NVIDIA Jetson AGX Orin edge computing node. After installation, perform necessary configuration and optimization of the operating system to improve system stability and performance.

[0038] Algorithm and Model Deployment: Algorithms and models for visual feature extraction (YOLOv8, ResNet-50), weight data processing, spatial feature extraction (UWB localization, DBSCAN clustering), radio frequency feature processing, and multimodal feature fusion are deployed to edge computing nodes. During deployment, the algorithms are optimized and adjusted according to the performance of the hardware devices and the requirements of the algorithms to ensure that they can run efficiently on the edge computing nodes.

[0039] Cloud Management Platform Configuration: Create projects and devices on the Alibaba Cloud IoT platform and register the edge computing nodes to the cloud management platform. Configure data upload rules and alarm rules for the devices to ensure that data from the edge computing nodes can be uploaded to the cloud management platform in real time and that alarms are triggered promptly in case of anomalies. Simultaneously, configure parameters such as the product feature library, conditional probability table, and prior probability on the cloud management platform to support system decision-making.

[0040] Based on the above system, this embodiment also discloses a product recognition and settlement method based on multimodal feature fusion, such as... Figure 2 As shown, the specific steps include the following: I. Customer Entry and Sensor Network Initialization (a) Obtaining Customer Identity When customers enter the unmanned store, they can obtain a customer ID in two ways: 1. Facial Recognition: A high-definition facial recognition camera with a resolution of at least 1080P is installed at the store entrance, equipped with a fast face detection and recognition algorithm. It is linked to Alipay's facial authentication system; when a customer enters the camera's field of view, the system automatically captures their facial image and obtains the corresponding customer ID.

[0041] 2. Mobile phone scanning: A store-specific QR code is posted at the entrance. After customers scan the QR code with their mobile phones, they are redirected to the store's login page, where they can log in by entering their mobile phone number or account password. After successful login, the system synchronizes the customer ID corresponding to the account to the store's sensor network.

[0042] (ii) Establishment of customer-sensor mapping table After obtaining the customer ID, the system synchronizes the ID to the in-store sensor network to establish a customer-sensor mapping table. This mapping table is stored in the memory of the edge computing nodes and records the sensor information currently associated with each customer ID, including sensor type (vision, weight, UWB, RFID), sensor number, and sensor location (shelf number, aisle number, etc.). Through this mapping table, the system can achieve full tracking of customer behavior in the store.

[0043] (III) Activation of UWB base stations UWB base stations are deployed at the four corners of the store, covering a radius of ≥20m with a positioning accuracy of ±5cm. When a customer enters the store, the UWB base station tracks the customer's location in real time. Once a customer is detected entering a certain shelf area, the system will trigger the activation of the corresponding shelf's sensors, including visual sensors (RGB-D cameras), weight sensors, etc., to ensure timely collection of relevant data on customer-product interactions.

[0044] II. Data Collection and Processing During the Shopping Process (a) Multimodal data acquisition 1. Visual data acquisition Equipment: The vision sensor is deployed on the top of the shelf and uses an RGB-D camera (resolution ≥4K, depth accuracy ±2mm).

[0045] Data Acquisition Process: The camera outputs real-time 3D point cloud and color features of the products. After a customer enters the shelf area, the camera acquires image data at a frequency of 30 frames per second and transmits the data to the edge computing node.

[0046] 2. Weight data acquisition Equipment: Each shelf unit is independently equipped with a weight sensor with a range of 0-10kg, an accuracy of ±1g, and a sampling frequency of ≥200Hz.

[0047] Data Acquisition Process: The weight sensor records the initial weight W0 of the shelf in real time. When a customer picks up or puts down an item, the sensor calculates the weight change ΔW in real time and transmits the data to the edge computing node.

[0048] 3. Spatial Data Acquisition Equipment: UWB positioning base stations are deployed at the four corners of the store.

[0049] Data collection process: The distance D between the customer and the shelf, the dwell time P, and the movement trajectory are calculated using UWB positioning. The base station collects customer location data 5 times per second and transmits the data to the edge computing node.

[0050] 4. Radio Frequency Data Acquisition Equipment: The RFID reader is integrated into the exit channel, supports the EPCC1G2 protocol, has a reading rate of ≥80 tags / second, and a false reading rate of <0.05%.

[0051] Data collection process: During the customer's shopping process, the RFID reader is in standby mode. When the customer enters the exit channel, the reader reads the product tag in real time, outputs the tag ID and signal strength (RSSI), and transmits the data to the cloud management layer.

[0052] (ii) Multimodal data processing, such as Figure 3 As shown.

[0053] 1. Visual Feature Extraction Product contour detection: The acquired RGB images are input into the YOLOv8 model, and the model outputs the category confidence V (0-1) and 3D bounding box coordinates for each detected product. The YOLOv8 model runs on edge computing nodes and performs real-time detection on the images using a pre-trained weight file.

[0054] Color and texture feature extraction: ResNet-50 is used to extract the color and texture features of the product. The RGB image is input into the ResNet-50 model, and the output of the intermediate layer of the model is extracted as a 2048-dimensional feature vector.

[0055] 2. Weight Feature Extraction Weight change calculation: Calculate the weight change ΔW based on the real-time collected weight data.

[0056] Matching degree calculation: Output matching degree W = ΔW 理论 / ΔW 实际 ΔW 理论 ΔW is calculated based on the known weight of the goods and the possible quantity to be taken. 实际 This refers to the weight change measured in real time.

[0057] 3. Spatial Feature Extraction Distance and duration calculation: The distance D between the customer and the shelf and the dwell time P are calculated using UWB positioning data.

[0058] Movement trajectory analysis: The DBSCAN clustering algorithm is used to divide customer activity areas and associate them with shelf numbers. Based on customer location data, customer movement trajectories are calculated, and the points on the trajectories are clustered to determine the main shelf areas where customers are active.

[0059] 4. Radio Frequency Feature Extraction Tag information reading: The RFID reader reads product tags in real time and outputs the tag ID and signal strength (RSSI). The tag ID is processed at the edge computing node to determine whether it matches the product tags in the system.

[0060] (III) Feature-level fusion Construct the product feature vector F: F=[V,W,P,D,R,T] Where: R: RFID tag matching degree (R=1 if a tag is detected; otherwise R=0). During the customer's shopping process, the value of R is updated in real time based on the reading results of the RFID reader.

[0061] T: Timestamp (milliseconds), used for time alignment of multimodal data. The timestamp is recorded simultaneously when collecting data from each modality to ensure accurate correspondence between data from different modalities.

[0062] (iv) Decision-level integration The probability of item presence, P(A|F), is calculated using a weighted Bayesian network: P(A|F)= Conditional probability P(Fi|A): Obtained through training on historical data. For example, by analyzing a large amount of product picking data, the probability that "when product A is picked up, the visual confidence level V is usually > 0.8" is calculated and used as the value of P(V|A).

[0063] Prior probability P(A): Represents the initial probability of the existence of product A, which is dynamically adjusted based on product inventory. When product inventory is sufficient, the value of P(A) is higher; when product inventory is low, the value of P(A) decreases accordingly.

[0064] The denominator is a normalization term to ensure that the sum of probabilities is 1. B⊆Θ iterates through all possible sets of goods B (Θ is the set of all goods). For each B, its corresponding joint probability is calculated and then summed.

[0065] Dynamic weight allocation: The weights of each mode are adjusted in real time according to the dynamic adjustment conditions of different modes. The specific dynamic weight allocation is shown in Table 1.

[0066] Table 1 Dynamic Weight Allocation (v) Product-Customer Association and Shopping Cart Update 1. Product Pickup Detection: When P(A|F)≥0.9, the system determines that the customer has picked up the product and adds it to the shopping cart. The shopping cart information is stored in the memory of the edge computing node, recording information such as customer ID, product ID, and product quantity.

[0067] 2. Product Return Detection: If a customer returns a product, it triggers a reverse change in the weight sensor. When the weight sensor detects an increase in weight that matches the weight of the previously picked-up product, the system updates the shopping cart and removes the product from the cart.

[0068] (vi) Behavioral Analysis Predicting customer intent using an LSTM model. Customer behavior data (such as picking up, examining, and putting back actions) is used as input to train the LSTM model, enabling it to predict the customer's next action. For example, if a customer's "picking up-examining-putting back" action sequence is detected, the model predicts the customer may not be interested in the item; if a "picking up-putting in shopping bag" action sequence is detected, the model predicts the customer may purchase the item. If abnormal behavior is detected (such as quickly hiding items), RFID channel pre-verification is triggered.

[0069] III. Check-out and RFID secondary verification (a) RFID secondary verification 1. Channel Reading: As customers pass through the exit channel, the RFID reader reads the product tags. The reader operates in high-speed reading mode to ensure that all product tags can be read quickly and accurately.

[0070] 2. Result Comparison: The RFID identification result is compared with the multimodal fusion result. If they match, the system automatically deducts the payment; if they do not match, a manual review process is triggered.

[0071] (ii) Data Cleaning After checkout, the system deletes the customer's facial image and shopping history, retaining only the transaction record. The transaction record is stored in a cloud-based management database, including information such as customer ID, product ID, product quantity, transaction amount, and transaction time, for subsequent querying and statistical analysis.

[0072] IV. Anomaly Handling Mechanism (a) Sensor Fault Handling 1. Redundancy Design: The system employs a redundancy design, dynamically adjusting the weights when a sensor fails. For example, if the vision sensor fails, the weight is increased to 0.6 to ensure the system continues to operate normally.

[0073] 2. Status Detection: Sensor status is monitored in real time using the Heartbeat Protocol. Each sensor periodically sends a heartbeat signal to the edge computing node. If the edge computing node does not receive a heartbeat signal from a sensor within a certain period of time, the sensor is considered to have failed.

[0074] (ii) Data Conflict Handling 1. Conflict Matrix Establishment: A conflict matrix is ​​established, prioritizing the use of high-confidence modal data. For example, when the visual confidence level is >0.9, weight sensor fluctuations are ignored. The conflict matrix is ​​stored in the memory of the edge computing nodes, recording the conflict relationships and processing strategies between different modal data.

[0075] Manual intervention: If the conflict cannot be resolved, manual intervention is triggered. The system sends an alarm message to administrators, who can then manually assess and handle the conflict by reviewing relevant data and video surveillance.

[0076] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A product recognition and settlement method based on multimodal feature fusion, characterized in that, Includes the following steps: Step 1: Obtain customer ID, establish customer-sensor mapping table, track customer location via UWB base station, and trigger activation of corresponding shelf sensors based on the location of the customer in the shelf area. Step 2: Multimodal data is acquired using visual sensors, weight sensors, UWB base stations, and radio frequency sensors, and the acquired data is extracted and processed. Step 3, construct the product feature vector F: F=[V,W,P,D,R,T] Where V represents the confidence level of the product category extracted from the data collected by the vision sensor, W represents the matching degree extracted from the data collected by the weight sensor, P represents the time the customer spends in front of the shelf, D represents the distance between the customer and the shelf, R represents the matching degree of the RFID tag, and T represents the timestamp. Step 4: Calculate the probability of item presence P(A|F) using a weighted Bayesian network, and dynamically adjust the modal weights: P(A∣F)= ; Where P(Fi|A) represents the conditional probability, and P(A) represents the initial probability of the existence of item A. Represents conditional probability. This represents the initial probability that product B exists. Let F represent the set of all goods, where i represents the feature index in the product feature vector F, i=1,2,……6; and n represents the number of features in the product feature vector F, n=6. Step 5: Determine the value of P(A|F). When P(A|F)≥0.9, the system determines that the customer has taken the product and adds the product to the shopping cart. Step 6: The customer reads the product tag through the exit channel using an RFID reader and compares the identification result with the result obtained in steps 1-5. If they match, the payment is automatically deducted; otherwise, a manual review channel is triggered.

2. The product identification and settlement method based on multimodal feature fusion according to claim 1, characterized in that, In step 2, the visual sensor acquires images of the products and inputs the acquired images into the YOLOv8 model. The model outputs the category confidence score and three-dimensional bounding box coordinates of each detected product. The weight sensor records the initial weight of the shelf in real time. When a customer picks up or puts down a product, the weight sensor calculates the weight change ΔW in real time. The UWB base station calculates the distance D between the customer and the shelf and the dwell time P, and uses a clustering algorithm to divide the customer activity area and associate it with the shelf number; The radio frequency sensor reads the product label in real time and outputs the label ID and signal strength.

3. The product identification and settlement method based on multimodal feature fusion according to claim 2, characterized in that, The images acquired by the visual sensor are processed using ResNet-50 to extract the color and texture features of the products.

4. The product identification and settlement method based on multimodal feature fusion according to claim 1, characterized in that, The formula for calculating the matching degree W obtained from the data collected by the weight sensor in step 3 is as follows: W=ΔW 理论 / ΔW 实际 ΔW 理论 ΔW is calculated based on the known weight of the goods and the possible quantity to be taken. 实际 This refers to the weight change measured in real time.

5. The product identification and settlement method based on multimodal feature fusion according to claim 1, characterized in that, In step 2, the visual sensor, weight sensor, UWB base station, and radio frequency sensor periodically send heartbeat signals to the edge computing layer. If the edge computing layer does not receive a heartbeat signal from a certain sensor within a certain period of time, it determines that the sensor is faulty, and the sensor weights are dynamically adjusted during the calculation in steps 3 and 4.

6. A commodity recognition and settlement system based on multimodal feature fusion, characterized in that, A product identification and settlement method based on multimodal feature fusion as described in any one of claims 1-5 includes a multimodal sensor layer, an edge computing layer, and a cloud management layer. The multimodal sensor layer is used for data acquisition, and the edge computing layer is used for extracting and processing the data acquired by the multimodal sensor layer. The cloud management layer is communicatively connected to the edge computing layer, and the data from the edge computing layer is uploaded to the cloud management layer in real time. An alarm is triggered when an abnormal situation occurs. A database and parameters are configured on the cloud management layer.

7. A commodity recognition and settlement system based on multimodal feature fusion according to claim 6, characterized in that, The multimodal sensor layer includes a vision sensor installed on top of the shelf, a weight sensor independently configured for each shelf unit, UWB base stations deployed at the four corners of the store, and a radio frequency sensor integrated into the exit aisle. The data collected by the vision sensor is used to extract the confidence level of the product category, the data collected by the weight sensor is used to calculate the matching degree, the UWB base station is used to calculate the distance between the customer and the shelf and the dwell time, and to perform motion trajectory analysis; the radio frequency sensor outputs the tag ID and signal strength.

8. A commodity recognition and settlement system based on multimodal feature fusion according to claim 6, characterized in that, It also includes a status detection module, which monitors the sensor status in real time and dynamically adjusts the weights of features extracted from the data collected by the sensor when the sensor fails.

Images