Supermarket cash register commodity intelligent settlement method and system based on multi-modal identification

Abstract

The invention discloses a supermarket cash register commodity intelligent settlement method and system based on multi-mode identification. According to the invention, the accuracy and environmental adaptability of commodity identification are significantly improved through a dynamic multi-modal weight distribution algorithm. The system can automatically adjust the contribution proportion of modes such as image RFID weight according to real-time environment parameters, the identification stability is ensured by enhancing the weight of a reliable mode in complex scenes such as insufficient light and signal interference, and the identification deviation caused by failure of single data is reduced by integrating the advantages of each mode through the multi-mode feature fusion technology. And the cross validation mechanism further reduces the error recognition rate, so that the judgment of the commodity category and price is more accurate, and the robustness of the settlement system and the user experience are effectively enhanced. The dynamic weight compensation function realizes cooperation between modes, and when a certain mode is abnormal, other modes automatically improve the weight to maintain continuous operation of the system, so that the manual intervention frequency is reduced, and the recognition efficiency and adaptability of the system are continuously improved in long-term use.

Description

Technical Field

[0001] This invention belongs to the field of supermarket checkout technology, specifically a smart settlement method and system for supermarket checkout machines based on multimodal recognition. Background Technology

[0002] Smart checkout in supermarkets refers to a modern retail payment method that leverages technologies such as computer vision, barcode recognition, RFID, and sensor fusion to achieve rapid product identification, automatic pricing, and intelligent checkout. This system automatically reads product information using smart cameras or scanning devices, eliminating the need for manual input and significantly improving checkout efficiency and accuracy. Simultaneously, the smart checkout system supports multiple payment methods, including QR code payment, facial recognition payment, and contactless payment, optimizing the customer checkout experience. Furthermore, the system can synchronize inventory data in real time, automatically update product prices and promotional information, and seamlessly integrate with membership systems and ERP systems to achieve integrated management of sales data and user behavior. With the deepening application of artificial intelligence and IoT technologies, smart checkout in supermarkets is gradually developing towards unmanned, self-service, and personalized operations, becoming a crucial infrastructure driving the digital transformation of the retail industry.

[0003] However, existing technologies often rely on a single recognition modality or fixed weight allocation, which can easily lead to recognition failure in complex environments such as changes in light and signal interference. The lack of a cross-modal dynamic compensation mechanism results in insufficient system robustness. In the settlement process, manual intervention is frequent and the matching efficiency of preferential rules is low. Overall adaptability and user experience need to be improved. Summary of the Invention

[0004] The purpose of this invention is to provide a smart settlement method and system for supermarket cash registers based on multimodal recognition in order to solve the problems mentioned above.

[0005] The technical solution adopted in this invention is as follows: a smart settlement method for supermarket cash registers based on multimodal recognition, the method comprising the following steps:

[0006] S1: System startup and multimodal device calibration, initializing modules such as camera, RFID reader, and weight sensor and performing self-test. The calibration parameters will be synchronized to the data acquisition module of S2 to ensure that the devices work together.

[0007] S2: When goods are placed in the checkout area, multimodal data acquisition is triggered. After the infrared sensor detects the object, image capture, RFID signal reading, and weight measurement are initiated. The collected raw data is transmitted to S3 in real time for preprocessing.

[0008] S3: Perform multimodal data preprocessing and spatiotemporal alignment, denoise the image, filter the RFID signal, calibrate the weight data, and integrate them into a unified data frame according to the timestamp. The processing results are sent to S4 for feature extraction.

[0009] S4: Perform dynamic multimodal weight allocation and commodity feature extraction. Automatically adjust the recognition weight of image RFID weight and other modalities according to environmental parameters. For example, enhance RFID weight when there is insufficient light. The extracted feature vector is fused and passed to S5 for matching.

[0010] S5: Perform intelligent product recognition and database matching. Compare the feature vector generated in S4 with the product database, and determine the product category and price based on the weight allocation results. The matching results are synchronized to S6 and S7.

[0011] S6: When S5 encounters recognition ambiguity, auxiliary data such as residual information of weight barcode are called for verification. If it cannot be confirmed, the user is prompted to intervene and the corrected result is fed back to S5 for rematch.

[0012] S7: Settlement amount is calculated in real time and matched with discount rules. The total price is calculated based on the product price confirmed in S5, combined with promotion and membership rules, and the result is pushed to the payment interface in S8.

[0013] S8: Performs multi-channel payment integration and transaction confirmation, supports payment methods such as QR code scanning, face recognition, and NFC, and synchronizes transaction information to S9 and updates the inventory system after payment is completed;

[0014] S9: Perform transaction data synchronization and user behavior analysis, store the transaction data in the database and analyze user purchasing habits. The analysis results are used to optimize the dynamic weight allocation model of S4.

[0015] In a preferred embodiment, in step S1, the system startup phase first performs a multimodal hardware self-test. Through a built-in diagnostic program, core modules such as the camera, RFID reader, and weight sensor are activated sequentially to check the communication links and data interfaces of each device. During calibration, the system collects reference environmental parameters, such as the white balance parameters of the camera, the signal gain value of the RFID reader, and the zero-point offset of the weight sensor. These parameters are synchronized to the data acquisition module in real time via the internal bus to ensure that the sampling frequency and accuracy of each device remain consistent in subsequent data acquisition phases. The calibration cycle is set to once every 24 hours. If a device malfunction is detected, such as camera lens contamination or RFID antenna failure, the system will automatically trigger a maintenance prompt and suspend the settlement function until calibration is complete.

[0016] In a preferred embodiment, in step S2, an infrared array sensor deployed at the edge of the checkout area monitors the entry of objects in real time. Upon detecting the placement of a product, it immediately sends a trigger signal to the central controller. The controller then simultaneously activates three sub-modules: image acquisition, RFID signal reading, and weight measurement. The camera captures three consecutive frames within 0.5 seconds to ensure multi-angle image capture of the product. The RFID reader continuously scans the tag at 1-second intervals until the signal stabilizes. The weight sensor begins sampling after the product is placed stably, with a sampling frequency of 10 times per second, and the average value is taken as valid data. All raw data is transmitted to the preprocessing module via a high-speed data bus, with the transmission delay controlled within 200 milliseconds to ensure data timeliness.

[0017] In a preferred embodiment, in step S3, the preprocessing stage first performs grayscale processing and edge enhancement on the image data, and filters out interference caused by changes in ambient light using a dynamic thresholding algorithm; the RFID signal undergoes bandpass filtering to remove high-frequency noise, and key information such as signal strength and tag ID is extracted; the weight data is processed using a sliding window algorithm to eliminate instantaneous fluctuations and retain stable weight readings. The system adds a timestamp accurate to the millisecond level to each data stream, and integrates the three modalities into a unified data frame based on the timestamp, ensuring that the image features, RFID information, and weight data of the same product are aligned in the time dimension. The processed data frame is verified to ensure integrity; if data is missing, such as an unidentified RFID tag, it is automatically marked as needing to be supplemented.

[0018] In a preferred embodiment, in step S4, during the product feature extraction stage, the system collects environmental parameters in real time, including: light intensity I, RFID signal-to-noise ratio S, and weight sensor stability W, and adjusts the contribution ratio of the three modes of image, RFID, and weight through a dynamic weight allocation model.

[0019] The algorithm first normalizes the environmental parameters: light intensity I is mapped to the [0,1] interval (I=0 represents complete darkness, I=1 represents a bright light environment), RFID signal-to-noise ratio SS is normalized to [0,1] (S=0 represents signal loss, S=1 represents a clear signal), and weight stability WW is normalized to [0,1] through variance calculation (W=1 represents no fluctuation in weight data). Then, the basic weights are calculated based on the modal reliability model: image modal weight w. img Initially set to I, RFID weight w rfid Let S be the weight w weight Let W be the modality. To avoid recognition interruption due to the failure of a single modality, a cross-compensation mechanism is introduced:

[0020] When any modal weight falls below a threshold θ (default θ = 0.3), the weights of other modalities are increased proportionally. Finally, all weights are normalized using the Softmax function to ensure that ∑(w img ,w rfid ,w weigh t) = 1.

[0021] In the feature fusion stage, the feature vectors extracted from each modality are: image features F img ∈R256, RFID feature F rfid ∈R64, weight characteristic F weight ∈R16, summed according to dynamic weights, to generate a fused feature vector:

[0022] F fusion =w img ·F img +w rfid ·F rfid +w weight ·F weight The data is then transmitted to the product recognition module for matching.

[0023] When the reliability of a certain mode is lower than the threshold, cross-modal weight compensation is achieved through the following formula:

[0024]

[0025] in:

[0026] wm′ is the compensated weight of mode m, m∈{img,rfid,weight};

[0027] wm represents the basic weight of modality mm;

[0028] α is the compensation coefficient, with a default value of 0.5, which controls the compensation intensity;

[0029] n is the index of other modalities.

[0030] In a preferred embodiment, in step S5, the fused feature vector output by the feature extraction module is first retrieved from the product feature database. The database uses a hierarchical index structure to categorize products, first matching broad category features and then gradually refining to specific products. During the matching process, the system adjusts the priority of each modality feature based on the dynamic weight allocation result. For example, when RFID has a higher weight, tag ID information is compared first; when image has a higher weight, product appearance features are matched first. The matching algorithm determines the candidate product list by calculating the cosine similarity between feature vectors. A similarity threshold of 0.85 is set; if the similarity is lower than this value, the cross-validation process begins. After determining the product category, the system retrieves real-time price information from the price database and synchronizes the product ID and price to the cross-validation module and the settlement module.

[0031] In a preferred embodiment, in step S6, when the similarity returned by the product identification module is lower than a threshold, the system automatically initiates a cross-validation process, comparing the weight data with the standard weight range in the product database. If the weight deviation is within the allowable range, the matching priority of the product is increased. If an RFID signal exists but no tag information is matched, the system triggers secondary image recognition to focus on capturing the barcode area on the product packaging, extracting the encoding information through optical character recognition technology to supplement the match. When all verification methods fail to confirm the product, the checkout interface displays a product image and feature description, prompting the cashier to manually enter the product code. The corrected product information is fed back to the identification module in real time to update the feature database and optimize subsequent matching accuracy.

[0032] In a preferred embodiment, in step S7, after receiving the product identification result, the settlement module calculates the base amount based on the product quantity and unit price, and simultaneously calls the discount rule engine to match currently valid promotional activities. The system queries the member database in real time to verify whether the user is a member, automatically applies member discounts and accumulates points. If the product is within the scope of a limited-time promotion, a promotional discount is applied, following the rule of single-item discounts first, followed by full-amount discounts. During the amount calculation process, the total settlement amount is updated in real time and details are displayed, including product name, unit price, quantity, discount amount, etc. The calculation result is refreshed every 0.5 seconds to ensure that the user can keep track of the payment amount in real time. After the final amount is confirmed, it is pushed to the payment interface for the user to complete the payment.

[0033] In a preferred embodiment, in step S8, the payment interface supports multiple payment methods such as QR code payment, facial recognition payment, and NFC payment. After the user selects a payment method, the system calls the corresponding payment interface. The interface communication uses an encryption protocol to ensure transaction security. After payment is completed, a transaction success signal is received from the payment platform. The central controller synchronously writes the transaction information into the database, including data such as product details, payment amount, transaction time, and payment method, and triggers the inventory management system to update the inventory quantity of the corresponding product. After the transaction is confirmed, the indicator light in the settlement area changes from red to green, prompting the user to take the product. At the same time, a transaction receipt is printed, containing all the key data of this transaction for the user to verify.

[0034] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0035] 1. In this invention, the accuracy and environmental adaptability of product identification are significantly improved through a dynamic multimodal weight allocation algorithm. The system can automatically adjust the contribution ratio of modalities such as image RFID weight according to real-time environmental parameters. In complex scenarios such as insufficient light and signal interference, the system ensures identification stability by enhancing the weight of reliable modalities. The multimodal feature fusion technology combines the advantages of each modality to reduce identification deviation caused by single data failures, and the cross-validation mechanism further reduces the false recognition rate, making the judgment of product category and price more accurate.

[0036] 2. This invention effectively enhances the robustness and user experience of the settlement system. The dynamic weight compensation function enables collaborative cooperation between modalities. When one modality malfunctions, other modalities automatically increase their weights to maintain continuous system operation, reducing the frequency of manual intervention. The real-time calculation and preferential rule matching mechanism accelerates the settlement process. Multi-channel payment integration meets the payment habits of different users. Continuous analysis of transaction data optimizes the dynamic weight model, enabling the system to continuously improve its recognition efficiency and adaptability over long-term use. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the process principle of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] Example:

[0040] Reference Figure 1 A smart settlement method for supermarket cash registers based on multimodal recognition, the method includes the following steps:

[0041] S1: System startup and multimodal device calibration, initializing modules such as camera, RFID reader, and weight sensor and performing self-test. The calibration parameters will be synchronized to the data acquisition module of S2 to ensure that the devices work together.

[0042] S2: When goods are placed in the checkout area, multimodal data acquisition is triggered. After the infrared sensor detects the object, image capture, RFID signal reading, and weight measurement are initiated. The collected raw data is transmitted to S3 in real time for preprocessing.

[0043] S3: Perform multimodal data preprocessing and spatiotemporal alignment, denoise the image, filter the RFID signal, calibrate the weight data, and integrate them into a unified data frame according to the timestamp. The processing results are sent to S4 for feature extraction.

[0044] S4: Perform dynamic multimodal weight allocation and commodity feature extraction. Automatically adjust the recognition weight of image RFID weight and other modalities according to environmental parameters. For example, enhance RFID weight when there is insufficient light. The extracted feature vector is fused and passed to S5 for matching.

[0045] S5: Perform intelligent product recognition and database matching. Compare the feature vector generated in S4 with the product database, and determine the product category and price based on the weight allocation results. The matching results are synchronized to S6 and S7.

[0046] S6: When S5 encounters recognition ambiguity, auxiliary data such as residual information of weight barcode are called for verification. If it cannot be confirmed, the user is prompted to intervene and the corrected result is fed back to S5 for rematch.

[0047] S7: Settlement amount is calculated in real time and matched with discount rules. The total price is calculated based on the product price confirmed in S5, combined with promotion and membership rules, and the result is pushed to the payment interface in S8.

[0048] S8: Performs multi-channel payment integration and transaction confirmation, supports payment methods such as QR code scanning, face recognition, and NFC, and synchronizes transaction information to S9 and updates the inventory system after payment is completed;

[0049] S9: Perform transaction data synchronization and user behavior analysis, store the transaction data in the database and analyze user purchasing habits. The analysis results are used to optimize the dynamic weight allocation model of S4.

[0050] In step S1, the system startup phase first performs a multimodal hardware self-test. Through the built-in diagnostic program, core modules such as the camera, RFID reader, and weight sensor are activated sequentially to check the communication links and data interfaces of each device. During calibration, the system collects baseline environmental parameters, such as the camera's white balance parameters, the RFID reader's signal gain, and the weight sensor's zero-point offset. These parameters are synchronized to the data acquisition module in real time via the internal bus to ensure consistent sampling frequency and accuracy across devices in subsequent data acquisition phases. The calibration cycle is set to once every 24 hours. If a device malfunction is detected, such as camera lens contamination or RFID antenna failure, the system will automatically trigger a maintenance prompt and suspend the settlement function until calibration is complete.

[0051] In step S2, infrared array sensors deployed at the edge of the checkout area monitor the entry of objects in real time. Upon detecting the placement of an item, a trigger signal is immediately sent to the central controller, which simultaneously activates three sub-modules: image acquisition, RFID signal reading, and weight measurement. The camera captures three consecutive frames within 0.5 seconds to ensure multi-angle image capture of the item. The RFID reader continuously scans the tag at 1-second intervals until the signal stabilizes. The weight sensor begins sampling after the item is placed stably, with a sampling frequency of 10 times per second, and the average value is taken as valid data. All raw data is transmitted to the preprocessing module via a high-speed data bus, with transmission latency controlled within 200 milliseconds to ensure data timeliness.

[0052] In step S3, the preprocessing stage first performs grayscale conversion and edge enhancement on the image data, and filters out interference caused by changes in ambient light using a dynamic thresholding algorithm. The RFID signal undergoes bandpass filtering to remove high-frequency noise, and key information such as signal strength and tag ID is extracted. Weight data is processed using a sliding window algorithm to eliminate instantaneous fluctuations, retaining stable weight readings. The system adds millisecond-accurate timestamps to each data stream, and integrates the three modalities into a unified data frame based on these timestamps, ensuring that the image features, RFID information, and weight data of the same product are aligned in the time dimension. The processed data frame is verified for integrity; if data is missing, such as an unidentified RFID tag, it is automatically marked as needing to be supplemented.

[0053] In step S4, during the product feature extraction stage, the system collects environmental parameters in real time, including: light intensity I, RFID signal-to-noise ratio S, and weight sensor stability W. The contribution ratios of the three modes—image, RFID, and weight—are adjusted through a dynamic weight allocation model.

[0054] The algorithm first normalizes the environmental parameters: light intensity I is mapped to the [0,1] interval (I=0 represents complete darkness, I=1 represents a bright light environment), RFID signal-to-noise ratio SS is normalized to [0,1] (S=0 represents signal loss, S=1 represents a clear signal), and weight stability WW is normalized to [0,1] through variance calculation (W=1 represents no fluctuation in weight data). Then, the basic weights are calculated based on the modal reliability model: image modal weight w. img Initially set to I, RFID weight w rfid Let S be the weight w weight Let W be the modality. To avoid recognition interruption due to the failure of a single modality, a cross-compensation mechanism is introduced:

[0055] When any modal weight falls below a threshold θ (default θ = 0.3), the weights of other modalities are increased proportionally. Finally, all weights are normalized using the Softmax function to ensure that ∑(w img ,w rfid ,wweigh t) = 1.

[0056] In the feature fusion stage, the feature vectors extracted from each modality are: image features F img ∈R256, RFID feature F rfid ∈R64, weight characteristic F weight ∈R16, summed according to dynamic weights, to generate a fused feature vector:

[0057] F fusion =w img ·F img +w rfid ·F rfid +w weight ·F weight The data is then transmitted to the product recognition module for matching.

[0058] When the reliability of a certain mode is lower than the threshold, cross-modal weight compensation is achieved through the following formula:

[0059]

[0060] in:

[0061] wm′ is the compensated weight of mode m, m∈{img,rfid,weight};

[0062] wm represents the basic weight of modality mm;

[0063] α is the compensation coefficient, with a default value of 0.5, which controls the compensation intensity;

[0064] n is the index of other modalities;

[0065] In step S5, the fused feature vector output by the feature extraction module is first retrieved from the product feature database. The database uses a hierarchical index structure to categorize products, first matching broad category features and then gradually refining to specific products. During the matching process, the system adjusts the priority of each modality feature based on the dynamic weight allocation results. For example, when RFID has a higher weight, tag ID information is compared first; when image has a higher weight, product appearance features are prioritized. The matching algorithm determines the candidate product list by calculating the cosine similarity between feature vectors. A similarity threshold of 0.85 is set; values ​​below this value trigger cross-validation. After determining the product category, the system retrieves real-time price information from the price database and synchronizes the product ID and price to the cross-validation module and the settlement module.

[0066] In step S6, when the similarity returned by the product identification module is lower than the threshold, the system automatically initiates a cross-validation process, comparing the weight data with the standard weight range in the product database. If the weight deviation is within the allowable range, the matching priority of the product is increased. If an RFID signal exists but no tag information is matched, the system triggers secondary image recognition to focus on capturing the barcode area on the product packaging, extracting the encoding information through optical character recognition technology to supplement the match. When all verification methods fail to confirm the product, the checkout interface displays a product image and feature description, prompting the cashier to manually enter the product code. The corrected product information is fed back to the identification module in real time to update the feature database and optimize subsequent matching accuracy.

[0067] In step S7, after receiving the product identification result, the settlement module calculates the base amount based on the product quantity and unit price, and simultaneously calls the discount rule engine to match currently valid promotional activities. The system queries the member database in real time to verify whether the user is a member, automatically applies member discounts and accumulates points. If the product is within the scope of a limited-time promotion, the promotional discount is applied, following the rule of single-item discount first, then full-amount discount. During the amount calculation process, the total settlement amount is updated in real time and details are displayed, including product name, unit price, quantity, discount amount, etc. The calculation result is refreshed every 0.5 seconds to ensure that the user can keep track of the payment amount in real time. After the final amount is confirmed, it is pushed to the payment interface to wait for the user to complete the payment.

[0068] In step S8, the payment interface supports multiple payment methods, including QR code payment, facial recognition payment, and NFC payment. After the user selects a payment method, the system calls the corresponding payment interface. The interface communication uses an encryption protocol to ensure transaction security. After payment is completed, a transaction success signal is received from the payment platform. The central controller synchronously writes the transaction information into the database, including data such as product details, payment amount, transaction time, and payment method, and triggers the inventory management system to update the inventory quantity of the corresponding product. After the transaction is confirmed, the indicator light in the settlement area changes from red to green, prompting the user to take the product. At the same time, a transaction receipt is printed, containing all the key data of this transaction for the user to verify.

[0069] A smart settlement system for supermarket cash registers based on multimodal recognition is provided, which is applied to the aforementioned smart settlement method for supermarket cash registers based on multimodal recognition.

[0070] From the above, we can conclude that:

[0071] In this invention, a dynamic multimodal weight allocation algorithm significantly improves the accuracy and environmental adaptability of product identification. The system can automatically adjust the contribution ratio of modalities such as image RFID weight based on real-time environmental parameters. In complex scenarios such as insufficient light and signal interference, the system ensures identification stability by enhancing the weight of reliable modalities. The multimodal feature fusion technology combines the advantages of each modality to reduce identification deviations caused by single data failures, and the cross-validation mechanism further reduces the false recognition rate, making the judgment of product category and price more accurate.

[0072] This invention effectively enhances the robustness and user experience of the settlement system. The dynamic weight compensation function enables inter-modal collaboration; when one modality malfunctions, other modalities automatically increase their weights to maintain continuous system operation, reducing the frequency of manual intervention. Real-time calculation and preferential rule matching mechanisms accelerate the settlement process. Multi-channel payment integration caters to different users' payment habits. Continuous analysis of transaction data optimizes the dynamic weight model, enabling the system to continuously improve its recognition efficiency and adaptability over long-term use.

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smart settlement method for supermarket cash registers based on multimodal recognition, characterized in that: The method includes the following steps: S1: System startup and multimodal device calibration, initializing modules such as camera, RFID reader, and weight sensor and performing self-test. The calibration parameters will be synchronized to the data acquisition module of S2 to ensure that the devices work together. S2: When goods are placed in the checkout area, multimodal data acquisition is triggered. After the infrared sensor detects the object, image capture, RFID signal reading, and weight measurement are initiated. The collected raw data is transmitted to S3 in real time for preprocessing. S3: Perform multimodal data preprocessing and spatiotemporal alignment, denoise the image, filter the RFID signal, calibrate the weight data, and integrate them into a unified data frame according to the timestamp. The processing results are sent to S4 for feature extraction. S4: Perform dynamic multimodal weight allocation and commodity feature extraction. Automatically adjust the recognition weight of image RFID weight and other modalities according to environmental parameters. For example, enhance RFID weight when there is insufficient light. The extracted feature vector is fused and passed to S5 for matching. S5: Perform intelligent product recognition and database matching. Compare the feature vector generated in S4 with the product database, and determine the product category and price based on the weight allocation results. The matching results are synchronized to S6 and S7. S6: When S5 encounters recognition ambiguity, auxiliary data such as residual information of weight barcode are called for verification. If it cannot be confirmed, the user is prompted to intervene and the corrected result is fed back to S5 for rematch. S7: Settlement amount is calculated in real time and matched with discount rules. The total price is calculated based on the product price confirmed in S5, combined with promotion and membership rules, and the result is pushed to the payment interface in S8. S8: Integrates and confirms multi-channel payments, supports payment methods such as QR code scanning, facial recognition, and NFC, and synchronizes transaction information to S9 and updates the inventory system after payment is completed; S9: Perform transaction data synchronization and user behavior analysis, store the transaction data in the database and analyze user purchasing habits. The analysis results are used to optimize the dynamic weight allocation model of S4.

2. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S1, the system startup phase first performs a multimodal hardware self-test, and activates core modules such as cameras, RFID readers, and weight sensors in sequence through the built-in diagnostic program to check whether the communication links of each device are smooth and whether the data interfaces respond normally.

3. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S2, an infrared array sensor deployed at the edge of the checkout area monitors the entry of objects in real time. When a product is detected being placed, a trigger signal is immediately sent to the central controller. The controller then simultaneously activates three sub-modules: image acquisition, RFID signal reading, and weight measurement. The camera completes three consecutive frame captures within 0.5 seconds to ensure the capture of multi-angle images of the product. The RFID reader continuously scans the tag at 1-second intervals until the signal stabilizes. The weight sensor begins sampling after the product is placed stably, with a sampling frequency of 10 times per second and the average value is taken as valid data.

4. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S3, the preprocessing stage first performs grayscale processing and edge enhancement on the image data, and filters out interference caused by changes in ambient light through a dynamic threshold algorithm; the RFID signal is filtered by bandpass filtering to remove high-frequency noise, and key information such as signal strength and tag ID is extracted; the weight data is processed by a sliding window algorithm to remove instantaneous fluctuation values ​​and retain stable weight readings.

5. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S4, during the product feature extraction stage, the system collects environmental parameters in real time, including: light intensity I, RFID signal signal-to-noise ratio S, and weight sensor stability W. The contribution ratios of the three modes—image, RFID, and weight—are adjusted through a dynamic weight allocation model. The algorithm first normalizes the environmental parameters: light intensity I is mapped to the [0,1] interval, RFID signal-to-noise ratio SS is normalized to [0,1], and weight stability WW is normalized to [0,1] through variance calculation; subsequently, the basic weights are calculated based on the modal reliability model: image modal weight w img Initially set to I, RFID weight w rfid Let S be the weight w weight Let W be the modality; to avoid recognition interruption due to single-modal failure, a cross-compensation mechanism is introduced: When any modal weight falls below a threshold θ (default θ = 0.3), the weights of other modalities are increased proportionally. Finally, all weights are normalized using the Softmax function to ensure that ∑(w img ,w rfid ,w weigh t) = 1; In the feature fusion stage, the feature vectors extracted from each modality are: image features F img ∈R256, RFID feature F rfid ∈R64, weight characteristic F weight ∈R16, summed according to dynamic weights, to generate a fused feature vector: F fusion =w img ·F img +w rfid ·F rfid +w weight ·F weight The data is then transmitted to the product recognition module for matching. When the reliability of a certain mode is lower than the threshold, cross-modal weight compensation is achieved through the following formula: in: wm′ is the compensated weight of mode m, m∈{img,rfid,weight}; wm represents the basic weight of modality mm; α is the compensation coefficient, with a default value of 0.5, which controls the compensation intensity; n is the index of other modalities.

6. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S5, the fused feature vector output by the feature extraction module is first entered into the product feature database for retrieval. The database uses a hierarchical index structure to classify products by category, first matching the major category features and then gradually refining to specific products. During the matching process, the system adjusts the priority of each modality feature according to the dynamic weight allocation result. For example, when the RFID weight is high, the tag ID information is compared first, and when the image weight is high, the product appearance features are matched first. The matching algorithm determines the candidate product list by calculating the cosine similarity between feature vectors. The similarity threshold is set at 0.

85. If the similarity is lower than this value, the cross-validation process is initiated. After determining the product category, the system retrieves real-time price information from the price database and synchronizes the product ID and price to the cross-validation module and the settlement module.

7. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S6, when the similarity returned by the product identification module is lower than the threshold, the system automatically starts the cross-validation process, calls the weight data and compares it with the standard weight range in the product database. If the weight deviation is within the allowable range, the matching priority of the product is increased. If the RFID signal exists but no tag information is matched, the system will trigger secondary image recognition to focus on capturing the barcode area on the product packaging, and extract the encoding information through optical character recognition technology to supplement the matching. When all verification methods fail to confirm the product, the checkout interface displays the product image and feature description to prompt the cashier to manually enter the product code. The corrected product information is fed back to the identification module in real time to update the feature database and optimize the subsequent matching accuracy.

8. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S7, after receiving the product identification result, the settlement module calculates the basic amount based on the product quantity and unit price, and simultaneously calls the discount rule engine to match the currently valid promotional activities. The system queries the member database in real time to verify whether the user is a member, automatically applies member discounts and accumulates points. If the product is within the scope of a limited-time promotion, the promotional discount is superimposed. The discount order follows the rule of first single-item discount and then full reduction discount. During the amount calculation process, the total settlement amount is updated in real time and details are displayed, including information such as product name, unit price, quantity, and discount amount. The calculation result is refreshed every 0.5 seconds to ensure that the user can keep track of the payment amount in real time. After the final amount is confirmed, it is pushed to the payment interface to wait for the user to complete the payment.

9. The intelligent settlement method for supermarket cash registers based on multimodal recognition as described in claim 1, characterized in that: In step S8, the payment interface supports multiple payment methods such as QR code payment, facial recognition payment, and NFC payment. After the user selects a payment method, the system calls the corresponding payment interface. The interface communication uses an encryption protocol to ensure transaction security. After the payment is completed, the system receives a transaction success signal from the payment platform. The central controller synchronously writes the transaction information into the database, including data such as product details, payment amount, transaction time, and payment method, and triggers the inventory management system to update the inventory quantity of the corresponding product. After the transaction is confirmed, the indicator light in the settlement area changes from red to green, prompting the user to take the product. At the same time, a transaction receipt is printed, which contains all the key data of this transaction for the user to verify.

10. A smart settlement system for supermarket cash registers based on multimodal recognition, characterized in that: The system is applied to the intelligent settlement method for supermarket cash registers based on multimodal recognition as described in any one of claims 1 to 9.

Images