Self-service settlement method and system for articles

Abstract

This application discloses a self-service checkout method and system for goods. The method includes: acquiring images of a user's actions while holding an item within a preset range around the self-service checkout device using a global image acquisition device to obtain a behavior trajectory image; acquiring images of the user's checkout actions while holding the item using a local depth image acquisition device to obtain a checkout behavior image; monitoring barcode scanning data using an item scanner and weight sensing data using a weight sensor; performing multi-source data joint detection on the behavior trajectory image, checkout behavior image, barcode scanning data, and weight sensing data to determine the result of the user's checkout behavior; and invoking an artificial intelligence digital human to generate corresponding interactive scenarios based on the checkout behavior result to assist in self-service checkout. This solution, by integrating multimodal data, compensates for the shortcomings of single-modal data and monitoring methods, improving loss prevention effectiveness.

Description

Technical Field

[0001] This application relates to the field of self-service checkout technology, and in particular to a self-service checkout method and system for goods. Background Technology

[0002] In the retail industry, the application of self-service checkout machines is increasing. The aim is to reduce manual checkout points and increase self-service checkout points, thereby reducing merchants' operating costs and labor costs, improving merchants' operational efficiency, enhancing customers' shopping experience, and improving merchants' technological image.

[0003] However, most current self-checkout machines only have scanning, calculation, and payment functions, and cannot effectively monitor the entire checkout process. This leads to issues such as unpaid goods being taken away or price tags being swapped, causing losses for merchants. Merchants and the industry currently lack effective solutions to this problem. Industry data shows that the theft rate at self-checkout points averages 3%, significantly impacting merchant profits. Consequently, merchants' enthusiasm for implementing unmanned self-checkout lanes has drastically decreased. To address this, most merchants using self-checkout machines assign personnel to monitor the lanes or check and question shoppers' receipts to prevent losses. However, this often creates a sense of suspicion among customers, sometimes leading to disputes and conflicts, ultimately harming the merchant's business. Summary of the Invention

[0004] This application provides a self-service checkout method and system for goods, so as to achieve comprehensive and accurate monitoring of goods checkout and guide the checkout process.

[0005] According to one aspect of this application, a self-service checkout method for goods is provided, the method comprising:

[0006] The behavior trajectory image is obtained by capturing images of the user's handheld items within a preset range around the self-checkout device using a global image acquisition device, and the checkout behavior image is obtained by capturing images of the checkout behavior performed by the user's handheld items using a local depth image acquisition device.

[0007] The barcode scanner monitors the barcode scanning data, and the weight sensor monitors the weight sensing data.

[0008] Multi-source data joint detection is performed on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the result of the user's settlement behavior for the item;

[0009] An AI-powered digital human is invoked, and based on the settlement behavior results, a corresponding interactive scenario is generated to assist in self-service settlement of goods.

[0010] According to one aspect of this application, a self-service checkout device for goods is provided, the device comprising:

[0011] The image acquisition module is used to acquire images of the user's handheld items within a preset range around the self-service checkout device through a global image acquisition device to obtain a behavior trajectory image, and to acquire images of the checkout behavior performed by the user's handheld items through a local depth image acquisition device to obtain a checkout behavior image.

[0012] The sensor data acquisition module is used to monitor barcode scanning data through the barcode scanner and to monitor weight sensing data through the weight sensor.

[0013] The settlement behavior result determination module can be used to perform multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode sensing data, and the weight sensing data to determine the settlement behavior result of the user for the item;

[0014] The interactive scene generation module is used to call upon the artificial intelligence digital human, and based on the settlement behavior results, generate corresponding interactive scenes to assist in self-service settlement of goods.

[0015] According to another aspect of this application, an embodiment of this application provides a self-service checkout system, which includes a global image acquisition unit, a local depth image acquisition unit, an item barcode scanner, a weight sensor, and a self-service checkout device;

[0016] The global image acquisition device is used to capture images of the user's handheld object's movement trajectory.

[0017] A local depth image acquisition device is used to acquire images of the payment actions performed by a user holding an item.

[0018] Item barcode scanners are used to scan and monitor the barcode sensor data of the identification codes on the outer packaging of items;

[0019] The weight sensor is used to detect and monitor the weight of items placed on the platform.

[0020] The self-service checkout device includes:

[0021] At least one processor; and,

[0022] A memory that is communicatively connected to at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the self-checkout method for items according to any embodiment of this application.

[0024] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the self-service checkout method for items according to any embodiment of this application.

[0025] According to another aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the self-service checkout method for items according to any embodiment of this application.

[0026] The technical solution of this application embodiment captures the user's behavior of holding an item within a preset range around the self-service checkout device using a global image acquisition device to obtain a behavior trajectory image, and captures the user's checkout behavior using a local depth image acquisition device to obtain a checkout behavior image. A barcode scanner monitors barcode sensing data, and a weight sensor monitors weight sensing data. Multi-source data joint detection is performed on the behavior trajectory image, the checkout behavior image, the barcode sensing data, and the weight sensing data to determine the user's checkout behavior result. An AI digital human is invoked, and based on the checkout behavior result, a corresponding interactive scene is generated to assist in self-service checkout. This solution captures the user's behavior trajectory using a global image acquisition device, focuses on the details of the checkout action using a local depth image acquisition device, and uses a barcode scanner and weight sensor to obtain item recognition and weight data, forming a multi-dimensional verification system of "vision + sensing." The multi-source data joint detection technology can effectively identify abnormal behaviors (such as unscanned barcodes, incorrect weights, etc.), reduce the checkout error rate, and significantly improve transaction reliability. Based on the settlement behavior results, AI digital humans can dynamically generate personalized interactive scenarios, such as guiding users to adjust the position of items through voice prompts or demonstrating the settlement process with 3D animations, further improving the convenience and accuracy of settlement. This solution achieves closed-loop processing from data collection to result feedback, reducing manual intervention. Through technological integration and scenario innovation, it not only solves the pain points of "inaccurate recognition and awkward interaction" in traditional self-service checkout, but also builds a virtuous cycle of "accurate recognition - intelligent guidance - data value-added," providing a reusable technological paradigm for the digital transformation of the retail industry.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a self-service checkout method for goods provided in this application embodiment;

[0030] Figure 2 A flowchart illustrating a self-service checkout method for goods, provided as another embodiment of this application;

[0031] Figure 3 A flowchart of a self-service checkout method for goods is provided in another embodiment of this application;

[0032] Figure 4 A flowchart of a self-service checkout method for goods is provided in another embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the structure of a self-service checkout device for goods provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the structure of a self-service checkout device provided in an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. The acquired data is obtained with authorization and will not be disclosed without permission, used for illegal purposes, purposes detrimental to the interests of others, or for personalized analysis or product promotion. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary and intended only to illustrate the feasibility of implementing the technical solution of this application, but do not imply that the applicant has already used or necessarily used the relevant content of such solutions.

[0038] Figure 1 This is a flowchart illustrating a self-service checkout method for goods, provided as an embodiment of this application. This embodiment is applicable to situations where goods are checked out independently. The method can be executed by a self-service checkout device, which can be implemented in hardware and / or software and can be configured within a self-service checkout device. Figure 1 As shown, the method includes:

[0039] S110. The behavior trajectory image is obtained by capturing images of the user's handheld items within a preset range around the self-service checkout device through a global image acquisition device, and the settlement behavior image is obtained by capturing images of the settlement behavior performed by the user's handheld items through a local depth image acquisition device.

[0040] The global image acquisition device can be positioned above the self-checkout device and the user, with a wide field of view, capable of capturing images of the user's behavioral trajectory; for example, it could be a wide-angle image acquisition device. The local depth image acquisition device can be positioned near or on the self-checkout device, capable of capturing images of the user's local hand movements at the self-checkout device. The field of view requirement for the local depth image acquisition device is lower than that of the global image acquisition device; the field of view of the local depth image acquisition device can be smaller than that of the global image acquisition device. The local depth image acquisition device can be a binocular image acquisition device.

[0041] For example, during the image acquisition process of the global image acquisition device, facial recognition and tracking of the acquired user can be performed to capture the user's behavioral trajectory images, which are generally multiple consecutive frames of the user's image. The local depth image acquisition device can capture images of the user's local actions at the self-checkout device to determine the checkout behavior images performed by the user, such as whether the barcode is obscured or whether the user is deliberately faking scanning. The local depth image acquisition device can detect whether the user is scanning an item correctly in front of the scanning window. If the user only makes a cursory movement and is not aligned with the item scanner, the local depth image acquisition device will detect the positional discrepancy and thus indicate improper scanning behavior. The behavioral trajectory images captured by the global image acquisition device and the checkout behavior images captured by the local depth image acquisition device can be used for user matching and tracking to extract behavioral trajectory images and checkout behavior images belonging to the same user.

[0042] In this embodiment, the user monitored by the global image acquisition device and the local depth image acquisition device can be a user holding an item. That is, if the global image acquisition device and the local depth image acquisition device identify that a user is holding an item, then the user is tracked and identified to determine the user's behavioral trajectory image and payment behavior image. If the user is not holding an item, there is no payment requirement or abnormal behavior of not paying, so tracking and monitoring are not required.

[0043] S120: Monitors barcode scanning data via an item barcode scanner and weight sensing data via a weight sensor.

[0044] The item scanner is used to scan and identify the identification codes on the outer packaging of items to determine item information and price. When a user scans an item in their hand, the scanner generates scan data. This data can include accurate item and price information if the scan is correct, or error messages if the scan is incorrect. The weight sensor is installed on the self-checkout platform to monitor the weight of items placed on it. It senses the weight of the items and generates weight data reflecting their weight.

[0045] S130. Perform multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the settlement behavior result of the user for the item.

[0046] For example, during the process of a user checking out a handheld item, a series of related changes occur in the global image, local image, barcode sensing, and weight sensing. For instance, if the global image detects the user walking towards the self-checkout device, the local image will detect the user's checkout action, the barcode sensor will detect the scanned information, and the weight sensor will detect the weight of the item. However, if the global image capture device detects that the user bypasses the self-checkout device and does not check out at it, the local image will not detect the user's hand movement, the barcode sensor will not detect the scanned information, and the weight sensor will not detect the weight of the item. Based on these correlations, multi-source data joint detection can be performed on the behavior trajectory image, checkout behavior image, barcode sensing data, and weight sensing data, thus jointly verifying from multiple perspectives whether the user actually performed the action of scanning and checking out the item.

[0047] In this embodiment, during the multi-source data joint detection of behavior trajectory images, settlement behavior images, barcode scanning data, and weight sensing data, the behavior trajectory images, settlement behavior images, barcode scanning data, and weight sensing data can be identified separately. For example, behavior trajectory features can be identified from the behavior trajectory images, settlement behavior features from the settlement behavior images, barcode scanning behavior features from the barcode scanning data, and the weight of the item can be identified from the weight sensing data. Joint detection is then performed based on the separately identified data to determine the user's settlement behavior result for the item. Alternatively, features can be extracted from the behavior trajectory images, settlement behavior images, barcode scanning data, and weight sensing data separately to obtain corresponding features. These multi-source features are then fused and input into the recognition model for identification to obtain the user's settlement behavior result for the item.

[0048] S140. Call the artificial intelligence digital human, and based on the settlement behavior result, generate corresponding interactive scenarios to assist in self-service settlement of items.

[0049] Artificial intelligence digital humans refer to virtual avatars that simulate the appearance, voice, movements, and interactive abilities of real people using artificial intelligence technology. Their core technologies encompass computer graphics, natural language processing, and machine learning, and they are primarily composed of four modules: humanoid appearance, interactive capabilities, multi-terminal deployment capabilities, and decision-making capabilities. Artificial intelligence digital humans can interact with users in real time through voice, facial expressions, and gestures, understand and respond to complex commands; combine multimodal data analysis to provide personalized services or decision-making suggestions; and flexibly adapt to different scenarios, such as customer service, education, and entertainment, improving user experience and efficiency.

[0050] In this embodiment, an AI-powered digital human can be invoked to generate corresponding interactive scenarios based on the settlement behavior result, assisting in self-service item settlement. The AI-powered digital human generates corresponding interactive scenarios based on the settlement behavior result. For example, if the settlement result is successful, it can issue a voice interaction such as "You have completed your settlement, welcome back again," which helps to create a pleasant atmosphere. If the settlement result is unsuccessful, it can issue a voice interaction such as "Is it impossible to scan the code for the item in your hand? Do you need to contact staff for assistance?", thereby guiding or prompting the user to settle the transaction as soon as possible. Through the intelligent prompts of the AI-powered digital human, a tactful and friendly reminder can be issued when a user has not settled the transaction or is suspected of trying to avoid paying, reminding the user to scan the code to settle the transaction. This avoids staff directly questioning the user about the unsettled behavior, which could lead to misunderstandings and negatively impact the user experience.

[0051] The technical solution of this application embodiment captures the user's behavior of holding an item within a preset range around the self-service checkout device using a global image acquisition device to obtain a behavior trajectory image, and captures the user's checkout behavior using a local depth image acquisition device to obtain a checkout behavior image. A barcode scanner monitors barcode sensing data, and a weight sensor monitors weight sensing data. Multi-source data joint detection is performed on the behavior trajectory image, the checkout behavior image, the barcode sensing data, and the weight sensing data to determine the user's checkout behavior result. An AI digital human is invoked, and based on the checkout behavior result, a corresponding interactive scene is generated to assist in self-service checkout. This solution captures the user's behavior trajectory using a global image acquisition device, focuses on the details of the checkout action using a local depth image acquisition device, and uses a barcode scanner and weight sensor to obtain item recognition and weight data, forming a multi-dimensional verification system of "vision + sensing." The multi-source data joint detection technology can effectively identify abnormal behaviors (such as unscanned barcodes, incorrect weights, etc.), reduce the checkout error rate, and significantly improve transaction reliability. Based on the settlement behavior results, AI digital humans can dynamically generate personalized interactive scenarios. For example, they can guide users to adjust the position of items through voice prompts, demonstrate the settlement process with 3D animations, or recommend preferential schemes based on users' historical data, further improving the convenience and accuracy of settlement. The above solutions achieve closed-loop processing from data collection to result feedback, reducing human intervention. Through technological integration and scenario innovation, not only are the pain points of "inaccurate recognition and awkward interaction" in traditional self-checkout solved, but a virtuous cycle of "accurate recognition - intelligent guidance - data value-added" is also built, providing a reusable technological paradigm for the digital transformation of the retail industry.

[0052] Figure 2 This is a flowchart illustrating a self-service checkout method for goods, provided as another embodiment of this application. This embodiment is an optimization based on the above embodiment; solutions not described in detail in this embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0053] S210. The behavior trajectory image is obtained by capturing images of the user's handheld items within a preset range around the self-service checkout device through a global image acquisition device, and the checkout behavior image is obtained by capturing images of the checkout behavior performed by the user's handheld items through a local depth image acquisition device.

[0054] S220: Monitors barcode scanning data via an item barcode scanner and weight sensing data via a weight sensor.

[0055] S230. Identify the user's behavioral trajectory data relative to the self-service checkout device based on the behavioral trajectory image.

[0056] For example, behavioral trajectory images can be identified to determine the user's behavioral trajectory data relative to the self-service checkout device. Specifically, a global image acquisition device can be used to track and detect the user, acquiring a multi-frame image sequence containing behavioral trajectory images. The image sequence is then identified to determine the user's position points and corresponding time points in each image frame. Based on the continuity of time points, the position points are combined and connected to form the user's behavioral trajectory data. From this data, the user's behavioral trajectory data relative to the self-service checkout device can be selectively extracted. That is, behavioral trajectory data appearing within a preset range around the self-service checkout device can be selected.

[0057] User behavior trajectory data relative to self-service checkout machines can reflect the user's movement relative to the self-service checkout machines, whether the user is walking towards the self-service checkout machines to pay for goods, or bypassing the self-service checkout machines to avoid payment.

[0058] S240. Based on the settlement behavior image, identify the user's settlement behavior data for the item at the self-service checkout device.

[0059] For example, user payment behavior data at self-service checkout machines can be identified based on payment behavior images. This involves recognizing the user's hand movements in the payment behavior images to determine if the user has scanned and paid for the items they are holding. The process of recognizing payment behavior data also involves identifying a continuous sequence of images captured of the user's hands to determine the hand movements and thus the payment behavior data.

[0060] In this embodiment of the application, the basis for identifying the user's settlement behavior data for the item is whether the user has used a handheld item barcode scanner to scan the identification code on the outer packaging of the item.

[0061] S250. Identify the user's scan-to-pay result for the item based on the scan-sensing data.

[0062] For example, the user's scan-to-pay result for an item can be identified based on the barcode scanning data. If the user can successfully scan the barcode on the item's outer packaging using a handheld barcode scanner, then the barcode scanning data indicates that the item's information, including its price, can be correctly identified, and the corresponding scan-to-pay result should be a successful scan. If the barcode scanning data confirms that a scan was triggered, but no item information or price was scanned, it indicates that the scan was unsuccessful. This could be due to the user not correctly aligning the barcode scanner with the barcode on the item's outer packaging, resulting in an unintentional error, or it could be a deliberate attempt by the user to create the illusion of a scan. However, the final scan result will always be an objective failure.

[0063] S260. Identify the weight data of the item held by the user based on the weight sensing data.

[0064] For example, the weight of items placed by the user can be identified based on weight sensor data. Specifically, currently, users typically hold items while scanning their barcodes using an item scanner, place the scanned items on a shelf, and then remove and scan another item. Therefore, the shelf typically holds already scanned items. If the weight sensor detects an increase in weight, this increase represents the weight of the item placed after the user scanned it. Based on this principle, weight sensor data can be used to determine the original weight of the scanned items the user held and placed on the shelf.

[0065] S270. Based on the joint detection of the behavior trajectory data, settlement behavior data, the scanning result and the weight data, determine the user's settlement behavior result for the item.

[0066] For example, the user's actual payment for the item can be determined by the joint detection of behavioral trajectory data, settlement behavior data, scanning results and weight data.

[0067] In this embodiment of the application, the user's settlement behavior result for the item is determined based on the joint detection of the behavior trajectory data, settlement behavior data, the scanning result, and the weight data, including:

[0068] If the behavior trajectory data determines that the user bypassed all self-service checkout devices and did not check out at any of the self-service checkout devices, and the checkout behavior data, the scanning result, and the weight data determine that the user did not check out at any of the self-service checkout devices, then the checkout behavior result is determined to be no checkout.

[0069] If the user is determined to be at any self-service checkout device based on the behavior trajectory data, then the user's checkout behavior result for the item is detected as unsettled based on the checkout behavior data, the scanning result, and the weight data.

[0070] In this embodiment of the application, the process of jointly detecting multi-source data to determine the settlement behavior result of the item can be as follows: if the behavior trajectory data determines that the user bypassed all self-service checkout devices and did not go to any self-service checkout device to settle the payment, and the settlement behavior data, scan results, and weight data identified by the local depth image acquisition device determine that the user was not at any self-service checkout device, and no local depth image acquisition device at any self-service checkout device monitored the user's settlement behavior data, that is, the settlement behavior data identified that the user did not settle the payment, and no scan result reflects that the user scanned the item, and no weight sensor at any self-service checkout device detected the weight data of the item, then it can be determined that the settlement behavior result of the user holding the item is not settled.

[0071] If the behavioral trajectory data determines that the user is at any self-checkout machine to make a payment, then the payment behavior data, scan results, and weight data are used to check whether the user's payment behavior result for the item is unpaid. In other words, although the behavioral trajectory data determines that the user has walked to a self-checkout machine to make a payment, it does not guarantee that the user has actually performed the payment behavior and that the payment was successful. Therefore, it is necessary to combine the payment behavior data, scan results, and weight data to jointly verify whether the user has actually made a successful payment.

[0072] The above solution significantly improves the ability to identify abnormal behavior in self-service checkout scenarios through a dual verification mechanism of behavioral trajectory and settlement data. First, behavioral trajectory data is used to determine whether a user has entered the checkout area. If the system detects that a user bypasses all devices, it directly determines that the user has not settled, effectively preventing missed detections caused by device blind spots in traditional solutions. Once the user enters the checkout area, the system further combines settlement behavior images, scanning results, and weight data to accurately identify unsettled behavior through multi-dimensional cross-verification, reducing the false positive rate. This layered detection logic of "location first, verification later" avoids the limitations of a single data source and improves system response speed.

[0073] In this embodiment of the application, detecting whether the user's settlement behavior result for the item is unsettled based on the settlement behavior data, the scanning result, and the weight data includes:

[0074] If the settlement behavior data determines that a user has operated the item scanner to scan and settle the item, then the scanning results are used to determine whether there is item scanning information within a target time period and / or the weight data is used to determine whether there is a change in item weight within the target time period; wherein, the target time period is the duration of the user's scanning and settlement behavior determined by the settlement behavior data.

[0075] If at least one of the following conditions is met, the user's settlement behavior for the item is determined to be unsettled:

[0076] The scan results are used to determine the scan information of items that do not exist within the target time period;

[0077] The weight data is used to determine that there is no change in the weight of the items during the target time period;

[0078] The scanning results determine the scanning information of the items present within the target time period, the weight data determines the weight changes of the items within the target time period, and the difference between the preset weight of the items determined based on the scanning information and the predicted weight of the items determined based on the weight data is greater than the preset difference.

[0079] For example, in the process of further jointly detecting whether the user's settlement behavior result for the item is unsettled based on settlement behavior data, scanning results, and weight data, specifically, if it is determined from the settlement behavior data that the user has operated the item scanner to scan the item for settlement, that is, by monitoring the local behavior of the user's hand through a local depth image acquisition device to determine that the user actually scanned the item, then the scanning result is used to determine whether there is item scanning information in the target time period and / or the weight data is used to determine whether there is a change in the weight of the item in the target time period. In other words, under normal circumstances, if it is determined through local behavior detection that the user has scanned the item, then the item scanner's scanning result will contain item scanning information in the target time period, and the weight data will determine whether there is a change in the weight of the item in the target time period. If the scan results and weight data meet at least one of the following conditions, the user's settlement behavior for the item is determined to be unsettled: The scan results determine that there is no scanned information for the item within the target time period; the weight data determines that there is no change in the item's weight within the target time period; the scan results determine that there is scanned information for the item within the target time period, and the weight data determines that there is a change in the item's weight within the target time period. However, if the difference between the preset weight of the item determined based on the scan information and the predicted weight of the item determined based on the weight data is greater than a preset difference, that is, the weight data determines that there is a change in the item's weight, and this change in weight is the predicted weight of the item. The specific information of the item determined based on the scan information includes the preset weight of the item. Theoretically, the predicted weight should be consistent with the preset weight, or the difference should be less than or equal to the preset difference. If the difference is small, and the difference is greater than the preset difference, it reflects that the item scanned by the user may not be consistent with the item placed, therefore, the settlement behavior can be determined to be unsettled.

[0080] S280. Call the artificial intelligence digital human, and based on the settlement behavior result, generate corresponding interactive scenarios to assist in self-service settlement of items.

[0081] The solution in this embodiment identifies the user's behavioral trajectory data relative to the self-service checkout device based on the behavioral trajectory image; identifies the user's checkout behavior data at the self-service checkout device based on the checkout behavior image; identifies the user's checkout result based on the barcode sensing data; identifies the weight data of the item held by the user based on the weight sensing data; and determines the user's checkout behavior result based on the joint detection of the behavioral trajectory data, checkout behavior data, scan result, and weight data. This solution achieves accurate determination of the user's self-service checkout behavior through multimodal data fusion and joint detection technology. It captures the complete movement line of the user from item selection to device operation through the behavioral trajectory image, identifies specific action details (such as scanning posture and placement position) by combining the checkout behavior image, verifies the accuracy of item recognition using barcode sensing data, and verifies the matching between the item and the scan result using weight sensor data, forming a four-fold verification mechanism of "behavior-action-recognition-verification". This multi-source data joint detection method can effectively identify abnormal scenarios (such as missed scanning, misplacement, and weight discrepancies), reduce settlement error rates, and significantly improve transaction reliability. At the same time, by dynamically analyzing the correlation between user behavior trajectories and settlement actions, the system can provide early warnings of potential operational errors (such as leaving before completing the scan), reducing repetitive user operations and improving self-service settlement efficiency.

[0082] Figure 3 This is a flowchart illustrating a self-service checkout method for goods, provided as another embodiment of this application. This embodiment is an optimization based on the above embodiments; solutions not described in detail in this embodiment are found in the above embodiments. Figure 3 As shown, the method in this embodiment of the application specifically includes the following steps:

[0083] S310. The global image acquisition device captures images of the user's actions of holding items within a preset range around the self-service checkout device to obtain a behavior trajectory image, and the local depth image acquisition device captures images of the checkout actions performed by the user holding items to obtain a checkout behavior image.

[0084] S320 monitors barcode scanning data via an item barcode scanner and weight sensing data via a weight sensor.

[0085] S330. Perform feature extraction on the behavior trajectory image to obtain trajectory features, perform feature extraction on the settlement behavior image to obtain settlement behavior features, perform feature extraction on the barcode sensing data to obtain barcode sensing features, and perform feature extraction on the weight sensing data to obtain weight sensing features.

[0086] In this embodiment of the application, feature extraction can also be performed on the behavior trajectory image to obtain trajectory features, on the settlement behavior image to obtain settlement behavior features, on the barcode sensing data to obtain barcode sensing features, and on the weight sensing data to obtain weight sensing features. The feature extraction process can be input into the corresponding features obtained in the feature extraction model.

[0087] S340. Perform multi-source feature fusion on the trajectory features, the settlement behavior features, the barcode scanning features, and the weight sensing features to obtain settlement recognition features.

[0088] For example, after extracting multi-source features, trajectory features, settlement behavior features, barcode scanning features, and weight sensing features can be fused to obtain settlement recognition features. Multi-source feature fusion can be achieved by directly concatenating the multi-source features to obtain the settlement recognition features. Alternatively, a weighted fusion based on an attention mechanism can be used to obtain the settlement recognition features.

[0089] S350. Input the settlement recognition features into the pre-trained settlement recognition model to obtain the settlement behavior result.

[0090] For example, by inputting the settlement recognition features into a pre-trained settlement recognition model, the settlement behavior result can be obtained. The settlement recognition model can be pre-trained and capable of recognizing the settlement recognition features fused from multiple sources, and outputting the settlement behavior result of whether the user has settled the payment for the item correctly.

[0091] In this embodiment of the application, the training process of the settlement recognition model includes:

[0092] Training data is pre-acquired for both normal and unsettled settlement scenarios for the user's payment behavior towards the item; the training data includes images of the user's hand-held item trajectory, settlement behavior images, barcode scanning data, and weight sensing data.

[0093] The training data is used to extract features to obtain training features. These training features are then input into a neural network model to predict settlement behavior results. A loss function is determined based on the predicted and actual settlement behavior results. The neural network model is then trained and optimized until the iteration stops, resulting in the settlement recognition model.

[0094] For example, during the training of the checkout recognition model, training data can be pre-acquired for both normal and non-normal checkout behavior. Specifically, in a real-world application scenario, multi-source data is acquired during normal detection using a global image acquisition device, a local depth image acquisition device, an item scanner, and a weight sensor. The user's checkout behavior is then manually verified, and the labels indicating whether checkout was normal are mapped to the corresponding multi-source data as training data. Feature extraction is performed on the multi-source training data to obtain training features. These features are then input into a neural network model to predict checkout behavior. A loss function is determined based on the deviation between the predicted and actual checkout behavior, and the neural network model is trained and optimized until the iteration conditions are met, resulting in the checkout recognition model.

[0095] S360: Call upon the AI ​​digital human, and based on the AI ​​digital human, generate corresponding interactive scenarios according to the settlement behavior results to assist in self-service settlement of items.

[0096] The solution of this application embodiment involves extracting features from the behavior trajectory image to obtain trajectory features, extracting features from the settlement behavior image to obtain settlement behavior features, extracting features from the barcode sensing data to obtain barcode sensing features, and extracting features from the weight sensing data to obtain weight sensing features; performing multi-source feature fusion on the trajectory features, settlement behavior features, barcode sensing features, and weight sensing features to obtain settlement recognition features; and inputting the settlement recognition features into a pre-trained settlement recognition model to obtain the settlement behavior result. The above-mentioned solution constructs a high-precision self-service checkout behavior recognition system through multi-source feature fusion and pre-trained model collaboration. It extracts features from behavior trajectory images, checkout behavior images, barcode scanning data, and weight sensing data to obtain feature vectors reflecting user movement patterns, operation details, barcode scanning validity, and weight consistency. Subsequently, feature fusion technology unifies heterogeneous data into structured checkout recognition features, solving the recognition bias problem caused by differences in data dimensions in traditional solutions. After inputting the fused features into the pre-trained checkout recognition model, the model can achieve millisecond-level determination of checkout behavior based on the spatiotemporal correlation patterns learned from massive historical data, maintaining a high recognition accuracy even in complex scenarios.

[0097] Figure 4 This is a flowchart illustrating a self-service checkout method for goods, provided as another embodiment of this application. This embodiment is an optimization based on the above embodiments; solutions not described in detail in this embodiment are found in the above embodiments. Figure 4 As shown, the method in this embodiment of the application specifically includes the following steps:

[0098] S410. The global image acquisition device captures images of the user's actions of holding items within a preset range around the self-service checkout device to obtain a behavior trajectory image, and the local depth image acquisition device captures images of the checkout actions performed by the user holding items to obtain a checkout behavior image.

[0099] S420 monitors barcode scanning data via an item barcode scanner and weight sensing data via a weight sensor.

[0100] S430. Perform multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the settlement behavior result of the user for the item.

[0101] S440. Determine whether the user has any unsettled behavior based on the settlement behavior result and predict the confidence level of the user generating unsettled behavior.

[0102] For example, the settlement behavior outcome may include a judgment on settlement results that have already occurred, and a prediction of settlement behavior that has not yet occurred. For instance, if a user does not go to the self-service checkout machine to settle their account, it is identified that the user has not yet settled their account, meaning the user has an unsettled transaction. Alternatively, if the settlement behavior outcome determines that the user has not settled their account, and this is based on the user bypassing the self-service checkout machine to identify the unsettled transaction, then it is predicted that the user has an unsettled transaction. In this case, the confidence level for determining that the user has an unsettled transaction or predicting that the user will engage in an unsettled transaction is determined.

[0103] Based on the settlement behavior results, predict the confidence level of the user's unsettled behavior, including:

[0104] If the user is at a self-service checkout device and no barcode scanning information and / or changes in item weight are detected within a preset time period, a confidence level higher than the preset confidence threshold is assigned.

[0105] Otherwise, assign a confidence level lower than or equal to the preset confidence threshold.

[0106] For example, if it is determined that a user has already engaged in unpaid behavior, a confidence level higher than a preset confidence threshold is assigned; otherwise, a confidence level lower than or equal to the preset confidence threshold is assigned. Additionally, when predicting unpaid behavior, if the user is at a self-service checkout device and no barcode scanning information and / or item weight changes are detected within a preset time period—that is, if it is predicted that the user may engage in unpaid behavior—a confidence level higher than the preset confidence threshold is assigned. Otherwise, a confidence level lower than or equal to the preset confidence threshold is assigned.

[0107] S450. Based on artificial intelligence digital human, determine the graded guidance interaction scenario according to whether the user has unsettled behavior and the confidence level of predicting that the user will generate unsettled behavior.

[0108] For example, tiered interactive scenarios can be determined based on the confidence level of whether a user has unpaid behavior and the confidence level of predicting that the user will engage in unpaid behavior. In other words, different levels of interactive scenarios are set according to different confidence levels. If the confidence level is low, meaning the user has no unpaid behavior, an interactive voice message to adjust the atmosphere is generated. If the confidence level is high, meaning the user may have unpaid behavior, or has already engaged in unpaid behavior, a reminder voice message is generated.

[0109] In this embodiment of the application, based on artificial intelligence digital humans, a tiered guidance interaction scenario is determined according to whether the user has any unpaid behavior and the confidence level of predicting that the user will have unpaid behavior, including:

[0110] If the user has any unsettled transactions, the system will prompt the user that the settlement was unsuccessful or failed, and will also provide the user with the correct settlement method.

[0111] If the confidence level of predicting that the user will generate unsettled behavior is higher than the preset confidence threshold, a prompt will be issued asking if help is needed.

[0112] If the confidence level of predicting that the user will generate unsettled behavior is lower than or equal to a preset confidence threshold, a settlement guidance prompt will be issued.

[0113] For example, in a tiered interactive scenario, if a user has unsettled behavior, the system will prompt the user that the settlement was unsuccessful or failed. If the confidence level of predicting unsettled behavior is higher than a preset confidence threshold, a prompt will be issued asking if the user needs assistance, reminding and assisting the user to complete the settlement. If the confidence level of predicting unsettled behavior is lower than or equal to the preset confidence threshold, a settlement guidance prompt will be issued, indicating that the user does not have unsettled behavior and that settlement guidance can be provided to guide the user to complete the settlement process.

[0114] Specifically, the system can be configured with multi-stage guidance, including routine guidance, preventative reminders, collaborative correction, and seamless upgrades to human customer service. Routine guidance includes standard process instructions ("Please align the product barcode with the scanning area"). Preventative reminders proactively intervene when potential confusion or risk is detected. For example, if a customer hesitates while picking up a fresh produce item, the digital human smiles and says, "This is an avocado. Would you like me to find the barcode for you?" The location of the barcode is highlighted on the screen. Collaborative correction occurs when a clear anomaly is detected (such as a weight mismatch). For example, "Hello, the system indicates that the last item placed, the 'chocolate,' may not have been successfully scanned. Can we check it together?" Simultaneously, the interface circles the image of the suspicious item and plays back a clip (anonymized) of it being placed for the customer to confirm. Seamless upgrades to human customer service: For complex questions, the digital human says, "I'll have a colleague answer this for you in detail," and then smoothly synchronizes the video stream and transaction context to a remote human customer service representative, achieving a "digital human-human" relay service.

[0115] Additionally, the global image acquisition unit detected a customer placing a "wine-shaped" object into a "juice box." If the weight sensor recorded a weight significantly greater than the expected weight of the juice, the edge unit instantly fused the information, determining it as a "high-risk tampering attempt" and triggering a guided loss prevention process. The digital human, showing concern, pointed to the scanning area: "I noticed the packaging of the product you're holding is quite unusual. For accurate pricing, could we directly scan the barcode on the product itself? If it needs to be opened, I can guide you through the process." This both prevented fraud and gave the customer a dignified opportunity to correct the situation.

[0116] The solution in this application embodiment determines whether the user has unsettled behavior based on the settlement behavior result and predicts the confidence level of the user's potential unsettled behavior. Based on the AI ​​digital human, it determines a tiered guidance interaction scenario based on whether the user has unsettled behavior and the confidence level of the predicted unsettled behavior. This solution constructs a tiered guidance intelligent interaction system through real-time settlement behavior analysis and confidence level prediction. It dynamically identifies whether a user has unsettled behavior based on the settlement behavior result and predicts the confidence level through a machine learning model, forming a two-dimensional assessment of "current state - risk level." The AI ​​digital human automatically triggers a tiered guidance strategy based on the assessment results—providing lightweight prompts (such as voice reminders to scan codes) for low-risk users and initiating enhanced interventions (such as 3D animation demonstrations of the correct operation process or generating personalized guidance dialogues) for medium- and high-risk users. This differentiated interaction mechanism improves user acceptance and increases the unsettled behavior correction rate. The above solution achieves precise control of the intervention intensity through confidence quantification, avoiding user resentment caused by traditional "one-size-fits-all" prompts. In scenarios such as supermarkets and convenience stores, it can reduce conflicts and complaints caused by misjudgments, providing a humanized and efficient behavior guidance solution for smart retail.

[0117] Figure 5 This is a schematic diagram of a self-service checkout device for goods provided in an embodiment of this application. This device can execute the self-service checkout method for goods provided in any embodiment of this application, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 5 As shown, the device includes:

[0118] The image acquisition module 510 is used to acquire images of the user's handheld items within a preset range around the self-service checkout device through a global image acquisition device to obtain a behavior trajectory image, and to acquire images of the checkout behavior performed by the user's handheld items through a local depth image acquisition device to obtain a checkout behavior image.

[0119] The sensing data acquisition module 520 is used to monitor scanning sensing data through the item barcode scanner and to monitor weight sensing data through the weight sensor.

[0120] The settlement behavior result determination module 530 can be used to perform multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode sensing data, and the weight sensing data to determine the settlement behavior result of the user for the item;

[0121] The interactive scene generation module 540 is used to call the artificial intelligence digital human and generate corresponding interactive scenes based on the settlement behavior results to assist in self-service settlement of items.

[0122] In this embodiment, the settlement behavior result determination module 530 performs multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the settlement behavior result of the user for the item, including:

[0123] The user's behavioral trajectory data relative to the self-service checkout device is identified based on the behavioral trajectory image.

[0124] The user's payment behavior data for the items at the self-service checkout device is identified based on the settlement behavior image.

[0125] The user's payment result for the item is identified based on the scanned data.

[0126] The weight data of the item held by the user is identified based on the weight sensing data;

[0127] Based on the joint detection of the behavior trajectory data, settlement behavior data, scanning results, and weight data, the user's settlement behavior result for the item is determined.

[0128] In this embodiment of the application, the settlement behavior result determination module 530 determines the user's settlement behavior result for the item based on the joint detection of the behavior trajectory data, settlement behavior data, the scan result, and the weight data, including:

[0129] If the behavior trajectory data determines that the user bypassed all self-service checkout devices and did not check out at any of the self-service checkout devices, and the checkout behavior data, the scanning result, and the weight data determine that the user did not check out at any of the self-service checkout devices, then the checkout behavior result is determined to be no checkout.

[0130] If the user is determined to be at any self-service checkout device based on the behavior trajectory data, then the user's checkout behavior result for the item is detected as unsettled based on the checkout behavior data, the scanning result, and the weight data.

[0131] In this embodiment of the application, the settlement behavior result determination module 530 detects whether the user's settlement behavior result for the item is unsettled based on the settlement behavior data, the scan result, and the weight data, including:

[0132] If the settlement behavior data determines that a user has operated the item scanner to scan and settle the item, then the scanning results are used to determine whether there is item scanning information within a target time period and / or the weight data is used to determine whether there is a change in item weight within the target time period; wherein, the target time period is the duration of the user's scanning and settlement behavior determined by the settlement behavior data.

[0133] If at least one of the following conditions is met, the user's settlement behavior for the item is determined to be unsettled:

[0134] The scan results are used to determine the scan information of items that do not exist within the target time period;

[0135] The weight data is used to determine that there is no change in the weight of the items during the target time period;

[0136] The scanning results determine the scanning information of the items present within the target time period, the weight data determines the weight changes of the items within the target time period, and the difference between the preset weight of the items determined based on the scanning information and the predicted weight of the items determined based on the weight data is greater than the preset difference.

[0137] In this embodiment, the settlement behavior result determination module 530 performs multi-source data joint detection on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the settlement behavior result of the user for the item, including:

[0138] Trajectory features are obtained by extracting features from the behavior trajectory image, settlement behavior features are obtained by extracting features from the settlement behavior image, QR code sensing features are obtained by extracting features from the QR code sensing data, and weight sensing features are obtained by extracting features from the weight sensing data.

[0139] The settlement recognition features are obtained by multi-source feature fusion of the trajectory features, the settlement behavior features, the barcode scanning features, and the weight sensing features.

[0140] The settlement recognition features are input into a pre-trained settlement recognition model to obtain the settlement behavior result.

[0141] In this embodiment of the application, the apparatus further includes the settlement recognition model training module, used for:

[0142] Training data is pre-acquired for both normal and unsettled settlement scenarios for the user's payment behavior towards the item; the training data includes images of the user's hand-held item trajectory, settlement behavior images, barcode scanning data, and weight sensing data.

[0143] The training data is used to extract features to obtain training features. These training features are then input into a neural network model to predict settlement behavior results. A loss function is determined based on the predicted and actual settlement behavior results. The neural network model is then trained and optimized until the iteration stops, resulting in the settlement recognition model.

[0144] In this embodiment, the interactive scene generation module 540 generates a corresponding interactive scene based on the settlement behavior result using an artificial intelligence digital human, including:

[0145] Based on the settlement behavior results, determine whether the user has any unsettled behavior and the confidence level for predicting that the user will have unsettled behavior;

[0146] Based on the artificial intelligence digital human, the system determines the tiered guidance interaction scenario according to whether the user has any unpaid behavior and the confidence level of predicting that the user will have unpaid behavior.

[0147] In this embodiment, the interactive scene generation module 540, based on an artificial intelligence digital human, determines a tiered guided interactive scene according to whether the user has any unpaid behavior and the confidence level of predicting that the user will have unpaid behavior, including:

[0148] If the user has any unsettled transactions, the system will prompt the user that the settlement was unsuccessful or failed, and will also provide the user with the correct settlement method.

[0149] If the confidence level of predicting that the user will generate unsettled behavior is higher than the preset confidence threshold, a prompt will be issued asking if help is needed.

[0150] If the confidence level of predicting that the user will generate unsettled behavior is lower than or equal to a preset confidence threshold, a settlement guidance prompt will be issued.

[0151] In this embodiment of the application, the interactive scene generation module 540 predicts the confidence level of the user's unsettled behavior based on the settlement behavior result, including:

[0152] If the user is at a self-service checkout device and no barcode scanning information and / or changes in item weight are detected within a preset time period, a confidence level higher than the preset confidence threshold is assigned.

[0153] Otherwise, assign a confidence level lower than or equal to the preset confidence threshold.

[0154] The self-service checkout device provided in this application can execute a self-service checkout method for items provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0155] This application provides a self-service checkout system, which includes a global image acquisition unit, a local depth image acquisition unit, an item barcode scanner, a weight sensor, and a self-service checkout device.

[0156] The global image acquisition device is used to capture images of the user's handheld object's movement trajectory.

[0157] A local depth image acquisition device is used to acquire images of the payment actions performed by a user holding an item.

[0158] Item barcode scanners are used to scan and monitor the barcode sensor data of the identification codes on the outer packaging of items;

[0159] The weight sensor is used to detect and monitor the weight of items placed on the platform.

[0160] The self-service checkout device includes:

[0161] At least one processor; and,

[0162] A memory communicatively connected to the at least one processor; wherein,

[0163] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the self-service checkout method for items as described in any of the above embodiments.

[0164] Figure 6 A schematic diagram of a self-checkout device 10, which can be used to implement embodiments of this application, is shown. The self-checkout device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital processors, servers, blade servers, mainframe computers, and other suitable computers. The self-checkout device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0165] like Figure 6As shown, the self-checkout device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the self-checkout device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0166] Multiple components in the self-checkout device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the self-checkout device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0167] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as self-checkout methods for items.

[0168] In some embodiments, the self-checkout method for items may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the self-checkout device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the self-checkout method for items described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the self-checkout method for items by any other suitable means (e.g., by means of firmware).

[0169] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable self-checkout device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0172] To provide user interaction, the systems and techniques described herein can be implemented on self-checkout devices, which include: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a touchscreen, mouse, or trackball) through which the user provides input to the self-checkout device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0173] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0174] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0175] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the self-service checkout method for items as provided in any embodiment of this application.

[0176] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0178] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A self-service checkout method for goods, characterized in that, The method includes: The behavior trajectory image is obtained by capturing images of the user's handheld items within a preset range around the self-checkout device using a global image acquisition device, and the checkout behavior image is obtained by capturing images of the checkout behavior performed by the user's handheld items using a local depth image acquisition device. The barcode scanner monitors the barcode scanning data, and the weight sensor monitors the weight sensing data. Multi-source data joint detection is performed on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the result of the user's settlement behavior for the item; An AI-powered digital human is invoked, and based on the settlement behavior results, a corresponding interactive scenario is generated to assist in self-service settlement of goods.

2. The method according to claim 1, characterized in that, Multi-source data joint detection is performed on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the result of the user's settlement behavior for the item, including: The user's behavioral trajectory data relative to the self-service checkout device is identified based on the behavioral trajectory image. The user's payment behavior data for the items at the self-service checkout device is identified based on the settlement behavior image. The user's payment result for the item is identified based on the scanned data. The weight data of the item held by the user is identified based on the weight sensing data; Based on the joint detection of the behavior trajectory data, settlement behavior data, scanning results, and weight data, the user's settlement behavior result for the item is determined.

3. The method according to claim 2, characterized in that, Based on the joint detection of the behavioral trajectory data, settlement behavior data, the scanning results, and the weight data, the user's settlement behavior result for the item is determined, including: If the behavior trajectory data determines that the user bypassed all self-service checkout devices and did not check out at any of the self-service checkout devices, and the checkout behavior data, the scanning result, and the weight data determine that the user did not check out at any of the self-service checkout devices, then the checkout behavior result is determined to be no checkout. If the user is determined to be at any self-service checkout device based on the behavior trajectory data, then the user's checkout behavior result for the item is detected as unsettled based on the checkout behavior data, the scanning result, and the weight data.

4. The method according to claim 3, characterized in that, Detecting whether the user's settlement behavior result for the item is unsettled based on the settlement behavior data, the scan result, and the weight data includes: If the settlement behavior data determines that a user has operated the item scanner to scan and settle the item, then the scanning results are used to determine whether there is item scanning information within a target time period and / or the weight data is used to determine whether there is a change in item weight within the target time period; wherein, the target time period is the duration of the user's scanning and settlement behavior determined by the settlement behavior data. If at least one of the following conditions is met, the user's settlement behavior for the item is determined to be unsettled: The scan results are used to determine the scan information of items that do not exist within the target time period; The weight data is used to determine that there is no change in the weight of the items during the target time period; The scanning results determine the scanning information of the items present within the target time period, the weight data determines the weight changes of the items within the target time period, and the difference between the preset weight of the items determined based on the scanning information and the predicted weight of the items determined based on the weight data is greater than the preset difference.

5. The method according to claim 1, characterized in that, Multi-source data joint detection is performed on the behavior trajectory image, the settlement behavior image, the barcode scanning data, and the weight sensing data to determine the result of the user's settlement behavior for the item, including: Trajectory features are obtained by extracting features from the behavior trajectory image, settlement behavior features are obtained by extracting features from the settlement behavior image, QR code sensing features are obtained by extracting features from the QR code sensing data, and weight sensing features are obtained by extracting features from the weight sensing data. The settlement recognition features are obtained by multi-source feature fusion of the trajectory features, the settlement behavior features, the barcode scanning features, and the weight sensing features. The settlement recognition features are input into a pre-trained settlement recognition model to obtain the settlement behavior result.

6. The method according to claim 5, characterized in that, The training process of the settlement recognition model includes: Training data is pre-acquired for both normal and unsettled settlement scenarios for the user's payment behavior towards the item; the training data includes images of the user's hand-held item trajectory, settlement behavior images, barcode scanning data, and weight sensing data. The training data is used to extract features to obtain training features. These training features are then input into a neural network model to predict settlement behavior results. A loss function is determined based on the predicted and actual settlement behavior results. The neural network model is then trained and optimized until the iteration stops, resulting in the settlement recognition model.

7. The method according to claim 1, characterized in that, Based on the settlement behavior results, the AI-powered digital human generates corresponding interactive scenarios, including: Based on the settlement behavior results, determine whether the user has any unsettled behavior and the confidence level for predicting that the user will have unsettled behavior; Based on the artificial intelligence digital human, the system determines the tiered guidance interaction scenario according to whether the user has any unpaid behavior and the confidence level of predicting that the user will have unpaid behavior.

8. The method according to claim 7, characterized in that, Based on artificial intelligence and digital human intelligence, the system determines tiered guidance and interaction scenarios according to whether the user has any unpaid behavior and the confidence level in predicting that the user will engage in unpaid behavior. These scenarios include: If the user has any unsettled transactions, the system will prompt the user that the settlement was unsuccessful or failed, and will also provide the user with the correct settlement method. If the confidence level of predicting that the user will generate unsettled behavior is higher than the preset confidence threshold, a prompt will be issued asking if help is needed. If the confidence level of predicting that the user will generate unsettled behavior is lower than or equal to a preset confidence threshold, a settlement guidance prompt will be issued.

9. The method according to claim 7, characterized in that, Based on the settlement behavior results, predict the confidence level of the user's unsettled behavior, including: If the user is at a self-service checkout device and no barcode scanning information and / or changes in item weight are detected within a preset time period, a confidence level higher than the preset confidence threshold is assigned. Otherwise, assign a confidence level lower than or equal to the preset confidence threshold.

10. A self-service checkout system, characterized in that, The self-service checkout system includes a global image acquisition unit, a local depth image acquisition unit, an item barcode scanner, a weight sensor, and a self-service checkout device, wherein the self-service checkout device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the self-service checkout method for items as described in any one of claims 1-9.

Images