A natural interaction method and system for AR glasses QR code payment scenarios

Abstract

This disclosure provides a natural interaction method and system for AR glasses QR code payment scenarios, applied in the field of AR technology. The method includes: responding to a user's activation command, the AR glasses acquire an environmental image of the area to be collected; wherein the user's hand is placed in the area to be collected with a specific gesture; based on the specific gesture, a lightweight image recognition model is used to identify whether the user in the environmental image has payment intent; if so, the corresponding payment code is identified; it is determined whether the identified payment code conforms to the static payment protocol specification; if so, a voice monitoring module is activated to acquire environmental sound, and the amount information is obtained based on the environmental sound; payment is performed after the user confirms the amount information. This disclosure integrates the payment process with natural offline interpersonal interaction, avoids accidental triggering through precise pointing selection, and embeds the payment amount input into the necessary dialogue between the user and the merchant, eliminating the awkwardness of the user publicly reciting the amount; finally, authorization is completed through physical triggering.

Description

Technical Field

[0001] This disclosure relates to the field of AR technology, and in particular to a natural interaction method and system for AR glasses QR code payment scenarios. Background Technology

[0002] Currently, AR headsets that support payments primarily rely on a "voice command" interaction mode to complete transactions. The standard process includes: the user activates the AR headset's QR code / barcode scanning function via voice activation; the device's camera scans and recognizes the payment code; the user confirms the payment and the specific amount (e.g., 58 yuan) via voice; the device confirms the command via voice recognition; and the user grants final authorization again via voice or gesture.

[0003] However, existing technical solutions have the following inherent flaws, resulting in a lengthy and unnatural payment experience that can easily cause psychological discomfort to users in public places:

[0004] (1) The triggering and selection phase is inefficient and error-prone: In typical payment scenarios such as merchant counters where multiple QR codes / barcodes exist (e.g., payment codes typically include Alipay and WeChat, and some may also include bank QR codes / barcodes), see [reference needed]. Figure 4 Common "wake-up scan" commands (whether voice or gesture) cannot accurately locate the target to be scanned. Even using the mature image recognition models built into mobile phones, the output is only multiple location points (the number of location points corresponds to the number of codes in the image). Since AR headsets typically lack eye tracking and sequential hand motion capture, they cannot select which QR codes / barcodes to scan. This necessitates either voice description or physical covering of unwanted QR codes / barcodes, disrupting the payment process and causing inconvenience for users, leading more people to return to mobile payments.

[0005] (2) Shame associated with voice-based task execution: The most crucial part of the process—payment confirmation—requires users to clearly state the specific amount and confirm payment. In public and semi-public social spaces such as cafes, restaurants, and convenience stores, the act of reading aloud "Pay XX yuan" or "Confirm payment" creates significant psychological pressure of "shame associated with voice-based task execution" for most users, leading to an awkward experience and reluctance to use the service.

[0006] (3) The interaction process is lengthy and unnatural: The entire process from triggering to completion is driven by the user actively issuing multiple explicit commands to the device, which is completely disconnected from the natural offline communication process between the user and the merchant (such as checking the bill, nodding to confirm, and taking out the phone to pay). The user needs to interact with the device like operating a walkie-talkie. The process is rigid and has many steps, which does not meet the inherent requirements of efficiency, low profile and smoothness in real-world payment scenarios.

[0007] Therefore, there is an urgent need to develop a natural interaction method and system for AR glasses QR code payment scenarios to solve the above problems. Summary of the Invention

[0008] This disclosure provides a natural interaction method and system for AR glasses QR code payment scenarios to solve the above problems. It addresses the issues that existing technologies struggle to balance temporal dynamics and spatial correlation, and that model update mechanisms are mostly static and offline, making them unable to adapt to data distribution drift caused by climate change.

[0009] According to a first aspect of this disclosure, a natural interaction method for AR glasses QR code payment scenarios is provided. The method includes:

[0010] The pre-action module is used to respond to the user's activation command, and the AR glasses acquire environmental images of the area to be captured; wherein, the user's hand is placed in the area to be captured with a specific gesture;

[0011] The QR code confirmation module is connected to the pre-action module and is used to identify whether the user has payment intent in the environmental image based on the specific gesture using a lightweight image recognition model. If so, the corresponding payment code is identified.

[0012] The payment confirmation module, connected to the QR code confirmation module, is used to determine whether the recognized payment code conforms to the static payment protocol specification. If so, the voice monitoring module is activated to obtain ambient sound, and the amount information is obtained based on the ambient sound. Payment is then made after the user confirms the amount information.

[0013] Furthermore, the lightweight image recognition model includes a model mapping module, an intent determination module, and a geometric affinity cropping module;

[0014] The model mapping module is used to generate a decoding guidance vector based on the environmental image;

[0015] The intent determination module is used to determine whether the user has a payment intent based on the decoded guidance vector.

[0016] The geometric affinity cropping module is used to crop and identify regions of interest based on the user's payment intent; wherein...

[0017] The decoding guidance vector includes intent confidence, normalized coordinates of fingertip keypoints, and normalized region of interest.

[0018] Furthermore, the model mapping module adopts the MobileNetV3-Small network;

[0019] The MobileNetV3-Small network comprises, in sequence, an input backbone layer, a bottleneck residual block stacking layer, and a multi-task output layer; wherein...

[0020] The input backbone layer sequentially includes a first 3×3 convolutional layer, a batch normalization layer, and a Hard-Swish activation function layer;

[0021] The bottleneck residual block stack layer includes several cascaded inverted residual structures, each of which sequentially includes a first 1×1 convolutional layer, a 3×3 or 5×5 depth-separable convolutional layer, an attention mechanism module, and a second 1×1 convolutional layer.

[0022] The multi-task output layer includes a multi-task output header, a global average pooling layer, and a branch processing layer, wherein the branch processing layer includes an intent classification branch and a location regression branch.

[0023] Furthermore, the determination process of the intent determination module includes:

[0024] Determine whether the confidence level of the intent is greater than or equal to a preset threshold;

[0025] If so, the normalized coordinates of the fingertip keypoints and the normalized region of interest are restored to the absolute pixel coordinates corresponding to the environment image;

[0026] Otherwise, it will be judged as a mistaken touch and terminated.

[0027] Furthermore, the geometric affinity cutting module specifically includes the following steps:

[0028] Detect all potential QR code center points within the normalized region of interest;

[0029] For each QR code center point, its affinity score relative to the fingertip is calculated;

[0030] The region of interest corresponding to the center point of the QR code with the highest affinity score is selected and cropped to obtain a result image containing the payment code. The result image is then input into a standard decoder for recognition.

[0031] The affinity score is obtained by multiplying the distance weight and the direction weight by their respective adjustment coefficients and then adding them together.

[0032] The distance weight is obtained based on the distance between the center point of each QR code and the fingertip;

[0033] The pointing weight is obtained based on the angle between the line connecting the fingertip direction vector and the fingertip, and the angle between the line connecting the fingertip direction vector and the center point of the QR code.

[0034] Furthermore, obtaining monetary information based on the ambient sound specifically includes the following steps:

[0035] The environmental sound is input into the large language model, which directly determines the prepayment amount in the environmental sound.

[0036] The large language model then outputs a set of candidate amounts in the environmental sound, which includes the candidate amount, the end timestamp corresponding to each candidate amount, and the original semantic probability corresponding to each candidate amount.

[0037] Based on the candidate amount set, a comprehensive confidence score is calculated for each candidate amount. The final payment amount is then determined based on the comprehensive confidence score and the pre-payment amount.

[0038] The overall confidence score is obtained by multiplying the original semantic probability by a time decay factor, and then by a contextual intent gain; the time decay factor is calculated based on the end timestamp corresponding to the candidate amount; the contextual intent gain is calculated based on positive intent indicator variables and negative intent indicator variables.

[0039] The positive intent indicator variable is determined by judging whether positive words appear before and after the candidate amount; the negative intent indicator variable is confirmed by judging whether negative words appear before and after the candidate amount.

[0040] Furthermore, the final payment amount is determined based on the comprehensive confidence score and the pre-payment amount, specifically including the following steps:

[0041] If the highest overall confidence score is greater than the autofill threshold, and the candidate amount corresponding to the highest overall confidence score is consistent with the prepayment amount, then the prepayment amount will be filled into the payment agreement field, and the system will enter the physical button waiting for payment authorization state.

[0042] If the highest overall confidence score is less than the autofill threshold but greater than the warning threshold, or if the candidate amount corresponding to the highest overall confidence score is inconsistent with the pre-payment amount, the AR glasses will display the amount information with a confirmation prompt on the display interface, and the user can confirm by nodding or pressing a physical button.

[0043] If the highest overall confidence score is less than or equal to the warning threshold, and the candidate amount corresponding to the highest overall confidence score is inconsistent with the pre-payment amount, then the task will be terminated.

[0044] According to a second aspect of this disclosure, a natural interaction system for AR glasses QR code payment scenarios is provided, comprising:

[0045] The pre-action module is used to respond to the user's activation command, and the AR glasses acquire environmental images of the area to be captured; wherein, the user's hand is placed in the area to be captured with a specific gesture;

[0046] The QR code confirmation module is connected to the pre-action module and is used to identify whether the user has payment intent in the environmental image based on the specific gesture using a lightweight image recognition model. If so, the corresponding payment code is identified.

[0047] The payment confirmation module, connected to the QR code confirmation module, is used to determine whether the recognized payment code conforms to the static payment protocol specification. If so, the voice monitoring module is activated to obtain ambient sound, and the amount information is obtained based on the ambient sound. Payment is then made after the user confirms the amount information.

[0048] According to a third aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement a natural interaction method in an AR glasses QR code payment scenario.

[0049] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon; the computer program is executed by a processor to implement a natural interaction method in an AR glasses QR code payment scenario.

[0050] The beneficial effects of this disclosure are:

[0051] By integrating the payment process with natural offline interpersonal interactions, precise targeting avoids accidental triggering, and innovatively embedding the payment amount input into the necessary dialogue between users and merchants, the awkwardness of users publicly reading out the amount is eliminated, and authorization is finally completed through physical triggering.

[0052] This disclosure solves the problems of multi-target selection and accidental touch through an image recognition model, and resolves the contradiction between privacy and efficiency in inputting amounts in public places through dialogue consensus extraction driven by a large language model.

[0053] First, a single-frame environmental image captured by the AR glasses' camera is acquired and input into a lightweight image recognition model, which directly outputs a decoding guidance vector containing intent confidence and target spatial parameters. When the intent confidence exceeds a preset threshold, a region of interest containing the user's hand pointing at the target is cropped from the environmental image, and the region is decoded as a QR code / barcode. When the decoding result is a static payment agreement requiring manual input of the amount, the voice monitoring module is automatically triggered to collect the natural dialogue audio between the user and the merchant. A large language model is used to perform semantic analysis and consensus extraction on the transcribed dialogue text to obtain the final transaction amount value after secondary confirmation, which is then automatically filled into the amount field of the payment agreement.

[0054] By using a finger to point naturally, the target QR code or barcode will be highlighted individually, eliminating interference from other QR codes or barcodes.

[0055] The amount input relies entirely on natural language dialogue with the merchant, requiring only a click on a physical button near the ear. The movement distance is extremely short, and the button itself has a large surface area; the movement distance can be defined as the time for physical button confirmation and the large model's understanding, and the movement distance is extremely short.

[0056] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0057] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0058] Figure 1 A flowchart of the natural interaction method in the AR glasses QR code payment scenario provided by the embodiments of this disclosure is shown;

[0059] Figure 2 This illustration shows a user-triggered process for AR glasses to acquire environmental images according to an embodiment of the present disclosure;

[0060] Figure 3 This document illustrates both incorrect and correct demonstrations of capturing environmental images according to embodiments of this disclosure.

[0061] Figure 4 This illustration shows a schematic diagram of scanning multiple QR codes using a frame-selection method, as provided in an embodiment of this disclosure.

[0062] Figure 5 A schematic diagram of the interaction process provided in an embodiment of this disclosure is shown;

[0063] Figure 6 A flowchart illustrating the natural interaction method for AR glasses QR code payment scenarios provided in this embodiment of the present disclosure is shown.

[0064] Figure 7 This diagram illustrates a framework of a natural interaction system for AR glasses QR code payment scenarios provided in this embodiment of the disclosure.

[0065] Figure 8 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0067] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0068] This disclosure provides a natural interaction method for AR glasses QR code payment scenarios. See [link to relevant documentation]. Figures 1 to 6 The method specifically includes the following steps:

[0069] S1. In response to the user's activation command, the AR glasses acquire an environmental image of the area to be captured; wherein, the user's hand is placed in the area to be captured with a specific gesture;

[0070] See Figure 2 The user first manually triggers the front-facing camera of the AR glasses to capture an image of the environment. Figure 1 This demonstrates how user gestures trigger the camera to capture an image. Typically, AR glasses only capture an image once for recognition to conserve battery life. Therefore, after activating the AR glasses, the finger should be positioned as soon as possible. Alternatively, a preset time can be set for delayed shooting to minimize the possibility of capturing invalid images. See also... Figure 3 This demonstrates both incorrect and correct methods for capturing environmental images.

[0071] S2. Based on the specific gesture, use a lightweight image recognition model to identify whether the user in the environmental image has the intention to pay. If so, identify the corresponding payment code.

[0072] The lightweight image recognition model constructed in this disclosure is used to perform recognition tasks. The collected environmental images are processed to determine whether the user intends to scan the code. If not, the task is stopped directly (considered as a false touch). If the user intends to scan the code, the location of the specific gesture and payment code in the environmental image is used to determine the user's intention to scan the corresponding payment code (QR code / barcode).

[0073] The aforementioned process is based on visual algorithms and does not require the user to issue any voice or large gesture commands, thus achieving silent and precise triggering of payment intent.

[0074] Specifically, the lightweight image recognition model includes a model mapping module, an intent determination module, and a geometric affinity cropping module. The model mapping module is used to generate a decoding guidance vector based on the environmental image. The intent determination module is used to determine whether the user has a payment intent based on the decoding guidance vector. The geometric affinity cropping module is used to crop the region of interest based on the user's payment intent and perform recognition. The decoding guidance vector includes intent confidence, normalized coordinates of fingertip keypoints, and the normalized region of interest.

[0075] (1) Model mapping module

[0076] The environmental image is fed into an end-to-end lightweight inference model, which uses an improved MobileNetV3-Small network as the backbone feature extraction network. The network structure consists of an input backbone layer, a bottleneck residual block stacking layer, and a multi-task output layer.

[0077] The input backbone layer includes a first 3×3 convolutional layer with a stride of 2, which is used to perform preliminary downsampling on the input environmental image, followed by a batch normalization layer and a Hard-Swish activation function layer.

[0078] The bottleneck residual block stacking layer consists of several cascaded inverted residual structures. Each inverted residual structure includes, in sequence, a first 1×1 convolutional layer for dimensionality upscaling, a 3×3 or 5×5 depthwise separable convolutional layer for feature extraction, an attention mechanism module for channel weighting, and a second 1×1 convolutional layer for dimensionality reduction output.

[0079] The multi-task output layer includes a multi-task output head, a global average pooling layer, and a branch processing layer. After passing through the global average pooling layer, the multi-task output head branches into an intent classification branch and a location regression branch. The output dimension of the intent classification branch is 1, and the intent confidence is obtained after activation by the Sigmoid function. The output dimension of the location regression branch is 6 (corresponding to the fingertip coordinates and parameters of the region of interest), which is used to generate the decoding guidance vector.

[0080] The optimal parameters of the model mapping module are obtained by minimizing the following joint loss function:

[0081] ,

[0082] in, Indicates the intention to classify losses. This represents the position regression loss. Indicates a positive sample mask. and This represents the weighting coefficient.

[0083] The output of the model mapping module is the decoding guide vector. Please refer to the following formula for details:

[0084] ,

[0085] in, , representing the confidence probability of the scanning intention; This represents the normalized coordinates of the identified fingertip key points. These represent the normalized x-coordinate and normalized y-coordinate of the key points at the fingertips, respectively. , represents the parameters of the normalized region of interest, where, and These represent the normalized x-coordinate and normalized y-coordinate of the center point of the region of interest, respectively. and These represent the normalized width and height of the region of interest, respectively. The values ​​for these parameters are all within the range of... Normalize relative to the overall resolution of the environmental image; Let I represent the mapping function, and let I represent the environment image.

[0086] In summary, shallow downsampling is performed on the input backbone layer, followed by extraction of lightweight features with both spatial and channel information from the bottleneck residual block stacking layer. Finally, the multi-task output layer outputs the intention confidence and location vector, providing a basis for subsequent calculations.

[0087] (2) Intent determination module

[0088] Based on the relationship between the intention confidence and the preset threshold, if the intention confidence is less than the preset threshold, it is considered a mistaken touch and the task is terminated immediately; otherwise, the normalized coordinates of the fingertip key points and the normalized region of interest are restored to the absolute pixel coordinates corresponding to the environment image.

[0089] The calculation of absolute pixel coordinates includes the following steps:

[0090] Let the width of the environment image be Gao Wei The parameters for absolute pixel coordinates are shown in the following formulas:

[0091]

[0092] in, This represents the absolute horizontal pixel coordinates of the center point of the region of interest in the environmental image. This represents the absolute vertical pixel coordinates of the center point of the region of interest in the environmental image. This represents the absolute pixel width of the region of interest in the environment image. This represents the pixel height of the region of interest in the original environment image. Furthermore, a similar principle can be used to restore the fingertip coordinates from the normalized coordinates of the fingertip keypoints. .

[0093] (3) Geometric Affinity Cutting Module

[0094] To solve the positioning ambiguity problem caused by the coexistence of multiple QR codes in complex backgrounds (i.e.) (Possibly covering multiple adjacent QR codes), this disclosure introduces a geometric affinity trimming module for final determination, including the following steps:

[0095] Detect all potential QR code center points within the normalized region of interest;

[0096] For each QR code center point, its affinity score relative to the fingertip is calculated;

[0097] The region of interest corresponding to the center point of the QR code with the highest affinity score is selected and cropped to obtain a result image containing the payment code. The result image is then input into a standard decoder for recognition.

[0098] Specifically, system detection All potential QR code center points are included to form a set of QR code center points. For the first Calculate the coordinates of the center point of the QR code relative to the fingertip. Affinity score The calculation is performed using the following formula:

[0099] ,

[0100] in, This represents the Euclidean distance; the closer the distance, the higher the score. This represents the function for calculating geometric affinity. This represents a numerical stability constant to prevent the denominator from being zero (e.g., a value that can take any value). ). This represents the angle between the line connecting the fingertip direction vector and the fingertip, and between the line connecting the fingertip direction vector and the center point of the QR code. The fingertip direction vector is derived from the recognized fingertip coordinates. The coordinates are calculated by vector difference with the coordinates of key points of adjacent finger joints (such as the distal interphalangeal joint). The smaller the angle, the more accurate the pointing and the higher the affinity score. and For example, the adjustment coefficient. ; The overall representation represents distance weight. This indicates the weight.

[0101] Select The corresponding region of interest is cropped to obtain the final image. And input it into a standard decoder for recognition:

[0102] ,

[0103] in, This represents an image cropping operator used to extract local sub-images from the original environment image based on determined region of interest parameters; This indicates the index corresponding to the maximum value in the affinity score set. This is used to lock onto the QR code that best matches the user's pointing.

[0104] In summary, the lightweight image recognition model first outputs a decoding guidance vector. This vector contains not only confidence in whether the user intends to scan the code, but also spatial parameters of the target region of interest. When the confidence level is high enough, the resulting image containing the payment code is directly cropped from the original high-resolution image (i.e., the environment image) based on these region of interest parameters and then fed into a standard barcode scanner for analysis and recognition.

[0105] This disclosure employs transfer learning to train the aforementioned lightweight image recognition model, and combines a multi-task loss function and a comprehensive decision method based on selection affinity to ensure the recognition accuracy of payment codes. It can also be adapted to other scenarios that require determining finger pointing.

[0106] S3. Determine whether the recognized payment code conforms to the static payment protocol specification. If so, start the voice monitoring module to obtain ambient sound, obtain the amount information based on the ambient sound, and make payment after the user confirms the amount information.

[0107] After recognizing the QR code / barcode, if the decoded information conforms to the preset static payment protocol specifications (such as passing the verification of a specific URL Scheme prefix or merchant identifier format), and it is determined that the current scenario is a transfer or counter payment (usually a static QR code / barcode that requires manual input of values), the context-aware voice monitoring module will be activated.

[0108] S41. Input the ambient sound into the large language model, and the large language model directly determines the prepayment amount in the ambient sound.

[0109] The voice monitoring module acquires ambient sound and uses a built-in large language model to recognize natural conversations between users and merchants within the ambient sound. (See also...) Figure 5 For example, if a user asks "Is 70 okay?" or a merchant says "75, please", the large language model can also capture and understand the voice dialogue that is used to correct errors or bargaining, in order to obtain a final amount information, which is used as the input value for static code payment and filled into the "amount" field in the static protocol.

[0110] The amount input process has been transformed from a command-based, publicly dictated human-machine interface to an amount confirmation-based interpersonal dialogue. Users do not need to change their natural social behavior to operate the device, while ensuring both efficiency and privacy of the input.

[0111] For example, when the large language model is kimi, Tongyi Qianwen, or Gemini, after the voice monitoring module is triggered, a dialogue state is maintained, and the physical button trigger is used as the end signal.

[0112] The dialogue that will take place (i.e., ambient sounds) and fixed-point prompts splicing (symbolized as) At the end of the dialogue context to form a complete reasoning input. For n, please refer to the following formula:

[0113] .

[0114] in, It is a set of optimized natural language instructions used to constrain the output format of a large language model and specify specific information extraction targets. In one specific embodiment, Configured with explicit extraction instructions, such as "Based on the above dialogue, what is the final confirmed payment amount? Please output only the number." This is achieved by... spliced ​​in the context of the dialogue The tail of the function can guide the large language model to ignore small talk or redundant information in the dialogue, force the large language model to focus on the recognition of monetary values, and output in a format that contains only numbers.

[0115] Input the reasoning n Input the large language model to obtain the original answer Numerical extraction regular expressions are applied to the original answers. Receive the prepayment amount :

[0116] ,

[0117] ,

[0118] Combining the above formulas, we get the following formula:

[0119] .

[0120] The above method directly identifies the pre-payment amount using a large language model. However, considering the interference of ambient noise in complex environments and the potential for amount recognition errors in dynamic dialogue scenarios such as bargaining and slips of the tongue, this disclosure adds a dynamic confidence scoring mechanism for double verification. Specifically, it maintains candidate amounts and determines the final payment amount by combining three dimensions: semantic probability, time decay factor, and contextual intent verification, thereby minimizing the error probability. The mechanism includes the following steps:

[0121] S42. The large language model then outputs a set of candidate amounts in the environmental sound. The set of candidate amounts includes candidate amounts, the end timestamp corresponding to each candidate amount, and the original semantic probability corresponding to each candidate amount.

[0122] After entering the voice monitoring module, set it to The system slices ambient sound in real time and inputs it into a large language model for processing. Typically, the large language model connected to the AR glasses and the phone can very accurately understand the final amount of most transactions. After processing by the large language model, the output includes... A candidate set M of candidate amounts:

[0123] ,

[0124] in, This represents the j-th candidate amount identified; This represents the end timestamp corresponding to the j-th candidate amount that appears in the dialogue; This represents the raw semantic probability extracted by the large language model for the j-th input value, typically taken as the output value of the Softmax layer, and its range is... .

[0125] S43. Calculate the comprehensive confidence score corresponding to each candidate amount based on the candidate amount set, and determine the final payment amount based on the comprehensive confidence score and the pre-payment amount.

[0126] In one specific embodiment, the comprehensive confidence score is obtained by multiplying the original semantic probability by a time-decay factor, and then by a contextual intent gain; the time-decay factor is calculated based on the end timestamp corresponding to the candidate amount; the contextual intent gain is calculated based on positive intent indicator variables and negative intent indicator variables.

[0127] The positive intent indicator variable is determined by judging whether positive words appear before and after the candidate amount; the negative intent indicator variable is confirmed by judging whether negative words appear before and after the candidate amount.

[0128] Specifically, calculate each candidate amount Overall confidence score Please refer to the following formula for details:

[0129]

[0130] in, Indicates the current moment. Indicates the attenuation constant; Represents the time-series decay factor. Overall, it represents the contextual intent gain. Indicator variables that indicate a positive intent Indicator variables that indicate negation. This indicates a positive reward coefficient. This indicates a negative penalty coefficient.

[0131] The time decay factor decreases as the interval between the current moment and the timestamp of the candidate amount increases, reflecting the priority of the most recently confirmed amount during the transaction. Based on trading conventions, the latest candidate amount overwrites the older candidate amounts. For example, in the query "Is 80 okay?" followed by "No, at least 90," the timestamp of the second candidate amount, 90, will be displayed. Timestamp later than the first candidate amount of 80 Therefore, the attenuation factor is larger, resulting in a higher score.

[0132] Determine candidate amount Does it follow immediately with affirmative words such as "okay," "scan this," or "sold"? If so, then... Otherwise, it is 0.

[0133] Determine candidate amount Does it contain negative words such as "no," "miscalculated," or "too expensive"? If so, then... Otherwise, it is 0.

[0134] Determine the candidate amount with the highest overall confidence score ,but Corresponding overall confidence score for:

[0135] ,

[0136] At this point, the dual insurance mechanism introduced in this disclosure can enter the determination stage, based on... The corresponding comprehensive confidence score determines the payment status, let's assume... To automatically fill in the threshold, such as 0.85. See below for details:

[0137] (1) If the highest overall confidence score is greater than the autofill threshold and the candidate amount corresponding to the highest overall confidence score is consistent with the prepayment amount, then the prepayment amount will be filled into the payment agreement field and the system will enter the physical button waiting for payment authorization state.

[0138] If the highest overall confidence score at the same time ( (This refers to the prepayment amount directly determined by the preceding large language model). To automatically fill in the threshold, such as 0.85, the system determines that the amount is the final amount agreed upon by both parties, automatically fills it into the payment agreement field, and enters the physical button waiting for authorization state.

[0139] (2) If the highest overall confidence score is less than the autofill threshold and greater than the warning threshold, or if the candidate amount corresponding to the highest overall confidence score is inconsistent with the pre-payment amount, the AR glasses will display the amount information with a confirmation prompt on the display interface, and the user can confirm by nodding or pressing a physical button.

[0140] like or ( (Indicating a warning threshold), although the amount is recognized, a risk is considered, such as in a noisy environment or if two amounts are similar. In this case, the AR glasses will display a number with a question mark on the screen, such as "80 yuan?". The user confirms by nodding or pressing a physical button, instead of directly entering "payment authorization." The user completes payment authorization by nodding or pressing the physical button on the AR glasses once or twice. The authorization process is extremely private and quick, and due to the deterministic and accidental touch-proof features of the physical button, it offers both high security and reliability.

[0141] (3) If the highest overall confidence score is less than or equal to the warning threshold, and the candidate amount corresponding to the highest overall confidence score is inconsistent with the prepayment amount, then the task shall be terminated.

[0142] like at the same time If the request is not accepted, the conversation will be considered invalid and no payment will be processed.

[0143] It should be noted that any other situations not included above will also be considered invalid dialogues, and the task will end immediately.

[0144] The efficiency of the method disclosed herein is judged based on Fitts's Law in the field of human-computer interaction, as detailed in the following formula:

[0145] ,

[0146] in, Difficulty level For the distance traveled, The target size is [value]. Calculations show that this disclosure significantly outperforms traditional mobile phone QR code scanning and existing AR interaction modes in terms of interaction efficiency:

[0147] Compared to the traditional mobile payment process, mobile payments involve a lengthy process: unlocking the phone, finding the app, opening the scanner, aligning the screen, tapping the input box, entering the numbers, and tapping confirm. This requires the finger to move frequently across different parts of the screen. (High value), leading to The value is high, and the operation takes a long time.

[0148] Compared to existing AR voice input payment processes, the length of voice commands and the time spent thinking and speaking can be considered as the distance traveled. Not only does it have a high It is valuable, but the error rate is relatively high. If a mistake is made, it is necessary to start over from scratch using physical methods.

[0149] Compared with the existing AR physical input payment process, it should be noted that due to the extremely limited physical operation, no AR glasses currently support virtual keyboard input. This disclosure assumes that AR glasses can achieve physical input through limited sliding input of numbers: Traditional AR glasses require users to select the corresponding numbers in the virtual panel by sliding up, down, left, and right, which has an extremely high D value and an extremely high error rate. The difficulty of sliding to delete after inputting an error is the same as the difficulty of inputting numbers.

[0150] Based on the above technical solution, this disclosure has the following advantages:

[0151] This disclosure solves the problems of multi-target selection and accidental touch through an image recognition model, and resolves the contradiction between privacy and efficiency in inputting amounts in public places through dialogue consensus extraction driven by a large language model.

[0152] First, a single-frame environmental image captured by the AR glasses' camera is acquired and input into a lightweight image recognition model, which directly outputs a decoding guidance vector containing intent confidence and target spatial parameters. When the intent confidence exceeds a preset threshold, a region of interest containing the user's hand pointing at the target is cropped from the environmental image, and the region is decoded as a QR code / barcode. When the decoding result is a static payment agreement requiring manual input of the amount, the voice monitoring module is automatically triggered to collect the natural dialogue audio between the user and the merchant. A large language model is used to perform semantic analysis and consensus extraction on the transcribed dialogue text to obtain the final transaction amount value after secondary confirmation, which is then automatically filled into the amount field of the payment agreement.

[0153] By using a finger to point naturally, the target QR code or barcode will be highlighted individually, eliminating interference from other QR codes or barcodes.

[0154] The amount input relies entirely on natural language dialogue with the merchant, requiring only a click on a physical button near the ear. The movement distance is extremely short, and the button itself has a large surface area; the movement distance can be defined as the time for physical button confirmation and the large model's understanding, and the movement distance is extremely short.

[0155] This disclosure simplifies the interaction process and reduces the interaction difficulty index to the theoretical minimum, achieving a seamless payment experience.

[0156] Through experimental testing and research, compared with explicit voice command-based AR payment methods, users showed a very high preference for the implicit dialogue confirmation payment method proposed in this disclosure.

[0157] In public testing scenarios such as convenience stores and elevators, explicit voice commands can cause stares from those around the user, creating significant social pressure. This invention, however, utilizes natural conversation between the user and the merchant as input to achieve invisible interaction.

[0158] It should be noted that, for the foregoing embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0159] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0160] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing embodiments, and will not be repeated here.

[0161] This disclosure also provides a natural interaction system 700 for AR glasses QR code payment scenarios. See [link to relevant documentation] Figure 7 Specifically, it includes:

[0162] The pre-action module 710 is used to respond to the user's activation command, and the AR glasses acquire an environmental image of the area to be captured; wherein the user's hand is placed in the area to be captured with a specific gesture;

[0163] The QR code confirmation module 720 is connected to the pre-action module 710 and is used to identify whether the user in the environmental image has payment intention based on the specific gesture using a lightweight image recognition model. If so, the corresponding payment code is identified.

[0164] The payment confirmation module 730, connected to the QR code confirmation module 720, is used to determine whether the recognized payment code conforms to the static payment protocol specification. If so, the voice monitoring module is activated to obtain ambient sound, and the amount information is obtained based on the ambient sound. Payment is made after the user confirms the amount information.

[0165] For other details, please refer to the methods described earlier; they will not be repeated here.

[0166] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0167] Electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in ROM 802 or a computer program loaded into RAM 803 from storage unit 808. RAM 803 can also store various programs and data required for the operation of electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. I / O interface 805 is also connected to bus 804.

[0168] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 805, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0169] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various processes described above. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809.

[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] The program code used to implement the schemes of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0175] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0176] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A natural interaction method for AR glasses QR code payment scenarios, characterized in that, Includes the following steps: In response to the user's activation command, the AR glasses acquire an environmental image of the area to be captured; wherein, the user's hand is placed in the area to be captured with a specific gesture; Based on the specific gesture, a lightweight image recognition model is used to identify whether the user in the environmental image has the intention to make a payment. If so, the corresponding payment code is identified. The system determines whether the recognized payment code conforms to the static payment protocol specification. If so, it activates the voice monitoring module to acquire ambient sound and obtains the amount information based on the ambient sound. Payment is then made after the user confirms the amount information.

2. The natural interaction method in an AR glasses QR code payment scenario according to claim 1, characterized in that, The lightweight image recognition model includes a model mapping module, an intent determination module, and a geometric affinity cropping module; The model mapping module is used to generate a decoding guidance vector based on the environmental image; The intent determination module is used to determine whether the user has a payment intent based on the decoded guidance vector. The geometric affinity cropping module is used to crop and identify regions of interest based on the user's payment intent; wherein... The decoding guidance vector includes intent confidence, normalized coordinates of fingertip keypoints, and normalized region of interest.

3. The natural interaction method in an AR glasses QR code payment scenario according to claim 2, characterized in that, The model mapping module uses the MobileNetV3-Small network; The MobileNetV3-Small network comprises, in sequence, an input backbone layer, a bottleneck residual block stacking layer, and a multi-task output layer; wherein... The input backbone layer sequentially includes a first 3×3 convolutional layer, a batch normalization layer, and a Hard-Swish activation function layer; The bottleneck residual block stack layer includes several cascaded inverted residual structures, each of which sequentially includes a first 1×1 convolutional layer, a 3×3 or 5×5 depth-separable convolutional layer, an attention mechanism module, and a second 1×1 convolutional layer. The multi-task output layer includes a multi-task output header, a global average pooling layer, and a branch processing layer, wherein the branch processing layer includes an intent classification branch and a location regression branch.

4. The natural interaction method in an AR glasses QR code payment scenario according to claim 2, characterized in that, The determination process of the intent determination module includes: Determine whether the confidence level of the intent is greater than or equal to a preset threshold; If so, the normalized coordinates of the fingertip keypoints and the normalized region of interest are restored to the absolute pixel coordinates corresponding to the environment image; Otherwise, it will be judged as a mistaken touch and terminated.

5. The natural interaction method in an AR glasses QR code payment scenario according to claim 1, characterized in that, The geometric affinity cutting module specifically includes the following steps: Detect all potential QR code center points within the normalized region of interest; For each QR code center point, its affinity score relative to the fingertip is calculated; The region of interest corresponding to the center point of the QR code with the highest affinity score is selected and cropped to obtain a result image containing the payment code. The result image is then input into a standard decoder for recognition. The affinity score is obtained by multiplying the distance weight and the direction weight by their respective adjustment coefficients and then adding them together. The distance weight is obtained based on the distance between the center point of each QR code and the fingertip; The pointing weight is obtained based on the angle between the line connecting the fingertip direction vector and the fingertip, and the angle between the line connecting the fingertip direction vector and the center point of the QR code.

6. The natural interaction method in an AR glasses QR code payment scenario according to claim 1, characterized in that, Obtaining monetary information based on the ambient sound specifically includes the following steps: The environmental sound is input into the large language model, which directly determines the prepayment amount in the environmental sound. The large language model then outputs a set of candidate amounts in the environmental sound, which includes the candidate amount, the end timestamp corresponding to each candidate amount, and the original semantic probability corresponding to each candidate amount. Based on the candidate amount set, a comprehensive confidence score is calculated for each candidate amount. The final payment amount is then determined based on the comprehensive confidence score and the pre-payment amount. The overall confidence score is obtained by multiplying the original semantic probability by a time decay factor, and then by a contextual intent gain; the time decay factor is calculated based on the end timestamp corresponding to the candidate amount; the contextual intent gain is calculated based on positive intent indicator variables and negative intent indicator variables. The positive intent indicator variable is determined by judging whether positive words appear before and after the candidate amount; the negative intent indicator variable is confirmed by judging whether negative words appear before and after the candidate amount.

7. The natural interaction method in an AR glasses QR code payment scenario according to claim 6, characterized in that, The final payment amount is determined based on the comprehensive confidence score and the pre-payment amount, specifically including the following steps: If the highest overall confidence score is greater than the autofill threshold, and the candidate amount corresponding to the highest overall confidence score is consistent with the prepayment amount, then the prepayment amount will be filled into the payment agreement field, and the system will enter the physical button waiting for payment authorization state. If the highest overall confidence score is less than the autofill threshold but greater than the warning threshold, or if the candidate amount corresponding to the highest overall confidence score is inconsistent with the pre-payment amount, the AR glasses will display the amount information with a confirmation prompt on the display interface, and the user can confirm by nodding or pressing a physical button. If the highest overall confidence score is less than or equal to the warning threshold, and the candidate amount corresponding to the highest overall confidence score is inconsistent with the pre-payment amount, then the task will be terminated.

8. A natural interaction system for AR glasses QR code payment scenarios, used to implement the natural interaction method for AR glasses QR code payment scenarios as described in any one of claims 1-7, characterized in that, The system includes: The pre-action module is used to respond to the user's activation command, and the AR glasses acquire environmental images of the area to be captured; wherein, the user's hand is placed in the area to be captured with a specific gesture; The QR code confirmation module is connected to the pre-action module and is used to identify whether the user has payment intent in the environmental image based on the specific gesture using a lightweight image recognition model. If so, the corresponding payment code is identified. The payment confirmation module, connected to the QR code confirmation module, is used to determine whether the recognized payment code conforms to the static payment protocol specification. If so, the voice monitoring module is activated to obtain ambient sound, and the amount information is obtained based on the ambient sound. Payment is then made after the user confirms the amount information.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement a natural interaction method in an AR glasses QR code payment scenario as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program; the computer program is executed by a processor to implement a natural interaction method in an AR glasses QR code payment scenario as described in any one of claims 1 to 7.

Images