An interaction control method, system and device for smart glasses

Abstract

The application discloses an interactive control method, system and device for smart glasses, and relates to the field of smart glasses interaction, and the method comprises the following steps: acquiring multi-modal interaction information through a multi-modal data acquisition device; extracting interaction features from the multi-modal interaction information, and identifying an interaction intention trigger point; analyzing a co-occurrence frequency matrix between modal pairs according to the trigger point, and obtaining a co-occurrence frequency vector; inputting the co-occurrence frequency vector into a credibility evaluation module, calculating a credibility score of the interaction intention, and if the score is greater than a preset threshold, triggering an interaction control instruction, and the smart glasses execute a control program. The technical problem that the existing smart glasses interaction mode relies on a single mode, resulting in a high false triggering rate and unstable user experience, is solved, the technical effect that the multi-modal cross verification and dynamic intention credibility evaluation reduce the false operation rate of the smart glasses and improve the multi-modal interaction experience is achieved.

Description

Technical Field

[0001] This application relates to the field of smart glasses interaction, and in particular to an interaction control method, system and device for smart glasses. Background Technology

[0002] With the continuous development of smart glasses technology, more and more application scenarios require smart glasses to have efficient and accurate interactive control capabilities. Traditional smart glasses interaction methods usually rely on single-modal input, such as voice commands or touch operations. However, single-modal interaction methods are often affected by environmental noise, user movement restrictions, or external interference, leading to frequent false triggers and interaction failures. To solve this problem, multimodal interaction control solutions have emerged in recent years. These solutions combine multiple input methods such as voice, eye tracking, head posture, and facial expressions, and improve the accuracy and reliability of interaction by comprehensively analyzing various information.

[0003] However, existing technologies still face some challenges, especially in the fusion and analysis of multimodal information. How to extract effective interaction features from multimodal data, accurately identify user interaction intentions, and prevent accidental triggering are important issues in current smart glasses interaction technology. Traditional methods often lack in-depth analysis of the synergistic effects of multimodal information, making the interaction control system susceptible to errors and environmental factors, thus affecting the user experience.

[0004] Therefore, there is an urgent need for a new interactive control method that can fully utilize the complementarity of multimodal data to improve the accuracy of interactive control of smart glasses, enhance their resistance to accidental touches, and thus meet the increasingly complex and varied usage needs. Summary of the Invention

[0005] This application provides an interactive control method, system, and device for smart glasses, which solves the technical problem that existing smart glasses interaction methods rely on a single modality, resulting in a high false trigger rate and unstable user experience. It achieves the technical effect of reducing the false operation rate of smart glasses and improving the multimodal interactive experience through multimodal cross-validation and dynamic intent credibility assessment.

[0006] This application provides an interactive control method for smart glasses, wherein the smart glasses are embedded with a multimodal data acquisition device, comprising: acquiring multimodal interaction information by the multimodal data acquisition device, the multimodal interaction information including eye movement trajectory, head posture data, voice input signal and facial expression changes stored in time sequence; extracting interaction features from the multimodal interaction information to identify multimodal interaction intent trigger points; analyzing the co-occurrence frequency matrix between modal pairs based on the multimodal interaction intent trigger points to obtain a modal co-occurrence frequency vector; inputting the modal co-occurrence frequency vector into an interaction intent credibility evaluation module to calculate the credibility score of the current interaction intent; if the credibility score is greater than a first preset credibility threshold, triggering an interaction control command, and the smart glasses executing a control program according to the interaction control command.

[0007] This application also provides an interactive control system for smart glasses, wherein the smart glasses are embedded with a multimodal data acquisition device, comprising: a data acquisition unit: acquiring multimodal interaction information by the multimodal data acquisition device, the multimodal interaction information including eye movement trajectory, head posture data, voice input signal and facial expression changes stored in time sequence; a feature extraction unit: extracting interaction features from the multimodal interaction information and identifying multimodal interaction intent trigger points; a modality analysis unit: analyzing the co-occurrence frequency matrix between modal pairs based on the multimodal interaction intent trigger points to obtain a modality co-occurrence frequency vector; and an interaction control unit: inputting the modality co-occurrence frequency vector into an interaction intent credibility evaluation module to calculate the credibility score of the current interaction intent. If the credibility score is greater than a first preset credibility threshold, an interaction control command is triggered, and the smart glasses execute a control program according to the interaction control command.

[0008] This application also provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing an interactive control method for smart glasses when executing the executable instructions stored in the memory.

[0009] This application proposes an interactive control method, system, and device for smart glasses. First, a multimodal data acquisition device acquires multimodal interaction information, including time-series stored eye movement trajectories, head posture data, voice input signals, and facial expression changes. Then, interactive features are extracted from the multimodal interaction information to identify multimodal interaction intent trigger points. Next, the co-occurrence frequency matrix between modal pairs is analyzed based on the multimodal interaction intent trigger points to obtain a modal co-occurrence frequency vector. Finally, the modal co-occurrence frequency vector is input into an interaction intent credibility assessment module to calculate the credibility score of the current interaction intent. If the credibility score is greater than a first preset credibility threshold, an interaction control command is triggered, and the smart glasses execute the control program according to the interaction control command. This achieves the technical effect of reducing the error rate of smart glasses and improving the multimodal interaction experience. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0011] Figure 1 This is a flowchart illustrating an interactive control method for smart glasses provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the structure of an interactive control system for smart glasses provided in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0014] Explanation of reference numerals in the attached figures: Data acquisition unit 11, Feature extraction unit 12, Modal analysis unit 13, Interactive control unit 14, Processor 21, Memory 22, Input device 23, Output device 24. Detailed Implementation

[0015] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below.

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0018] This application provides an interactive control method for smart glasses, wherein the smart glasses are embedded with a multimodal data acquisition device, such as... Figure 1 As shown, the method includes:

[0019] Multimodal interaction information is acquired by the multimodal data acquisition device, including eye movement trajectories, head posture data, voice input signals, and facial expression changes stored in time series.

[0020] Specifically, the smart glasses integrate a multimodal data acquisition device. This device simultaneously collects various interaction signals from the user during use, encompassing multi-dimensional information input, including but not limited to eye movement tracking, head posture data, voice input signals, and facial expression changes. Eye movement tracking is achieved through real-time recording of the user's eye movement path by eye-tracking sensors, reflecting their gaze direction, gaze duration, and saccade pattern. Head posture data utilizes inertial sensors such as gyroscopes and accelerometers to detect the user's head rotation angle and movement state, used to identify natural postures such as nodding and shaking the head. Voice input signals are collected through a microphone array to capture user voice commands, used to identify keywords or semantic information. Facial expression changes are captured by an embedded camera to show changes in the user's facial muscles, used to determine emotions or specific facial expression intentions (such as opening the mouth, blinking, smiling, etc.). All multimodal information collected by the multimodal data acquisition device is recorded synchronously in chronological order, forming a continuous interactive data stream with time labels—the multimodal interaction information. This provides a comprehensive and real-time input foundation for subsequent interaction feature extraction and intent recognition. This approach significantly improves the contextual integrity of the interaction, enabling the system to make more accurate interaction judgments.

[0021] Interaction features are extracted from the multimodal interaction information to identify the trigger points of multimodal interaction intentions.

[0022] Specifically, after collecting multimodal interaction information, the collected information is divided into blocks according to information type and stored. Each block is responsible for storing a type of data, such as eye movement trajectory and head posture data. Then, multiple interaction feature extraction blocks are invoked, and each interaction feature extraction block is matched with an information storage block to ensure a one-to-one correspondence between the interaction feature extraction blocks and the information storage blocks. The interaction feature extraction blocks are then used to extract key interaction-related features from the modal interaction information stored in the information storage blocks. These features reflect the user's interaction intent. For example, the gaze persistence feature of eye movement trajectory can reveal the user's attention focus area, and the directional features of head posture can reflect whether the user's intent is related to a certain direction or target. By extracting these modal features, the trigger points of the user's multimodal interaction intent can be identified. These trigger points serve as the basis for subsequent interaction processing, helping the system to more accurately understand user needs and respond accordingly.

[0023] In one possible implementation, the method involves extracting interaction features from the multimodal interaction information and identifying multimodal interaction intent trigger points, including:

[0024] The multimodal interaction information is divided into blocks and stored to obtain multiple interaction information storage blocks; multiple interaction feature extraction blocks are set, and the multiple interaction feature extraction blocks are correspondingly connected to the multiple interaction information storage blocks. The multiple interaction feature extraction blocks include pre-stored eye-tracking interaction features, head posture interaction features, voice keyword interaction features, and facial expression interaction features; wherein, each interaction feature extraction block is used to receive the modal data of the interaction information storage blocks, extract interaction features, and output multimodal interaction intent trigger points.

[0025] Specifically, in order to improve the efficiency and accuracy of multimodal interaction information processing, the continuously collected multimodal interaction information is first segmented according to content features and divided into multiple interaction information storage blocks. Each block stores any one of the following data: eye movement trajectory, head posture data, voice input signal, and facial expression changes. Subsequently, multiple interaction feature extraction blocks are configured, each corresponding one-to-one with the interaction information storage block. These blocks are used to process and analyze the modal data within their respective blocks. Each interaction feature extraction block pre-defines feature extraction logic for different modalities. For the interaction information storage block storing eye-tracking trajectories, the interaction feature extraction block pre-stored with eye-tracking interaction features is used to analyze the block. Eye-tracking interaction features include gaze persistence and rapid lookback. Gazing persistence can be determined by accumulating the timestamps of consecutive gaze coordinates in the eye-tracking trajectory. When a user's gaze lingers in a small area for an extended period, such as more than 300ms, that area is marked as a candidate intent trigger point. Rapid lookback can be determined by recording the time and number of times the gaze enters a certain area. If, within a set time window (e.g., 2 seconds), the user's gaze enters the same area multiple times (e.g., 2 times or more), that area is marked as a candidate intent trigger point. For the interactive information storage block that stores head posture data, an interactive feature extraction block pre-stored with head posture interaction features is used to analyze the interactive information storage block. Head posture interaction features include features such as rotation direction and rotation amplitude. When the rotation direction in the head posture data is up or down, the number of times the rotation amplitude meets the amplitude threshold for upward or downward movement is counted. If the count meets the nodding threshold, it indicates that the user is relatively satisfied with the currently gazed area, and this area is marked as a candidate intent trigger point. For the interactive information storage block that stores voice input signals, an interactive feature extraction block pre-stored with voice keyword interaction features is used to analyze the interactive information storage block. Voice keyword interaction features include multiple keywords, such as "confirm," "like," "dislike," and "open." When the voice input signal shows that the user has responded with a keyword to a certain area, this area is marked as a candidate intent trigger point. For the interaction information storage block containing facial expression changes, an interaction feature extraction block pre-stored with facial expression interaction features is used to analyze this block. Facial expression interaction features include multiple expression information, such as raising eyebrows, smiling, and opening the mouth. When a user looks at a certain area and smiles, it may indicate that the user has an intention towards that area. In this case, the area is marked as a candidate intention trigger point. After all interaction feature extraction blocks have processed the stored data, all candidate intention trigger points are summarized to obtain multimodal interaction intention trigger points, providing input for subsequent modal co-occurrence analysis and credibility assessment.

[0026] Based on the analysis of the co-occurrence frequency matrix between the modal pairs according to the multimodal interaction intent trigger point, the modal co-occurrence frequency vector is obtained.

[0027] Specifically, by utilizing the identified multimodal interaction intent trigger points, the study analyzes whether synchronous or collaborative triggering behaviors exist between different modalities. The aim is to assess whether these modalities frequently occur simultaneously in the same interaction, thereby constructing a more reliable basis for judging interaction intent. Specifically, the multimodal interaction intent trigger points are processed based on their timestamps, retaining trigger point information within a preset time window (e.g., 10 seconds). Subsequently, all modal interaction intent trigger points are paired to form several modal pairs, such as eye-tracking-speech, speech-facial expression, etc., to statistically analyze whether these modal combinations are frequently triggered simultaneously. Then, these modal pairs are analyzed using defined co-occurrence criteria to calculate their co-occurrence frequency. Finally, the co-occurrence frequencies of all modal pairs are arranged in rows and columns to generate a symmetric co-occurrence frequency matrix, where each element represents the collaborative strength of the corresponding modal pair. To facilitate subsequent credibility assessment, the co-occurrence frequency matrix is ​​transformed into a modality co-occurrence frequency vector based on multimodal intent weights. This vector is a concise representation of the degree of collaboration between multimodalities; a higher value indicates that the modalities are more likely to belong to the same interaction intent. Through this series of processes, not only are collaborative relationships between modalities established, but a data foundation is also laid for subsequent credibility assessment of interaction intents, thereby improving the accuracy and robustness of multimodal fusion decision-making.

[0028] In one possible implementation, analyzing the co-occurrence frequency matrix between modal pairs based on the multimodal interaction intent trigger point includes the following method:

[0029] The system acquires multimodal interaction intent trigger points, including trigger points for each modality within a preset time window; constructs modal pairs, which are acquired by combining all modalities in pairs; sets co-occurrence determination conditions, counts the number of co-occurrences for each modal pair according to the co-occurrence determination conditions, calculates the co-occurrence frequency based on the counted number of co-occurrences, and generates a co-occurrence frequency matrix.

[0030] Specifically, after processing the obtained multimodal interaction intent trigger points, the trigger points of each modality within the multimodal interaction intent trigger points are all within the same preset time window. Then, all modalities are randomly paired to form multiple modal pairs, such as eye-tracking-head posture modal pairs, eye-tracking-speech modal pairs, etc. Next, a co-occurrence determination condition is set, and based on this condition, it is determined whether each modal pair co-occurs within the same time window. Co-occurrence is typically defined based on temporal proximity; that is, if the time difference between the trigger points of two modalities is less than a preset threshold, then the two modal trigger points are considered to co-occur, meaning they interacted within a relatively short period and may represent the same interaction intent. For each modal pair, the number of times they co-occur in all interactions is counted; that is, each modal pair is checked to see if it meets the co-occurrence determination condition within the time window, and the number of co-occurrence events is recorded. For example, if the eye-tracking-speech modal pair is co-triggered 5 times within the preset time window, then their co-occurrence count is 5. Next, the co-occurrence frequency (COF) of each modality pair is calculated. COF is the ratio of the number of times a modality pair co-occurs to the number of its trigger points. After obtaining the COF for each modality pair, a COF matrix is ​​generated based on these COFs. Each element of this matrix represents the COF between a pair of modalities. This matrix provides data support for subsequent analysis and helps assess the degree of synergy between different modal combinations. In summary, through the above steps, the interaction patterns between different modalities can be effectively analyzed, potential trigger points of user intent can be identified, and a precise basis can be provided for subsequent credibility assessment and interaction control.

[0031] In one possible implementation, it is determined whether the interaction intent trigger point of each modal pair satisfies the co-occurrence determination condition. If the co-occurrence determination condition is satisfied, a co-occurrence event is recorded. The co-occurrence determination condition is that the time difference between the interaction intent trigger points of the modal pair is less than a preset time difference threshold.

[0032] Specifically, for each modality pair, the interaction intent trigger point data within a preset time window is first acquired, including the timestamps of the trigger points. Then, the time difference between the trigger points of the two modalities in each pair is calculated and compared with a preset time difference threshold (e.g., 300ms). When the time difference between the trigger points of the modal pair is less than the preset time difference threshold, the two modalities are considered to meet the co-occurrence condition in time, meaning they are triggered synchronously. Conversely, if the time difference is greater, the two modalities are considered not to have triggered within the same time period and do not meet the co-occurrence condition. For each modal pair that meets the co-occurrence condition, a co-occurrence event is recorded, indicating that the two modalities have a common interaction intent trigger within the same time period. This process is repeated for all modal pairs, performing the above checks and statistics, and recording co-occurrence events, thus providing data support for subsequent modal co-occurrence frequency calculations and interaction intent credibility assessments.

[0033] In one possible implementation, the co-occurrence frequency is calculated based on the statistically obtained co-occurrence counts. The co-occurrence frequency is the ratio of the co-occurrence counts of the mode pair to the total number of trigger points, where the total number of trigger points is min(|T). i |,|T j |); where T i T represents the number of interaction intent trigger points corresponding to the i-th modality. j This represents the number of interaction intent trigger points corresponding to the j-th modality.

[0034] Specifically, the number of times C, for each pair of modalities, satisfies the co-occurrence condition throughout the entire interaction process is statistically obtained. i，j Afterwards, the total number of trigger points for the i-th mode and the j-th mode will be counted separately, denoted as T. i and T j These two values ​​represent the total number of times the interaction intent trigger points occurred independently for each of the two modalities throughout the entire acquisition time. Subsequently, to avoid frequency calculation imbalances caused by differences in trigger frequencies between different modalities, min(|T) is used. i |,|T j |) is used as the normalization base for the co-occurrence frequencies of mode pairs, by using C i，j with min(|T) i |,|T j |) Perform ratio calculation to obtain the co-occurrence frequency of mode i and mode j.

[0035] For example, if the eye-tracking modality is triggered 50 times, the speech modality is triggered 30 times, and the eye-tracking and speech modalities co-occur 15 times, then their co-occurrence frequency is 15 ÷ min(50, 30) = 0.5.

[0036] In one possible implementation, the modal co-occurrence frequency vector can be obtained by methods including:

[0037] Construct a multimodal intent weight matrix; perform matrix calculations on the co-occurrence frequency matrix according to the multimodal intent weight matrix to obtain the modal co-occurrence frequency vector, which is used to represent the synchronization degree of interaction intent trigger points between modalities.

[0038] Specifically, a multimodal intent weight matrix is ​​constructed based on prior knowledge and domain experience. This matrix represents the relative importance between different modalities, with each element representing the weight between modality i and modality j, reflecting the contribution of each modality pair to the interaction intent. Subsequently, based on this multimodal intent weight matrix, a matrix calculation is performed on the co-occurrence frequency matrix. Specifically, matrix multiplication is used to apply the multimodal intent weight matrix to the co-occurrence frequency matrix, resulting in a modal co-occurrence frequency vector. This vector is obtained after weighted calculation, where each element represents the degree of synchronization of the interaction intent between modality i and other modalities. This reflects the overall synergy between modal pairs, providing data support for the accuracy and anti-misclick capability of interactive control.

[0039] The modal co-occurrence frequency vector is input into the interaction intent credibility assessment module to calculate the credibility score of the current interaction intent. If the credibility score is greater than the first preset credibility threshold, an interaction control command is triggered, and the smart glasses execute the control program according to the interaction control command.

[0040] Specifically, after obtaining the modal co-occurrence frequency vector, it is input into the interaction intent credibility assessment module. This module calculates the data in the modal co-occurrence frequency vector based on the connected fuzzy rule engine, analyzes the degree of coordination between different modalities, and derives a credibility score. The credibility score reflects the degree of acceptance of the current interaction intent judgment; the higher the score, the more consistent the trigger points of multiple modalities. Subsequently, the calculated credibility score is compared with a preset first credibility threshold. If the calculated credibility score is higher than the threshold, it indicates that the recognition of the current interaction intent is accurate and reliable. At this time, a corresponding interaction control command is triggered, which instructs the smart glasses to perform a specific operation or task, such as adjusting the display content, launching an application, or executing a voice command. Finally, the smart glasses perform the actual operation according to the triggered interaction control command and the predetermined control program. In this way, it can make intelligent responses based on the user's multimodal interaction intent, thereby improving the accuracy and smoothness of the interaction experience, while avoiding false triggers and improving the reliability and intelligence of the interaction.

[0041] In one possible implementation, the modal co-occurrence frequency vector is input into the interaction intent credibility assessment module to calculate the credibility score of the current interaction intent. The method includes:

[0042] A fuzzy interval is defined, comprising a first co-occurrence frequency region, a second co-occurrence frequency region, and a third co-occurrence frequency region. The first co-occurrence frequency region represents high co-occurrence frequencies, the second co-occurrence frequency region represents common co-occurrence frequencies, and the third co-occurrence frequency region represents low co-occurrence frequencies. Based on the fuzzy interval, credibility rules are set, and a fuzzy rule engine is encapsulated using these credibility rules. This engine is then connected to the interaction intent credibility assessment module to calculate credibility scores for the modal co-occurrence frequency vector and output the credibility score results.

[0043] Specifically, a fuzzy interval is defined based on different modal co-occurrence frequencies. This fuzzy interval is used to divide different co-occurrence frequency values ​​into multiple regions to assess the credibility of interaction intent. The divided fuzzy intervals include a first co-occurrence frequency region, a second co-occurrence frequency region, and a third co-occurrence frequency region. The first co-occurrence frequency region (high co-occurrence frequency) indicates a strong synchronous cooperative relationship between modal pairs, typically corresponding to a high co-occurrence frequency. A high co-occurrence frequency indicates that multiple modalities are triggered simultaneously very frequently, making the identification of interaction intent relatively reliable. The second co-occurrence frequency region (moderate co-occurrence frequency) indicates a moderate cooperative relationship between modal pairs, typically corresponding to a moderate co-occurrence frequency. A moderate co-occurrence frequency means that modalities are triggered synchronously in some interactions, but not as strongly as in the high co-occurrence frequency region. The third co-occurrence frequency region (low co-occurrence frequency) indicates a weak synchronous relationship between modal pairs, typically corresponding to a low co-occurrence frequency. A low co-occurrence frequency indicates less cooperative interaction between modalities, making the identification of interaction intent less certain. This division allows for the differentiation of modal cooperative relationships of different strengths, thus providing a basis for credibility calculation. Subsequently, credibility rules are established based on the aforementioned fuzzy intervals. These rules define how to calculate credibility scores based on different co-occurrence frequency regions. Specifically, the credibility score is determined based on the region where the co-occurrence frequency of each modality pair is located. For the first co-occurrence frequency region, the credibility score is high, possibly close to the full score, indicating a relatively accurate confirmation of the interaction intent. For the second co-occurrence frequency region, the credibility score is moderate, indicating a generally average confirmation of the interaction intent. For the third co-occurrence frequency region, the credibility score is low, indicating a weak confirmation of the interaction intent. These rules help assess the relationship between the degree of collaboration between modalities and the credibility score. Next, the aforementioned credibility rules are encapsulated into fuzzy rules, and a fuzzy rule engine is constructed. The fuzzy rule engine is an intelligent decision-making component that can output a credibility score for the interaction intent based on the input co-occurrence frequency vector and according to the fuzzy rules. The engine categorizes the co-occurrence frequency values ​​of each modality pair, determines its co-occurrence frequency region, and then matches it according to the credibility rules of the corresponding region to obtain a credibility score. Then, the fuzzy rule engine is connected to the interaction intent credibility assessment module. The interaction intent credibility assessment module can receive the modal co-occurrence frequency vector from the system, and can also encapsulate the credibility score calculated by the fuzzy rule engine into a credibility score result and send it back to the system to reflect the reliability of the current interaction intent, ensuring that the corresponding control command can only be triggered under high credibility conditions, and reducing the probability of erroneous operation events.

[0044] In one possible implementation, after calculating the credibility score of the current interaction intent, the method further includes:

[0045] If the credibility score is less than or equal to the first preset credibility threshold, determine whether the credibility score is greater than the second preset credibility threshold, wherein the second preset credibility threshold is less than the first preset credibility threshold; if the credibility score is greater than the second preset credibility threshold, send a waiting confirmation instruction to the user wearing the smart glasses to prompt the user whether to trigger an interactive control instruction.

[0046] Specifically, after the interaction intent credibility assessment module outputs the credibility score of the current interaction intent, it compares the credibility score with a first preset credibility threshold. If the credibility score is less than or equal to the first preset credibility threshold, it indicates that the system's judgment of the current interaction intent is uncertain, and the interaction control command cannot be directly triggered. To avoid directly ignoring potentially valid interaction intents due to slight uncertainty, a second preset credibility threshold is set, which is lower than the first preset credibility threshold. At this point, it is determined whether the current credibility score is higher than the second preset credibility threshold to further distinguish between potentially valid and obviously invalid interaction intents. If the credibility score is lower than or equal to the second preset credibility threshold, it indicates that the credibility of the interaction intent is very low, and it is highly likely to be a false trigger; therefore, no response is made, and it is directly ignored. If the credibility score is higher than the second preset credibility threshold but lower than the first preset credibility threshold, it indicates that the interaction intent has a certain probability. Therefore, the user confirmation mechanism will be initiated. This involves sending a confirmation request to the user wearing the smart glasses, typically through visual prompts (e.g., a confirmation prompt popping up in the lens HUD), voice announcements, or vibration alerts. The prompts may include messages such as "Possible interaction intent detected, confirm execution?" or "Please nod / say 'yes' / tap to confirm to continue." Users can confirm or cancel the operation through preset methods, such as nodding, voice confirmation (e.g., saying "yes"), or tapping the confirmation area. If the user confirms, the corresponding interaction control command will be triggered; if the user cancels, or there is no confirmation action within a preset time, the system will not execute the interaction control operation by default. This mechanism allows the system to avoid accidental triggering when facing interaction intents of moderate credibility, ensuring the responsiveness and flexibility of the interaction, thereby achieving a safe, controllable, and user-friendly smart glasses interaction experience.

[0047] In the above text, refer to Figure 1 An interactive control method for smart glasses according to an embodiment of the present invention has been described in detail. Next, reference will be made to... Figure 2 An interactive control system for smart glasses is described according to an embodiment of the present invention.

[0048] An interactive control system for smart glasses according to an embodiment of the present invention addresses the technical problem that existing smart glasses interaction methods rely on a single modality, resulting in a high false trigger rate and unstable user experience. It achieves the technical effect of reducing the false operation rate of smart glasses and improving the multimodal interaction experience through multimodal cross-validation and dynamic intent credibility assessment. The interactive control system for smart glasses includes: a data acquisition unit 11, a feature extraction unit 12, a modality analysis unit 13, and an interaction control unit 14.

[0049] Data acquisition unit 11: Acquires multimodal interaction information from the multimodal data acquisition device. The multimodal interaction information includes eye movement trajectories, head posture data, voice input signals, and facial expression changes stored in a time series. Feature extraction unit 12: Extracts interaction features from the multimodal interaction information and identifies multimodal interaction intent trigger points. Modality analysis unit 13: Analyzes the co-occurrence frequency matrix between modal pairs based on the multimodal interaction intent trigger points to obtain a modality co-occurrence frequency vector. Interaction control unit 14: Inputs the modality co-occurrence frequency vector into the interaction intent credibility assessment module, calculates the credibility score of the current interaction intent, and if the credibility score is greater than a first preset credibility threshold, triggers an interaction control command. The smart glasses then execute the control program according to the interaction control command.

[0050] The specific configuration of the feature extraction unit 12 will be described in detail below. As mentioned above, to extract interaction features from the multimodal interaction information and identify multimodal interaction intent trigger points, the feature extraction unit 12 may further include: storing the multimodal interaction information in blocks to obtain multiple interaction information storage blocks; setting multiple interaction feature extraction blocks, which are correspondingly connected to the multiple interaction information storage blocks, and the multiple interaction feature extraction blocks include pre-stored eye-tracking interaction features, head posture interaction features, voice keyword interaction features, and facial expression interaction features; wherein, each interaction feature extraction block is used to receive the modal data from the interaction information storage block, extract interaction features, and output multimodal interaction intent trigger points.

[0051] The specific configuration of the modal analysis unit 13 will be described in detail below. As mentioned above, based on the analysis of the co-occurrence frequency matrix between modal pairs according to the multimodal interaction intent trigger points, the modal analysis unit 13 may further include: obtaining multimodal interaction intent trigger points, including the trigger points of each modality within a preset time window; constructing modal pairs, wherein the modal pairs are obtained by combining all modalities pairwise; setting co-occurrence determination conditions, performing co-occurrence counts for each modal pair according to the co-occurrence determination conditions, calculating the co-occurrence frequency according to the counted co-occurrence counts, and generating a co-occurrence frequency matrix.

[0052] The modal analysis unit 13 may further include: determining whether the interaction intent trigger point of each modal pair meets the co-occurrence determination condition; if the co-occurrence determination condition is met, recording a co-occurrence event; wherein the co-occurrence determination condition is that the time difference between the interaction intent trigger points of the modal pair is less than a preset time difference threshold.

[0053] The modal analysis unit 13 may further include: calculating the co-occurrence frequency based on the statistically obtained co-occurrence counts, wherein the co-occurrence frequency is the ratio of the co-occurrence counts of the modal pair to the total number of trigger points, and the total number of trigger points is min(|T i |,|T j |); where T i T represents the number of interaction intent trigger points corresponding to the i-th modality. j This represents the number of interaction intent trigger points corresponding to the j-th modality.

[0054] The modal analysis unit 13 may further include: constructing a multimodal intent weight matrix; performing matrix calculations on the co-occurrence frequency matrix according to the multimodal intent weight matrix to obtain the modal co-occurrence frequency vector, which is used to represent the degree of synchronization of interaction intent trigger points between modalities.

[0055] The specific configuration of the interaction control unit 14 will be described in detail below. As mentioned above, the modal co-occurrence frequency vector is input into the interaction intent credibility assessment module to calculate the credibility score of the current interaction intent. The interaction control unit 14 may further include: defining fuzzy intervals, the fuzzy intervals including a first co-occurrence frequency region, a second co-occurrence frequency region, and a third co-occurrence frequency region, wherein the first co-occurrence frequency region is a high co-occurrence frequency, the second co-occurrence frequency region is a common co-occurrence frequency, and the third co-occurrence frequency region is a low co-occurrence frequency; setting credibility rules based on the fuzzy intervals, encapsulating the credibility rules to obtain a fuzzy rule engine, connecting it to the interaction intent credibility assessment module, calculating the credibility score of the modal co-occurrence frequency vector, and outputting the credibility score result.

[0056] After calculating the credibility score of the current interaction intent, the interaction control unit 14 may further include: if the credibility score is less than or equal to the first preset credibility threshold, determining whether the credibility score is greater than the second preset credibility threshold, wherein the second preset credibility threshold is less than the first preset credibility threshold; if the credibility score is greater than the second preset credibility threshold, sending a waiting confirmation instruction to the user wearing the smart glasses to prompt the user whether to trigger the interaction control instruction.

[0057] The interactive control system for smart glasses provided in this embodiment of the invention can execute the interactive control method for smart glasses provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0058] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0059] Based on the foregoing embodiments, this application also provides an electronic device. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present invention. Figure 3 The illustrated electronic device is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention. This electronic device is presented as a general-purpose computing device, and its components may include, but are not limited to, a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21 may be one or more; the memory 22 may include a computer-readable medium and at least one program product having a set (at least one) of program modules configured to perform the functions of the embodiments of this application.

[0060] The memory 22 shown in this embodiment of the invention can be any combination of one or more computer-readable media. The computer-readable storage medium can be, but is not limited to, an infrared, semiconductor system, device or apparatus, or any combination thereof, used to store software programs, computer-executable programs and modules, such as the program instructions / modules corresponding to an interactive control method for smart glasses in this embodiment of the invention. The processor 21 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned interactive control method for smart glasses.

[0061] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An interaction control method for smart glasses, characterized in that, The smart glasses are embedded with a multi-modal data acquisition device, and the method comprises: obtaining multi-modal interaction information by the multi-modal data acquisition device, the multi-modal interaction information comprising eye movement trajectories, head posture data, voice input signals and facial expression changes stored in time sequence; extracting interaction features from the multi-modal interaction information to identify multi-modal interaction intent trigger points; analyzing the co-occurrence frequency matrix between modal pairs according to the multi-modal interaction intent trigger points to obtain a modal co-occurrence frequency vector; inputting the modal co-occurrence frequency vector into an interaction intent credibility evaluation module to calculate the credibility score of the current interaction intent, and if the credibility score is greater than a first preset credibility threshold, triggering an interaction control instruction, and the smart glasses executing a control program according to the interaction control instruction; wherein the method for analyzing the co-occurrence frequency matrix between modal pairs according to the multi-modal interaction intent trigger points comprises: obtaining multi-modal interaction intent trigger points, including trigger points of each modality within a preset time window; forming modal pairs, which are obtained by combining all modalities two by two; setting a co-occurrence judgment condition, counting the number of co-occurrences of each modal pair according to the co-occurrence judgment condition, calculating the co-occurrence frequency according to the counted number of co-occurrences, and generating a co-occurrence frequency matrix; determining whether the interaction intent trigger points of each modal pair meet the co-occurrence judgment condition, and recording a co-occurrence event if the co-occurrence judgment condition is met; wherein the co-occurrence judgment condition is that the time difference between the interaction intent trigger points of the modal pair is less than a preset time difference threshold; The co-occurrence frequency is calculated according to the counted co-occurrence times, and the co-occurrence frequency is a ratio of the co-occurrence times of the modal pair to the total trigger point quantity ; wherein, is the number of interaction intent trigger points corresponding to the i-th modality, is the number of interaction intent trigger points corresponding to the j-th modality; wherein the method for obtaining the modal co-occurrence frequency vector comprises: constructing a multi-modal intent weight matrix; performing matrix calculation on the co-occurrence frequency matrix according to the multi-modal intent weight matrix to obtain the modal co-occurrence frequency vector, which is used to represent the degree of synchronization of interaction intent trigger points between modalities.

2. The method of claim 1, wherein, The method for extracting interaction features from the multi-modal interaction information to identify multi-modal interaction intent trigger points comprises: storing the multi-modal interaction information in blocks to obtain a plurality of interaction information storage blocks; setting a plurality of interaction feature extraction blocks, which are connected to the plurality of interaction information storage blocks, the plurality of interaction feature extraction blocks comprising pre-stored eye movement interaction features, head posture interaction features, voice keyword interaction features and facial expression interaction features; wherein each interaction feature extraction block is used to receive modal data of an interaction information storage block for interaction feature extraction and output multi-modal interaction intent trigger points.

3. The method of claim 1, wherein, The method for inputting the modal co-occurrence frequency vector into an interaction intent credibility evaluation module to calculate the credibility score of the current interaction intent comprises: defining a fuzzy interval, the fuzzy interval comprising a first co-occurrence frequency region, a second co-occurrence frequency region and a third co-occurrence frequency region, the first co-occurrence frequency region being a high co-occurrence frequency, the second co-occurrence frequency region being a medium co-occurrence frequency, and the third co-occurrence frequency region being a low co-occurrence frequency; A trustworthiness rule is set based on the fuzzy interval, a fuzzy rule engine is encapsulated with the trustworthiness rule, and the fuzzy rule engine is connected with the interactive intention trustworthiness evaluation module to perform trustworthiness score calculation on the modality co-occurrence frequency vector and output a trustworthiness score result.

4. The method of claim 1, wherein, After calculating the trustworthiness score of the current interactive intention, the method further comprises: If the trustworthiness score is less than or equal to the first preset trustworthiness threshold, it is determined whether the trustworthiness score is greater than a second preset trustworthiness threshold, wherein the second preset trustworthiness threshold is less than the first preset trustworthiness threshold. If the trustworthiness score is greater than the second preset trustworthiness threshold, a waiting confirmation instruction is sent to a user wearing the smart glasses, prompting the user whether to trigger an interactive control instruction.

5. An interaction control system for smart glasses, characterized in that, The smart glasses are embedded with a multi-modal data acquisition device, and the system is used to implement the interactive control method for smart glasses according to any one of claims 1-4, and the system comprises: A data acquisition unit acquires multi-modal interactive information from the multi-modal data acquisition device, and the multi-modal interactive information includes eye movement trajectories, head posture data, voice input signals and facial expression changes stored in time sequence; A feature extraction unit extracts interactive features from the multi-modal interactive information and identifies multi-modal interactive intention trigger points; A modality analysis unit analyzes the co-occurrence frequency matrix between modalities according to the multi-modal interactive intention trigger points to obtain a modality co-occurrence frequency vector; An interactive control unit inputs the modality co-occurrence frequency vector into an interactive intention trustworthiness evaluation module, calculates the trustworthiness score of the current interactive intention, and triggers an interactive control instruction if the trustworthiness score is greater than a first preset trustworthiness threshold, and the smart glasses execute a control program according to the interactive control instruction.

6. An electronic device, comprising: The electronic device comprises: a memory for storing executable instructions; a processor for executing the executable instructions stored in the memory to implement the interactive control method for smart glasses according to any one of claims 1-4.

Images