Voice control method, device and smart glasses

Abstract

The application discloses a voice control method, device and intelligent glasses. The method comprises the following steps: starting a dialogue state machine when the intelligent glasses are switched from a standby state to an instruction receiving state; identifying a voice signal input by a user in current environmental sound to obtain target identification text; determining a current instruction of the user according to the target identification text and dialogue context information stored in the dialogue state machine; wherein the dialogue context information comprises historical dialogue content and intention information of the user; and controlling the intelligent glasses according to the current instruction.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, and more specifically, to a voice control method, a voice control device, and a smart glasses. Background Technology

[0002] Smart glasses, as a core wearable device to assist visually impaired individuals in traveling and perceiving their environment, have received widespread attention in recent years. The interactive experience of smart glasses directly affects the user experience and safety, especially in complex application scenarios such as daily travel navigation, obstacle recognition, and scene-based inquiries, where interaction design is particularly crucial.

[0003] Currently, assistive devices for the visually impaired remain relatively limited in their interaction methods. Traditional devices often rely on tactile feedback (such as vibrating canes) or simple voice prompts, providing only basic obstacle alerts and failing to offer rich contextual information. While existing smartphone voice navigation systems have achieved some degree of voice interaction, their technical solutions are mostly single-turn modes where the user issues a command, the cloud recognizes and converts it, and then the command is executed and broadcast. These existing interaction methods have significant drawbacks: firstly, they are highly dependent on network connectivity and cannot be used offline; secondly, they lack the ability to understand dialogue context, resulting in a stiff and unnatural interactive experience. Summary of the Invention

[0004] One objective of this application is to provide a new voice control scheme to solve the problems of existing voice control methods being unusable in offline environments and unable to accurately understand user voice commands.

[0005] According to a first aspect of this application, a voice control method is provided, comprising: When the smart glasses switch from standby mode to command receiving mode, the dialogue state machine is activated; Identify the user's voice signal in the current ambient sound to obtain the target text; The user's current instruction is determined based on the target recognition text and the dialogue context information stored in the dialogue state machine; wherein, the dialogue context information includes historical dialogue content and the user's intent information; Control the smart glasses according to the current instructions.

[0006] Optionally, the step of recognizing the user-inputted speech signal in the current ambient sound to obtain the target recognition text includes: Extract the user's voice signal from the current ambient sound; The speech signal is input into the first speech recognition model to obtain the first recognized text; If the accuracy of the first recognized text does not meet the accuracy requirements, the voice signal is sent to the cloud server, and the second recognized text returned by the cloud server is received. The second identified text is used as the target identified text.

[0007] Optionally, controlling the smart glasses according to the current instruction includes: Based on the type of the current instruction, the corresponding functional engine is invoked to execute the task corresponding to the current instruction, and the task execution result is obtained; Based on the current status information collected by the smart glasses, the current scene information of the user is determined; wherein, the current status information includes the current location, current IMU data, current ambient sound, and current ambient image, and the current scene information includes the current scene type, current scene sub-state, user motion state, and ambient noise value; Obtain the user's historical preference information; wherein, the historical preference information includes explicit preference information and implicit preference information; The current scene information and the implicit preference information are fused to determine the current scene fusion information; The current scene fusion information is matched with a preset scene rule base to obtain target broadcast rules; wherein, the target broadcast rules include target content generation rules, target broadcast adjustment parameters, and target broadcast control parameters; Based on the explicit preference information, the target broadcasting rules, and the task execution results, broadcast the target feedback content.

[0008] Optionally, the target broadcast adjustment parameters include a first speech rate coefficient and a first volume offset; the explicit preference information includes a second speech rate coefficient, a second volume offset, and a detail preference; and the step of broadcasting the target feedback content according to the explicit preference information, the target broadcast rules, and the task execution result includes: The first speech rate coefficient is corrected based on the second speech rate coefficient to obtain the third speech rate coefficient; The first volume offset is corrected based on the second volume offset to obtain the third volume offset; Based on the level of detail preference, a target content template with the corresponding level of detail is selected from the preset content template library, and the target content template is filled in according to the target content generation rules and the task execution results to generate target feedback content; The target feedback content is broadcast based on the third speech rate coefficient, the third volume offset, and the target broadcast control parameters.

[0009] Optionally, controlling the smart glasses according to the current instruction includes: Based on the type of the current instruction, the corresponding functional engine is invoked to execute the task corresponding to the current instruction, and the task execution result is obtained; Based on the current status information collected by the smart glasses, it is determined whether the user is in a preset dangerous scenario; wherein, the current status information includes the current location, current IMU data, current ambient sound, and current ambient image; When the user is in the preset dangerous scenario, the target feedback content corresponding to the task execution result is broadcast according to the broadcast rules corresponding to the safety priority broadcast mode.

[0010] Optionally, the method further includes: The configuration scenario rules are received from user input. The configuration scenario rules include configuration trigger conditions and configuration broadcast parameters corresponding to the configuration trigger conditions. If the current scene information matches the configured triggering conditions, the target feedback content is broadcast according to the configured broadcast parameters; If the current scene information matches the triggering conditions of multiple scene rules in the preset scene rule library, the broadcast parameters corresponding to one of the scene rules are executed according to the priority of the multiple scene rules.

[0011] Optionally, broadcasting the target feedback content includes: Upon receiving an interrupt command, the broadcast of the target feedback content is stopped, and corresponding processing is performed according to the type of the interrupt command.

[0012] Optionally, determining the user's current instruction based on the target-recognized text and the dialogue context information stored in the dialogue state machine includes: The target-recognized text is input into the dialogue state machine to obtain the target instruction; If the target instruction is incomplete, output follow-up questioning voice information; Recognize the user's feedback voice input in response to the follow-up question voice information, obtain supplementary recognition text, and correct the target instruction based on the supplementary recognition text and the current dialogue context information to obtain the corrected target instruction, which is used as the user's current instruction; If the target instruction is complete, the target instruction will be used as the user's current instruction.

[0013] According to a second aspect of this application, a voice control device is also provided, including a memory and a processor, the memory being used to store executable instructions; the processor being used to operate according to the control of the instructions to perform the method as described in the first aspect.

[0014] According to a third aspect of this application, smart glasses are also provided, including a voice control device as described in the second aspect and an image acquisition device communicatively connected to the voice control device.

[0015] One beneficial effect of this application is that by storing dialogue context information through a dialogue state machine and recognizing user-input voice signals locally, it can solve the problems of existing voice control methods being unusable in offline environments and unable to accurately understand user voice commands. This improves the accuracy of voice recognition and enhances the user's voice control experience in offline scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0017] Figure 1 This is a schematic diagram of the structure of smart glasses according to an embodiment of this application; Figure 2 This is a flowchart illustrating a voice control method according to an embodiment of this application. Detailed Implementation

[0018] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0019] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0020] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0021] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0023] <Hardware Configuration> Figure 1 This is a schematic diagram of the structure of smart glasses according to an embodiment of this application.

[0024] like Figure 1 As shown, the smart glasses 10 includes a voice control device 100 and a sound acquisition device 200.

[0025] The sound acquisition device 200 can be a microphone array located at the front of the two temples of the smart glasses. Furthermore, the sound acquisition device 200 has beamforming and noise suppression functions. The sound acquisition device 200 is used to acquire ambient sound at a sampling rate of 16kHz. This current ambient sound may include the user's voice, environmental sounds (such as vehicle horns), etc. The acquisition frequency of this current ambient sound is adapted to the requirements of ambient sound recognition.

[0026] The voice control device 100 includes at least an integrated processor, memory, and feedback device. The feedback device outputs target feedback content. The feedback device includes at least one of a voice feedback module and a vibration feedback module. The voice feedback module can be a bone conduction speaker, which can be located at the back of the temple of the smart glasses to avoid obstructing the ear canal. The voice feedback module outputs the target feedback content via voice, and the vibration feedback module outputs the target feedback content via vibration.

[0027] In one example, the vibration feedback module consists of three vibration motors. These motors are arranged as follows: one linear vibration motor is placed in the middle-rear part of the left and right temples (behind the ears) of the smart glasses to provide directional cues (such as left / right turn, object on the left / right). A miniature vibration motor is integrated into the nose pad of the smart glasses to provide neutral global cues (such as confirmation, warning, progress reminder). This arrangement achieves directional perception through symmetrical distribution, ensures clear signal by being close to the skin, and allows different vibration frequencies, intensities, and durations to correspond to different target feedback content.

[0028] The voice control device 100 is used to execute the voice control method of the present application embodiment, that is, when the smart glasses switch from standby state to command receiving state, the dialogue state machine is activated; the voice signal input by the user in the current ambient sound is identified to obtain the target recognition text; the user's current command is determined according to the target recognition text and the dialogue context information stored in the dialogue state machine; and finally, the smart glasses are controlled according to the current command.

[0029] In this embodiment, refer to Figure 1 As shown, the voice control device 100 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a feedback device 1700, a microphone 1800, etc.

[0030] Processor 1100 may be a mobile processor. Memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 1300 includes, for example, a USB interface, a headphone jack, and a charging interface. Communication device 1400 may be capable of wired or wireless communication. Communication device 1400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 1400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 1500 is, for example, an LCD screen or a touch screen. Input device 1600 may include, for example, a touch screen or a keyboard. The user can know the execution status of the task corresponding to the current command through feedback device 1700. The user can input voice signals through microphone 1800.

[0031] In this embodiment, the memory 1200 of the voice control device 100 is used to store instructions for controlling the processor 1100 to operate in order to at least execute the voice control method according to any embodiment of this application. Those skilled in the art can design instructions based on the solutions disclosed in this application. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0032] Despite Figure 1 The document shows multiple devices of the voice control device 100; however, this application may only relate to some of these devices. For example, the voice control device 100 may only relate to the memory 1200 and the processor 1100.

[0033] In one embodiment of this application, the smart glasses further include a coprocessor. The coprocessor runs a lightweight network wake-up model specifically designed for real-time detection of ambient sound signals to determine if a set wake-up word is present. If a set wake-up word is present, the coprocessor sends a wake-up trigger signal to the processor of the voice control device.

[0034] In one embodiment of this application, the smart glasses further include an image acquisition device, a location acquisition device, and an IMU measurement unit.

[0035] The image acquisition device is a binocular camera, deployed at the front of the smart glasses, used to acquire images and depth information of the current environment. Its acquisition frequency is set according to the real-time perception requirements of the blind user scenario. Specifically, the image acquisition device can acquire images of the current environment at a first image acquisition frequency of once every 30ms.

[0036] The location acquisition device is a GPS / BeiDou positioning module, integrated inside the smart glasses, used to collect the current location.

[0037] The IMU measurement unit is a six-axis inertial measurement unit (integrating an accelerometer and a gyroscope), integrated inside the smart glasses, used to collect the current IMU data (current acceleration and angular velocity data) of the smart glasses to help perceive the pose and motion state of the smart glasses.

[0038] The current ambient sound, current ambient image, current location, and current IMU data can be collectively referred to as the current state information of the smart glasses. The voice control device 100 determines the user's current scene information based on this current state information, and determines the feedback information and its corresponding broadcast parameters based on the execution result of the task corresponding to the current command, the current scene information, and the user's historical preference information. Finally, the voice control device 100 controls the smart glasses to broadcast the target feedback content according to the broadcast parameters.

[0039] <Method Implementation> Figure 2 This is a flowchart illustrating a voice control method according to an embodiment of the present application, which can be implemented by a voice control device 100.

[0040] according to Figure 2 As shown, the voice control method of this embodiment may include the following steps S2100~S2400: Step S2100: When the smart glasses switch from standby mode to command receiving mode, the dialogue state machine is started.

[0041] In this embodiment, the smart glasses can be smart assistive glasses for the blind. Standby state refers to a low-power state where the smart glasses are running only a lightweight neural network wake-up model on the coprocessor. Command receiving state refers to the working state where the smart glasses can receive user voice commands. The dialogue state machine can be used to maintain dialogue context information, parse user intent, and manage the interaction process. After startup, it can synchronize user dialogue data in real time, providing support for subsequent command recognition and parsing.

[0042] In one embodiment of this application, before starting the dialogue state machine, the method further includes steps S1100 to S1200.

[0043] Step S1100: Receive a wake-up trigger signal, which is triggered when a set wake-up word is detected in the ambient sound information.

[0044] In this embodiment, the wake-up word can be a specific keyword used to wake up the voice control device of the glasses. It can be the system default wake-up word or a user-defined wake-up word, which is not limited here.

[0045] The wake word can be something like "Help, Help," "Hello Glasses," or "Start Navigation," and there are no restrictions here.

[0046] The coprocessor first divides the ambient sound captured by the sound acquisition device into multiple sound frames at set time intervals (e.g., 25 milliseconds). These multiple sound frames can also be referred to as multiple audio frames. MFCC features are extracted from each sound frame to obtain a 40-dimensional feature vector. These multiple 40-dimensional feature vectors are input into a lightweight neural network wake-up model. This model determines whether a set wake-up word exists in each sound frame and outputs the confidence score of its presence. If the confidence score of the presence of the set wake-up word is greater than a wake-up confidence threshold (e.g., 0.8) for three consecutive frames in the multi-frame sound sequence, it is determined that the set wake-up word exists in the current ambient sound. The coprocessor then sends a wake-up trigger signal to the voice control device.

[0047] The lightweight neural network wake-up model employs a multi-layer convolutional neural network structure consisting of three convolutional layers and two fully connected layers, with a total parameter count of approximately 300K. Furthermore, practical testing demonstrates that the model maintains a wake-up accuracy of 95% even under 70dB ambient noise, with no more than one false wake-up per day.

[0048] In step S1200, in response to the wake-up trigger signal, the smart glasses are controlled to switch from standby state to command receiving state, and a state switching prompt message is output.

[0049] In this embodiment, after receiving the wake-up trigger signal sent by the coprocessor, the voice control device of the smart glasses controls the glasses to switch from standby mode to command receiving mode, ensuring that the user can input voice commands in a timely manner.

[0050] Status transition prompts are used to prompt the user to input commands.

[0051] The status switching prompts may include at least one of auditory and tactile prompts. Auditory prompts may be preset short, sharp sounds (such as a short "beep") or voice prompts (such as "I am here"). Tactile prompts may be slight vibration feedback emitted by the vibration motor of the eyeglass temple.

[0052] In noisy environments, auditory and tactile cues can be output simultaneously, preventing users from making mistakes due to not hearing the auditory cues in noisy environments. For example, when a user says "Help! Help!", they will simultaneously hear a "beep" sound and feel the temple of the glasses vibrate, making it easy for the user to confirm that the smart glasses have entered command receiving mode.

[0053] Step S2200: Identify the user-inputted voice signal in the current ambient sound to obtain the target recognition text.

[0054] In this embodiment, the current ambient sound is a mixture of sounds collected by the glasses' sound acquisition device, including user voice, ambient noise, echoes, etc. The user-input voice signal refers to the specific operational command spoken by the user after confirming that the smart glasses have entered the command receiving state (such as "navigate to the nearest supermarket," "what obstacles are ahead," etc.). Target recognition text refers to the recognizable text converted from the user's voice signal, used for subsequent analysis of the user's intent.

[0055] In one embodiment of this application, step S2200 identifies the voice signal input by the user in the current ambient sound to obtain the target recognition text, including: steps S2200.1 to S2200.4.

[0056] Step S2200.1: Extract the user's voice signal from the current ambient sound.

[0057] In this embodiment, the current ambient sound is processed to obtain the user's voice signal. The audio processing includes at least beamforming, echo cancellation, and noise suppression.

[0058] Beamforming focuses the user's voice source in the direction of the sound source, filtering out irrelevant ambient sounds. Echo cancellation removes interference from the sound emitted by the glasses, preventing interference with microphone input. Noise suppression reduces the impact of environmental noise (such as street traffic and crowd noise), ultimately extracting the enhanced user's voice signal from the current ambient sound to ensure the accuracy of subsequent recognition. For example, in a noisy street (70dB noise), the voice signal of the user saying "Navigate to the subway station" can be effectively extracted.

[0059] Step S2200.2: Input the speech signal into the first speech recognition model to obtain the first recognized text.

[0060] In this embodiment, the first speech recognition model is the local speech recognition model of the smart glasses.

[0061] For example, the first speech recognition model can be built on the TensorFlow Lite lightweight machine learning inference framework. It is a lightweight version of Conformer-CTC with 15M parameters and a vocabulary covering 2,000 commonly used command words, covering scenarios such as daily navigation (such as "go home" and "go to the supermarket"), device control (such as "turn up the volume" and "pause broadcast"), and information query. With an inference time of less than 300ms, it can achieve fast offline recognition.

[0062] After the extracted user voice signal is input into the model, the model outputs the corresponding first recognized text.

[0063] For example, if a user says "Navigate to the nearest supermarket", the first recognized text will be "Navigate to the nearest supermarket".

[0064] Step S2200.3: If the recognition accuracy of the first recognized text does not meet the accuracy requirements, the voice signal is sent to the cloud server, and the second recognized text returned by the cloud server is received.

[0065] In this embodiment, when the recognition confidence of the first recognized text is lower than the recognition confidence threshold (e.g., 0.7), or when the first recognized text contains complex sentence structures, the recognition accuracy is determined to be unsatisfactory. In this case, the user's voice signal is sent to the cloud server, so that the cloud server can recognize the voice signal based on the second speech recognition model to obtain the second recognized text.

[0066] The confidence threshold can be dynamically adjusted based on the risk level of the scenario and the network connection status.

[0067] In one example, the recognition confidence threshold for safe scenarios (such as indoors or in a library) is set to 0.65, in which case the local first speech recognition model is used first. The recognition confidence threshold for dangerous scenarios (such as crossing the road or approaching a traffic lane) is set to 0.8 to ensure recognition accuracy and avoid misleading.

[0068] In one example, when the voice control device and the cloud server have a good network connection, the recognition confidence threshold is appropriately reduced to 0.65, meaning the cloud server recognizes commands and improves the recognition range. When the network connection between the voice control device and the cloud server is poor or interrupted, the recognition confidence threshold is 0.7 to ensure the reliability of the local first speech recognition model.

[0069] The recognition confidence threshold can also be adjusted based on historical recognition success rates. For example, if the historical recognition success rate of the first speech recognition model is greater than the success rate threshold, the recognition confidence threshold is lowered (e.g., -0.05). If the historical recognition success rate of the first speech recognition model is less than or equal to the success rate threshold, the recognition confidence threshold is increased (e.g., +0.05).

[0070] The recognition confidence threshold is stored in the configuration file of the voice control device and can be updated via OTA.

[0071] In another embodiment, different instruction categories correspond to different recognition confidence thresholds.

[0072] For example, the recognition confidence threshold for navigation commands is 0.75, and the recognition confidence threshold for control commands is 0.65.

[0073] Complex sentences can be those exceeding the preset word count, such as "What are some good Sichuan restaurants nearby, within 1 kilometer of me?"

[0074] The second speech recognition model could be, for example, a general speech recognition model with hundreds of billions of parameters, a vocabulary covering millions of general words, no limitation on the recognition range, and support for any colloquial expression, but it requires a network connection with a latency of approximately 500-1500ms.

[0075] Step S2200.4: Use the second identified text as the target identified text.

[0076] In this embodiment, the second recognized text is the text result returned by the cloud server after recognizing the user's voice signal. It has a wider recognition range and is suitable for complex sentence structures. When the accuracy of the first recognized text does not meet the requirements, the second recognized text is used as the target recognized text to ensure that the user's instructions can be accurately recognized.

[0077] Based on the above steps S2200.1 to S2200.4, the accuracy of recognition can be improved in noisy environments.

[0078] Step S2300: Determine the user's current instruction based on the target recognition text and the dialogue context information stored in the dialogue state machine; The dialogue context information includes historical dialogue content and the user's intent information.

[0079] In one embodiment of this application, step S2300 determines the user's current instruction based on the target recognition text and the dialogue context information stored in the dialogue state machine, including steps S2300.1 to S2300.4.

[0080] Step S2300.1: Input the target recognition text into the dialogue state machine to obtain the target instruction.

[0081] In this embodiment, the dialogue state machine stores the current dialogue context information (including historical dialogue content and user intent). After the target recognition text is input into the dialogue state machine, the dialogue state machine parses the user's true intent in the target recognition text by associating it with the context information, and then determines the corresponding target instruction.

[0082] For example, if the user said "Navigate to the supermarket" in the previous round (historical dialogue content), and the target recognition text in this round is "nearest" (target recognition text), the dialogue state machine combines the current dialogue context information to determine the target instruction as "Navigate to the nearest supermarket".

[0083] Step S2300.2: If the target instruction is incomplete, output follow-up questioning voice information.

[0084] In this embodiment, an incomplete target instruction may occur when the dialogue state machine, based on the target recognition text and the current context, cannot clearly understand the user's specific operational needs; that is, the target instruction lacks key object information. For example, the target instruction may lack a navigation destination or information about the target item to be found.

[0085] The follow-up voice prompts can be preset voice prompts used to obtain key object information missing from the target instruction. These prompts are broadcast through the smart glasses' voice feedback module (such as a bone conduction speaker). Simultaneously, a vibration feedback module is triggered to ensure the user receives the follow-up prompts.

[0086] For example, the user inputs the target recognition text "go there," without specifying the exact location corresponding to "there," and the current dialogue context does not mention any relevant location. In this case, the target instruction is determined to be incomplete. The follow-up voice message could be "Which location do you mean by 'there'?" or "Please provide more details about the specific operation you wish to perform," without limitation here.

[0087] Step S2300.3: Identify the user's feedback voice input in response to the follow-up question voice information, obtain supplementary identification text, and correct the target instruction based on the supplementary identification text and the target dialogue context information to obtain the corrected target instruction, which is used as the user's current instruction.

[0088] In this embodiment, the user inputs feedback voice in response to follow-up questions, and the feedback voice includes supplementary key object information.

[0089] Continuing with the example above, a follow-up voice message could be, "Which location do you mean by 'there'?", and the user could input the voice message "supermarket".

[0090] The feedback voice is processed through the speech recognition process of step S2200 above to obtain supplementary recognition text (such as "supermarket").

[0091] The dialogue state machine modifies the target instruction obtained in step S2300.1 based on the supplementary recognized text, resulting in a modified target instruction. For example, the modified target instruction is "Navigate to the supermarket". This modified target instruction is then substituted into step S2400 as the user's current instruction, and the step of controlling the smart glasses according to the current instruction is executed.

[0092] Step S2300.4: If the target instruction is complete, the target instruction is taken as the user's current instruction.

[0093] Based on the above, accurate recognition of user commands can be achieved, improving the user's voice control experience.

[0094] Step S2400: Control the smart glasses according to the current instruction.

[0095] In this embodiment, the current instruction can be, for example, navigation, query, device control, etc., and is not limited here.

[0096] Based on the current instruction, the corresponding functional module is invoked to execute the current instruction, and the execution result of the current instruction is fed back to the user, thus realizing a closed loop of voice control and control result feedback.

[0097] For example, if the current instruction is "navigate to the nearest supermarket", the navigation engine is invoked to plan the navigation route and broadcast the navigation information to the user to complete the navigation task corresponding to the current instruction.

[0098] In one embodiment of this application, step S2400 controls the smart glasses according to the current instruction, including steps S2400.1 to S2400.6.

[0099] Step S2400.1: Based on the type of the current instruction, call the corresponding function engine to execute the task corresponding to the current instruction and obtain the task execution result.

[0100] In this embodiment, the current instruction can be categorized into different instruction types based on its purpose. Instruction types can include navigation, query, control, obstacle detection, etc., and are not limited here.

[0101] Different types of current instructions correspond to different functional engines.

[0102] For example, navigation commands correspond to a navigation engine, query commands correspond to a voice interaction engine (corresponding to query commands), control commands such as volume adjustment and mode switching correspond to a device control engine, and obstacle detection commands correspond to an environmental perception engine.

[0103] The result of task execution can be the data or status information output by the function engine after completing the task corresponding to the current instruction.

[0104] For example, when the navigation engine is invoked to execute the command "navigate to the nearest supermarket," the task execution result includes the name of the nearest supermarket, distance, planned route, estimated arrival time, etc. When the environmental perception engine is invoked to execute the command "detect obstacles ahead," the task execution result includes obstacle type, distance, and direction, etc.

[0105] Step S2400.2: Determine the current scene information of the user based on the current state information collected by the smart glasses; The current status information includes the current location, current IMU data, current ambient sound, and current ambient image. The current scene information includes the current scene type, current scene sub-state, user motion state, and ambient noise value.

[0106] In this embodiment, scene information is determined once according to a set scene recognition frequency (e.g., once per second).

[0107] The current location refers to the smart glasses' current location. Typically, the smart glasses are activated when the user wears them. Therefore, the current location can also be considered the user's current location.

[0108] Current IMU data includes acceleration data, angular velocity data, etc., which are used to determine the user's motion state.

[0109] First, a semantic point cloud is constructed based on the current environmental image. Then, the environmental noise value in the current ambient sound is identified. Finally, the current location, current IMU data, current semantic point cloud, and environmental noise value are matched with the preset scene type in the preset scene correspondence to determine the target scene type.

[0110] The preset scene correspondence can refer to a set of rules pre-stored by the system for scene matching. The preset scene correspondence reflects at least one feature from the location, IMU data, semantic point cloud, and environmental noise value corresponding to each preset scene type.

[0111] The system includes three preset scene types: indoor scenes, outdoor scenes, and traffic scenes. Indoor scenes refer to enclosed or semi-enclosed indoor spaces (such as shopping malls, offices, and residences), while outdoor scenes can include non-traffic areas such as parks and squares. Traffic scenes can include scenarios such as crossing the street, pedestrian crossings, and traffic lights.

[0112] Each scenario type corresponds to a specific combination of features.

[0113] For example, the feature combination corresponding to an indoor scene could be a weak signal at the current location and indoor features (such as a home or library) identified in the current environmental image. The feature combination corresponding to an outdoor scene could be a strong signal at the current location and outdoor non-traffic features (such as a square, park, or lawn) identified in the current environmental image. The feature combination corresponding to a traffic scene could be the presence of traffic features (such as traffic lights, zebra crossings, or sidewalks) in the current environmental image and an environmental noise level greater than a preset decibel value.

[0114] The target scene type includes one of the following: indoor scene (such as home, library, etc.), outdoor scene (such as street, park), and transportation scene (such as crossing the road, waiting for a bus, pedestrian crossing, intersection, subway station, bus stop).

[0115] If the matching degree of the target scene type is greater than the matching degree threshold, the target scene type will be used as the current scene type of the user.

[0116] The matching threshold can be a pre-set critical value used to determine whether scene matching is effective.

[0117] If the matching degree of the target scene type is less than or equal to the matching degree threshold, the current semantic point cloud is input into the scene recognition model to obtain the current scene type.

[0118] Specifically, firstly, based on the number, density, spatial distribution, and depth information of semantic labels in the current semantic point cloud (such as the spatial distribution characteristics of the "zebra crossing" label and the proportion of each semantic label), features are extracted from the current semantic point cloud to generate feature vectors. Then, the extracted feature vectors are input into the scene recognition model. The model, through inference calculations, outputs the probability distribution of each preset scene type (e.g., 60% for indoor scenes, 30% for outdoor scenes, and 10% for traffic scenes). Finally, the scene type with the highest probability in the probability distribution is selected as the current scene type, completing the scene recognition. Simultaneously, the system records the scene recognition process and results, feeding them back to the incremental learning module to optimize the preset scene correspondence and the recognition accuracy of the scene recognition model.

[0119] Scene recognition models can be lightweight scene classifiers.

[0120] The current scene sub-state is related to the current scene type. The current scene sub-state is used to describe the details of the current scene type.

[0121] For example, if the current scene type is indoor, the current scene sub-state is whether the indoor environment is quiet. If the current scene type is outdoor, the current scene sub-state is whether the outdoor environment is noisy. If the current scene type is traffic, the current scene sub-state is whether the traffic is busy.

[0122] User activity status can include being stationary, walking, running, etc.

[0123] Ambient noise levels can be used to determine whether the current environment is quiet or noisy.

[0124] For example, if the ambient noise level is "50dB", which is less than the preset decibel value (e.g., 60dB), then the current environment is determined to be quiet. If the ambient noise level is "75dB", which is greater than the preset decibel value (e.g., 60dB), then the current environment is determined to be noisy.

[0125] Step S2400.3: Obtain the user's historical preference information; The historical preference information includes explicit preference information and implicit preference information.

[0126] In this embodiment, historical preference information refers to personalized configuration and behavioral tendency data related to user voice interaction stored locally on the smart glasses. This data is used to adapt to user habits and improve the interactive experience. Explicit preferences are personalized parameters actively set by the user, while implicit preferences are preference labels extracted from the user's historical interactions through machine learning. Explicit preference information may include a second speech rate factor (e.g., 1.2 times the base speech rate), a second volume offset (e.g., +3dB), a level of detail preference (e.g., detailed mode or concise mode), and a feedback mode preference (e.g., voice priority). Users can manually adjust and save these settings through the glasses' "Settings - Voice" option or the accompanying mobile app.

[0127] Implicit preference information may include bias labels extracted from user behavior, such as normal mode, safety-first mode, etc., which are not limited here.

[0128] In one example, the broadcast rules for the safety priority mode and the normal mode are as follows:

[0129] Step S2400.4: The current scene information and the implicit preference information are fused to determine the current scene fusion information.

[0130] For example, "the scenario is crossing the street, the traffic light is green, and the user's implicit preference is safety first" constitutes a set of scenario-based information. This information can provide a more user-friendly scenario basis for subsequent rule matching.

[0131] Step S2400.5: Match the current scene fusion information with the preset scene rule base to obtain the target broadcast rule; The target broadcasting rules include target content generation rules, target broadcasting adjustment parameters, and target broadcasting control parameters.

[0132] In this embodiment, the current scene fusion information is matched with multiple scene rules in a preset scene rule base, and the scene rule with the highest matching degree is selected as the target broadcast rule. The target broadcast rule is the core rule that guides the system to generate broadcast content and parameters, including target content generation rules (such as the core elements and expression standards of the broadcast content), target broadcast adjustment parameters (such as the first speech rate coefficient and the first volume offset), and target broadcast control parameters (such as the number of repetitions and the content priority). Among them, the target broadcast adjustment parameters and the target broadcast control parameters are both target broadcast parameters.

[0133] The number of repetitions refers to the number of times key information in the broadcast content (such as the warning "Watch out for oncoming traffic" or the navigation prompt "Turn left ahead") is repeated. The specific number is determined by the risk level of the scenario and user preferences.

[0134] For example, in high-risk scenarios (traffic scenarios) where users prioritize safety, key information is repeated twice; in low-risk scenarios (indoor scenarios) where users prefer normal mode, key information is not repeated; and in medium-risk scenarios (outdoor scenarios) where users prioritize safety, key information is repeated once.

[0135] Content priority refers to the order in which broadcast content is played, and it is related to the risk level of the scene and the implicit preferences of users.

[0136] For example, when a user is crossing the street and prefers the safety-first mode, the system will first announce "Watch out for vehicles on your right!", then announce "Please use the crosswalk," with navigation information announced during safe intervals. If the user is crossing the street and prefers the normal mode, the system will announce "Turn right in 200 meters ahead" while the user is crossing.

[0137] The preset scenario rule base is a set of broadcast rules pre-stored by the system and formulated based on scenario risk level and user preferences. The rules adopt the correspondence form of "scenario-preference-action".

[0138] For example, if the scenario is crossing the street and the user's implicit preference is safety first, then the broadcast mode is emergency mode. The target broadcast rules corresponding to emergency mode include: target content generation rules (such as prioritizing warning and navigation information), target broadcast adjustment parameters (such as first speech rate coefficient 1.3, first volume offset +5dB), and broadcast control parameters (such as repeat count 1 time, content priority: warning information > navigation information > normal information).

[0139] The preset scenario rule base can be dynamically adjusted based on real-time scenario information and user preferences, rather than remaining fixed.

[0140] Step S2400.6: Broadcast the target feedback content according to the explicit preference information, the target broadcasting rules, and the task execution results.

[0141] Prioritize the explicit preferences set by the user and adjust the relevant parameters and content in the target broadcasting rules to ensure that the broadcasting effect meets the user's specific personalized needs and avoid conflicts between the target broadcasting rules obtained by rule matching and the preferences set by the user.

[0142] In one embodiment of this application, the target broadcast adjustment parameters include a first speech rate coefficient and a first volume offset; the explicit preference information includes a second speech rate coefficient, a second volume offset, and a detail preference. Step S2400.6 broadcasts the target feedback content according to the explicit preference information, the target broadcast rules, and the task execution result, including: steps S2400.61 to S2400.64.

[0143] Step S2400.61: Correct the first speech rate coefficient according to the second speech rate coefficient to obtain the third speech rate coefficient.

[0144] In this embodiment, the second speech rate coefficient is a personalized speech rate parameter actively set by the user. For example, if the user selects a speech rate of 1.2 times the base speech rate in the settings, then 1.2 is the second speech rate coefficient. The first speech rate coefficient is the basic speech rate coefficient obtained by rule matching in step S2400.5. For example, the first speech rate coefficient obtained after scene matching is 1.3.

[0145] The third speech rate coefficient is the parameter ultimately used to control the speech rate of the voice broadcast.

[0146] Prioritize the second speech rate coefficient and adjust the first speech rate coefficient to ensure that the speech rate meets the user's active needs. The correction method is to integrate and adapt the two (such as taking a reasonable superposition of the two or giving priority to the user's set value).

[0147] For example, if a user sets the second speech rate coefficient to 1.2, the rule matching yields a first speech rate coefficient of 1.3, and the corrected third speech rate coefficient is 1.2 (the user-defined coefficient takes precedence). If the user does not set a second speech rate coefficient, the third speech rate coefficient is equal to the first. The value range of the third speech rate coefficient remains 0.8 to 1.5.

[0148] Step S2400.62: Correct the first volume offset according to the second volume offset to obtain the third volume offset.

[0149] In this embodiment, the second volume offset is a personalized volume adjustment value actively set by the user. For example, if the user selects to increase the volume by 3dB in the settings, this +3dB is the second volume offset. The first volume offset is the basic volume adjustment value obtained by rule matching in step S2400.5. For example, the first volume offset obtained after scene matching is +5dB.

[0150] The first volume offset is adjusted with the second volume offset as the priority. The correction method is to add the two together (base gain + user offset) to ensure that the volume meets the user's active needs.

[0151] For example, if the first volume offset is +5dB and the second volume offset is +3dB, the corrected third volume offset is +8dB. However, it needs to be controlled within the range of -3dB to +6dB. Therefore, the final third volume offset is +6dB (taking the maximum value). The third volume offset is the parameter ultimately used to control the volume of the voice broadcast.

[0152] Step S2400.63: Based on the level of detail preference, select a target content template with the corresponding level of detail from the preset content template library, and fill the target content template according to the target content generation rules and the task execution results to generate target feedback content.

[0153] In this embodiment, the preset content template library is a collection of broadcast text templates with different levels of detail that are pre-stored by the system. It is divided into concise templates (suitable for high-risk scenarios, containing only key instructions) and detailed templates (suitable for low-risk scenarios, containing complete context and supplementary prompts).

[0154] The target content template is a suitable template selected from the template library based on the user's level of detail preference. For example, if the user prefers a concise mode, a concise template is selected; if they prefer a detailed mode, a detailed template is selected.

[0155] The target content generation rule is the specification that guides the content generation in the target broadcasting rule (such as priority warning information for high-risk scenarios). The fill operation refers to filling the target content template with the task execution results according to the target content generation rule to generate the final target feedback content.

[0156] For example, if the user prefers the concise mode and the task execution result is "Turn right in 200 meters to reach the supermarket entrance," filling in the concise template will generate the target feedback content "Turn right in 200 meters." If the user prefers the detailed mode, filling in the detailed template will generate the target feedback content "The light is green now. Turn right in 200 meters to reach the supermarket entrance. Please observe your surroundings."

[0157] Step S2400.64: ​​Based on the third speech rate coefficient, the third volume offset, and the target broadcast control parameters, broadcast the target feedback content.

[0158] In this embodiment, the third speech rate coefficient and the third volume offset jointly control the speed and volume of the speech playback. The repetition count in the target playback control parameters is the number of times key information is repeated, and the content priority is the order in which the content is played.

[0159] The target feedback content is synthesized into a speech waveform by applying a third speech rate coefficient and a third volume offset, and then broadcast through the speech feedback module (bone conduction speaker). Furthermore, the broadcast strictly adheres to content priority, repeating key information a prescribed number of times.

[0160] For example, if the target feedback is "Caution: oncoming traffic on the right, turn right in 200 meters ahead", with a third speech rate of 1.2, a third volume offset of +4dB, and one repetition, and the content priority warning message is greater than the navigation, then first announce "Caution: oncoming traffic on the right" at 1.2 times the speech rate and +4dB volume (repeated once), and then announce "Turn right in 200 meters ahead".

[0161] Through the aforementioned dynamic broadcasting strategy, smart glasses can adaptively adjust the broadcasting method according to the user's personalized settings and real-time scenarios, thereby improving user experience and security.

[0162] In one embodiment of this application, step S2400 controls the smart glasses according to the current instruction, including steps S3100 to S3300.

[0163] Step S3100: Based on the type of the current instruction, call the corresponding function engine to execute the task corresponding to the current instruction and obtain the task execution result.

[0164] This step is basically the same as step S2400.1 above, and will not be elaborated here.

[0165] Step S3200: Based on the current status information collected by the smart glasses, determine whether the user is in a preset dangerous scenario; The current status information includes the current location, current IMU data, current ambient sound, and current ambient image.

[0166] In this embodiment, the preset dangerous scenarios are a set of scenarios that are predefined by the system and may pose a threat to the personal safety of users. These include, but are not limited to, three categories: first, traffic-related scenarios, such as crossing the road, approaching the motor vehicle lane, and turning at an intersection; second, environment-related scenarios, such as construction sites, stairwells, and potholes; and third, dynamic dangerous scenarios, such as detecting rapidly approaching vehicles or crowds. The current status information has already been explained above and will not be elaborated upon here.

[0167] By identifying visual markers such as zebra crossings, traffic lights, and construction signs in the current environmental image, it determines whether the user is in a dangerous scenario such as a traffic intersection or construction site. By identifying characteristic audio features in the current environmental audio, such as vehicle engine sounds, horns, and construction noise, it assesses the dynamic dangers around the user. Based on the user's current location, it determines whether the user is in a pre-defined danger zone such as a road intersection. If any of the above three determinations are true, the user is determined to be in a pre-defined dangerous scenario. Otherwise, it is determined to be a non-dangerous scenario.

[0168] Step S3300: When the user is in the preset dangerous scenario, broadcast the target feedback content corresponding to the task execution result according to the broadcast rules corresponding to the safety priority broadcast mode.

[0169] In this embodiment, when a user is determined to be in a dangerous situation, the system automatically switches to the "Safety First" broadcast mode. This mode switch requires no manual intervention from the user; a short "beep" sound is played during the switch to inform the user that they have entered the Safety First mode, ensuring the user is aware of the mode change. The "Safety First" broadcast mode is specifically designed for dangerous scenarios. Its core logic is to prioritize broadcasting warning information to protect the user's personal safety. Compared to the normal broadcast mode, it has a clear difference in broadcast strategy, specifically reflected in the five dimensions shown in the table above, which will not be elaborated upon here.

[0170] In safety-first mode, the task execution result is adjusted according to the corresponding broadcasting rules to obtain the target feedback content, and then the target feedback content is broadcast according to the broadcasting parameters corresponding to safety-first mode.

[0171] For example, if a user issues a navigation command "Go to the nearest bus stop," the task execution result is "Turn right in 200 meters to reach the bus stop." If the system detects that the user is crossing the road at this time, it automatically switches to a safety-priority mode, prioritizing the broadcast of warning messages such as "Watch out for vehicles on your right!" and "Please use the crosswalk," repeating them every 5 seconds with a 20% faster speech rate and a 6dB higher volume. After the user crosses the road and the danger level decreases, the navigation information "Turn right in 200 meters to reach the bus stop" is broadcast again, fully balancing user safety and command execution needs, aligning with the core safety adaptation goals of the MTE smart assistive glasses.

[0172] In one embodiment of this application, step S2400.6, broadcasting the target feedback content, includes: Upon receiving an interrupt command, the broadcast of the target feedback content is stopped, and corresponding processing is performed according to the type of the interrupt command.

[0173] In this embodiment, during the broadcast of the target feedback content, various interruption commands are continuously monitored to ensure that the broadcast process can be flexibly controlled and adapted to user needs and emergency scenarios.

[0174] The interruption commands are mainly divided into two categories. One category consists of user-initiated voice interruption commands, such as "replay," "pause," and "stop," which users can issue directly via voice. The other category consists of automatically detected emergency interruption commands, such as those triggered by an approaching vehicle detected by current status information, which automatically trigger an interruption without user intervention. Upon receiving either type of interruption command, the broadcast will immediately stop. The overall interruption response delay from the user issuing the voice command to the broadcast stopping does not exceed 50ms, ensuring timely interruption. After the broadcast stops, a standardized reset process is executed, consisting of five steps: Step 1: Immediately stop the target feedback content synthesis and audio output from the voice feedback module, ensuring no residual audio and controlling the response time to within 10ms. Step 2: Save the broadcast progress (e.g., the current broadcast up to a specific sentence of the target feedback content, the number of times it has been repeated), providing a basis for subsequent resumption of broadcasting. Step 3: Clear the audio output buffer to prevent residual audio data from affecting subsequent broadcasting or interruption processing. Step 4: Jump to the corresponding state based on the type of interruption command. Specifically, if the interruption command is of the "rebroadcast" type, jump to the broadcast start state. If the interruption command is of the "pause" or "stop" type, jump to the pause state. If the interruption command is due to the detection of an emergency event, jump to the emergency alert state, prioritizing the broadcast of emergency warning information. Step 5: Continue execution from the new state after the jump, completing the post-interruption processing. The method of resumption after an interruption is determined by user commands, specifically in two ways: If the user issues a "replay" command, the playback will resume from the beginning of the target feedback content without requiring the user to re-trigger the playback command. If the user issues a "pause" command, resumption can be triggered later via voice command, continuing playback from the previous breakpoint without repeating completed content, improving user convenience. Simultaneously, multi-level interrupt priority is supported. Emergency interruptions (such as those triggered by an approaching vehicle) can interrupt normal interruptions (such as those corresponding to user-initiated "pause" or "replay" commands), ensuring that users receive warning information first in emergencies, protecting user safety, and aligning with the core adaptation requirements of smart glasses.

[0175] In one embodiment of this application, the method further includes steps S2500 to S2700.

[0176] Step S2500: Receive configuration scenario rules input by the user. The configuration scenario rules include configuration trigger conditions and configuration broadcast parameters corresponding to the configuration trigger conditions.

[0177] In this embodiment, users can customize input configuration scene rules through the rule editing interface of the mobile app that comes with the smart glasses.

[0178] For example, reducing the broadcast volume in "indoor scenes" and "automatically lowering the volume in indoor scenes after 10 pm".

[0179] Configuring trigger conditions refers to the scene, time, and other conditions that trigger the effect of this custom scene rule. Configuring trigger conditions can be scenes selected by the user from the App's predefined scenes (such as indoor, outdoor, traffic, etc.) or custom conditions set by the user (such as the time range "22:00-08:00").

[0180] Configuring broadcast parameters refers to the parameters related to the voice broadcast that will be applied after the trigger conditions are met. Configuring broadcast parameters include volume, speech rate, level of detail, whether to repeat, and may also include the on / off status of haptic feedback.

[0181] The configuration of the broadcast parameters can be, for example, reducing the volume by 6dB or slowing the speech rate to 0.9 times the base speech rate.

[0182] After receiving the configuration scenario rule, it will be stored in the preset scenario rule library in JSON format. The rule is uniquely identified, which facilitates subsequent identification and retrieval.

[0183] Step S2600: If the current scene information matches the configured triggering conditions, broadcast the target feedback content according to the configured broadcast parameters.

[0184] In this embodiment, the concept of current scene information is the same as that in step S2400.2, and will not be elaborated here.

[0185] The system compares the current scene information with the configured trigger conditions in the preset scene rule library. If the current scene, time, etc., meet the configured trigger conditions (e.g., the system detects that the current scene is "indoor" and the time is between 22:00 and 08:00), then it is considered a successful match. After a successful match, the system will automatically apply the configured broadcast parameters in the configured scene rule to replace the current broadcast parameters, and generate and broadcast the target feedback content.

[0186] For example, a user sets a configuration trigger condition for "automatically lowering the volume in indoor scenes after 10 PM". When the current scene type is detected as "indoor" at 22:00, the current scene information matches the configuration trigger condition, and the volume will be automatically adjusted according to the configuration broadcast parameters (e.g., from 5 to 2). The speech rate, level of detail, and other parameters will also be adjusted according to the configuration broadcast parameters to broadcast the target feedback content, without requiring manual adjustment by the user, thus improving user comfort.

[0187] Step S2700: If the current scene information matches the triggering conditions of multiple scene rules in the preset scene rule library, execute the broadcast parameters corresponding to one of the scene rules according to the priority of the multiple scene rules.

[0188] In this embodiment, the preset rule priority is "user-defined rules take precedence over system default rules," meaning that when multiple rules conflict, the configuration broadcast parameters corresponding to the user-defined configuration scenario rules are executed first. If multiple user-defined configuration scenario rules conflict, the user can manually set the priority of the custom rules in the App.

[0189] For example, a user defines a scenario rule for "lowering volume in indoor mode," while the voice control device presets a rule for "normal volume and detailed playback in indoor scenes." When the current scenario type is detected as "indoor scene," the trigger conditions of both rules match, resulting in a rule conflict. In this case, the user-defined "lower volume in indoor mode" rule will be executed first, playing the target feedback content according to the user-set volume, speech rate, and other parameters, instead of executing the preset indoor scene rule. This ensures that the user's customized needs are prioritized, while avoiding rule conflicts that could lead to inconsistent playback parameters. Based on the above, by storing dialogue context information through a dialogue state machine and recognizing user-input voice signals locally, the accuracy of voice recognition can be improved in offline scenarios, thus enhancing the user's voice control experience.

[0190] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above-described method embodiments.

[0191] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0192] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0193] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0194] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0195] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0196] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0197] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0198] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0200] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A voice control method, characterized in that, include: When the smart glasses switch from standby mode to command receiving mode, the dialogue state machine is activated; Identify the user's voice signal in the current ambient sound to obtain the target text; The user's current instruction is determined based on the target recognition text and the dialogue context information stored in the dialogue state machine; wherein, the dialogue context information includes historical dialogue content and the user's intent information; Control the smart glasses according to the current instructions.

2. The method according to claim 1, characterized in that, The process of identifying the user's input speech signal in the current ambient sound to obtain the target recognition text includes: Extract the user's voice signal from the current ambient sound; The speech signal is input into the first speech recognition model to obtain the first recognized text; If the accuracy of the first recognized text does not meet the accuracy requirements, the voice signal is sent to the cloud server, and the second recognized text returned by the cloud server is received. The second identified text is used as the target identified text.

3. The method according to claim 1, characterized in that, The step of controlling the smart glasses according to the current instruction includes: Based on the type of the current instruction, the corresponding functional engine is invoked to execute the task corresponding to the current instruction, and the task execution result is obtained; Based on the current status information collected by the smart glasses, the current scene information of the user is determined; wherein, the current status information includes the current location, current IMU data, current ambient sound, and current ambient image, and the current scene information includes the current scene type, current scene sub-state, user motion state, and ambient noise value; Obtain the user's historical preference information; wherein, the historical preference information includes explicit preference information and implicit preference information; The current scene information and the implicit preference information are fused to determine the current scene fusion information; The current scene fusion information is matched with a preset scene rule base to obtain target broadcast rules; wherein, the target broadcast rules include target content generation rules, target broadcast adjustment parameters, and target broadcast control parameters; Based on the explicit preference information, the target broadcasting rules, and the task execution results, broadcast the target feedback content.

4. The method according to claim 3, characterized in that, The target broadcast adjustment parameters include a first speech rate coefficient and a first volume offset; the explicit preference information includes a second speech rate coefficient, a second volume offset, and a detail preference; the step of broadcasting target feedback content according to the explicit preference information, the target broadcast rules, and the task execution result includes: The first speech rate coefficient is corrected based on the second speech rate coefficient to obtain the third speech rate coefficient; The first volume offset is corrected based on the second volume offset to obtain the third volume offset; Based on the level of detail preference, a target content template with the corresponding level of detail is selected from the preset content template library, and the target content template is filled in according to the target content generation rules and the task execution results to generate target feedback content; The target feedback content is broadcast based on the third speech rate coefficient, the third volume offset, and the target broadcast control parameters.

5. The method according to claim 1, characterized in that, The step of controlling the smart glasses according to the current instruction includes: Based on the type of the current instruction, the corresponding functional engine is invoked to execute the task corresponding to the current instruction, and the task execution result is obtained; Based on the current status information collected by the smart glasses, it is determined whether the user is in a preset dangerous scenario; wherein, the current status information includes the current location, current IMU data, current ambient sound, and current ambient image; When the user is in the preset dangerous scenario, the target feedback content corresponding to the task execution result is broadcast according to the broadcast rules corresponding to the safety priority broadcast mode.

6. The method according to claim 3, characterized in that, The method further includes: The configuration scenario rules are received from user input. The configuration scenario rules include configuration trigger conditions and configuration broadcast parameters corresponding to the configuration trigger conditions. If the current scene information matches the configured triggering conditions, the target feedback content is broadcast according to the configured broadcast parameters; If the current scene information matches the triggering conditions of multiple scene rules in the preset scene rule library, the broadcast parameters corresponding to one of the scene rules are executed according to the priority of the multiple scene rules.

7. The method according to claim 3, characterized in that, The broadcast of the target feedback content includes: Upon receiving an interrupt command, the broadcast of the target feedback content is stopped, and corresponding processing is performed according to the type of the interrupt command.

8. The method according to claim 1, characterized in that, Determining the user's current command based on the target-recognized text and the dialogue context information stored in the dialogue state machine includes: The target-recognized text is input into the dialogue state machine to obtain the target instruction; If the target instruction is incomplete, output follow-up questioning voice information; Recognize the user's feedback voice input in response to the follow-up question voice information, obtain supplementary recognition text, and correct the target instruction based on the supplementary recognition text and the current dialogue context information to obtain the corrected target instruction, which is used as the user's current instruction; If the target instruction is complete, the target instruction will be used as the user's current instruction.

9. A voice control device, characterized in that, The voice control device includes a memory and a processor, the memory being used to store executable instructions; the processor being used to operate according to the control of the instructions to perform the method as described in any one of claims 1 to 8.

10. A type of smart glasses, characterized in that, It includes the voice control device as described in claim 9 and the voice acquisition device that is communicatively connected to the voice control device.

Images