Wearable device and semantic recognition method and apparatus therefor, storage medium and computer program product

Abstract

This application relates to a wearable device and its semantic recognition method, apparatus, storage medium, and computer program product. The method includes: in response to detecting a user's voice request, determining whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes requests for at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, all indicating a translation intent; if the voice request is a voice-only translation request, invoking a local voice recognition module to perform voice translation and generating a voice translation result; if the voice request is a mixed translation request, sending the user data corresponding to the voice request to a server, and receiving a mixed intent translation result returned by the server based on the user data after performing mixed translation. This method can reduce cloud dependency, optimize complex intent processing flows, and improve user experience.

Description

Technical Field

[0001] This application relates to the field of semantic recognition technology, and in particular to a wearable device and its semantic recognition method, apparatus, storage medium and computer program product. Background Technology

[0002] With the deep integration of artificial intelligence and real-time translation technology, wearable devices are gradually becoming core tools in cross-language communication scenarios, widely used in cross-border travel, business communication, and living abroad. The core requirements of these devices are low latency and high semantic coherence. Users expect to quickly obtain translation results in natural conversations while avoiding communication interruptions caused by information fragmentation.

[0003] Existing related technologies rely entirely on cloud computing for speech recognition, text processing, and translation. When using these headphone-type translators, the end-to-end translation process suffers from significant latency, which severely disrupts conversational flow, resulting in a poor experience, especially in fast-paced interactive scenarios. While some next-generation translation devices have reduced latency for pure speech translation through localized optimizations, information gaps still occur during multi-round interactions, requiring users to wait repeatedly and creating a fragmented experience. Summary of the Invention

[0004] Therefore, it is necessary to provide a wearable device and its semantic recognition method, apparatus, storage medium and computer program product that can reduce cloud dependence, optimize complex intent processing flow and improve user experience in response to the above technical problems.

[0005] Firstly, this application provides a semantic recognition method for wearable devices. The method includes:

[0006] In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0007] If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result;

[0008] In the case that the voice request is a mixed translation request, the user data corresponding to the voice request is sent to the server, and the server returns the mixed intent translation result returned by performing mixed translation based on the user data.

[0009] In some embodiments of the method, the wearable device includes a voice playback device and a display device, and the method further includes:

[0010] Semantic segments from the speech translation result or the mixed intent translation result are added to a feedback queue in chronological order, and semantic feedback is performed based on the feedback queue, including:

[0011] The voice playback device performs first-word feedback according to the feedback queue;

[0012] The display device synchronizes the first word feedback according to the feedback queue and displays the corresponding complete sentence's subtitles and / or image markers.

[0013] In some embodiments of the method, performing semantic feedback based on the feedback queue further includes:

[0014] The voice input features of the user are acquired in real time, and the voice input features include at least one of the following: number of syllables per unit time, sentence interval time, and sentence pause nodes;

[0015] The size of the semantic segmentation window is dynamically adjusted based on the speech input features, and semantic segments are output based on the adjusted semantic segmentation window.

[0016] In some embodiments of the method, the method further includes:

[0017] Receive user touch control commands and / or voice control commands, and execute corresponding processing strategies in the feedback queue according to the touch control commands and / or the voice control commands;

[0018] The processing strategy is configured to include at least one of the following: adjust the translation language, view detailed explanation, pause translation, and continue translation.

[0019] In some embodiments of the method, the method further includes:

[0020] The system stores user interaction data, which includes at least one of interaction log information, translation results, and error correction feedback information.

[0021] When the current network connection is idle, the interaction data is encrypted and sent to the server, and the updated offline translation model and intent recognition parameters are returned by the server based on the encrypted interaction data.

[0022] In some embodiments of the method, the method further includes:

[0023] User data is collected through sensors, and noise reduction and endpoint detection processing are performed on the user data.

[0024] Timestamps are generated synchronously based on the processed user data.

[0025] The step of adding semantic segments from the speech translation result or the mixed intent translation result to the feedback queue in chronological order includes:

[0026] Semantic segments from the speech translation results or the mixed intent translation results are added to the feedback queue according to the timestamp order.

[0027] According to a second aspect of the present disclosure, a semantic recognition device for a wearable device is provided, the device comprising:

[0028] An intent recognition module is used to respond to the detection of a user's voice request and determine whether the voice request is a voice-only translation request or a mixed translation request. The mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0029] The local speech translation module is used to call the local speech recognition module to perform speech translation and generate a speech translation result when the speech request is a speech translation-only request;

[0030] The communication module is used to send the user data corresponding to the voice request to the server when the voice request is a mixed translation request, and to receive the mixed intent translation result returned by the server after performing mixed translation based on the user data.

[0031] According to a third aspect of the present disclosure, a wearable device is provided. The wearable device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0032] In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0033] If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result;

[0034] In the case that the voice request is a mixed translation request, the user data corresponding to the voice request is sent to the server, and the server returns the mixed intent translation result returned by performing mixed translation based on the user data.

[0035] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0036] In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0037] If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result;

[0038] In the case that the voice request is a mixed translation request, the user data corresponding to the voice request is sent to the server, and the server returns the mixed intent translation result returned by performing mixed translation based on the user data.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0040] In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0041] If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result;

[0042] In the case that the voice request is a mixed translation request, the user data corresponding to the voice request is sent to the server, and the server returns the mixed intent translation result returned by performing mixed translation based on the user data.

[0043] The semantic recognition solution for wearable devices provided in this application can closely align with user interaction habits through wearable devices, optimize resource allocation, and process only voice translation requests through local modules, avoiding network latency in cloud interactions. Simultaneously, it can improve processing efficiency in complex scenarios, enabling joint analysis of multimodal data through a single cloud request for mixed translation requests, eliminating the need for multiple requests, improving the efficiency of multi-turn dialogues, avoiding frequent device operations by users, reducing user operating costs, and allowing the device to automatically match processes through intent recognition, reducing the sense of disjointed experience, adapting to multi-scenario needs, and thus comprehensively improving the user experience.

[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0046] Figure 1 This is an application environment diagram illustrating a semantic recognition method for a wearable device according to an exemplary embodiment;

[0047] Figure 2 This is a flowchart illustrating a semantic recognition method for a wearable device according to an exemplary embodiment.

[0048] Figure 3 This is a schematic diagram illustrating the specific process of a semantic recognition method for a wearable device according to an exemplary embodiment;

[0049] Figure 4 This is a structural block diagram of a semantic recognition device for a wearable device according to an exemplary embodiment;

[0050] Figure 5 This is a diagram illustrating the internal structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., to denote names does not indicate any specific order.

[0053] In some embodiments provided in this disclosure, the execution of the semantic recognition method for wearable devices can be controlled by a unified controller or by multiple controllers. These controllers may include controllers on local terminals, such as the semantic recognition controller for the wearable device, or controllers on remote servers, such as controllers in a server that can communicate with the semantic recognition of the wearable device. In some embodiments, the controllers on local terminals and servers may work together to complete the semantic recognition control processing of the wearable device. The local terminal mentioned in this disclosure may include, but is not limited to, various robotic devices, in-vehicle devices, personal computers, laptops, smartphones, tablets, wearable devices, medical devices, VR (Virtual Reality) devices, etc. The server may also be a server, server cluster, distributed subsystem, cloud processing platform, server containing blockchain nodes, or a combination thereof. The controllers described in this disclosure may include various control units capable of implementing logic processing functions, including but not limited to CPU (Central Processing Unit), PLC (Programmable Logic Controller), ECU (Electronic Control Unit), MCU (Microcontroller Unit), FPGA (Field Programmable Gate Array), and CPLD (Complex Programmable Logic Device), as well as controllers composed of one or more logic function units, chips, etc.

[0054] The semantic recognition method for wearable devices provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers. Terminal 102 can be a wearable device, such as a smartwatch, smart bracelet, or head-mounted device. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0055] In some embodiments of this disclosure, a semantic recognition method for wearable devices is provided, such as... Figure 2 As shown, it includes the following steps:

[0056] S20. In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent.

[0057] In some embodiments of this disclosure, wearable devices generally refer to portable electronic devices that can be worn directly on a user's body or integrated into a user's clothing or accessories. Wearable devices may include headphone devices, augmented reality (AR) glasses, and VR glasses, etc. Headphone devices may include over-ear headphones, in-ear headphones, bone conduction headphones, and wireless Bluetooth headphones, etc. Headphone devices and / or glasses may integrate data acquisition sensors to collect user data. Voice requests generally refer to instructions or requests initiated by a user to the wearable device through spoken commands. The wearable device can capture voice signals through a voice acquisition unit and further analyze the user's intent. Voice requests may include voice-only translation requests and mixed translation requests. Among them, a voice-only translation request means that the user only needs to translate the voice content from one language to another without combining it with other information such as images and text, for example, the user speaks "Please translate 'What's the weather like today?' into English." Hybrid translation requests refer to user requests involving two or more translation intentions, requiring the combined processing of voice, image, and text information. For example, when a user holds a foreign language menu and says, "Please translate the names of the dishes on this menu into Chinese," the device needs to perform both voice recognition and image recognition simultaneously.

[0058] S22. If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result.

[0059] In some embodiments of this disclosure, voice translation requests may include requests for translating everyday conversations, short command phrases, or vocabulary lookup.

[0060] A local speech recognition module typically refers to a functional module integrated into the local hardware of a wearable device. It consists of a low-power processor, an offline speech recognition model, an offline translation model, and a data buffer. It can independently complete the entire process of speech signal acquisition, preprocessing, speech-to-text conversion, text translation, and result output, without relying on cloud server computing resources. Its core advantages are strong real-time performance and high privacy, avoiding security risks and latency issues during data transmission. The processor may include an NPU (Neural Processing Unit). A lightweight speech intent recognition network can run through the NPU. In some implementations, the lightweight speech intent recognition network can employ hierarchical recognition logic to recognize basic commands, such as waking up the device or switching languages. If it is a basic command, the corresponding operation is executed directly. The lightweight speech intent recognition network can also determine the type of translation intent. If the speech request does not involve mixed translation intents such as images or text, it is determined to be a speech-only translation request. If it involves at least two of the following translation intents: speech recognition translation, image recognition translation, and text recognition translation, it is determined to be a mixed translation request, triggering the image acquisition unit to capture the current scene's image and / or text information.

[0061] Lightweight speech intent recognition networks can convert preprocessed speech signals into source language text. Offline translation models can convert source language text into target language text based on embedded machine translation models (such as lightweight Transformer models). Calling the local speech recognition module to perform speech translation optimizes resource allocation, eliminates the need for cloud requests, and reduces network latency.

[0062] S24. If the voice request is a mixed translation request, send the user data corresponding to the voice request to the server and receive the mixed intent translation result returned by the server based on the user data.

[0063] In some embodiments of this disclosure, the hybrid translation request may include requests for combined text and image translation, multi-scene text translation, and cross-modal semantic association translation. User data typically refers to various types of information collected and generated by wearable devices during user interaction. User data may include raw collected data, processed data, and user feedback data. In this application, user data undergoes preprocessing and encryption to ensure both data validity and security. The server typically refers to a cloud server cluster that deploys multimodal large-scale model inference services, databases, and model optimization modules. It possesses powerful computing power and multimodal data collaborative analysis capabilities, can receive user data uploaded by wearable devices, perform hybrid translation processing, and return results. The server may include a multimodal processing module, a translation engine module, a database module, and a model optimization module. The multimodal processing module is responsible for performing deep intent analysis on uploaded voice data and performing OCR (Optical Character Recognition) text extraction and object recognition on image data. The translation engine module, based on the multimodal semantic understanding results, calls multilingual translation models to generate translated text that integrates voice intent and image information. The database module stores user interaction data and multimodal model parameters. The model optimization module analyzes user data to improve the recognition and translation accuracy of the multimodal model, and can generate local model update packages adapted for wearable devices. Hybrid intent translation results typically refer to the structured, multi-dimensional translation output generated by the server after performing multimodal joint processing on user data from mixed translation requests. Hybrid intent translation results can include core translated text, image-related tags, and auxiliary information.

[0064] In some embodiments of this disclosure, wearable devices can closely align with user interaction habits, optimize resource allocation, and process only voice translation requests through local modules, avoiding network latency in cloud interactions. Simultaneously, they can improve processing efficiency in complex scenarios, enabling joint analysis of multimodal data through a single cloud request for mixed translation requests, eliminating the need for multiple requests, improving the efficiency of multi-turn dialogues, avoiding frequent device operations by users, reducing user operating costs, and allowing the device to automatically match processes through intent recognition, reducing jarring experiences, adapting to multi-scenario needs, and comprehensively enhancing the user experience.

[0065] The following is in conjunction with the appendix Figure 3 Further explanation is needed.

[0066] In some embodiments of this disclosure, the wearable device includes a voice playback device and a display device; based on this, the method further includes:

[0067] Semantic segments from the speech translation result or the mixed intent translation result are added to a feedback queue in chronological order, and semantic feedback is performed based on the feedback queue, including:

[0068] The voice playback device performs first-word feedback according to the feedback queue;

[0069] The display device synchronizes the first word feedback according to the feedback queue and displays the corresponding complete sentence's subtitles and / or image markers.

[0070] In some embodiments of this disclosure, the voice playback device typically refers to a hardware module integrated into a wearable device for transmitting audio information from semantic feedback to the user. The voice playback device may include bone conduction headphones, miniature speakers, and in-ear headphones. Bone conduction headphones transmit sound to the auditory nerve by vibrating the skull, eliminating the need for insertion into the ear and making them suitable for use in noisy environments while avoiding the impact of ear canal blockage on the user's perception of surrounding sounds. Miniature speakers are small in size and can be integrated into devices such as smartwatches and translation bracelets, outputting audio via external speakers, suitable for close-range, low-noise scenarios. In-ear headphones, with their in-ear design, improve sound clarity and are suitable for scenarios requiring high audio quality.

[0071] Display devices typically refer to hardware modules integrated into wearable devices to present visual information about translation results. These devices can include waveguide displays, micro-OLED (Organic Electroluminescence Display) screens, and flexible AMOLED (Active-matrix organic light-emitting diode) screens. Waveguide displays use optical refraction to overlay subtitles and image markers onto a real-world scene, making them suitable for devices like translation glasses. Micro-OLED screens are small and high-resolution, allowing integration into the control panels of smartwatches and translation headsets to display complete translated text via pop-up windows, making them ideal for lightweight devices. Flexible AMOLED screens are bendable, fitting snugly to the wrist or arm, and can be integrated into flexible smart bracelets.

[0072] Semantic segments typically refer to short sentences or phrases derived from speech translation results or mixed intent translation results, broken down according to semantic coherence and user voice input characteristics. Semantic segments can include keyword segments, phrase segments, short sentence segments, etc. A feedback queue typically refers to a buffered data structure in wearable devices used to store and sort semantic segments. In some implementations, a timestamp can be added to each semantic segment (synchronized with the user's voice input time or the server's result return time), and the segments are added to the feedback queue in the order of their timestamps. The voice playback device can perform first-word feedback based on the feedback queue, prioritizing the output of core words from keyword segments. For example, when translating "Grilled Steak, 25 Euros," the keyword "steak" can be played first to reduce the delay of first-word feedback, achieving a delay of <0.3 seconds. The control module of the display device receives the synchronization signal from the feedback queue and extracts the visual data of the corresponding semantic segment from the feedback queue simultaneously with the output of the first word by the voice playback device. For example, the display device can synchronously display the complete sentence subtitle of "Grilled Steak, 25 Euros," using pre- and post-translation subtitles, such as bilingual (Chinese and English) subtitles. In some implementations, in addition to complete subtitles, image marker data can also be extracted. The text area corresponding to the translation object, such as "pan-fried steak", can be selected in the menu image through an optical waveguide display. The subtitles are superimposed and blended with the image scene, and the user can view them without switching their gaze.

[0073] In some embodiments of this disclosure, the first word priority output significantly reduces the user's waiting time, which can shorten the time for users to obtain core information in long sentence translation scenarios and avoid the interruption of dialogue due to waiting for the complete sentence; visual feedback and audio feedback are executed synchronously to avoid information fragmentation caused by audio-visual asynchrony, thereby improving the accuracy of user information comprehension and user viewing efficiency, and thus improving the user experience.

[0074] In some embodiments of this disclosure, the method further includes:

[0075] User data is collected through sensors, and noise reduction and endpoint detection processing are performed on the user data.

[0076] Timestamps are generated synchronously based on the processed user data.

[0077] Based on this, the step of adding the semantic segments in the speech translation result or the mixed intent translation result to the feedback queue in chronological order further includes: adding the semantic segments in the speech translation result or the mixed intent translation result to the feedback queue according to the order of the timestamps.

[0078] In some embodiments of this disclosure, user data may include raw acquired data. Raw acquired data may include speech signals captured by a microphone sensor, scene image information and / or text image information captured by a camera sensor. Noise reduction to remove ambient noise and endpoint detection to identify the start and end points of speech can be used for data preprocessing. Timestamps can be synchronously generated based on the processed user data to associate the time dimension of speech and images.

[0079] Semantic segments from speech translation results or mixed intent translation results can be added to the feedback queue according to the order of timestamps. In some examples, the speech acquisition unit captures user speech through a microphone and bone conduction pickup, simultaneously performing noise reduction processing and endpoint detection. If a mixed translation request is subsequently detected, the image acquisition unit activates a miniature camera to capture images of the current scene (such as menus, road signs, etc.), and generates timestamps synchronously with the speech signal to ensure the temporal correlation between speech and images, providing a basis for subsequent semantic segmentation.

[0080] In some embodiments of this disclosure, hardware parameter optimization and preprocessing reduce interference from invalid data, thereby improving the signal-to-noise ratio of speech signals and the accuracy of image-text recognition. Timestamp synchronization ensures the consistency of multimodal data, avoids the problem of speech and image mismatch during subsequent mixed translation, ensures the temporal correlation between speech and image, and provides a basis for subsequent semantic segmentation.

[0081] In some embodiments of this disclosure, the step of performing semantic feedback according to the feedback queue further includes:

[0082] The voice input features of the user are acquired in real time, and the voice input features include at least one of the following: number of syllables per unit time, sentence interval time, and sentence pause nodes;

[0083] The size of the semantic segmentation window is dynamically adjusted based on the speech input features, and semantic segments are output based on the adjusted semantic segmentation window.

[0084] In some embodiments of this disclosure, voice input features typically refer to key data extracted by the wearable device during the collection of user voice requests, reflecting the user's speaking habits and rhythm. Voice input features need to be collected and analyzed in real time to ensure that the semantic segmentation window size is highly matched to the user's voice rhythm, avoiding semantic breaks due to overly fine segmentation or feedback delays due to overly coarse segmentation. Voice input features may include the number of syllables per unit time, sentence interval time, and sentence pause nodes. The number of syllables per unit time typically refers to the number of syllables spoken by the user per second, used to determine the speaking speed. For example, the normal speaking speed of a Chinese user is approximately 3-4 syllables per second, reaching 5-6 syllables per second when speaking quickly, and possibly less than 2 syllables per second when speaking slowly. Sentence interval time refers to the duration of the pause between two consecutive sentences. For example, the sentence interval in daily conversation is usually 0.5-1 second, while the thinking interval may be extended to greater than or equal to 2 seconds. Sentence pause nodes refer to the short pause positions within a single sentence, usually related to grammatical structure. Sentence pause nodes can reflect the semantic logical segmentation of a sentence.

[0085] The low-power processor in wearable devices can dynamically adjust the size of the semantic segmentation window based on preset window adjustment rules according to voice input characteristics. In some examples, for fast speech rates (e.g., greater than or equal to a first preset speech rate, such as 5 syllables per second), combined with high-frequency pauses (determined by a combination of sentence interval time and sentence pause nodes), a semantic segmentation window of the first size can be matched. This first-size semantic segmentation window is typically a small window. For example, if a user quickly says, "I'm going to the airport at 10 a.m. tomorrow," the first size can be set to one word, splitting it into semantic segments: tomorrow, morning, 10 a.m., and going to the airport. Similarly, in other examples, for medium speech rates (e.g., less than or equal to the first preset speech rate but greater than or equal to a second preset speech rate), combined with mid-frequency pauses, a semantic segmentation window of the second size can be matched. In other examples, for slow speech rates (e.g., less than or equal to the second preset speech rate), combined with low-frequency pauses, a semantic segmentation window of the third size can be matched. The first size is smaller than the second size, and the second size is smaller than the third size. It should be noted that the above-mentioned preset window adjustment rules are only illustrative examples and can be combined and adjusted according to actual needs, such as combining fast speech with mid-frequency pauses.

[0086] In some embodiments of this disclosure, the size of the semantic segmentation window can be dynamically adjusted according to the speech input features to ensure that the window size is highly adapted to the speech input features. This allows for real-time dynamic adjustment to respond to changes in speech rhythm, avoids feedback deviations caused by feature changes but window unchanged, ensures that semantic segments conform to language logic, avoids cross-breakpoint splitting, and balances semantic rationality, segmentation accuracy, and semantic scene adaptability.

[0087] In some embodiments of this disclosure, the method further includes:

[0088] The system receives touch control commands and / or voice control commands from the user, and executes corresponding processing strategies in the feedback queue according to the touch control commands and / or the voice control commands; the processing strategies are configured to include at least one of adjusting the translation language, viewing detailed explanations, pausing translation, and continuing translation.

[0089] In some embodiments of this disclosure, touch control commands typically refer to operation commands initiated by a user by touching the display screen or physical touch area of ​​the wearable device. Touch control commands can include single-point touch commands, swipe touch commands, long-press touch commands, etc. Voice control commands typically refer to operation commands initiated by a user by uttering specific keywords or short phrases to the wearable device. Voice control commands can include categories such as basic control commands, function adjustment commands, and detailed query commands. Basic control commands can be used to start or stop translation feedback, such as pausing translation, continuing translation, or terminating translation. Function adjustment commands can be used to modify core translation parameters, such as switching to Chinese-Japanese translation or changing the target language to French. Detailed query commands are used to obtain supplementary information about the translation results and can be associated with semantic segments in the current feedback queue to ensure the accuracy of the query target.

[0090] Feedback queue processing strategies typically refer to the operational logic executed by wearable devices on a feedback queue storing semantic segments based on touch control commands and / or voice control commands. Feedback queue processing strategies may include adjusting the translation language, viewing detailed explanations, pausing translation, and resuming translation. In some implementations, feedback queue processing strategies may also include combined strategies, such as pausing translation and viewing detailed explanations. The strategy of adjusting the translation language can be implemented by updating the translation rules of semantic segments in the feedback queue, such as changing the Chinese-to-English translation of subsequent segments to Chinese-to-Japanese, keeping the original translation of already generated segments, and processing new segments according to the new rules. The strategy of viewing detailed explanations inserts detailed explanation segments into the feedback queue. For example, after a user clicks "steak," the queue pauses the output of the current segment, inserts an explanation segment containing the keyword "steak," and resumes the original queue after output is complete. The strategy of pausing translation pauses the scheduled output of the feedback queue, retains the output progress of the current segment, and waits for related commands such as continuing translation. The strategy of resuming translation resumes the scheduling of the feedback queue and continues outputting subsequent segments from the paused position, ensuring semantic coherence.

[0091] In some embodiments of this disclosure, users can adjust the language according to the communication scenario, view details according to their understanding needs, and control the start and stop according to their rhythm needs, which can meet users' personalized needs and improve user personalization adaptability; it can also retain the output progress of segments, avoid repeated feedback caused by pausing and restarting, and automatically restore the original queue after viewing the detailed explanation, without requiring users to manually backtrack, thus ensuring semantic coherence; at the same time, it optimizes resource allocation, and adjusting the language strategy only updates the rules of subsequent segments, without reprocessing the generated segments, reducing the local hardware computing load and adapting to the low power consumption requirements of wearable devices.

[0092] In some embodiments of this disclosure, the method further includes:

[0093] The system stores user interaction data, which includes at least one of interaction log information, translation results, and error correction feedback information.

[0094] When the current network connection is idle, the interaction data is encrypted and sent to the server, and the updated offline translation model and intent recognition parameters are returned by the server based on the encrypted interaction data.

[0095] In some embodiments of this disclosure, user interaction data typically refers to various types of information related to the entire translation process collected and generated by the wearable device during semantic recognition interaction with the user. User interaction data may include interaction log information, translation results, error correction feedback information, etc. Interaction log information records key operations and environmental parameters during the interaction process, such as request type, network status, processing time, device model, etc., and can be used by the server to analyze translation performance in different scenarios. Translation results refer to the complete translation content generated locally or on the server, including source language text, target language text, and semantic segmentation records, and can be used to analyze translation accuracy and segmentation rationality. Error correction feedback information refers to the operation records of users actively correcting translation results, such as users marking omissions or correcting translation results, which can directly reflect translation deviations and / or user translation habits, and is key data for model optimization.

[0096] When the network is idle, wearable devices package locally encrypted interaction data according to time ranges and send it to the server. The server then fine-tunes the model based on the user interaction data, such as adding high-frequency word weights and optimizing semantic segmentation algorithms, generating a parameter configuration file containing an updated keyword library, intent classification thresholds, and scene adaptation rules. The model file and parameter file can then be packaged into an encrypted update package and sent to the wearable device for hot updating. In standby or non-interactive states, the old offline translation model and intent recognition parameters are automatically replaced without affecting normal user operation. After the update is complete, an update notification can pop up on the display device to provide update feedback.

[0097] In some embodiments of this disclosure, lightweight updates adapt to the storage and transmission needs of wearable devices, avoiding storage occupation and transmission delays caused by large-capacity updates. Hot updates and update feedback can ensure that users are unaware of the update and know the update status, which is more in line with the user's usage habits and does not affect the user's normal use, thus comprehensively improving the user experience.

[0098] The semantic recognition methods for wearable devices disclosed herein can closely align with user interaction habits, optimize resource allocation, and process voice translation requests locally, avoiding network latency in cloud interactions. Simultaneously, they can improve processing efficiency in complex scenarios, enabling joint analysis of multimodal data in a single cloud request for mixed translation requests, eliminating the need for multiple requests, thus improving the efficiency of multi-turn dialogues, reducing frequent device operations, lowering user operating costs, and allowing devices to automatically match processes through intent recognition, minimizing jarring experiences, adapting to diverse scenario needs, and comprehensively enhancing the user experience.

[0099] It is understood that the various embodiments of the methods described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Related details can be found in the descriptions of other method embodiments.

[0100] It should be understood that although the steps in the flowcharts shown in the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0101] Based on the description of the semantic recognition method embodiments for wearable devices described above, this disclosure also provides a semantic recognition device for wearable devices used to implement the semantic recognition method for wearable devices involved above. The device may include a system (including a distributed system), software (application), module, component, controller, server, terminal, etc., using the method described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the devices in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the devices are similar, the implementation of specific devices in the embodiments of this specification can refer to the implementation of the foregoing method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0102] Figure 4 This is a schematic block diagram illustrating a semantic recognition device for a wearable device according to an exemplary embodiment. The device can be the aforementioned terminal, a server, or a module, component, device, control unit, etc., integrated into the terminal. For details, please refer to... Figure 4 The semantic recognition device 100 of the wearable device may include: an intent recognition module 120, a local speech translation module 140, and a communication module 160. The intent recognition module 120 is used to determine, in response to detecting a user's voice request, whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes requests for at least two of the following: speech recognition translation, image recognition translation, and text recognition translation. The local speech translation module 140 is used to, when the voice request is a voice-only translation request, invoke the local speech recognition module to perform speech translation and generate a speech translation result. The communication module 160 is used to, when the voice request is a mixed translation request, send the user data corresponding to the voice request to the server and receive the mixed intent translation result returned by the server after performing mixed translation based on the user data.

[0103] In some embodiments of the device, the wearable device includes a voice playback device and a display device. The semantic recognition device 100 of the wearable device further includes a semantic feedback module, used to add semantic segments from the voice translation result or the mixed intent translation result to a feedback queue in chronological order, and to perform semantic feedback according to the feedback queue. The semantic feedback module is also used to perform first-word feedback through the voice playback device according to the feedback queue; and to synchronize the first-word feedback through the display device according to the feedback queue, displaying subtitles and / or image markers corresponding to the complete sentence.

[0104] In some embodiments of the device, the semantic feedback module is also used to acquire the user's voice input features in real time, the voice input features including at least one of the number of syllables per unit time, the sentence interval time, and the sentence pause node; it is also used to dynamically adjust the size of the semantic segmentation window according to the voice input features, and output semantic segments according to the adjusted semantic segmentation window.

[0105] In some embodiments of the device, the semantic feedback module is further configured to receive touch control commands and / or voice control commands from the user, and execute corresponding processing strategies in the feedback queue according to the touch control commands and / or the voice control commands; the processing strategies are configured to be at least one of adjusting the translation language, viewing detailed explanations, pausing translation, and continuing translation.

[0106] In some embodiments of the device, the semantic recognition device 100 of the wearable device further includes a data storage module for storing user interaction data, which includes at least one of interaction log information, translation results, and error correction feedback information. The communication module 160 is also used to encrypt the interaction data and send it to the server when the current network connection is idle, and to receive an updated offline translation model and intent recognition parameters returned by the server based on the encrypted interaction data.

[0107] In some embodiments of the device, the semantic recognition device 100 of the wearable device further includes a data acquisition module for acquiring user data through sensors and performing noise reduction and endpoint detection processing on the user data. The intent recognition module 120 is also used to synchronously generate timestamps based on the processed user data. The semantic feedback module is also used to add semantic segments from the speech translation result or the mixed intent translation result to a feedback queue according to the order of the timestamps.

[0108] The modules in the semantic recognition device of the aforementioned wearable device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0109] In one embodiment, a wearable device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the wearable device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface allows for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a semantic recognition method for the wearable device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the wearable device's casing, or an external keyboard, touchpad, or mouse.

[0110] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the wearable device to which the present application is applied. A specific wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0111] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a wearable device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the semantic recognition method of the wearable device described in any embodiment of this specification.

[0112] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by the processor of a computer device, enables the computer device to implement the semantic recognition method for wearable devices as described in any embodiment of this disclosure.

[0113] Based on the foregoing description of the relevant methods and apparatus embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the semantic recognition method for wearable devices described in any embodiment of this specification.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0116] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0117] It should be noted that the apparatus, computer equipment, storage medium, and computer program products described above may also include other implementation methods according to the description of the method embodiments. Specific implementation methods can be found in the description of the relevant method embodiments. Furthermore, new embodiments formed by combinations of features from various methods, apparatuses, devices, and server embodiments still fall within the scope of this disclosure and will not be elaborated upon here.

[0118] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling and communication connections between the devices or units shown or described can be implemented through direct and / or indirect coupling / connection, through standard or custom interfaces or protocols, and can be implemented electrically, mechanically, or in other forms.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0120] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A semantic recognition method of a wearable device, the method comprising: The method includes: In response to detecting a user's voice request, determine whether the voice request is a voice-only translation request or a mixed translation request, wherein the mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent. If the voice request is a voice translation-only request, the local speech recognition module is invoked to perform the voice translation and generate the voice translation result; In the case that the voice request is a mixed translation request, the user data corresponding to the voice request is sent to the server, and the server returns the mixed intent translation result returned by performing mixed translation based on the user data; The wearable device includes a voice playback device and a display device, and the method further includes: Semantic segments from the speech translation result or the mixed intent translation result are added to a feedback queue in chronological order, and semantic feedback is performed based on the feedback queue, including: The voice playback device performs first-word feedback according to the feedback queue; The display device synchronizes the first word feedback according to the feedback queue and displays the corresponding complete sentence's subtitles and / or image markers.

2. The method of claim 1, wherein, The step of performing semantic feedback based on the feedback queue further includes: The voice input features of the user are acquired in real time, and the voice input features include at least one of the following: number of syllables per unit time, sentence interval time, and sentence pause nodes; The size of the semantic segmentation window is dynamically adjusted based on the speech input features, and semantic segments are output based on the adjusted semantic segmentation window.

3. The method of claim 1, wherein, The method further includes: Receive user touch control commands and / or voice control commands, and execute corresponding processing strategies in the feedback queue according to the touch control commands and / or the voice control commands; The processing strategy is configured to include at least one of the following: adjust the translation language, view detailed explanation, pause translation, and continue translation.

4. The method of claim 1, wherein, The method further includes: The system stores user interaction data, which includes at least one of interaction log information, translation results, and error correction feedback information. When the current network connection is idle, the interaction data is encrypted and sent to the server, and the updated offline translation model and intent recognition parameters are returned by the server based on the encrypted interaction data.

5. The method of claim 1, wherein, The method further includes: User data is collected through sensors, and noise reduction and endpoint detection processing are performed on the user data. Timestamps are generated synchronously based on the processed user data. The step of adding semantic segments from the speech translation result or the mixed intent translation result to the feedback queue in chronological order includes: Semantic segments from the speech translation results or the mixed intent translation results are added to the feedback queue according to the timestamp order. 6.A semantic recognition apparatus of a wearable device, characterized by, The device includes: An intent recognition module is used to respond to the detection of a user's voice request and determine whether the voice request is a voice-only translation request or a mixed translation request. The mixed translation request includes at least two of the following: voice recognition translation, image recognition translation, and text recognition translation, which identify the translation intent. The local speech translation module is used to call the local speech recognition module to perform speech translation and generate a speech translation result when the speech request is a speech translation-only request; The communication module is used to send the user data corresponding to the voice request to the server when the voice request is a mixed translation request, and to receive the mixed intent translation result returned by the server after performing mixed translation based on the user data; The wearable device includes a voice playback device and a display device. The device further includes a semantic feedback module, which is used to add semantic segments from the voice translation result or the mixed intent translation result to a feedback queue in chronological order, and to perform semantic feedback according to the feedback queue. The semantic feedback module is also used to perform first-word feedback through the voice playback device according to the feedback queue, and to synchronize the first-word feedback through the display device according to the feedback queue, and to display subtitles and / or image markers corresponding to the complete sentence.

7. A wearable device, comprising: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images