A cross-language conference translation method and smart glasses

Abstract

The embodiment of the present application discloses a cross-language conference translation method and smart glasses, which relate to the field of human-computer interaction technology and are used to solve the problem that existing translation is inaccurate and difficult to combine image rendering and translation functions. The method includes: extracting the human voice signal based on the sound signal characteristics collected by the smart glasses, and obtaining the translation scene corresponding to the smart glasses based on the human voice signal; determining whether the translation scene has corresponding linkage translation information; if not, extracting the human voice signal to be translated in the human voice signal in the translation scene to translate it into the translation result of the target language; rendering and displaying the translation result through the perspective information and image information detected by the smart glasses, as well as the orientation information corresponding to the human voice signal to be translated.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a cross-language conference translation method and smart glasses. Background Art

[0002] With the acceleration of globalization, cross-language conferences have become a vital form of international business, academic exchange, and multilateral cooperation. However, language barriers still severely constrain meeting efficiency and participant experience. In a multilingual meeting, miscommunication simply due to language barriers can lead to missed business opportunities, misunderstandings in academic exchanges, and even erroneous policy discussions. Therefore, in this context, the importance of translation equipment has become increasingly prominent.

[0003] While traditional simultaneous interpretation can provide basic language conversion, it's costly, inflexible, and requires extremely high professional skills from interpreters. Modern translation equipment, which primarily relies on handheld devices or software tools, can provide basic translation capabilities, but conference scenarios often present issues such as multiple speakers speaking at once and ambient noise interference. Existing equipment struggles to accurately separate the target voice from background noise, which can lead to translation confusion, especially in multi-person discussions. Traditional equipment relies solely on single voice input and output, and lacks the ability to integrate well with ambient imagery. This makes image rendering and translation functionality inflexible and unable to meet the diverse needs of users in cross-language meetings. Summary of the Invention

[0004] In order to solve the above technical problems, one or more embodiments of the present application provide a cross-language conference translation method and smart glasses.

[0005] One or more embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, one or more embodiments of the present application provide a cross-language conference translation method, the method comprising:

[0007] Extracting a human voice signal based on the sound signal features collected by the smart glasses, so as to obtain a translation scenario corresponding to the smart glasses according to the human voice signal;

[0008] Determining whether the translation scenario has corresponding linkage translation information;

[0009] If not, extracting the human voice signal to be translated from the human voice signal in the translation scenario to translate it into a translation result in the target language;

[0010] The translation result is rendered and displayed using the viewing angle information and image information detected by the smart glasses, as well as the orientation information corresponding to the human voice signal to be translated.

[0011] Optionally, in one or more embodiments of the present application, extracting a human voice signal based on the sound signal features collected by the smart glasses specifically includes:

[0012] Collecting sound signals based on preset microphone arrays distributed at different positions of the smart glasses;

[0013] Acquiring sound signal features of the sound signal; wherein the sound signal features include: spatial features and spectral features;

[0014] The sound signal features are classified using a pre-trained classifier, and a human voice signal is extracted from the sound signal according to the classification result.

[0015] Optionally, in one or more embodiments of the present application, obtaining a translation scenario corresponding to the smart glasses according to the human voice signal specifically includes:

[0016] Extracting voiceprint features and time series features corresponding to the human voice signal;

[0017] Obtaining a voiceprint feature vector corresponding to the voiceprint feature, and comparing the voiceprint feature vector with a voiceprint feature vector of a user of the smart glasses;

[0018] According to the comparison results and the human voice interaction features corresponding to the user voiceprint feature vector determined based on the time series features, the translation scenario corresponding to the smart glasses is determined; wherein the translation scenarios include: dialogue interaction scenarios and speech scenarios.

[0019] Optionally, in one or more embodiments of the present application, determining whether the translation scenario has corresponding linkage translation information specifically includes:

[0020] If it is determined that the translation scenario is a dialogue interaction scenario, determining whether the interlocutor voiceprint feature vector corresponding to the user voiceprint feature vector belongs to the current speaker;

[0021] If so, determining whether there is corresponding linkage translation information based on the venue information of the current meeting;

[0022] If it is determined that the translation scene is a speech scene, it is determined whether there is corresponding linkage translation information based on the venue information of the current meeting.

[0023] Optionally, in one or more embodiments of the present application, extracting the to-be-translated human voice signal from the human voice signal in the translation scenario to translate it into a translation result in the target language specifically includes:

[0024] Identifying the language type of the human voice signal using a pre-trained language recognition model, and comparing the language type with the system language type;

[0025] Based on the comparison result and the voiceprint feature vector contained in the translation scenario, screening the human voice signal to be translated;

[0026] The human voice signal to be translated is translated based on a pre-trained translation model to obtain a text translation result in the target language.

[0027] Optionally, in one or more embodiments of the present application, after extracting the human voice signal to be translated from the human voice signal in the translation scenario, the method further includes:

[0028] Extracting voiceprint feature parameters corresponding to the human voice signal to be translated to determine a voiceprint feature vector corresponding to the voiceprint feature parameters;

[0029] Binding the voiceprint feature vector to the identity information of the current speaker and storing the result in a storage unit corresponding to the smart glasses;

[0030] The voiceprint feature vector is compared with the real-time voiceprint feature vector to obtain the identity information of the new speaker for binding record.

[0031] Optionally, in one or more embodiments of the present application, rendering and displaying the translation result using the viewing angle information and image information detected by the smart glasses, and the orientation information corresponding to the human voice signal to be translated, specifically includes:

[0032] Initiate acquisition of the image information based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated;

[0033] determining a display position of the translation result according to the image information and the viewing angle information;

[0034] Adjusting the current display parameters of the translation result by using the display parameters recorded by the XR rendering engine of the smart glasses and the current environment parameters corresponding to the image information;

[0035] The translation result is rendered and displayed based on the display position and the current display parameters.

[0036] Optionally, in one or more embodiments of the present application, determining a display position of the translation result according to the image information and the viewing angle information specifically includes:

[0037] Locating a corresponding physical space area based on the viewing angle information, and performing real-time semantic segmentation on the image information to obtain an idle area;

[0038] Performing perspective transformation alignment based on the perspective focus coordinates corresponding to the perspective information and the two-dimensional image coordinates of the idle area to determine an intersection area between the physical space area and the idle area;

[0039] A display path of the translation result in the intersection area is planned by using an A* algorithm, so as to determine a display position of the translation result based on the display path.

[0040] Optionally, in one or more embodiments of the present application, if the translation scenario has corresponding linkage translation information, the method further includes:

[0041] extracting a human voice signal to be translated from the human voice signal in the translation scenario, and translating it into a first translation result in a target language;

[0042] Rendering and displaying the first translation result based on the viewing angle information and image information detected by the smart glasses, and the orientation information corresponding to the human voice signal to be translated;

[0043] Obtaining the linked translation information to translate it into a second translation result in the target language;

[0044] determining whether the linked translation information is within the user's line of sight based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the linked translation information;

[0045] If so, based on the duration of the linked translation information within the user's line of sight, the display of the second translation result is started, so as to render and display the second translation result based on the viewing angle information and image information detected by the smart glasses.

[0046] Second aspect. One or more embodiments of the present application provide smart glasses, comprising at least one processor, a memory, a camera module, a microphone module, and a display module, wherein the at least one processor is coupled to the memory and configured to read and execute instructions in the memory to perform the method of the first aspect and possible design methods thereof.

[0047] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0048] By extracting the human voice signal through sound signal features, the target voice can be accurately separated from background noise, avoiding the translation confusion caused by noise interference in traditional devices. By determining whether there is linked translation information in the translation scene, it helps to balance translation flexibility and accuracy and improve the translation scene adaptability. Based on multi-dimensional information integration, the translation results are deeply integrated with the environmental image, breaking through the problem that traditional devices cannot meet the user's needs for combining image rendering and translation functions, and improving the efficiency of information acquisition and the immersive experience in cross-language meetings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0050] Figure 1 A schematic diagram of an application scenario of a cross-language conference translation method provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of a cross-language conference translation method provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of an interactive scenario for cross-language conference translation provided in an embodiment of the present application;

[0053] Figure 4 A flowchart of a method for extracting a human voice signal provided in an embodiment of the present application;

[0054] Figure 5 A flowchart of a method for determining a translation scenario provided in an embodiment of the present application;

[0055] Figure 6 A flowchart of a method for determining linked translation information corresponding to a translation scenario provided in an embodiment of the present application;

[0056] Figure 7 A flowchart of a method for generating translation results provided in an embodiment of the present application;

[0057] Figure 8 A flowchart of an identity information binding method provided in an embodiment of the present application;

[0058] Figure 9 A flowchart of a method for rendering and displaying translation results provided in an embodiment of the present application;

[0059] Figure 10 A flowchart of a method for determining a display position of a translation result provided in an embodiment of the present application;

[0060] Figure 11 A flowchart of another method for determining linked translation information corresponding to a translation scenario provided in an embodiment of the present application;

[0061] Figure 12 A hardware structure block diagram of smart glasses provided in an embodiment of the present application;

[0062] Figure 13 A schematic diagram of the structure of a cross-language conference translation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] Embodiments of the present application provide a cross-language conference translation method and smart glasses.

[0064] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0065] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0066] The cross-language conference translation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1As shown, the application environment may include: smart glasses 101 worn by user 107, a communication network 102, a server 103, a language resource server 104, an image resource server 105, and a database 106. The communication network 102 can serve as a data transmission channel, providing a communication link for communication between the smart glasses 101 and the server 103. This allows the smart glasses 101 to collect various types of sound signals, visual angle information, image information, and other information and transmit them to the server 103 via the communication network 102. The information processed by the server 103 can also be fed back to the smart glasses 101 for rendering and display. The server 103 is a server that can provide various services. To improve processing efficiency, the server 103 is connected to the language resource server 104 and the image resource server 105. After receiving the information collected by the smart glasses 101 via the communication network 102, the server 103 can determine whether there is corresponding linked translation information for the translation scene. If so, it is sent to the image resource server 105 for processing. On the other hand, if there is no corresponding linked translation information, the human voice signal to be translated is extracted from the human voice signal and sent to the language resource server 104 for translation processing. After the translation is completed, the server 103 combines the perspective information, image information and the orientation information of the human voice signal to be translated detected by the smart glasses 101 to render and display the translation results, and finally feeds them back to the smart glasses 101 based on the communication network 102 to present them to the user.

[0067] The database 106 is connected to the language resource server 104 and the image resource server 105, respectively, and can be used to store and manage the language data of the language resource server 104 and the image data of the image resource server 105. The database 106 can be integrated on a server, or placed on a cloud or other network server.

[0068] like Figure 2 The embodiment of the present application shown is a flowchart of a cross-language conference translation method. Figure 2 The method shown can be applied to high-frequency scenarios that require multilingual real-time interaction and focus on immersive experience, including but not limited to business meetings, academic seminars, product launches, and meetings involved in education, entertainment, and medical fields. The wearer wears smart glasses, which can be XR glasses. The method includes the following steps: S201-S204.

[0069] S201: extracting a human voice signal based on the sound signal features collected by the smart glasses, so as to obtain a translation scenario corresponding to the smart glasses according to the human voice signal.

[0070] When users participate in cross-language conferences and want to understand the speaker's speech or PPT content, how can they Figure 3In the interactive process shown, after collecting various sounds, the smart glasses need to extract the human voice signal based on the sound signal characteristics. This allows the smart glasses to determine the corresponding translation scenario based on the human voice signal. For example, if the human voice signal is a single voice and the voiceprint feature vector corresponds to the speaker's voiceprint feature vector, it indicates a speech scenario. If there are multiple voice signals and the voiceprint feature vector corresponds to the user wearing the smart glasses, it indicates a conversational interaction scenario.

[0071] In some embodiments, the process of extracting the human voice signal based on the sound signal features collected by the smart glasses in step S201 is as follows: Figure 4 The process specifically includes steps S401-S403:

[0072] S401: collecting sound signals based on preset microphone arrays distributed at different positions of the smart glasses.

[0073] S402: Acquire sound signal features of the sound signal; wherein the sound signal features include: spatial features and spectral features.

[0074] S403: Classify the sound signal features using a pre-trained classifier, and extract the human voice signal from the sound signal according to the classification result.

[0075] Smart glasses can collect sound signals based on the microphone module, which can be a pre-set microphone array distributed in different positions of the smart glasses. Among them, the pre-set microphone array is a microphone module of the smart glasses equipped with multiple high-precision microphones to form a microphone array, which is distributed in different positions of the glasses. These microphones have high sensitivity and a wide frequency response range, and can clearly capture various sound signals at the conference site, thereby realizing all-round collection of sound signals. Then the sound signal features of the sound signal are obtained, including: spatial features and spectral features. The sound signal features are classified by a pre-trained classifier, and the human voice signal in the sound signal is extracted according to the classification results. That is, based on the spatial features such as the arrival time difference and phase difference of the sound, combined with the spectral features of the sound, the classifier trained by the machine learning algorithm can accurately and intelligently extract the speaker's voice from the complex sound environment, providing reliable data support for subsequent recognition.

[0076] In some embodiments, the added spatial features, namely the time difference and phase difference of the sound arriving at different microphones, can also be combined with beamforming technology to construct a spatial distribution model of the sound field, thereby accurately locking the speaker's position to obtain the sound source localization result, and then quickly determining the main speaker and background communication sound, thereby avoiding the problem of confusion of the translation object. After obtaining the sound source localization result, the sound in the target direction of the smart glasses can be amplified based on the sound source localization result, and the noise in other directions can be suppressed. At the same time, the sound reflection path corresponding to the sound signal and the spatial distribution of the microphone array are used to distinguish between the direct sound and the reflected sound in the sound signal, thereby filtering the reflected sound and obtaining the direct sound in the target direction. After obtaining the direct sound in the target direction, in order to avoid the interference of the natural sound mixed therein, considering that the human voice signal has a unique spectral structure with a fundamental frequency range of 50-500Hz and a resonance peak concentrated in 200-3000Hz, in the embodiment of the present application, the spectral characteristics of the direct sound in the target direction obtained after the above-mentioned filtering based on the spatial features are extracted by fast Fourier transform, thereby effectively distinguishing the human voice from the natural sound and realizing the extraction of the human voice signal.

[0077] In some embodiments, in order to improve the quality of the human voice signal, after the smart glasses collect the sound signal, the collected sound signal first enters the preprocessing stage, and uses digital signal processing technology to remove environmental noise interference. For example, a filtering algorithm is used to filter out high-frequency or low-frequency noise components to remove environmental noise interference and improve the quality of the human voice signal.

[0078] In some embodiments, the process of obtaining the translation scene corresponding to the smart glasses according to the human voice signal in step S201 is as follows: Figure 5 The following steps S501-S503 are implemented as shown:

[0079] S501: Extracting voiceprint features and time sequence features corresponding to the human voice signal;

[0080] S502: Obtain a voiceprint feature vector corresponding to the voiceprint feature, and compare the voiceprint feature vector with a voiceprint feature vector of the user of the smart glasses;

[0081] S503: Determine the translation scenario corresponding to the smart glasses based on the comparison results and the human voice interaction features corresponding to the user voiceprint feature vector determined based on the time series features; wherein the translation scenarios include: dialogue interaction scenarios and speech scenarios.

[0082] Among them, the human voice interaction feature takes into account the scenario where the user wearing smart glasses talks and communicates with the person next to him during the meeting or outside the meeting time. Therefore, the human voice interaction feature refers to the voiceprint feature vector that has a dialogue interaction relationship with the user's voiceprint feature vector determined based on the timestamp corresponding to the user's voiceprint feature vector in the time series feature and the interval between the timestamps corresponding to the adjacent voiceprint feature vectors.

[0083] In this embodiment, the translation scenario corresponding to the smart glasses is determined based on the comparison results and the human voice interaction features corresponding to the user voiceprint feature vector determined based on the time series features. When the comparison results determine that there is a voiceprint feature vector that is consistent with the user voiceprint feature vector of the smart glasses, it means that the wearer of the smart glasses has spoken. At this time, based on the human voice interaction features corresponding to the user voiceprint feature vector, it can be determined whether there is a person who is having a conversation with the wearer of the smart glasses, and whether the person who is having a conversation with the wearer of the smart glasses is the speaker, etc. Therefore, it can be determined based on these contents whether the scenario faced by the smart glasses is a conversation interaction scenario or a speech scenario.

[0084] Through the dual verification mechanism of determining identity through voiceprint features and judging mode through temporal features, accurate recognition of translation scenarios is achieved, which solves the problem that traditional translation systems have difficulty in dynamically distinguishing whether the user is in a dialogue interaction or a speech scenario, resulting in inappropriate timing or position of the translation content display. It significantly improves the translation efficiency and user experience of smart glasses in multiple scenarios, and achieves adaptability to complex scenarios that require rapid switching of interaction modes.

[0085] S202: Determine whether the translation scenario has corresponding linkage translation information.

[0086] Since cross-language conferences may involve the use of electronic screen displays, whiteboards, and other devices to display the speaker's PPT or meeting documents, in order to enhance the user's immersive experience in the meeting and achieve comprehensive translation of content that requires supplementary translation, it will be determined whether the translation scenario has corresponding linked translation information.

[0087] In some embodiments, the process of determining whether the translation scene has corresponding linkage translation information according to the human voice signal in step S202 is as follows: Figure 6 The following steps S601-S603 are implemented as shown:

[0088] S601: If it is determined that the translation scenario is a dialogue interaction scenario, determining whether the interlocutor voiceprint feature vector corresponding to the user voiceprint feature vector belongs to the current speaker;

[0089] S602: If yes, determine whether there is corresponding linkage translation information based on the venue information of the current meeting;

[0090] S603: If it is determined that the translation scene is a speech scene, it is determined whether there is corresponding linkage translation information based on the venue information of the current meeting.

[0091] When the system determines that the current scenario is a conversational interaction, it further verifies the identity of the conversation partner and, based on the venue information, determines whether to trigger linked translation. During this process, the system extracts the voiceprint feature vector of the conversation partner and compares it with the voiceprint feature vector of the current speaker to determine whether the conversation partner is the speaker. This takes into account the interaction between the user and the speaker during the meeting. If the conversation partner is the current speaker, the system retrieves the venue information, such as the conference room number, layout, and projection equipment location, to determine whether there are translation resources associated with that venue. For example, in a scenario where a person wearing smart glasses asks a question to the speaker during a meeting, the system identifies the conversation partner as the speaker and automatically associates the venue with the projection screen or other display device. If the current scenario is a lecture, linked translation is triggered directly based on the venue information. In this process, in a conversational interaction scenario, by comparing the user's voiceprint feature vector with the conversation partner's voiceprint feature vector, the conversation partner can be accurately determined to be the current speaker, thus avoiding the possibility of mishandling private communications. Retrieving pre-set linked resources based on the venue information helps ensure that all visual aids used by the speaker are presented synchronously with the translation results.

[0092] In some embodiments, after the display device is associated, the reverse translation of the voice of the smart glasses wearer can be displayed on the display device.

[0093] In this process, by determining whether there is corresponding linkage translation information based on the translation scenario, it can ensure that the venue linkage translation is only triggered when the user interacts with the speaker, avoiding irrelevant conversations from interfering with the meeting process. In addition, by combining the translation scenario type with the venue information, the linkage strategy is dynamically matched to ensure seamless coordination between the translation content and the venue equipment. By adding the comparison of voiceprint feature vectors, it is possible to accurately locate effective voice in noisy, multi-speaker environments, reducing the interference of environmental noise on the decision-making of whether there is linkage translation information, which helps to improve the practicality and intelligence level of smart glasses in professional scenarios.

[0094] S203: If not, extracting the human voice signal to be translated from the human voice signal in the translation scenario to translate it into a translation result in the target language.

[0095] If the translation scenario does not have corresponding linked translation information, the system extracts the voice signal to be translated from the voice signal within the translation scenario and translates it into the target language that the user can understand. If the voice signal to be translated is detected to be different from the user's system language, it will automatically translate into the user's system language or the target language specified by the user.

[0096] In some embodiments, the process of extracting the human voice signal to be translated from the human voice signal in the translation scenario according to the human voice signal in step S203 to translate it into the translation result of the target language is as follows: Figure 7 The following steps S701-S703 are implemented as shown:

[0097] S701: Identify the language type of the human voice signal using a pre-trained language recognition model, and compare the language type with the system language type;

[0098] S702: Filtering a human voice signal to be translated based on the comparison result and the voiceprint feature vector included in the translation scenario;

[0099] S703: Translate the human voice signal to be translated based on the pre-trained translation model to obtain a text translation result in the target language.

[0100] In cross-language conferences, speech in multiple languages ​​may be present. Traditional translation devices cannot distinguish between "speech to be translated" and "irrelevant speech," resulting in a mixture of invalid information in the translation results. Therefore, by using a pre-trained speech recognition model to identify the language type of the voice signal, it is possible to filter out speech that aligns with the system language. The voiceprint features contained in the translation scenario are then used to further identify the valid speaker, eliminating ambient noise and non-critical individuals. This means that, based on steps S501-S503, the voiceprint feature vectors contained in each translation scenario can be determined. By triggering the translation process on demand to filter speech in non-target languages ​​and non-critical scenarios, the voice signal to be translated is extracted and translated. This allows for the rapid elimination of native language content already understood by the user in a multilingual conference, reducing computing resources consumed by ineffective translation. Furthermore, by combining the recorded voiceprint feature vectors in the translation scenario to filter the voice signal to be translated, this dual filtering of language type and voiceprint features reduces the amount of translation data, thereby minimizing real-time translation latency. By translating only the selected high-quality voice signals to be translated, the translation model can more efficiently generate accurate results, reducing recalculation and error correction costs.

[0101] The pre-trained language recognition model is pre-trained on a large-scale multilingual speech dataset and can quickly and accurately identify the speaker's language. The pre-trained translation model is trained on a large multilingual parallel corpus and can accurately translate speech content into the user's system language or a user-specified target language based on the context. The translation results are output as text.

[0102] In some embodiments, for the complex environment of the smart glasses wearer's native language, multilingual speeches, and other participants communicating in minority languages, a hierarchical screening mechanism can be used to screen various language types layer by layer, and irrelevant human voice signals can be filtered in combination with voiceprint features to achieve the extraction of key voice signals that need to be translated and obtain the human voice signals to be translated.

[0103] In some embodiments, after extracting the human voice signal to be translated from the human voice signal in the translation scenario, the method further includes: Figure 8 The following steps S801-S803 are shown:

[0104] S801: Extracting voiceprint feature parameters corresponding to the human voice signal to be translated to determine a voiceprint feature vector corresponding to the voiceprint feature parameters.

[0105] S802: Bind the voiceprint feature vector with the identity information of the current speaker and store the result in a storage unit corresponding to the smart glasses.

[0106] S803: Compare the voiceprint feature vector with the real-time voiceprint feature vector to obtain the identity information of the new speaker for binding record.

[0107] During the vocal signal acquisition process, in order to subsequently identify the person corresponding to the vocal signal to be translated, this application extracts the voiceprint feature parameters corresponding to the vocal signal to be translated to determine the voiceprint feature vector corresponding to these voiceprint feature parameters. Specifically, the voiceprint feature parameters of each speaker are combined into a feature vector to obtain a voiceprint feature vector. The voiceprint feature vector is then bound to the current speaker's identity information. At this point, whether the conference speaker or anyone else conversing with the user corresponds to the vocal signal to be translated, they are considered the current speaker. This voiceprint feature vector is bound to the current speaker's identity information, such as name and profile picture, and stored in the smart glasses' local storage unit or a cloud server connected to the smart glasses. Due to the possibility of the current speaker changing, the smart glasses will display the name, profile picture, and other information of another speaker based on the voiceprint recognition and identity binding records. Therefore, in this embodiment of the application, the voiceprint feature vector is compared with the real-time voiceprint feature vector to obtain the new speaker's identity information for binding.

[0108] In one embodiment, the process of obtaining the identity information of a new speaker for binding and recording can include extracting the voiceprint feature vector of the new voice in real time when a new voice input is detected. The newly extracted voiceprint feature vector is then compared with the voiceprint feature vectors of each previously recorded speaker. This comparison uses metrics such as Euclidean distance and cosine similarity to calculate the similarity between the new voiceprint feature vector and the previously recorded voiceprint feature vectors. If the similarity exceeds a preset threshold, the speaker of the new voice matches a previously recorded speaker. Based on the identity binding record, the smart glasses display the speaker's name, profile picture, and other information on the glasses' display screen. If the similarity does not exceed the threshold, the speaker is considered new, the voiceprint feature vector of the new speaker is recorded, and the user is prompted to enter the new speaker's identity information or obtain the new speaker's identity information through the glasses' ID for binding and recording, after which the relevant information about the new speaker is displayed.

[0109] S204: Rendering and displaying the translation result based on the viewing angle information and image information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated.

[0110] In order to fix the position of the translation image and realize the continuous display of information when the viewing angle is out of sync, an immersive viewing experience is provided to users so that they can be more focused and obtain information more conveniently during the meeting. In the embodiment of the present application, the viewing angle information and image information detected by the smart glasses, as well as the azimuth information corresponding to the human voice signal to be translated, are used to render and display the translation results. That is, the smart glasses have a built-in camera to monitor the user's viewing angle in real time. When it is detected that the user's viewing angle is on the PPT or the speaker, the camera collects the corresponding image information in real time. The camera has high resolution and a wide-angle field of view, which can clearly capture the content of the PPT and the full view of the speaker. The collected image information is then input into the image recognition module, which uses a target detection and recognition algorithm based on deep learning to identify the text area on the PPT or the position of the speaker. In combination with the azimuth information corresponding to the human voice signal to be translated, the display position of the translation result is screened out, so that the translation result is superimposed on the corresponding display position with a suitable font, color and size.

[0111] In some embodiments, step S204 specifically includes the following steps: Figure 9 Steps S901-S904 shown:

[0112] S901: Initiate the collection of the image information based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated.

[0113] S902: Determine a display position of the translation result according to the image information and the viewing angle information.

[0114] S903: Adjust the current display parameters of the translation result according to the display parameters recorded by the XR rendering engine of the smart glasses and the current environment parameters corresponding to the image information.

[0115] S904: Render and display the translation result based on the display position and the current display parameters.

[0116] Due to the subjective initiative of the wearer of smart glasses, they may discuss with people around them during the meeting, that is, their eyes are not on the meeting speaker. In this case, it is necessary to initiate the collection of image information based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated. That is, the collection of image information will be initiated when the viewing angle information contains this orientation information. By triggering the camera to collect image information when the orientation of the human voice signal to be translated overlaps with the user's viewing angle, the problem of power waste caused by indiscriminate collection at all times is avoided, and the strong correlation between the image information and the current translation needs is ensured. Among them, the viewing angle information detected by the smart glasses in the above steps can be determined based on a common visual SLAM algorithm. Then, in order to fix the display position of the translation result and enhance the immersive viewing experience, the display position of the translation result will be determined based on the image information and the viewing angle information. For example, based on the viewing angle range of the viewing angle information, the edge area of ​​the screen is selected as the default display position, and then whether the default display position is blocked is determined based on the image information. The default display position is moved and adjusted to determine the display position of the translation result. To improve rendering quality and make the translation results clearer and easier to read, the smart glasses' XR rendering engine records the display parameters and the current environment parameters corresponding to the image information, and gradually adjusts the current display parameters of the translation results. The translation results are then rendered and displayed based on the display position and current display parameters.

[0117] In some embodiments, to improve accuracy, the viewing angle information detected by the smart glasses in the above steps can also be obtained based on the following methods: using the inertial measurement device (i.e., gyroscope and accelerometer) built into the smart glasses to acquire real-time head movement data of the smart glasses wearer, and determining the wearer's head orientation vector based on the head movement data; using the eye tracking camera built into the smart glasses to collect the wearer's eye movement data, and converting the two-dimensional pupil position of the eye movement data into a three-dimensional gaze direction vector to determine the wearer's eye gaze vector; wherein the conversion of the two-dimensional pupil position into the three-dimensional gaze direction vector can be obtained by projecting the two-dimensional pupil position into the three-dimensional gaze direction vector based on the optical parameters of the smart glasses, and the conversion process is not described in detail here. With the optical center of the smart glasses as the origin, a device coordinate system is established, which can be understood as having the x-axis pointing right, the y-axis pointing downward, and the z-axis pointing forward. The viewing angle direction of the smart glasses wearer is determined by combining the head orientation vector and the eye gaze vector. With the viewing angle direction vector as the center, the left and right and upper and lower boundary vectors of the viewing angle range are calculated based on the physical field of view of the smart glasses to determine the viewing angle range. The viewing angle information detected by the smart glasses can be determined based on the viewing angle direction and viewing angle range.

[0118] In some embodiments, as Figure 10 As shown, the above step S902 includes the following steps S1001-S1003:

[0119] S1001: Locating a corresponding physical space area based on the viewing angle information, and performing real-time semantic segmentation on the image information to obtain an idle area;

[0120] S1002: performing perspective transformation alignment on the perspective focus coordinates corresponding to the perspective information and the two-dimensional image coordinates of the idle area to determine an intersection area between the physical space area and the idle area;

[0121] S1003: Planning a display path of the translation result in the intersection area by using an A* algorithm, so as to determine a display position of the translation result based on the display path.

[0122] Real-time semantic segmentation of image information to obtain free areas can be achieved by training a lightweight neural network model, such as ENet, BiSeNet, or MobileNet-DeepLab, to segment the image in real time, thereby obtaining a semantic label map. Based on the annotation categories corresponding to each pixel in the semantic label map, the free areas without occlusions can be determined. Then, after perspective transformation and alignment of the viewpoint focus coordinates corresponding to the viewpoint information with the two-dimensional image coordinates of the free areas, spatial consistency is ensured. The intersection of the physical space area and the free areas is determined. This ensures that the translation results are displayed in the overlap between the user's gaze direction and the available space in the scene, which conforms to the user's visual habits and avoids the problem of unclear display of the translation results in non-free areas with occlusions. Furthermore, the A* algorithm is used to plan the display path of the translation results within the intersection area. The display position of the translation results is determined based on the display path. This further avoids dynamic objects identified by semantic segmentation and prevents the translation results from jumping between frames, ensuring a stable transition of display positions and a natural display of the translation results.

[0123] In the process of planning the display path of the translation result within the intersection area using the A* algorithm and determining the display position of the translation result based on the display path, in order to avoid the problems of path rigidity and lack of scene semantic understanding in traditional A* algorithm planning, in some embodiments, it can also be implemented based on the following methods:

[0124] Based on the variance of the pixel values ​​corresponding to the spatial regions within the intersection region, the intersection region is adaptively meshed. This means that a larger variance indicates a more vacant space within the region, such as a large wall area requiring coarse meshing, while a smaller variance indicates the presence of complex background features, such as a table or a human head, requiring fine meshing. A global path is quickly generated at the top level of the adaptive mesh based on the mesh after meshing. The labels corresponding to the semantic segmentation are used as constraints, and the global path is updated using the A* algorithm under these constraints to obtain a display path for the translation result within the intersection region. This path then determines the display location of the translation result based on the display path.

[0125] In some embodiments, in addition to the gradual adjustment, the above step S703 can also be implemented in the following ways, thereby improving clarity while reducing time costs and improving processing efficiency:

[0126] The ambient light intensity and color temperature are obtained through the ambient light sensor, and the brightness, contrast and color RGB value of the translation result are calibrated based on the white balance algorithm. Then, the dynamic obstructions in the direction of the user's line of sight are detected through the depth sensor, and the display position coordinates of the translation result are dynamically corrected according to the position and movement speed of the obstructions. For example, based on the depth map and RGB image generated by the depth sensor, the type of obstruction is identified through the target detection model. If the obstruction is a human body part or an important object, the displacement vector of the translation result is calculated to avoid covering the obstruction. Otherwise, the transparency of the translation result is adjusted to be negatively correlated with the transparency of the obstruction. Then, in response to detecting that the ambient light intensity exceeds a preset threshold, the background of the translation result is switched to an inverted color mode, and the font edge anti-aliasing level is increased; the user's preset display preference parameters, including font size, transparency and dynamic obstacle avoidance sensitivity, are loaded, and the final display parameters are output after weighted fusion with the real-time environmental parameters.

[0127] In some embodiments, if the translation scenario has corresponding linkage translation information, then Figure 11 As shown, the method further includes S1101-S1105:

[0128] S1101: extracting a human voice signal to be translated from the human voice signal in the translation scenario, and translating it into a first translation result in a target language;

[0129] S1102: Rendering and displaying the first translation result based on the viewing angle information and image information detected by the smart glasses, and the orientation information corresponding to the human voice signal to be translated;

[0130] S1103: Acquire the linked translation information to translate it into a second translation result in the target language;

[0131] S1104: Determining whether the linked translation information is within the user's line of sight based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the linked translation information;

[0132] S1105: If yes, then based on the duration of the linked translation information within the user's line of sight, start displaying the second translation result, so as to render and display the second translation result based on the viewing angle information and image information detected by the smart glasses.

[0133] The specific implementation process of the above-mentioned step S1101 corresponds to the process of obtaining the translated human voice signal in S401-S403, and the specific implementation process of the first translation result corresponds to the process of S701-S703. The specific implementation process of other related steps S1102 can also be implemented with reference to the corresponding methods in the above-mentioned embodiments, and the repeated content will not be repeated here.

[0134] This process obtains a second translation result, which translates the linked translation information into the target language, to address the problem that existing devices cannot effectively integrate image rendering and translation functions, failing to meet the diverse needs of users in cross-language conferences, making it difficult to simultaneously translate PowerPoint presentations or other visual information in presentations. Then, based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the linked translation information, it is determined whether the linked translation information is within the user's line of sight. This is because the wearer of the smart glasses may shift their gaze away from the area where the linked translation information is located due to factors such as communication with others and attention shifts. In this case, when the user's view is away from the PowerPoint presentation or the speaker, the translated information remains displayed at a fixed position until the user returns to the preset position, ensuring that even if the user briefly shifts their view during the meeting, they will not miss important translated information. The display of the second translation result is initiated based on the duration of the linked translation information within the user's line of sight. Specifically, if the second translation result has been previously displayed, the duration of the linked translation information within the user's line of sight is determined to initiate the display process for the second translation result. If the second translation result has been previously displayed and the time spent away from the linked translation information area, as determined based on the duration of the user's line of sight, is less than a preset time threshold, the second translation result continues to be displayed. When displaying the second translation result, it needs to be rendered based on the perspective information and image information. For example, the translation information on the PPT can be fixed next to or below the PPT text; the speaker's translation information can be fixed near the speaker. This makes it easier for users to access the translation information while viewing, enhancing the immersive viewing experience.

[0135] The present application also provides a smart glasses, such as Figure 12 As shown, the smart glasses include a processor 1201 , a memory 1202 , a shooting module 1203 , a display module 1204 , an interaction module 1205 and a wireless communication module 1206 .

[0136] The memory 1202 can be used to store computer programs, such as software programs and modules of application software. The processor 1201 executes various functional applications and data processing by running the computer programs stored in the memory 1202, such as implementing the method provided in the embodiment of the present application.

[0137] The shooting module 1203 is mainly responsible for shooting image information of the conference environment, and the shooting module can be a camera.

[0138] The display module 1204 is used to display the translation results obtained by the method provided in the embodiment of the present application.

[0139] The interaction module 1205 includes a microphone array and the like.

[0140] The wireless communication module 1206 includes: 4G / 5G full network access, WiFi, Bluetooth, etc.

[0141] It can be understood by those skilled in the art that Figure 12 The structure shown is only for illustration and does not limit the structure of smart glasses. Figure 12 More or fewer components than shown, or with Figure 12 Different configurations shown.

[0142] The following is a brief introduction to the application scenarios of smart glasses.

[0143] Wearers of smart glasses can participate in cross-language meetings. During meetings, the smart glasses distinguish between the speaker's voice and natural sounds, remove ambient noise, and intelligently extract the speaker's voice. After intelligently extracting the voice, the glasses automatically identify the speaker's language, perform voiceprint analysis, and record identity binding. If the speaker's voice differs from the user's system language, the glasses automatically translate it into the user's system language or a specified target language. Voiceprint analysis is performed during the detection process, and each speaker's voiceprint characteristics are recorded for subsequent identity recognition. If the speaker changes during the meeting, the smart glasses will display the other speaker's name, profile picture, and other information based on the voiceprint recognition and identity binding records. During the recognition and translation process, conversations between the wearer and nearby speakers can be translated into the system language or target language if the languages ​​differ; if the languages ​​are the same, the translation is filtered out. When the smart glasses detect that the user's view is focused on a presentation or speaker, they render the image in real time. Using built-in image recognition technology, the rendered information and position are determined and the translated image is fixed in place, enhancing the immersive viewing experience. If the user's view is shifted away from the presentation or speaker, the translated information remains displayed in the fixed position until the user returns to the preset view. This way, users will not miss important translation information even if they shift their gaze briefly during the meeting.

[0144] like Figure 13 As shown in FIG, the embodiment of the present application provides a schematic diagram of the structure of a cross-language conference translation device. Figure 13 It can be seen that in one or more embodiments of the present application, a cross-language conference translation device includes:

[0145] An extraction module 1301 is configured to extract a human voice signal based on the sound signal features collected by the smart glasses, so as to obtain a translation scenario corresponding to the smart glasses based on the human voice signal;

[0146] Determination module 1302, used to determine whether the translation scenario has corresponding linkage translation information;

[0147] The translation module 1303 is configured to extract the human voice signal to be translated from the human voice signal in the translation scenario when there is no corresponding linkage translation information, so as to translate the human voice signal into a translation result in the target language;

[0148] The display module 1304 is configured to render and display the translation result based on the viewing angle information and image information detected by the smart glasses and the position information corresponding to the human voice signal to be translated.

[0149] In some embodiments, extracting a human voice signal based on the sound signal features collected by the smart glasses specifically includes:

[0150] Collecting sound signals based on preset microphone arrays distributed at different positions of the smart glasses;

[0151] Acquiring sound signal features of the sound signal; wherein the sound signal features include: spatial features and spectral features;

[0152] The sound signal features are classified using a pre-trained classifier, and a human voice signal is extracted from the sound signal according to the classification result.

[0153] In some embodiments, obtaining a translation scenario corresponding to the smart glasses according to the human voice signal specifically includes:

[0154] Extracting voiceprint features and time series features corresponding to the human voice signal;

[0155] Obtaining a voiceprint feature vector corresponding to the voiceprint feature, and comparing the voiceprint feature vector with a voiceprint feature vector of a user of the smart glasses;

[0156] According to the comparison results and the human voice interaction features corresponding to the user voiceprint feature vector determined based on the time series features, the translation scenario corresponding to the smart glasses is determined; wherein the translation scenarios include: dialogue interaction scenarios and speech scenarios.

[0157] In some embodiments, determining whether the translation scenario has corresponding linkage translation information specifically includes:

[0158] If it is determined that the translation scenario is a dialogue interaction scenario, determining whether the interlocutor voiceprint feature vector corresponding to the user voiceprint feature vector belongs to the current speaker;

[0159] If so, determining whether there is corresponding linkage translation information based on the venue information of the current meeting;

[0160] If it is determined that the translation scene is a speech scene, it is determined whether there is corresponding linkage translation information based on the venue information of the current meeting.

[0161] In some embodiments, extracting the to-be-translated human voice signal from the human voice signal in the translation scenario to translate it into a translation result in the target language specifically includes:

[0162] Identifying the language type of the human voice signal using a pre-trained language recognition model, and comparing the language type with the system language type;

[0163] Based on the comparison result and the voiceprint feature vector contained in the translation scenario, screening the human voice signal to be translated;

[0164] The human voice signal to be translated is translated based on a pre-trained translation model to obtain a text translation result in the target language.

[0165] In some embodiments, after extracting the human voice signal to be translated from the human voice signal in the translation scenario, the method further includes:

[0166] Extracting voiceprint feature parameters corresponding to the human voice signal to be translated to determine a voiceprint feature vector corresponding to the voiceprint feature parameters;

[0167] Binding the voiceprint feature vector to the identity information of the current speaker and storing the result in a storage unit corresponding to the smart glasses;

[0168] The voiceprint feature vector is compared with the real-time voiceprint feature vector to obtain the identity information of the new speaker for binding record.

[0169] In some embodiments, rendering and displaying the translation result based on the viewing angle information and image information detected by the smart glasses and the position information corresponding to the human voice signal to be translated specifically includes:

[0170] Initiate acquisition of the image information based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated;

[0171] determining a display position of the translation result according to the image information and the viewing angle information;

[0172] Adjusting the current display parameters of the translation result by using the display parameters recorded by the XR rendering engine of the smart glasses and the current environment parameters corresponding to the image information;

[0173] The translation result is rendered and displayed based on the display position and the current display parameters.

[0174] In some embodiments, determining a display position of the translation result based on the image information and the viewing angle information specifically includes:

[0175] Locating a corresponding physical space area based on the viewing angle information, and performing real-time semantic segmentation on the image information to obtain an idle area;

[0176] Performing perspective transformation alignment based on the perspective focus coordinates corresponding to the perspective information and the two-dimensional image coordinates of the idle area to determine an intersection area between the physical space area and the idle area;

[0177] A display path of the translation result in the intersection area is planned by using an A* algorithm, so as to determine a display position of the translation result based on the display path.

[0178] In some embodiments, if the translation scenario has corresponding linkage translation information, the method further includes:

[0179] extracting a human voice signal to be translated from the human voice signal in the translation scenario, and translating it into a first translation result in a target language;

[0180] Rendering and displaying the first translation result based on the viewing angle information and image information detected by the smart glasses, and the orientation information corresponding to the human voice signal to be translated;

[0181] Obtaining the linked translation information to translate it into a second translation result in the target language;

[0182] determining whether the linked translation information is within the user's line of sight based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the linked translation information;

[0183] If so, based on the duration of the linked translation information within the user's line of sight, the display of the second translation result is started, so as to render and display the second translation result based on the viewing angle information and image information detected by the smart glasses.

[0184] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0185] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0186] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0187] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A cross-language conference translation method, characterized in that: Applied to smart glasses, the method includes: Extracting a human voice signal based on the sound signal features collected by the smart glasses, so as to obtain a translation scenario corresponding to the smart glasses according to the human voice signal; Determining whether the translation scenario has corresponding linkage translation information; If not, extracting the human voice signal to be translated from the human voice signal in the translation scenario to translate it into a translation result in the target language; Rendering and displaying the translation result based on the viewing angle information and image information detected by the smart glasses, as well as the orientation information corresponding to the human voice signal to be translated; Determining whether the translation scenario has corresponding linkage translation information specifically includes: If it is determined that the translation scenario is a dialogue interaction scenario, determining whether the interlocutor voiceprint feature vector corresponding to the user voiceprint feature vector belongs to the current speaker; wherein the user voiceprint feature vector is obtained by extracting voiceprint features of a human voice signal, and the interlocutor voiceprint feature vector is obtained by extracting human voice interaction features corresponding to the user voiceprint feature vector; If so, determining whether there is corresponding linkage translation information based on the venue information of the current meeting; If it is determined that the translation scene is a speech scene, it is determined whether there is corresponding linkage translation information based on the venue information of the current meeting.

2. A cross-language conference translation method according to claim 1, characterized in that: Extracting a human voice signal based on the sound signal features collected by the smart glasses specifically includes: Collecting sound signals based on preset microphone arrays distributed at different positions of the smart glasses; Acquiring sound signal features of the sound signal; wherein the sound signal features include: spatial features and spectral features; The sound signal features are classified using a pre-trained classifier, and a human voice signal is extracted from the sound signal according to the classification result.

3. A cross-language conference translation method according to claim 1, characterized in that: Obtaining a translation scenario corresponding to the smart glasses according to the human voice signal, specifically including: Extracting voiceprint features and time series features corresponding to the human voice signal; Obtaining a voiceprint feature vector corresponding to the voiceprint feature, and comparing the voiceprint feature vector with a voiceprint feature vector of a user of the smart glasses; According to the comparison results and the human voice interaction features corresponding to the user voiceprint feature vector determined based on the time series features, the translation scenario corresponding to the smart glasses is determined; wherein the translation scenarios include: dialogue interaction scenarios and speech scenarios.

4. The cross-language conference translation method according to claim 1, characterized in that: Extracting the to-be-translated human voice signal from the human voice signal in the translation scenario to translate it into a translation result in the target language specifically includes: Identifying the language type of the human voice signal using a pre-trained language recognition model, and comparing the language type with the system language type; Based on the comparison result and the voiceprint feature vector contained in the translation scenario, screening the human voice signal to be translated; The human voice signal to be translated is translated based on a pre-trained translation model to obtain a text translation result in the target language.

5. The cross-language conference translation method according to claim 1, characterized in that: After extracting the to-be-translated human voice signal from the human voice signal in the translation scenario, the method further includes: Extracting voiceprint feature parameters corresponding to the human voice signal to be translated to determine a voiceprint feature vector corresponding to the voiceprint feature parameters; Binding the voiceprint feature vector to the identity information of the current speaker and storing the result in a storage unit corresponding to the smart glasses; The voiceprint feature vector is compared with the real-time voiceprint feature vector to obtain the identity information of the new speaker for binding record.

6. The cross-language conference translation method according to claim 1, characterized in that: Rendering and displaying the translation result based on the viewing angle information and image information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated specifically includes: Initiate acquisition of the image information based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the human voice signal to be translated; determining a display position of the translation result according to the image information and the viewing angle information; Adjusting the current display parameters of the translation result by using the display parameters recorded by the XR rendering engine of the smart glasses and the current environment parameters corresponding to the image information; The translation result is rendered and displayed based on the display position and the current display parameters.

7. A cross-language conference translation method according to claim 6, characterized in that: Determining a display position of the translation result according to the image information and the viewing angle information specifically includes: Locating a corresponding physical space area based on the viewing angle information, and performing real-time semantic segmentation on the image information to obtain an idle area; Performing perspective transformation alignment based on the perspective focus coordinates corresponding to the perspective information and the two-dimensional image coordinates of the idle area to determine an intersection area between the physical space area and the idle area; A display path of the translation result in the intersection area is planned by using an A* algorithm, so as to determine a display position of the translation result based on the display path.

8. The cross-language conference translation method according to claim 1, characterized in that: If the translation scenario has corresponding linkage translation information, the method further includes: extracting a human voice signal to be translated from the human voice signal in the translation scenario, and translating it into a first translation result in a target language; Rendering and displaying the first translation result based on the viewing angle information and image information detected by the smart glasses, and the orientation information corresponding to the human voice signal to be translated; Obtaining the linked translation information to translate it into a second translation result in the target language; determining whether the linked translation information is within the user's line of sight based on the viewing angle information detected by the smart glasses and the orientation information corresponding to the linked translation information; If so, based on the duration of the linked translation information within the user's line of sight, the display of the second translation result is started, so as to render and display the second translation result based on the viewing angle information and image information detected by the smart glasses.

9. A pair of smart glasses, characterized in that: The invention comprises at least one processor, a memory, a camera module, a microphone module and a display module. The at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the cross-language conference translation method according to any one of claims 1 to 8.

Images