Translation processing method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a translation processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a sound source feature of a current scene, and determining target pickup azimuth information according to the sound source feature; acquiring a target audio in the current scene according to the target pickup azimuth information; and obtaining scene features of the current scene, and translating the target audio according to the scene features to obtain a target translation result. Directional pickup is achieved, collection of invalid noise is reduced, and the accuracy and effectiveness of collected audios are improved. The translation process is optimized, and the accuracy of the target translation result is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a translation processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Extended Reality (XR) is a comprehensive term that covers virtual reality (VR), augmented reality (AR) and mixed reality (MR). XR technology expands the way humans perceive and interact, interacting with the real world through digital technology to create virtual, augmented and mixed reality experiences. XR devices are widely used in scenarios such as navigation, information display and augmented reality experience.

[0003] With the development of extended reality technology or extended reality devices, the function of language translation using extended reality devices has been realized. Through the extended reality devices, audio can be collected in real time for language translation.

[0004] However, the audio collected during translation is currently susceptible to interference. For example, when there is environmental background noise or multiple people speak at the same time, the collected audio will contain noise and cannot be effectively translated. The accuracy of the translation results cannot be guaranteed. In addition, current translation is mainly based on the collected audio, resulting in the accuracy of the translation results needs to be improved. Summary of the invention

[0005] The embodiments of the present application provide a translation processing method, device, electronic device and computer-readable storage medium, which can improve the accuracy of translation results.

[0006] In a first aspect, an embodiment of the present application provides a translation processing method, which is applied to an extended reality device, and the method includes:

[0007] Acquire the sound source characteristics of the current scene, and determine the target sound pickup direction information according to the sound source characteristics;

[0008] Collecting target audio in the current scene according to the target sound pickup orientation information;

[0009] The scene features of the current scene are obtained, and the target audio is translated according to the scene features to obtain a target translation result.

[0010] In a second aspect, an embodiment of the present application further provides a translation processing device, which is applied to an extended reality device, and the device includes:

[0011] An acquisition module is used to acquire the sound source characteristics of the current scene and determine the target sound pickup direction information according to the sound source characteristics;

[0012] A collection module, used to collect target audio in the current scene according to the target sound pickup orientation information;

[0013] The translation module is used to obtain the scene features of the current scene and translate the target audio according to the scene features to obtain a target translation result.

[0014] Optionally, in some embodiments of the present application, the acquiring the sound source features of the current scene includes:

[0015] Acquire audio information of the current scene, and perform feature extraction on the audio information to obtain sound source features;

[0016] The sound source feature includes at least one of audio source orientation information, audio language type information, audio duration information or audio level information.

[0017] Optionally, in some embodiments of the present application, the sound source feature further includes lip-sync audio state information, and the step of acquiring the sound source feature of the current scene includes:

[0018] Get the audio and image information of the current scene;

[0019] Determine lip shape information of at least one user in the current scene according to the audio information;

[0020] Matching processing is performed on each of the lip shape information and the audio information to determine lip shape audio synchronization state information.

[0021] Optionally, in some embodiments of the present application, translating the target audio according to the scene feature to obtain a target translation result includes:

[0022] Convert the target audio into text format to obtain original text information;

[0023] Performing semantic understanding processing on the original text information, determining current topic information, and determining text information to be filtered that is not related to the current topic information in the original text information;

[0024] Filtering the to-be-filtered text information from the original text information to obtain target text information;

[0025] The target text information is translated according to the scene features to obtain a target translation result.

[0026] Optionally, in some embodiments of the present application, the acquiring scene features of the current scene includes:

[0027] Acquire image information of the current scene, and perform feature extraction on the image information to obtain scene features;

[0028] The scene features include at least one of facial expression features, lip shape features, body movement features or scene visual features.

[0029] Optionally, in some embodiments of the present application, the scene feature includes a body movement feature, and the translating the target text information according to the scene feature to obtain a target translation result includes:

[0030] Determining a target object to be translated according to the body movement characteristics;

[0031] A target translation result for the target object is generated according to the target text information.

[0032] Optionally, in some embodiments of the present application, after acquiring the scene features of the current scene and translating the target audio according to the scene features to obtain a target translation result, the method further includes:

[0033] Determining target display position information from a virtual display screen of the extended reality device according to the target sound pickup orientation information;

[0034] The target translation result is displayed on the virtual display screen according to the target display position information.

[0035] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps in the above-mentioned translation processing method are implemented when the computer program is executed by the processor.

[0036] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned translation processing method are implemented.

[0037] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0038] The extended reality device of the embodiment of the present application obtains the sound source characteristics of the current scene, determines the target sound pickup orientation information based on the sound source characteristics, collects the target audio in the current scene based on the target sound pickup orientation information, obtains the scene characteristics of the current scene, and translates the target audio according to the scene characteristics to obtain the target translation result.

[0039] Among them, by determining the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collecting the target audio based on the target sound pickup orientation information, directional audio collection, i.e. directional sound pickup, is achieved, the collection of invalid noise is reduced, and the accuracy and effectiveness of the collected audio are improved.

[0040] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a schematic diagram of a scenario in which an extended reality device provided in an embodiment of the present application executes the translation processing method;

[0043] Figure 2 It is a flowchart of the translation processing method provided in the embodiment of the present application;

[0044] Figure 3 is a structural diagram of a translation processing device provided in an embodiment of the present application;

[0045] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] The embodiments of the present application provide a translation processing method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide a translation processing device suitable for electronic devices, which is used to improve the accuracy of translation results. Specifically, the electronic device at least includes a terminal device with sound pickup, text processing and translation capabilities, and the terminal device includes an extended reality device, which includes but is not limited to a head-mounted display, wearable glasses, and the extended reality device can also be an integrated extended reality device built into the computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, cars, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, an extended reality application (such as extended reality games, virtual tourism, telemedicine or virtual experiments, etc.) applied to handheld mobile devices such as mobile phones, laptops, and tablets, and a near-eye display system for wearable devices such as head-mounted displays and smart glasses.

[0048] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the translation processing method, wherein the specific execution process of the extended reality device performing the translation processing method is as follows:

[0049] The extended reality device 10 obtains the sound source characteristics of the current scene, determines the target sound pickup orientation information based on the sound source characteristics, collects the target audio in the current scene based on the target sound pickup orientation information, obtains the scene characteristics of the current scene, and translates the target audio based on the scene characteristics to obtain a target translation result.

[0050] In summary, the embodiments of the present application determine the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collect the target audio based on the target sound pickup orientation information, thereby realizing directional audio collection, i.e., directional sound pickup, reducing the collection of invalid noise, and improving the accuracy and effectiveness of the collected audio.

[0051] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results.

[0052] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0053] See also Figure 2 , Figure 2 A flowchart of a translation processing method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the flowchart. Specifically, the execution subject of the translation processing method is an extended reality device. Specifically, the process of the extended reality device executing the translation processing method specifically includes:

[0054] 101. Acquire sound source features of a current scene, and determine target sound pickup azimuth information according to the sound source features.

[0055] The current scene refers to the scene in which the extended reality device is currently used, and the current scene includes but is not limited to a meeting scene, a conversation scene, or an offline shopping scene.

[0056] The sound source feature refers to the feature of the sound or audio in the current scene. For example, the sound source feature includes but is not limited to timbre, volume, direction of sound source, language type, etc.

[0057] Among them, by determining the target sound pickup direction information based on the sound source characteristics, the specific sound source can be located, which is helpful for subsequent directional sound pickup based on the sound source direction.

[0058] 102. Collect target audio in the current scene according to the target sound pickup orientation information.

[0059] Wherein, directional sound pickup is achieved by collecting audio in a specific direction based on target sound pickup direction information.

[0060] For example, multiple high-sensitivity microphones are arranged in a specific array and controlled based on target sound pickup orientation information to collect sound sources in a specific orientation.

[0061] Among them, directional sound pickup is used to reduce the collection of noise in the environment and improve the accuracy of the collected target audio.

[0062] 103. Acquire scene features of the current scene, and translate the target audio according to the scene features to obtain a target translation result.

[0063] It can be understood that the scene feature is the feature of the current scene, which is information that describes the characteristics of the current scene. For example, the scene feature includes the real environment in which the extended reality device is used, as well as information such as the characters, actions, and expressions in the real environment.

[0064] Among them, the translation of the target audio is optimized by combining the scene features of the current scene, thereby improving the accuracy of the target translation result obtained by the translation.

[0065] In summary, the embodiments of the present application determine the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collect the target audio based on the target sound pickup orientation information, thereby realizing directional audio collection, i.e., directional sound pickup, reducing the collection of invalid noise, and improving the accuracy and effectiveness of the collected audio.

[0066] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results.

[0067] It can be understood that the sound source feature can be obtained by extracting features from audio information extracted from the environment. For example, the audio in the environment is collected and features are extracted from the audio to obtain the sound source feature. That is, optionally, in some embodiments of the present application, the step of "obtaining the sound source feature of the current scene" includes:

[0068] Acquire audio information of the current scene, and perform feature extraction on the audio information to obtain sound source features;

[0069] The sound source feature includes at least one of audio source orientation information, audio language type information, audio duration information or audio level information.

[0070] The audio information is all the audio that can be collected from the current scene, and the target audio is specific audio extracted from a specific position of the current scene based on the target sound pickup position information.

[0071] In the embodiments of the present application, the sound source characteristics of the audio information can be extracted by waveform analysis, Fourier transform, cepstral coefficients or deep learning.

[0072] The audio source location information includes the location of the currently collected audio information. For example, if there are multiple speakers in a real scene, the collected audio information includes multiple audio source location information.

[0073] The audio language type information refers to the language type of the audio information. For example, the audio language type information includes Chinese, English or Russian.

[0074] The audio duration information refers to the duration of the audio information, that is, the duration of the collected audio information, which reflects how long the speaker corresponding to the audio segment spoke.

[0075] The audio level information refers to the level of audio information, reflecting whether it is the main audio in the scene, for example, the audio of a certain speaker in the scene is mainly the audio, or the relatively clear audio collected in front of the user is mainly the audio.

[0076] Among them, by determining the target sound pickup orientation information based on the sound source characteristics and collecting the target audio based on the target sound pickup orientation information, the accuracy of the collected target audio is improved. For example, whether to collect is required based on the language type (the purpose is translation, and the same language type does not need to be collected), and whether to collect is required based on the audio source orientation information (for example, if it is not in front of the user in the conversation or the speaker is far away from the current user based on the orientation, then collection is not required, etc.), wherein the audio duration and audio level information can determine whether the audio information is the audio of the main speaker who is continuously speaking in the scene, etc. For example, taking a specific scene as an example, collect the target audio in front of the user and that is continuously speaking and whose language type is different from the user's native language.

[0077] It is understandable that when multiple people are speaking at the same time in a scene, in order to improve the accuracy or effectiveness of the collected target audio, the audio can also be screened in combination with the speaker's lip shape, and the target audio that is consistent with the lip shape of the speaker that the user is concerned about is selected. That is, optionally, in some embodiments of the present application, the sound source feature also includes lip shape audio synchronization state information, and the step of "obtaining the sound source feature of the current scene" includes:

[0078] Get the audio and image information of the current scene;

[0079] Determine lip shape information of at least one user in the current scene according to the audio information;

[0080] Matching processing is performed on each of the lip shape information and the audio information to determine lip shape audio synchronization state information.

[0081] The lip shape audio synchronization state information refers to whether the lip shape is synchronized with the audio, including synchronization or non-synchronization state, etc. For example, for a speaker that the user is paying attention to (such as a speaker in front of the user's field of vision), among the multiple audio information collected, each audio information has a synchronization state with the speaker's lip shape, and generally, at least one audio information is consistent with the speaker's lip shape.

[0082] Among them, the lip shape information can be analyzed based on the scene picture of the current scene. For example, the image information of the current scene is collected, and the lip shape information of the speaker in the current scene is extracted by analyzing the image information. The lip shape information includes open mouth, closed mouth, continuous open mouth or lip shape change information.

[0083] Among them, by comparing the lip shape and the audio, the lip shape audio synchronization status information can be analyzed.

[0084] Optionally, the target audio may be converted into text, and the text may be translated and then displayed. That is, optionally, in some embodiments of the present application, the step of “translating the target audio according to the scene feature to obtain a target translation result” includes:

[0085] Convert the target audio into text format to obtain original text information;

[0086] Performing semantic understanding processing on the original text information, determining current topic information, and determining text information to be filtered that is not related to the current topic information in the original text information;

[0087] Filtering the to-be-filtered text information from the original text information to obtain target text information;

[0088] The target text information is translated according to the scene features to obtain a target translation result.

[0089] The current topic information refers to the topic information corresponding to the target audio, for example, the subject content of the current conversation, etc., wherein the accuracy of the target translation result is improved by filtering the to-be-filtered text information irrelevant to the topic based on the current topic information. In the embodiment of the present application, the original text information can be semantically understood and processed based on methods such as deep learning or machine learning, and the to-be-filtered text information irrelevant to the current topic information can be extracted from the original text information.

[0090] In the embodiments of the present application, the image information of the current scene may be collected, and the current scene features may be analyzed based on the image information. That is, optionally, in some embodiments of the present application, the step of “obtaining the scene features of the current scene” includes:

[0091] Acquire image information of the current scene, and perform feature extraction on the image information to obtain scene features;

[0092] The scene features include at least one of facial expression features, lip shape features, body movement features or scene visual features.

[0093] The expression feature refers to the speaker's facial expression, including but not limited to happiness, sadness, anger or fear.

[0094] Among them, the lip shape feature, i.e. the aforementioned lip shape information, indicates the shape of the speaker's mouth.

[0095] Among them, body movements refer to body movements of the speaker or related to the speaker's speech content, such as hand pointing movements.

[0096] Among them, scene visual features refer to the real environment features of the scene, such as the room, road, building background, etc. where the two parties in the current conversation are located.

[0097] Among them, the target audio is translated by combining scene features to improve the accuracy of the target translation result obtained by translation. For example, the current speaking context can be understood through scene features, and the accuracy of the translation result can be improved based on the context.

[0098] Among them, in the embodiments of the present application, the object of the conversation can also be clarified based on the body movement features, and the translation is performed in combination with the object, which can improve the accuracy of the translation result. That is, optionally, in some embodiments of the present application, the scene features include body movement features, and the step of "translating the target text information according to the scene features to obtain the target translation result" includes:

[0099] Determining a target object to be translated according to the body movement characteristics;

[0100] A target translation result for the target object is generated according to the target text information.

[0101] For example, taking the target object as an apple, if the speaker points his finger at the apple and says "What's this?", the translation result is not "What is this" but "What is an apple?".

[0102] Accordingly, in the embodiment of the present application, after the target translation result is obtained by translation, the display of the target translation result can be controlled based on the target sound pickup orientation information, that is, optionally, in some embodiments of the present application, after the step of "obtaining the scene features of the current scene, and translating the target audio according to the scene features to obtain the target translation result", the method further includes:

[0103] Determining target display position information from a virtual display screen of the extended reality device according to the target sound pickup orientation information;

[0104] The target translation result is displayed on the virtual display screen according to the target display position information.

[0105] The virtual display screen is a virtual screen provided by the extended reality device. Usually, the virtual display screen is combined with the real scene screen to achieve the effect of virtual reality or augmented reality. By directly displaying the target translation result in the virtual display screen of the extended reality device, the user can browse the translation result as soon as he wears the extended reality device, without the need to translate through other tools such as mobile phones, or to know the translation result through voice broadcast.

[0106] Among them, the target display position information is determined based on the target sound pickup orientation information, and the display of the target translation result is controlled based on the target display position information, so that the display of the translation result matches the sound source orientation of the target audio. For example, if the sound source direction of the target audio is on the right, the target translation result is also displayed on the right side of the virtual display screen.

[0107] Among them, by determining the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collecting the target audio based on the target sound pickup orientation information, directional audio collection, i.e. directional sound pickup, is achieved, the collection of invalid noise is reduced, and the accuracy and effectiveness of the collected audio are improved.

[0108] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results.

[0109] In order to better implement the translation processing method of the present application, the present application also provides a translation processing device based on the above translation processing method. The meanings of the nouns are the same as those in the above translation processing method, and the specific implementation details can refer to the description in the method embodiment.

[0110] See also Figure 3 , Figure 3 : is a structural diagram of a translation processing device provided in an embodiment of the present application. The translation processing device is applied to an extended reality device. The translation processing device may be specifically as follows:

[0111] An acquisition module 201 is used to acquire the sound source characteristics of the current scene and determine the target sound pickup direction information according to the sound source characteristics;

[0112] A collection module 202 is used to collect target audio in the current scene according to the target sound pickup orientation information;

[0113] The translation module 203 is used to obtain the scene features of the current scene, and translate the target audio according to the scene features to obtain a target translation result.

[0114] Optionally, in some embodiments of the present application, the acquiring the sound source features of the current scene includes:

[0115] Acquire audio information of the current scene, and perform feature extraction on the audio information to obtain sound source features;

[0116] The sound source feature includes at least one of audio source orientation information, audio language type information, audio duration information or audio level information.

[0117] Optionally, in some embodiments of the present application, the sound source feature further includes lip-sync audio state information, and the step of acquiring the sound source feature of the current scene includes:

[0118] Get the audio and image information of the current scene;

[0119] Determine lip shape information of at least one user in the current scene according to the audio information;

[0120] Matching processing is performed on each of the lip shape information and the audio information to determine lip shape audio synchronization state information.

[0121] Optionally, in some embodiments of the present application, translating the target audio according to the scene feature to obtain a target translation result includes:

[0122] Convert the target audio into text format to obtain original text information;

[0123] Performing semantic understanding processing on the original text information, determining current topic information, and determining text information to be filtered that is not related to the current topic information in the original text information;

[0124] Filtering the to-be-filtered text information from the original text information to obtain target text information;

[0125] The target text information is translated according to the scene features to obtain a target translation result.

[0126] Optionally, in some embodiments of the present application, the acquiring scene features of the current scene includes:

[0127] Acquire image information of the current scene, and perform feature extraction on the image information to obtain scene features;

[0128] The scene features include at least one of facial expression features, lip shape features, body movement features or scene visual features.

[0129] Optionally, in some embodiments of the present application, the scene feature includes a body movement feature, and the translating the target text information according to the scene feature to obtain a target translation result includes:

[0130] Determining a target object to be translated according to the body movement characteristics;

[0131] A target translation result for the target object is generated according to the target text information.

[0132] Optionally, in some embodiments of the present application, after acquiring the scene features of the current scene and translating the target audio according to the scene features to obtain a target translation result, the method further includes:

[0133] Determining target display position information from a virtual display screen of the extended reality device according to the target sound pickup orientation information;

[0134] The target translation result is displayed on the virtual display screen according to the target display position information.

[0135] In the embodiment of the present application, the acquisition module 201 acquires the sound source characteristics of the current scene, and determines the target sound pickup orientation information according to the sound source characteristics. The acquisition module 202 collects the target audio in the current scene according to the target sound pickup orientation information. The translation module 203 acquires the scene characteristics of the current scene, and translates the target audio according to the scene characteristics to obtain a target translation result.

[0136] Among them, the embodiment of the present application determines the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collects the target audio based on the target sound pickup orientation information, thereby realizing directional audio collection, i.e. directional sound pickup, reducing the collection of invalid noise, and improving the accuracy and effectiveness of the collected audio.

[0137] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results.

[0138] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0139] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0140] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0141] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and translation processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0142] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0143] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0144] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in any one of the translation processing methods provided in the embodiments of the present application.

[0145] Among them, in the embodiment of the present application, the electronic device includes an extended reality device, and the extended reality device includes smart glasses in the form of natural glasses, and the smart glasses have at least the functions of extended reality display, wearing state detection, biometric recognition, human-computer interaction, data processing, etc. It includes a frame, temples, processors, sensors, optical display components, microphones, speakers, etc. There are cavities inside the temples and frames, and circuits and electronic components are placed in the cavities. Sensors include cameras, eye trackers, iris meters, IMUs, gyroscopes, etc. The optical display components include micro-projection optical machines and optical couplers. The micro-projection optical machines can be based on Micro-Oled, Micro-Led, LCOS, or LBS. The optical coupler can be an optical lens or an optical waveguide. The processor can be an XR professional processor or an ordinary general-purpose processor. The frame and temples are the supporting structure of the entire glasses. The temples have a certain elasticity, and the length and clamping force can be adjusted to suit the wearing of users with different head shapes; the camera can take pictures of the user's hands, face, eyes, etc., the microphone can listen to the user's voice, and the processor can calculate and process various data.

[0146] The extended reality device of the embodiment of the present application obtains the sound source characteristics of the current scene, determines the target sound pickup orientation information based on the sound source characteristics, collects the target audio in the current scene based on the target sound pickup orientation information, obtains the scene characteristics of the current scene, and translates the target audio according to the scene characteristics to obtain the target translation result.

[0147] Among them, by determining the target sound pickup orientation information of the collected audio based on the sound source characteristics, and collecting the target audio based on the target sound pickup orientation information, directional audio collection, i.e. directional sound pickup, is achieved, the collection of invalid noise is reduced, and the accuracy and effectiveness of the collected audio are improved.

[0148] Among them, by translating the target audio based on the scene features of the current scene, the translation process can be combined with the scene features. Compared with the traditional solution that only translates the collected audio, the embodiment of the present application combines other reference features other than audio for translation, thereby optimizing the translation process and improving the accuracy of the target translation results.

[0149] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0150] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0151] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any translation processing method provided in the present application.

[0152] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0153] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0154] Since the instructions stored in the computer-readable storage medium can execute the steps in any translation processing method provided in the present application, the beneficial effects that can be achieved by any translation processing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0155] The above is a detailed introduction to a translation processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A translation processing method, characterized in that: Applied to an extended reality device, the method comprises: Acquire the sound source characteristics of the current scene, and determine the target sound pickup direction information according to the sound source characteristics; Collecting target audio in the current scene according to the target sound pickup orientation information; The scene features of the current scene are obtained, and the target audio is translated according to the scene features to obtain a target translation result.

2. The translation processing method according to claim 1, characterized in that: The obtaining of the sound source features of the current scene includes: Acquire audio information of the current scene, and perform feature extraction on the audio information to obtain sound source features; The sound source feature includes at least one of audio source orientation information, audio language type information, audio duration information or audio level information.

3. The translation processing method according to claim 1, characterized in that: The sound source feature also includes lip-sync audio state information. The step of obtaining the sound source feature of the current scene includes: Get the audio and image information of the current scene; Determine lip shape information of at least one user in the current scene according to the audio information; Matching processing is performed on each of the lip shape information and the audio information to determine lip shape audio synchronization state information.

4. The translation processing method according to claim 1, characterized in that: The translating the target audio according to the scene feature to obtain a target translation result includes: Convert the target audio into text format to obtain original text information; Performing semantic understanding processing on the original text information, determining current topic information, and determining text information to be filtered that is not related to the current topic information in the original text information; Filtering the to-be-filtered text information from the original text information to obtain target text information; The target text information is translated according to the scene features to obtain a target translation result.

5. The translation processing method according to claim 4, characterized in that: The acquiring of scene features of the current scene includes: Acquire image information of the current scene, and perform feature extraction on the image information to obtain scene features; The scene features include at least one of facial expression features, lip shape features, body movement features or scene visual features.

6. The translation processing method according to claim 4, characterized in that: The scene feature includes a body movement feature, and the target text information is translated according to the scene feature to obtain a target translation result, including: Determining a target object to be translated according to the body movement characteristics; A target translation result for the target object is generated according to the target text information.

7. The translation processing method according to claim 1, characterized in that: After acquiring the scene features of the current scene and translating the target audio according to the scene features to obtain a target translation result, the method further includes: Determining target display position information from a virtual display screen of the extended reality device according to the target sound pickup orientation information; The target translation result is displayed on the virtual display screen according to the target display position information.

8. A translation processing device, characterized in that: Applied to an extended reality device, the device comprises: An acquisition module is used to acquire the sound source characteristics of the current scene and determine the target sound pickup direction information according to the sound source characteristics; A collection module, used to collect target audio in the current scene according to the target sound pickup orientation information; The translation module is used to obtain the scene features of the current scene and translate the target audio according to the scene features to obtain a target translation result.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the translation processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the translation processing method according to any one of claims 1 to 7 are implemented.

Images