Audio processing method and device, electronic equipment and readable medium

Abstract

The invention relates to an audio processing method and device, electronic equipment and a readable medium, and the method comprises the steps: obtaining a full-scene image corresponding to a session scene where a current user is located; constructing a geodetic coordinate system by taking AR glasses worn by the current user as an original point, and calibrating the full-scene image in the geodetic coordinate system to obtain a coordinate model; when it is detected that a speaker speaks in the session scene, determining a sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker; and determining a voice output parameter according to the audio information and the sound source position, and outputting the speaking voice of the speaker to the current user according to the voice output parameter so as to simulate playing of the speaking voice from the sound source position to the current user. The problem that the sound emitted by the noise reduction earphone cannot enable the user to locate the real orientation of the speaker in the AR scene is solved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an audio processing method, device, electronic device and readable medium. Background Art

[0002] In recent years, AR (Augmented Reality) technology has made great progress. AR technology uses computers to generate realistic virtual information of vision, hearing, force, touch and movement, and displays virtual information in the real world. AR glasses based on AR technology allow the wearer to see not only real-world objects, but also the generated virtual objects, as if they exist in the real world.

[0003] AR glasses can also be applied to conference scenarios. When users use AR glasses to attend meetings, they can see the meeting scene through the camera of the AR glasses they wear. Other people often speak at the meeting scene. If the participants wear noise-canceling headphones and the noise-canceling headphones are connected to AR glasses, then AR glasses can transmit external sounds to the user through the noise-canceling headphones. However, the sound transmitted through the noise-canceling headphones does not have the natural directionality in the real environment, so users cannot distinguish the current speaker's position information in the picture presented by AR glasses based on the sound transmitted by the noise-canceling headphones. This affects the user's experience of attending the meeting to a certain extent and cannot bring a better sense of presence to each user attending the meeting.

[0004] Currently, no effective solution has been proposed to the problem that the sound emitted by the noise reduction headphones cannot allow the user to locate the real position of the speaker in the AR scene. Summary of the invention

[0005] The present application provides an audio processing method, device, electronic device and readable medium to solve the above-mentioned technical problem that "the sound emitted by the noise reduction headphones cannot allow the user to locate the real position of the speaker in the AR scene".

[0006] According to one aspect of an embodiment of the present application, the present application provides an audio processing method, including: obtaining a full-scene image corresponding to a conversation scene in which a current user is located; constructing a geodetic coordinate system with AR glasses worn by the current user as the origin, and calibrating the full-scene image in the geodetic coordinate system to obtain a coordinate model; when a speaker is detected speaking in the conversation scene, determining a sound source position corresponding to the speaker in the coordinate model according to the speaker's audio information; determining speech output parameters through the audio information and the sound source position, and outputting the speaker's speech voice to the current user according to the speech output parameters, so as to simulate the speech voice being played from the sound source position to the current user.

[0007] Optionally, obtaining a full-scene image corresponding to the conversation scene where the current user is located includes: capturing a first video stream by at least one first camera device pre-set in the conversation scene, wherein the first video stream includes the current user; capturing a second video stream by a second camera device on AR glasses worn by the current user; and fusing the first video stream and the second video stream by a preset image processing strategy to generate a full-scene image corresponding to the conversation scene, wherein the full-scene image includes the current user.

[0008] Optionally, after generating a full-scene image corresponding to the conversation scene, the method further includes: generating a non-full-scene image from the perspective of the current user based on the second video stream; displaying the non-full-scene image in the display area of ​​the current user's AR glasses; and when a switching instruction is received, switching the display content of the display area from the non-full-scene image to the full-scene image.

[0009] Optionally, a geodetic coordinate system is constructed with the AR glasses currently worn by the user as the origin, and the full-scene image is calibrated in the geodetic coordinate system to obtain a coordinate model, including: setting the center of the AR glasses as the origin, and setting the direction based on the orientation of the AR glasses to construct a geodetic coordinate system; extracting each feature point in the full-scene image, and determining the three-dimensional coordinates of each feature point in the geodetic coordinate system; and using the three-dimensional coordinates of each feature point to construct a coordinate model corresponding to the full-scene image.

[0010] Optionally, when it is detected that a speaker is speaking in the conversation scene, the sound source position corresponding to the speaker in the coordinate model is determined based on the speaker's audio information, including: when it is detected that a speaker is speaking in the conversation scene, the speaker's audio information is collected by a microphone built into the AR glasses; the character positions of the speaker and the current user in the full-scene image are determined based on the audio information; and the character position of the speaker is converted into position coordinates in the coordinate model, wherein the position coordinates represent the sound source position of the speaker in the coordinate model.

[0011] Optionally, speech output parameters are determined through audio information and sound source position, and the speaker's speech is output to the current user according to the speech output parameters, including: obtaining the head posture of the current user; determining the relative position information of the speaker's head and the current user's head according to the sound source position and the head posture; determining speech output parameters according to the relative position information; extracting the speech voice from the audio information, and playing the speech voice in the headphones worn by the user according to the speech output parameters.

[0012] Optionally, the relative position information includes azimuth, elevation and distance, and the speech output parameters are determined according to the relative position information, including: determining convolution parameters according to the azimuth and elevation, wherein the convolution parameters are used to process the spoken voice to simulate the spatial direction effect of the sound source; determining the volume ratio of the left and right channels according to the azimuth; determining the delay time of the left and right channels according to the azimuth and distance; determining the volume attenuation amount according to the distance, and the speech output parameters include the convolution parameters, the volume ratio of the left and right channels, the delay time of the left and right channels, and the volume attenuation amount.

[0013] According to another aspect of an embodiment of the present application, the present application provides an audio processing device, including: an acquisition module, used to acquire a full-scene image corresponding to a conversation scene in which a current user is located; a construction module, used to construct a geodetic coordinate system with AR glasses worn by the current user as the origin, and calibrate the full-scene image in the geodetic coordinate system to obtain a coordinate model; a determination module, used to determine the sound source position corresponding to the speaker in the coordinate model according to the speaker's audio information when a speaker is detected speaking in the conversation scene; an output module, used to determine speech output parameters through audio information and sound source position, and output the speaker's speech voice to the current user according to the speech output parameters, so as to simulate the speech voice being played from the sound source position to the current user.

[0014] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0015] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application adopts an audio processing method, including: obtaining a full scene image corresponding to the conversation scene where the current user is located; constructing a geodetic coordinate system with the AR glasses worn by the current user as the origin, and calibrating the full scene image in the geodetic coordinate system to obtain a coordinate model; when a speaker is detected to be speaking in the conversation scene, determining the sound source position corresponding to the speaker in the coordinate model according to the speaker's audio information; determining the voice output parameters according to the audio information and the sound source position, and outputting the speaker's speech voice to the current user according to the voice output parameters, so as to simulate the current user playing the speech voice from the sound source position. First, the full scene image is calibrated to the geodetic coordinate system constructed with the user's AR glasses as the origin to obtain a coordinate model, and then the sound source position corresponding to the speaker is determined in the coordinate model according to the speaker's audio information, and finally the voice output parameters are adjusted according to the sound source position, and the speech voice is played using the voice output parameters, so as to achieve the effect of simulating the speech voice played from the sound source position, so that the user can perceive the real position of the speaker, and solve the problem that the sound emitted by the noise reduction headphones cannot allow the user to locate the real position of the speaker in the AR scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of an optional audio processing method provided according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an optional conference scene based on AR glasses provided according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of constructing an optional geodetic coordinate system provided according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of calibrating a sound source position within a user's field of view according to an optional embodiment of the present application;

[0024] Figure 5 A block diagram of an optional audio processing device provided according to an embodiment of the present application;

[0025] Figure 6A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0028] In recent years, AR (Augmented Reality) technology has made great progress. AR technology uses computers to generate realistic virtual information of vision, hearing, force, touch and movement, and displays virtual information in the real world. AR glasses based on AR technology allow the wearer to see not only real-world objects, but also the generated virtual objects, as if they exist in the real world.

[0029] AR glasses can also be applied to conference scenarios. When users use AR glasses to attend meetings, they can see the meeting scene through the camera of the AR glasses they wear. Other people often speak at the meeting scene. If the participants wear noise-canceling headphones and the noise-canceling headphones are connected to AR glasses, then AR glasses can transmit external sounds to the user through the noise-canceling headphones. However, the sound transmitted through the noise-canceling headphones does not have the natural directionality in the real environment, so users cannot distinguish the current speaker's position information in the picture presented by AR glasses based on the sound transmitted by the noise-canceling headphones. This affects the user's experience of attending the meeting to a certain extent and cannot bring a better sense of presence to each user attending the meeting.

[0030] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiment of the present application, an audio processing method is provided, such as Figure 1 As shown, including:

[0031] Step 101, obtaining a full scene image corresponding to the conversation scene where the current user is located;

[0032] Step 102, constructing a geodetic coordinate system with the AR glasses currently worn by the user as the origin, and calibrating the full scene image in the geodetic coordinate system to obtain a coordinate model;

[0033] Step 103, when it is detected that there is a speaker speaking in the conversation scene, determine the sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker;

[0034] Step 104, determining speech output parameters through audio information and the sound source position, and outputting the speaker's speech to the current user according to the speech output parameters, so as to simulate the speech being played from the sound source position to the current user.

[0035] The present application provides an audio processing method that can be applied to AR conversation scenarios, aiming to match the voice orientation of a speaker in a conversation scenario with the spatial position of a character in a virtual picture, thereby truly achieving a three-dimensional, immersive user experience.

[0036] First, the camera in the conversation scene captures the full scene image of the current user's environment, which is the complete visual information of the scene the current user and other speakers are in. The full scene image can include the current user himself and all people, objects and other environmental details around him.

[0037] Secondly, a three-dimensional geodetic coordinate system is constructed based on the AR glasses currently worn by the user, and then the acquired full-scene image is mapped into the geodetic coordinate system to obtain a coordinate model corresponding to the session scene.

[0038] Next, when a speaker is detected, the built-in microphone array of the AR glasses is used to capture audio information, and the audio information is combined with the constructed coordinate model to determine the corresponding sound source position of the speaker in the geodetic coordinate system.

[0039] Next, a set of speech output parameters is generated by combining the speaker's audio information and the corresponding sound source position. The speech output parameters of the present application include but are not limited to convolution parameters (such as HRTF (Head-Related Transfer Function) convolution kernel), the volume ratio of the left and right channels, the delay time of the left and right channels, and the volume attenuation.

[0040] Finally, according to the determined set of voice output parameters, the speech in the audio information is played to the current user, and the speaker's speech is played in a spatialized manner. Even if the sound is emitted from noise-canceling headphones, the user can feel that the sound is emitted from the sound source position in the real space.

[0041] In the embodiments provided in the present application, by spatializing the audio information, the user can not only see the speaker in the real conversation scene in the full-scene image displayed by the AR glasses, but also feel the sound coming from the actual sound source position of the speaker, providing a more immersive interactive experience.

[0042] The audio processing method provided in this application can be applied to a variety of AR conversation scenarios, including but not limited to AR remote conferences, multiplayer AR games, virtual social scenes or online activities. In these scenarios, by combining sound and visual information through this application, users can more naturally perceive the virtual or real sound sources around them, thereby improving the efficiency and comfort of interaction.

[0043] This application proposes a method for enhancing vision and hearing of AR glasses based on multimodal AI technology. Based on the image information collected by cameras at different positions, a complete scene graph is constructed through multimodal AI fusion processing. By constructing a coordinate model, combining the spatial position information and sound information of the AR user, and processing the image, position, and sound information through multimodal AI fusion, a three-dimensional scene including image, sound, and position is constructed. Combined with the sound-with-head movement technology, the speaker's voice position and the spatial position of the character in the virtual picture are matched, truly realizing a three-dimensional, immersive user experience.

[0044] As an optional embodiment, obtaining a full-scene image corresponding to the conversation scene in which the current user is located includes: capturing a first video stream through at least one first camera device pre-set in the conversation scene, wherein the first video stream includes the current user; capturing a second video stream through a second camera device on AR glasses worn by the current user; and fusing the first video stream and the second video stream through a preset image processing strategy to generate a full-scene image corresponding to the conversation scene, wherein the full-scene image includes the current user.

[0045] In the entire conversation scene, multiple people can wear AR glasses to participate in the meeting. Each AR glass will be equipped with a camera, and the glasses box corresponding to the AR glasses can also be equipped with a camera. In addition, additional cameras (which can be fixed or movable) can be set at appropriate locations in the conversation scene (such as on the wall) to obtain a perspective of the entire conversation scene.

[0046] Specifically, the second camera device represents a camera set on the current user's AR glasses, and the second camera device is used to capture what the current user sees; the first camera device represents other cameras except the second camera device, such as the camera on the glasses case, the cameras on the AR glasses of other users except the current user, and the cameras set on the wall, etc. Each first camera device is used to capture all participants in the entire session scene (including the current user).

[0047] It should be noted that the present application does not limit the number of first camera devices. For example, the present application can also be applied to a scenario where there is only one user (i.e., the current user) in the conversation scene, and the first camera device in this case is a camera on a glasses box and / or a camera on a wall that matches the AR glasses worn by the current user.

[0048] After the first video stream and the second video stream are obtained, the first video stream and the second video stream are fused through a preset image processing strategy to generate a full-scene image corresponding to the conversation scene. The preset image processing strategy provided in this application is a multimodal AI processing technology. Multimodal AI is an artificial intelligence technology that combines multiple sources of perceptual information. It uses multiple data modalities such as vision, voice, and text for information processing and analysis.

[0049] In the embodiments of the present application, multimodal AI fusion processing can be performed based on cloud-based multimodal AI to process remote video streams, audio streams, and location information, and the processed results are directly transmitted to AR glasses via 4G / 5G / WIFI for display. At this time, the AR glasses do not do a lot of computing work, which helps to improve battery life and reduce power consumption.

[0050] In the embodiments of the present application, the video stream, audio stream, and location information transmitted from the cloud can also be fused and processed based on the local multimodal AI of the AR glasses. However, if the local multimodal AI is responsible for data processing, power consumption will increase.

[0051] Figure 2 A schematic diagram of a conference scene based on AR glasses provided for this application. The conversation scene in the figure includes two users (user 1 and user 2), and any user can be the current user. As shown in the figure, each user's AR glasses are equipped with cameras (Cam1, Cam2) and microphones (MIC1, MIC2), and each AR glasses also has a corresponding glasses box, which is also equipped with cameras (Cam3, Cam4), and a camera (Cam5) is also set at a preset position in the scene. The matching AR glasses and the glasses box are transmitted via 4G / 5G / WIFI, and each AR glasses is transmitted via 4G / 5G / WIFI. The camera outside the AR glasses can also be transmitted to the AR glasses via 4G / 5G / WIFI. Data such as video streams and audio streams can be transmitted to the cloud through 4G / 5G / WIFI for multimodal AI processing. It should be noted that Cam3, Cam4, and Cam5 in the figure do not need to exist at the same time. This depends on the actual application scenario. When there are one or more people in the environment or in the meeting, the processing scheme is also different. The following is explained separately for different scenarios.

[0052] If there is only a single user in the current user environment or in the meeting, assuming there is only user 1, then you can use Cam3 on the AR glasses box 1 or Cam5 placed in the room to capture the image of the current user in the environment, and transmit the video stream captured by Cam3 or Cam5 to AR glasses 1 via 4G / 5G / WIFI. AR glasses 1 uses multimodal AI to fuse and process the received video stream of Cam3 or Cam5 and the video stream captured by its own Cam1 to generate a complete scene image including the user itself. AR glasses 1 can also transmit the received video stream of Cam3 or Cam5 and the video stream captured by its own Cam1 via 4G / 5G / WIFI to the multimodal AI in the cloud for fusion and processing, and can also generate a complete scene image including the user itself.

[0053] If there is more than one user in the current user environment or conference, such as Figure 2 In the example, there are two users, user 1 and user 2. At this time, Cam1 on AR glasses 1 worn by user 1 collects a video stream with the image of user 2. The video stream collected by Cam1 can be transmitted to AR glasses 2 via 4G / 5G / WIFI. AR glasses 2 receives the video data stream with its own image information collected by Cam1. AR glasses 2 uses multimodal AI to process the video stream collected by Cam2 and the video stream transmitted from Cam1. By fusing and processing the video streams through multimodal AI, a complete scene picture including user 2 can be obtained. AR glasses 2 can also transmit the received video streams collected by Cam1 and Cam2 to multimodal AI in the cloud through 4G / 5G / WIFI for fusion and processing, so that a complete scene picture including user 2 can be presented to user 2. Other multi-user scene processing methods are similar to this case and will not be repeated here.

[0054] As an optional embodiment, after generating a full-scene image corresponding to the conversation scene, the method also includes: generating a non-full-scene image from the current user's perspective based on the second video stream; displaying the non-full-scene image in the display area of ​​the current user's AR glasses; and when a switching instruction is received, switching the display content of the display area from the non-full-scene image to the full-scene image.

[0055] In addition to the full-scene image, the present application can also generate a non-full-scene image that does not include the current user based on the second video stream from the current user's perspective. The non-full-scene image is from the perspective of the current user and focuses on a specific area or object in front of the current user, rather than the entire session scene.

[0056] In addition, when generating non-full-scene images, the video streams collected by other cameras can also be used in combination. The entire image generation process may include technologies such as image segmentation, object recognition, and depth perception. The generated images are accurate and comprehensive.

[0057] Currently, users can see full-scene images or non-full-scene images in the display area (e.g., display screen) of AR glasses, and issue switching instructions according to their own needs to implement switching operations. Switching instructions can be issued in a variety of interactive ways, such as gesture control, voice commands, or buttons on the user interface.

[0058] The present application collects video streams by integrating a camera device on the glasses case, or by using the camera devices of other AR glasses users in the space, or by using a camera device fixed in the space, and then fuses and processes the video streams collected by the user's AR glasses themselves and the video streams collected by a camera device outside the AR glasses through multimodal AI to synthesize a complete visual image including the user in the current environment and present it to the AR user.

[0059] By using a flexible switching mechanism to view full-scene images or non-full-scene images, users can view the entire session scene at any time and maintain a comprehensive understanding of the current environment. This dynamic display method provides augmented reality applications with a higher sense of user participation and convenience in information acquisition.

[0060] As an optional embodiment, a geodetic coordinate system is constructed with the AR glasses currently worn by the user as the origin, and the full-scene image is calibrated in the geodetic coordinate system to obtain a coordinate model, including: setting the center of the AR glasses as the origin, and setting the direction based on the orientation of the AR glasses to construct a geodetic coordinate system; extracting each feature point in the full-scene image, and determining the three-dimensional coordinates of each feature point in the geodetic coordinate system; and using the three-dimensional coordinates of each feature point to construct a coordinate model corresponding to the full-scene image.

[0061] Specifically, the center of the AR glasses is set as the origin, and the direction is set based on the orientation of the AR glasses to construct a geodetic coordinate system. First, the center of the AR glasses is set as the origin of the coordinate system (usually the coordinates (0, 0, 0)), and then the three axes of the coordinate system are set based on the current orientation of the AR glasses: X-axis, Y-axis and Z-axis. The X-axis, Y-axis and Z-axis can be set according to actual needs, and the X-axis, Y-axis and Z-axis are perpendicular to each other.

[0062] Figure 3 The schematic diagram of the construction of the geodetic coordinate system provided for this application is shown in the figure. The midpoint of the frame of the AR glasses is used as the origin of the coordinate system (point O), and the X-axis, Y-axis and Z-axis are constructed based on this to form a geodetic coordinate system.

[0063] Use image processing algorithms to identify and extract important visual feature points from the full-scene image. Feature points refer to points in the image that can represent important locations or objects, such as corners, edges, or center points of specific objects. The identified feature points should cover all key information in the full-scene image.

[0064] Three-dimensional positioning is performed through the position of the feature points in the geodetic coordinate system. Each extracted feature point corresponds to a two-dimensional coordinate (x, y) in the full-scene image. It is necessary to combine the field of view depth information of the AR glasses to convert it into three-dimensional coordinates (X, Y, Z). The two-dimensional coordinates of each feature point are combined with its corresponding depth information, and the camera is calibrated using the internal and external parameters of the AR glasses (such as focal length, aperture, etc.), so as to obtain the accurate three-dimensional position of each feature point in the geodetic coordinate system.

[0065] Using the three-dimensional coordinates of all the determined feature points, a three-dimensional model containing these points is generated. The three-dimensional model is a spatial representation of the full scene image. The coordinate model not only contains the information of the feature points, but also can include the spatial relationship between these points to form a complete three-dimensional scene.

[0066] By constructing a geodetic coordinate system corresponding to the position and orientation of the user's AR glasses, the full-scene image is combined with the three-dimensional coordinates to achieve spatial audio processing and augmented reality interaction.

[0067] Optionally, in the process of using AR glasses in the remote cloud, the AR glasses display interface can simulate a usage scenario, in which the wearer of the AR glasses can see the full scene map including himself. When the person at the far end speaks, the AR user can hear the speaker's voice, but the sound heard cannot correspond to the person on the screen presented in the glasses, and the speaker's position information cannot be judged by the human ear as in reality. In this case, the AR glasses cannot provide the user with a fully immersive experience. Through the establishment of the above-mentioned coordinate model, in this case, the position of the character in the display screen simulated by the AR glasses can be determined in the established model. When the virtual character in the AR glasses display screen speaks, it is possible to know which character is speaking, and then know the spatial coordinates of the current speaker in the virtual screen.

[0068] Optionally, if the user wears noise-canceling headphones while using AR glasses, the noise-canceling headphones and AR glasses are connected via BLE (Bluetooth Low Energy), and the sound of the AR glasses is transmitted to the user through the headphones. At this time, for the AR user, he cannot directly hear the voices of people in the current environment, and it is necessary to use the embodiments provided in this application to determine the sound source position of the speaker, so that the voices of people in the environment and their spatial positions have a corresponding relationship in the same coordinate system. Of course, there are other scenarios that require the use of AR glasses to hear the voices of people in the environment, which are not listed here one by one.

[0069] As an optional embodiment, when it is detected that a speaker is speaking in a conversation scene, the sound source position corresponding to the speaker in the coordinate model is determined according to the audio information of the speaker, including: when it is detected that a speaker is speaking in the conversation scene, the audio information of the speaker is collected by a microphone built into the AR glasses; the character positions of the speaker and the current user in the full-scene image are determined according to the audio information; and the character position of the speaker is converted into position coordinates in the coordinate model, wherein the position coordinates represent the sound source position of the speaker in the coordinate model.

[0070] Monitor the audio signals in the conversation scene to detect when someone starts speaking. This can be achieved through the strength, frequency characteristics and pattern of the audio signal. Once a speaker is detected, the built-in microphone of the AR glasses is used to collect relevant audio information, including the content of the speech (i.e., speech voice), tone, volume and other characteristics.

[0071] Specifically, the speaker and the current user's position in the full scene image are determined based on the audio information. The speaker can be identified and their position in the full scene image can be determined by using the feature points detected in the full scene image and image processing technology (such as target recognition, deep learning, etc.).

[0072] Once the speaker's position in the full scene image is determined, it can be converted into coordinates in the coordinate model. Using the depth information of the AR glasses and the camera calibration parameters, the speaker's two-dimensional position in the full scene image (such as pixel coordinates) is mapped to the three-dimensional coordinate system through the coordinate model constructed above to obtain the position coordinates of the sound source position.

[0073] Figure 4 A schematic diagram of the calibration of the sound source position within the user's field of view provided for this application. At any time, in the scene that the current user can see, when someone speaks, the speaker's voice is picked up by the microphone on the AR glasses, and the acoustic positioning technology is used to calibrate and record the spatial position of the current speaker in the established coordinate system (i.e. (x, y, z) in the figure).

[0074] Accurately identify the speaker in a conversation scenario and combine their visual and spatial information to determine the location of their sound source, so that subsequent sound processing can be spatialized based on the actual speaker position, providing a more realistic and immersive user experience.

[0075] As an optional embodiment, speech output parameters are determined through audio information and sound source position, and the speaker's speech is output to the current user according to the speech output parameters, including: obtaining the head posture of the current user; determining the relative position information of the speaker and the current user's head according to the sound source position and the head posture; determining speech output parameters according to the relative position information; extracting the speech voice from the audio information, and playing the speech voice in the headphones worn by the user according to the speech output parameters.

[0076] Specifically, obtaining the current user's head posture includes: tracking the user's head position and orientation in real time through sensors in the AR glasses (such as accelerometers, gyroscopes, and depth cameras). The head posture is usually represented as a vector in a three-dimensional coordinate system, including the rotation angle of the user's head (pitch angle, yaw angle, and roll angle), which will affect the user's perception of surrounding sounds.

[0077] After obtaining the user's head posture, the relative position between the speaker and the current user's head can be calculated based on the sound source position corresponding to the speaker and the current user's head posture, including calculating the distance, direction and angle between them.

[0078] The speech output parameters are determined based on the relative position of the speaker to the user, thereby determining how to adjust the sound output to enhance the sense of space and realism.

[0079] The speech output parameters provided in this application include convolution parameters, volume ratio of left and right channels, delay time of left and right channels, and volume attenuation, but in actual process, there may be other parameters, such as reverberation. This application does not limit the content of speech output parameters, and only provides them as the best embodiment for reference.

[0080] After determining the appropriate speech output parameters, the speaker's speech can be extracted from the audio information and played according to the speech output parameters. When the speech is played using the speech output parameters, the spatial characteristics of the sound can be simulated, allowing users to experience the feeling that the sound is coming from the actual position of the speaker.

[0081] By combining audio information with the user's spatial perception and dynamically adjusting voice output parameters to achieve an immersive auditory experience, users can not only hear clear speaking voices, but also feel the spatial sense of the sound, making the interaction with the speaker more natural and real.

[0082] Optionally, after acquiring the video stream and audio stream, the multimodal AI first determines the coordinate position of the virtual character in the virtual screen based on the established coordinate model, and then determines which person is currently speaking based on the incoming sound information. Once it is determined who is speaking, the speaker's sound source position has been obtained in advance through calculation. At this time, the multimodal AI can send the calculated result of the position to the sound-following head movement algorithm. The sound-following head movement algorithm outputs the speaking voice at a predetermined spatial position according to the voice output parameters based on the input sound information and spatial position data. At this time, when the AR user hears the speaking voice, he can judge which virtual character is speaking based on the sound information heard by his own ears, thereby achieving correspondence and unification of the sound and picture positions.

[0083] Next, how to determine the speech output parameters is described.

[0084] As an optional embodiment, the relative position information includes azimuth, elevation and distance, and the speech output parameters are determined according to the relative position information, including: determining convolution parameters according to the azimuth and elevation, wherein the convolution parameters are used to process the spoken voice to simulate the spatial direction effect of the sound source; determining the volume ratio of the left and right channels according to the azimuth; determining the delay time of the left and right channels according to the azimuth and distance; determining the volume attenuation amount according to the distance, and the speech output parameters include the convolution parameters, the volume ratio of the left and right channels, the delay time of the left and right channels, and the volume attenuation amount.

[0085] The relative position information includes azimuth, elevation and distance, where the azimuth indicates the horizontal position of the speaker relative to the current user's head (usually expressed in degrees), the elevation indicates the vertical position of the speaker relative to the current user's head (usually expressed in degrees), and the distance indicates the straight-line distance between the speaker and the current user.

[0086] Specifically, based on the azimuth and elevation, convolution parameters can be calculated, and the convolution parameters are used to perform convolution processing on the speaker's voice to simulate the spatial direction effect of the sound source and ensure that the sound sounds like it is coming from a specific direction.

[0087] Specifically, the azimuth directly affects the volume ratio of the left and right channels. If the speaker is on the right side of the user, the volume of the right channel will increase, while the volume of the left channel will decrease accordingly.

[0088] Specifically, the sound propagates from the speaker to the user's ear, and may have different arrival times due to the speaker's position, so the delay time of the left and right channels can be calculated based on the azimuth and distance. For example, if the speaker is on the right side of the user, the right ear will hear the sound earlier than the left ear, so the right channel may play the sound slightly in advance, while the left channel will have a delay to simulate the real sound source positioning.

[0089] Specifically, since the volume of sound usually attenuates as the distance increases, the attenuation amount (usually in decibels) needs to be calculated according to the distance between the speaker and the user. For example, if the speaker is far away from the user, the sound will be attenuated more, and conversely, if the distance is close, the sound will be relatively loud.

[0090] By analyzing the relative position, the sound output parameters can be dynamically adjusted so that the user can perceive the direction, distance and spatial sense of the sound when hearing the speaker's voice. This not only enhances the user's sense of immersion, but also improves the naturalness and authenticity of the interaction, especially in augmented reality or virtual reality scenarios, and can provide a more attractive auditory experience.

[0091] By establishing a coordinate system, combining acoustic positioning technology and sound-following-head-motion technology, and processing different types of data through multimodal AI fusion, it is possible to correspond the speaker's voice with the person's position information in the virtual screen presented in the AR glasses in full / non-full scene graph mode, so that AR users can determine the current speaker's position in the virtual screen by the sound they hear.

[0092] The present application adopts an audio processing method, including: obtaining a full scene image corresponding to the conversation scene where the current user is located; constructing a geodetic coordinate system with the AR glasses worn by the current user as the origin, and calibrating the full scene image in the geodetic coordinate system to obtain a coordinate model; when a speaker is detected to be speaking in the conversation scene, determining the sound source position corresponding to the speaker in the coordinate model according to the speaker's audio information; determining the voice output parameters according to the audio information and the sound source position, and outputting the speaker's speech voice to the current user according to the voice output parameters, so as to simulate the current user playing the speech voice from the sound source position. First, the full scene image is calibrated to the geodetic coordinate system constructed with the user's AR glasses as the origin to obtain a coordinate model, and then the sound source position corresponding to the speaker is determined in the coordinate model according to the speaker's audio information, and finally the voice output parameters are adjusted according to the sound source position, and the speech voice is played using the voice output parameters, so as to achieve the effect of simulating the speech voice played from the sound source position, so that the user can perceive the real position of the speaker, and solve the problem that the sound emitted by the noise reduction headphones cannot allow the user to locate the real position of the speaker in the AR scene.

[0093] According to another aspect of the embodiment of the present application, the present application provides an audio processing device, such as Figure 5 As shown, including:

[0094] An acquisition module 501 is used to acquire a full scene image corresponding to the conversation scene where the current user is located;

[0095] A construction module 502 is used to construct a geodetic coordinate system with the AR glasses currently worn by the user as the origin, and calibrate the full scene image in the geodetic coordinate system to obtain a coordinate model;

[0096] A determination module 503 is used to determine the sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker when it is detected that there is a speaker speaking in the conversation scene;

[0097] The output module 504 is used to determine the speech output parameters according to the audio information and the sound source position, and output the speaker's speech to the current user according to the speech output parameters, so as to simulate the speech played from the sound source position to the current user.

[0098] It should be noted that the acquisition module 501 in this embodiment can be used to execute step 101 in the embodiment of the present application, the construction module 502 in this embodiment can be used to execute step 102 in the embodiment of the present application, the determination module 503 in this embodiment can be used to execute step 103 in the embodiment of the present application, and the output module 504 in this embodiment can be used to execute step 104 in the embodiment of the present application.

[0099] Optionally, the acquisition module 501 is also used to capture a first video stream through at least one first camera device pre-set in the conversation scene, wherein the first video stream includes the current user; capture a second video stream through a second camera device on the AR glasses worn by the current user; and fuse the first video stream and the second video stream through a preset image processing strategy to generate a full-scene image corresponding to the conversation scene, wherein the full-scene image includes the current user.

[0100] Optionally, the acquisition module 501 is also used to generate a non-full-scene image from the current user's perspective according to the second video stream after generating a full-scene image corresponding to the session scene; display the non-full-scene image in the display area of ​​the current user's AR glasses; and when a switching instruction is received, switch the display content of the display area from the non-full-scene image to the full-scene image.

[0101] Optionally, the construction module 502 is also used to set the center of the AR glasses as the origin, and set the direction based on the orientation of the AR glasses to construct a geodetic coordinate system; extract each feature point in the full-scene image, and determine the three-dimensional coordinates of each feature point in the geodetic coordinate system; use the three-dimensional coordinates of each feature point to construct a coordinate model corresponding to the full-scene image.

[0102] Optionally, the determination module 503 is also used to collect the audio information of the speaker through the built-in microphone of the AR glasses when a speaker is detected in the conversation scene; determine the character positions of the speaker and the current user in the full-scene image based on the audio information; and convert the character position of the speaker into position coordinates in the coordinate model, wherein the position coordinates represent the sound source position of the speaker in the coordinate model.

[0103] Optionally, the output module 504 includes:

[0104] The acquisition submodule is used to obtain the head posture of the current user;

[0105] The first determination submodule is used to determine the relative position information of the speaker and the current user's head according to the sound source position and the head posture;

[0106] A second determination submodule, used to determine the speech output parameters according to the relative position information;

[0107] The extraction submodule is used to extract the spoken voice from the audio information and play the spoken voice in the earphones worn by the user according to the voice output parameters.

[0108] Optionally, the relative position information includes azimuth, elevation and distance, and the second determination submodule is further used to determine convolution parameters based on the azimuth and elevation, wherein the convolution parameters are used to process the spoken voice to simulate the spatial direction effect of the sound source; determine the volume ratio of the left and right channels based on the azimuth; determine the delay time of the left and right channels based on the azimuth and distance; determine the volume attenuation amount based on the distance, and the voice output parameters include convolution parameters, the volume ratio of the left and right channels, the delay time of the left and right channels and the volume attenuation amount.

[0109] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0110] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 6 As shown, it includes a memory 601, a processor 603, a communication interface 605 and a communication bus 607. The memory 601 stores a computer program that can be run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. When the processor 603 executes the computer program, the steps of the above method are implemented.

[0111] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0112] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0113] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0114] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0115] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0116] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0117] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0118] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0119] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0124] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0125] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. An audio processing method, characterized in that: include: Obtain the full scene image corresponding to the current user's session scene; Constructing a geodetic coordinate system with the AR glasses worn by the current user as the origin, and calibrating the full scene image in the geodetic coordinate system to obtain a coordinate model; When it is detected that a speaker is speaking in the conversation scene, determining a sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker; The speech output parameters are determined by the audio information and the sound source position, and the speech of the speaker is output to the current user according to the speech output parameters, so as to simulate the speech being played from the sound source position to the current user.

2. The method according to claim 1, characterized in that The obtaining of a full scene image corresponding to the current user's session scene includes: Capturing a first video stream by at least one first camera device pre-set in the conversation scene, wherein the first video stream includes the current user; Capturing a second video stream through a second camera device on the AR glasses worn by the current user; The first video stream and the second video stream are fused through a preset image processing strategy to generate the full-scene image corresponding to the conversation scene, wherein the full-scene image includes the current user.

3. The method according to claim 2, characterized in that After generating the full-scene image corresponding to the conversation scene, the method further includes: Generate a non-full-scene image from the current user's perspective according to the second video stream; Displaying the non-full-scene image in a display area of ​​the AR glasses of the current user; When a switching instruction is received, the display content of the display area is switched from the non-full-scene image to the full-scene image.

4. The method according to claim 1, characterized in that: The method of constructing a geodetic coordinate system with the AR glasses worn by the current user as the origin and calibrating the full scene image in the geodetic coordinate system to obtain a coordinate model includes: The center of the AR glasses is set as the origin, and the direction is set based on the orientation of the AR glasses to construct a geodetic coordinate system; Extracting each feature point in the full scene image, and determining the three-dimensional coordinates of each feature point in the geodetic coordinate system; The coordinate model corresponding to the full scene image is constructed using the three-dimensional coordinates of each of the feature points.

5. The method according to claim 1, characterized in that When it is detected that a speaker is speaking in the conversation scene, determining the sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker includes: When it is detected that the speaker is speaking in the conversation scene, the audio information of the speaker is collected by using the built-in microphone of the AR glasses; Determine the positions of the speaker and the current user in the full scene image according to the audio information; The character position of the speaker is converted into position coordinates in the coordinate model, wherein the position coordinates represent the sound source position of the speaker in the coordinate model.

6. The method according to claim 1, characterized in that The step of determining the speech output parameters by using the audio information and the sound source position, and outputting the speech of the speaker to the current user according to the speech output parameters, comprises: Obtaining the head posture of the current user; Determining relative position information of the speaker and the head of the current user according to the sound source position and the head posture; Determining the speech output parameter according to the relative position information; The speech voice is extracted from the audio information, and the speech voice is played in the earphones worn by the user according to the voice output parameters.

7. The method according to claim 6, characterized in that The relative position information includes an azimuth, an elevation, and a distance, and determining the voice output parameter according to the relative position information includes: Determine a convolution parameter according to the azimuth angle and the elevation angle, wherein the convolution parameter is used to process the spoken speech to simulate the spatial direction effect of the sound source; Determining the volume ratio of the left and right channels according to the azimuth angle; Determine the delay time of the left and right channels according to the azimuth angle and the distance; The volume attenuation is determined according to the distance, and the speech output parameters include the convolution parameter, the volume ratio of the left and right channels, the delay time of the left and right channels, and the volume attenuation.

8. An audio processing device, characterized in that: include: An acquisition module is used to acquire a full scene image corresponding to the current user's session scene; A construction module, used to construct a geodetic coordinate system with the AR glasses worn by the current user as the origin, and calibrate the full scene image in the geodetic coordinate system to obtain a coordinate model; A determination module, configured to determine, when detecting that a speaker is speaking in the conversation scene, a sound source position corresponding to the speaker in the coordinate model according to the audio information of the speaker; The output module is used to determine the speech output parameters through the audio information and the sound source position, and output the speech of the speaker to the current user according to the speech output parameters, so as to simulate the speech being played from the sound source position to the current user.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 7.

Images