Audio signal correction method and apparatus, electronic device, and readable storage medium

Abstract

The application discloses an audio signal correction method and device, electronic equipment and a readable storage medium, which are applied to electronic equipment. The method comprises the following steps: in response to an audio-video recording request, collecting an angle change parameter of a head of a target user based on a wearable device; collecting a human voice uttered by the target user as a human voice audio signal; determining position information of the target user in a video picture based on the angle change parameter; determining a position distribution of a target sound source in a sound field where the electronic equipment is located based on the position information; and correcting the human voice audio signal based on the position distribution to obtain a corrected audio signal. By determining the position information of the target user in the video picture and then determining the position distribution of the target sound source in the sound field where the electronic equipment is located, the matching degree of the position distribution of the sound source corresponding to the corrected audio signal in the sound field where the electronic equipment is located and the position of the target user in the video picture can be improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more specifically, to an audio signal correction method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, with the development of electronic information technology, audio signals can be recorded using electronic devices. However, algorithms can correct situations where the target user's head may turn during recording, causing a mismatch between the recorded audio signal's location within the electronic device's sound field and the target user's location in the video frame. Nevertheless, even after algorithmic correction, the location of the audio signal's location within the electronic device's sound field may still not match the target user's location in the video frame, resulting in a relatively poor correction effect. Summary of the Invention

[0003] This application discloses an audio signal correction method, apparatus, electronic device, and readable storage medium.

[0004] In a first aspect, embodiments of this application provide an audio signal correction method applied to an electronic device. The method includes: responding to an audio / video recording request, acquiring angle change parameters of a target user's head based on a wearable device; acquiring the voice emitted by the target user as a voice audio signal; determining the target user's position information in the video frame based on the angle change parameters; determining the position distribution of a target sound source in the sound field of the electronic device based on the position information, wherein the target sound source is the equivalent sound source of the target user; and correcting the voice audio signal based on the position distribution to obtain a corrected audio signal.

[0005] Secondly, embodiments of this application also provide an audio signal correction device applied to an electronic device. The device includes: a first acquisition unit, configured to acquire angle change parameters of a target user's head based on a wearable device in response to an audio / video recording request; a second acquisition unit, configured to acquire the voice emitted by the target user as a voice audio signal; a positioning unit, configured to determine the position information of the target user in the video frame based on the angle change parameters; a position distribution determination unit, configured to determine the position distribution of a target sound source in the sound field of the electronic device based on the position information, wherein the target sound source is the equivalent sound source of the target user; and a correction unit, configured to correct the voice audio signal based on the position distribution to obtain a corrected audio signal.

[0006] Thirdly, embodiments of this application also provide an electronic device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the method described in the first aspect.

[0007] Fourthly, embodiments of this application also provide a computer-readable storage medium storing program code that can be invoked by a processor to execute the method described in the first aspect above.

[0008] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the method described in the first aspect above.

[0009] The audio signal correction method, apparatus, electronic device, and readable storage medium provided in this application first collect the head angle change parameters of a target user based on a wearable device; then, they collect the voice emitted by the target user as a voice audio signal; next, they determine the target user's position information in the video frame based on the angle change parameters, thereby determining the position distribution of the target sound source in the sound field of the electronic device based on the position information; finally, they correct the voice audio signal based on the position distribution to obtain a corrected audio signal. Since the target user's head may rotate, causing a change in head angle, the position distribution of the collected user's voice audio signal in the sound field of the corresponding sound source of the electronic device will not match the position of the target user in the video frame. Therefore, in this embodiment, the position distribution for correcting the voice audio signal is determined by the angle change parameters obtained from the target user's head movement, which determine the target user's position information in the video frame. By correcting the voice audio signal through position distribution, the matching degree between the position distribution of the corrected audio signal's corresponding sound source in the sound field of the electronic device and the position of the target user in the video frame can be improved.

[0010] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 The diagram illustrates an application scenario of the audio signal correction method provided in this application embodiment.

[0013] Figure 2 A flowchart of the audio signal correction method provided in an embodiment of this application is shown;

[0014] Figure 3 A schematic diagram of the specified data provided in the embodiments of this application is shown;

[0015] Figure 4 A flowchart of an audio signal correction method according to another embodiment of this application is shown;

[0016] Figure 5 A schematic diagram illustrating the calculation of angular velocity provided in an embodiment of this application is shown;

[0017] Figure 6 A schematic diagram illustrating the process of obtaining the first output signal and the second output signal provided in the embodiment itself is shown;

[0018] Figure 7 A flowchart of an audio signal correction method according to another embodiment of this application is shown;

[0019] Figure 8 A schematic diagram of a stereo audio signal obtained by downmixing according to an embodiment of this application is shown;

[0020] Figure 9 A structural block diagram of the audio signal correction device provided in an embodiment of this application is shown;

[0021] Figure 10 A structural block diagram of the electronic device provided in an embodiment of this application is shown;

[0022] Figure 11 This paper shows a structural block diagram of a computer-readable storage medium provided in an embodiment of this application;

[0023] Figure 12 A structural block diagram of a computer program product provided in an embodiment of this application is shown. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] Currently, with the development of electronic information technology, audio signals can be recorded using electronic devices. However, a common issue is that the target user may turn their head during recording, causing a mismatch between the recorded audio signal's position in the sound field of the electronic device and the target user's position in the video frame. Algorithms can correct this mismatch. However, even after algorithmic correction, the position of the audio signal in the sound field of the electronic device may still not match the target user's position in the video frame, resulting in poor correction effectiveness. Therefore, improving the matching degree between the audio signal's position in the sound field of the electronic device and the target user's position in the video frame is a problem that urgently needs to be solved.

[0027] Currently, electronic devices can be equipped with image acquisition modules. For example, these devices can be smartphones, tablets, etc., and thus can be equipped with cameras, allowing users to control the device to record. For instance, the image acquisition module of the electronic device can be controlled to record video information. Furthermore, users can also wear wearable devices that connect to the electronic device. An example of such a wearable device is a wireless headset, such as a True Wireless Stereo (TWS) headset, or a wireless headset. Specifically, the wearable device can establish a communication link with the electronic device through wireless communication technology, such as Bluetooth. Furthermore, the wearable device can generally also be equipped with an audio acquisition module, so that when the user controls the electronic device to record video signals, the wearable device, which has established a communication link with the electronic device, can acquire audio signals through the audio acquisition module.

[0028] It's easy to understand that during the recording of video and audio signals, if the user's head rotates, their position in the video frame will change. However, even with head rotation, the relative position between the wearable device and the user remains almost unchanged, resulting in minimal alteration to the audio signal captured by the wearable device. Consequently, a mismatch may occur between the positional distribution of the audio signal captured by the wearable device within the sound field of the electronic device and the user's position in the video frame. Therefore, audio signal correction is necessary to match the positional distribution of the audio signal within the sound field of the corresponding sound source on the electronic device with the user's position in the video frame. The audio signal captured by the wearable device can be the user's voice. A correction algorithm can be used to center the recorded user's voice audio signal, thereby correcting the audio signal and ensuring that the positional distribution of the corresponding sound source within the sound field of the electronic device matches the user's position in the video frame.

[0029] However, the inventors discovered in their research that for schemes that directly use algorithms to center human voice audio signals, if the user's head rotates to a large extent, the ability to correct the human voice audio signal will be weakened, resulting in a poor matching effect between the position distribution of the sound source corresponding to the human voice audio signal in the sound field of the electronic device and the position of the user in the video screen.

[0030] Therefore, in order to overcome the above-mentioned defects, this application provides an audio signal correction method, apparatus, electronic device, and readable storage medium.

[0031] Please see Figure 1 , Figure 1 An application scenario diagram of an audio signal correction method is shown, namely audio signal correction scenario 100, which includes an electronic device 110, a target user 120, and a wearable device 130.

[0032] Among them, Figure 1 In this context, the wearable device 130 can be a truly wireless stereo headset, which can be worn by the target user 120. The wearable device 130 can establish a wireless communication link with the electronic device 110, such as a Bluetooth communication link, thereby enabling information communication between the wearable device 130 and the electronic device 110.

[0033] In some embodiments, the electronic device 110 may be configured with an image acquisition module. The user can control the electronic device 110 to generate audio and video recording requests. In response to these requests, the electronic device 110 uses the configured image acquisition module to acquire video information of the target user 120, which can then be displayed in real-time on a display screen 190. The video frame 119 displayed on the display screen 190 represents the target user 120 captured by the image acquisition module. Furthermore, the electronic device 110 can also control the wearable device 130 to simultaneously acquire audio signals and the target angle of the target user 120's head. The wearable device 130 can then transmit the acquired audio signals and the target head angle to the electronic device 110 via a pre-established wireless communication link. The electronic device 110 can then determine the target user's head angle change parameters based on the target angle, and adjust the audio signals to match the position of the sound source corresponding to the audio signal in the sound field of the electronic device with the position of the user in the video frame. Specific methods can be found in the following embodiments.

[0034] Please see Figure 2 , Figure 2 This illustration shows a flowchart of an audio signal correction method provided in an embodiment of this application. This audio signal correction method can be applied to… Figure 1 In the electronic device 110 shown, the processor in the electronic device 110 can be used as the execution subject. The audio signal correction method specifically includes steps S110 to S150.

[0035] Step S110: In response to the audio and video recording request, collect the angle change parameters of the target user's head based on the wearable device.

[0036] Electronic devices can run different applications to perform different functions. For example, an electronic device can run an audio / video recording application to achieve audio and video recording. As an example, the audio / video recording application can be a "camera" application in the electronic device. The user can run the "camera" application and switch to the recording function in the application to record audio and video through the "camera" application.

[0037] The audio / video recording request can be a request triggered by the user based on the electronic device. For example, if the electronic device detects that the user has opened an audio / video recording application and switched to the recording function, and the user clicks the button corresponding to start recording, then it can be confirmed that an audio / video recording request has been received.

[0038] Upon detecting an audio / video recording request, the electronic device can capture both video footage and audio signals. For example, the electronic device may be equipped with an image acquisition module, allowing it to capture video footage in response to the request. Furthermore, the electronic device can also respond to the audio / video recording request by capturing the target user's voice audio signal through a wearable device; details will be provided in subsequent steps.

[0039] In the embodiments provided in this application, the target user can wear a wearable device, which may have wireless communication capabilities. Therefore, the wearable device can establish a wireless communication link with an electronic device, thereby allowing the electronic device to control the wearable device. For example, if the wearable device is a wireless headset, such as a true stereo wireless headset, the electronic device can control the wireless headset to play audio, control the wireless headset to collect the voice emitted by the target user, and control the wireless headset to collect the target angle of the target user's head, thereby determining the angle change parameters.

[0040] In this context, the target user can be any user appearing in the video feed. In some cases, there may only be one user in the video feed; in this case, the user making the voice is the target user. In other cases, there may be multiple users in the video feed; in this case, the user wearing a wearable device that has established a communication link with the electronic device can be considered the target user. For example, if there are a first user, a second user, and a third user, and the second user is wearing a wearable device that has established a communication link with the electronic device, then the second user can be considered the target user. In still other cases, the target user can also be the user who triggered the audio / video recording request. For instance, if the user has the electronic device fixed in one location, and after triggering the audio / video recording request, the user moves into the video feed captured by the electronic device, then that user is considered the target user.

[0041] It's easy to understand that during the process of an electronic device responding to an audio / video recording request and capturing images and audio signals, the target user's head may turn, causing their position in the video frame to change. However, the voice emitted by the target user, captured by the wearable device, hardly changes with head movement. Therefore, the positional distribution of the target user's voice in the sound field of the electronic device may mismatch with the target user's position in the video frame. In this case, the positional distribution of the target user's voice in the sound field of the electronic device can be further corrected by observing changes in the user's head angle, thus ensuring that the corrected positional distribution matches the target user's position in the video frame. Detailed explanations can be found in the following steps.

[0042] In some implementations, the angle change parameter of the target user's head can be the specific angle at which the user rotates their head. This angle change parameter can be determined by the target angle collected by the wearable device worn by the target user, and the target angle can be understood as the angle at which the target user's head rotates.

[0043] As an example, the wearable device may be configured with an Inertial Measurement Unit (IMU), which typically includes a gyroscope, an accelerometer, and a magnetometer. The gyroscope obtains the angular velocity of each axis, the accelerometer obtains the acceleration in different directions, and the magnetometer obtains information about the surrounding magnetic field. Thus, the gyroscope can be used to obtain the target angle of the user's head, and then, through a pre-established wireless communication link between the wearable device and the electronic device, the target angle is transmitted to the electronic device to determine the angle change parameters of the user's head. For details, please refer to the description of the following embodiments.

[0044] Step S120: Collect the voice emitted by the target user as a human voice audio signal.

[0045] Furthermore, after receiving an audio / video recording request, the device can capture the voice emitted by the target user as the audio signal. For example, the wearable device can also be configured with an audio acquisition component, such as a microphone or microphone array. Thus, after receiving the audio / video recording request, the electronic device can, based on the communication link established with the wearable device, control the wearable device to capture the voice emitted by the target user through the audio acquisition component, using it as the audio signal. Since the wearable device is generally worn on the target user's limbs or ears, the relative distance between the wearable device and the target user is relatively close, and the relative positional relationship between the wearable device and the target user generally does not change with the user's head movement, resulting in a high-quality audio signal captured by the wearable device. For example, high quality can be understood as a high signal-to-noise ratio for the audio signal; it can also be understood as low ambient noise for the audio signal.

[0046] After a wearable device collects human voice audio signals, it can send them to an electronic device. For example, it can send specified data by establishing a pre-defined wireless communication link with the electronic device. It's easy to understand that this specified data can include the human voice audio signal and the target angle. For an example, please refer to [link to example]. Figure 3 , Figure 3 This illustration shows a schematic diagram of the specified data provided in an embodiment of this application. Figure 3 The specified data 300 may include a header 310 and frame data 320. The header 310 may be a type of data pre-agreed by the electronic device and the wearable device to identify the identity of subsequent data following the header 310, and this application does not specifically limit this. The specified data 300 may also include frame data 320, which includes a frame sequence number, data length, data frame, and target angle, etc. In some embodiments, after the wearable device collects the user's voice audio signal, it can encode the voice audio signal to obtain at least a portion of the frame data consisting of a frame sequence number, data length, and data frame. Then, the target angle collected by the wearable device is encoded after the portion of frame data used to identify the voice audio signal to obtain frame data 320.

[0047] After acquiring the specified data, the electronic device can decode the specified data to obtain the human voice audio signal and the target angle, and then determine the angle change parameters of the target user's head based on the target angle.

[0048] In other embodiments, the human voice audio signal and the target angle can also be transmitted to the electronic device through a communication link, and the embodiments of this application do not specifically limit this.

[0049] Optionally, after decoding and obtaining the target angle, the electronic device can temporarily store the target angle in a buffer, so that the target angle can be retrieved directly from the buffer to calculate the angle change parameters.

[0050] Step S130: Determine the position information of the target user in the video frame based on the angle change parameters.

[0051] Furthermore, it's easy to understand that the positional distribution of the sound source corresponding to the audio signal in the sound field of an audio / video file generally needs to match the position of the sound source emitting that audio signal in the image frame of the same audio / video file. If they don't match, it will degrade the user's viewing experience. The user's head rotation causes a change in the head angle transformation parameters, resulting in a mismatch between the positional distribution of the sound source corresponding to the acquired human voice audio signal in the sound field and the position of the target user in the video frame. Therefore, the target user's position information in the video frame can be determined based on the aforementioned angle change parameters, and then the human voice audio signal can be further corrected using this position information. For details, please refer to the subsequent steps.

[0052] In some implementations, location information may specifically include location coordinates and the user's orientation. The location coordinates characterize the target user's specific position within a coordinate system, and the electronic device's coordinates within that system can be considered known quantities. Therefore, the electronic device's location coordinates can be determined based on the relative distance between the target user and the electronic device. For example, the electronic device can determine the relative distance between the target user and the electronic device by capturing video footage, thereby determining the electronic device's location coordinates.

[0053] The coordinate system can be the world coordinate system or a coordinate system established with the electronic device as the origin. This application does not impose any specific limitations on the embodiments.

[0054] It's easy to understand that if the target user's orientation changes in the video frame, the audio signal should change accordingly. For example, if the target user turns their head to the left in the video frame, the audio signal should have a higher volume on the left and a lower volume on the right. This allows us to determine the target user's orientation in the video frame. Specifically, the user's orientation can be determined using angle change parameters in a coordinate system. For example, this coordinate system could be a world coordinate system with x, y, and z axes, where the x, y, and z axes are mutually perpendicular. A reference direction can be predetermined, and the user's orientation can be determined based on the angle change parameters within that reference direction. The reference direction could be the direction from the target user's location in the world coordinate system to the location of the electronic device.

[0055] Step S140: Determine the location distribution of the target sound source in the sound field where the electronic device is located based on the location information, wherein the target sound source is the equivalent sound source of the target user.

[0056] Furthermore, after obtaining the location information, the positional distribution of the target sound source within the sound field of the electronic device can be determined. The target sound source can be a signal source emitting sound within this sound field, which is the sound field where the electronic device is located. Since the positional distribution of the target sound source within the sound field of the electronic device is determined using the aforementioned location information, the human voice audio signal can be subsequently corrected based on this positional distribution. This ensures that the positional distribution of the sound source corresponding to the corrected human voice audio signal within the sound field of the electronic device matches the location of the target user in the video frame.

[0057] The location distribution of the target sound source in the sound field can include the sound source's location and the direction of sound emission within the sound field. Specifically, the sound source location can be determined using the location distribution information above, and the direction of sound emission can be determined using the user's orientation information above.

[0058] The target sound source is the equivalent sound source of the target user. It should be noted that the target sound source can be a virtual sound source, i.e., a virtual equivalent sound source, which can be used to simulate the position distribution of the target user in the sound field. In other words, to determine the position distribution, a virtual sound source can be created such that its position distribution in the sound field of the electronic device conforms to this distribution. The target sound source can be a specific device that cannot emit sound. For example, the target sound source can be a wearable device worn by the target user. When the wearable device is a wireless headset, the target sound source can be the wireless headset, which then serves as a virtual equivalent sound source. Thus, the position of the target user can be simulated by the location of the equivalent sound source, and the direction in which the target user's voice is emitted can be simulated by the sound emission of the equivalent sound source. In other embodiments, the target sound source can also be a real device that can emit sound, such as a loudspeaker or horn.

[0059] Step S150: Correct the human voice audio signal based on the location distribution to obtain a corrected audio signal.

[0060] After determining the location distribution, the human voice audio signal can be corrected based on the location distribution to obtain a corrected audio signal. In some embodiments, correction parameters can be generated based on the location distribution, thereby correcting the human voice audio signal using these parameters. For example, a target impulse response parameter corresponding to the location distribution can be found using a pre-acquired lookup table of impulse response parameters and location distributions, and used as the correction parameter. This target impulse response parameter can be a head-related transfer function (HRTF), as detailed in subsequent embodiments.

[0061] In some embodiments, the human voice audio signal can be a stereo audio signal, thus the human voice audio signal may include a first sub-signal and a second sub-signal. The first sub-signal and the second sub-signal can then be corrected based on a correction parameter distribution obtained from the positional distribution to obtain a corrected audio signal. In other embodiments, the human voice audio signal can also be a multi-channel signal, such as a six-channel audio signal. At least some channels of the six-channel audio signal can then be corrected using a correction parameter distribution obtained from the positional distribution to obtain a corrected audio signal. The six-channel audio signal can be a 5.1-channel audio signal. Detailed descriptions can be found in subsequent embodiments.

[0062] Optionally, after obtaining the corrected audio signal, the corrected audio signal can be encoded and combined with the time-frequency image to obtain an audio-visual signal, which can then be used as a signal recorded in response to the aforementioned audio-visual recording request.

[0063] The audio signal correction method provided in this application first collects the angle change parameters of the target user's head based on a wearable device; then, it collects the voice emitted by the target user as the voice audio signal; next, it determines the target user's position information in the video frame based on the angle change parameters, thereby determining the position distribution of the target sound source in the sound field of the electronic device based on the position information; finally, it corrects the voice audio signal based on the position distribution to obtain a corrected audio signal. Since the target user's head may rotate, causing a change in head angle, the position distribution of the collected user's voice audio signal in the sound field of the corresponding sound source of the electronic device will not match the position of the target user in the video frame. Therefore, in this embodiment, the position distribution for correcting the voice audio signal is determined by the angle change parameters obtained from the target user's head movement, which determine the target user's position information in the video frame. By correcting the voice audio signal through position distribution, the matching degree between the position distribution of the corrected audio signal's sound source in the sound field of the electronic device and the position of the target user in the video frame can be improved.

[0064] Please see Figure 4 , Figure 4 This illustration shows a flowchart of an audio signal correction method provided in an embodiment of this application. This audio signal correction method can be applied to… Figure 1 In the electronic device 110 shown, the processor in the electronic device 110 can be used as the execution subject. The audio signal correction method specifically includes steps S210 to S2110.

[0065] Step S210: In response to the audio / video recording request, the linear velocity of the target user's head in a first specified direction is acquired based on the inertial measurement component.

[0066] Step S220: Based on the inertial measurement unit, acquire the Coriolis force on the head of the target user in the second specified direction.

[0067] Step S230: Determine the target angle of the target user's head around the third specified direction based on the linear velocity and the Coriolis force, wherein the first specified direction, the second specified direction and the third specified direction are perpendicular to each other.

[0068] As described above, after receiving an audio / video recording request, the electronic device can collect the angle change parameters of the target user's head based on the wearable device.

[0069] The gyroscope in the inertial measurement unit of the wearable device can calculate the angular velocity by measuring the linear velocity and the Coriolis force, and then calculate the target angle by the angular velocity. Based on the difference between the target angle and the pre-acquired standard angle, the angle change parameter is determined.

[0070] For example, when the target user's head rotates, such as when the target user's head rotates around a third specified direction, the inertial measurement unit configured in the wearable device can collect the linear velocity of the target user in the first specified direction and the Coriolis force in the second specified direction, thereby determining the target angle of the target user's head around the third specified direction, wherein the first specified direction, the second specified direction and the third specified direction are perpendicular to each other.

[0071] Please see Figure 5 , Figure 5 A schematic diagram illustrating the calculation of angular velocity provided in an embodiment of this application is shown. Figure 5 The system includes a measured object 510, a first specified direction 520, a second specified direction 530, and a third specified direction 540. The first specified direction 520 can be the direction of the linear velocity of the measured object 510, the second specified direction 530 is the direction of the Coriolis force acting on the measured object 510, and the third specified direction 540 is the direction around which the measured object 510 rotates.

[0072] It is easy to understand that the derivative of the target angle in which the target user's head rotates around the third specified direction with respect to time is the angular velocity obtained above. Therefore, in order to obtain the target angle in which the target user's head rotates around the third specified direction, the angular velocity can be integrated.

[0073] Specifically, step S230 may also include steps S231 and S232.

[0074] Step S231: Determine the angular velocity of the target user's head as it travels in the third specified direction based on the linear velocity and the Coriolis force.

[0075] Step S232: Integrate the angular velocity to obtain the target angle at which the target user's head orbits in the third specified direction.

[0076] The relationship between the Coriolis force, linear velocity, and angular velocity can be characterized by the following equation (1):

[0077] a=2ν×ω (1)

[0078] Here, 'a' can be used to characterize the Coriolis force, 'ν' to characterize the linear velocity, and 'ω' to characterize the angular velocity.

[0079] Therefore, after obtaining the linear velocity of the target user's head in the first specified direction and the Coriolis force on the target user's head in the second specified direction through the gyroscope in the inertial measurement unit, the angular velocity can be obtained through the above equation (1).

[0080] Furthermore, the angular velocity can be integrated to obtain the target angle at which the target user's head orbits in the third specified direction. Specifically, the target angle can be obtained by calculating the definite integral of the angular velocity with respect to time.

[0081] The relationship between the third specified direction and the target angle around which the target user's head rotates can conform to the right-hand rule. That is, the third specified direction can be the direction pointed by the right thumb, and thus the direction pointed by the other fingers of the right hand when the fist is clenched is the direction of rotation of the target user's head.

[0082] Step S240: Determine the angle change parameter based on the difference between the target angle and the pre-acquired standard angle.

[0083] Furthermore, the target angle can be sent to the electronic device via a pre-established communication link between the wearable device and the electronic device. The target angle collected by the wearable device can be understood as an angle value corresponding to the current position of the user's head. To obtain the angle change parameter, it is also necessary to compare the difference between the target angle and a pre-acquired standard angle. The standard angle can be pre-set; for example, it can be the current angle of the target user's head when an audio / video recording request is detected, which is then used as the standard angle. The angle change parameter can then be determined based on the difference between the target angle and the standard angle; for example, the difference between the target angle and the standard angle can be used as the angle change parameter. For instance, the current angle of the target user's head when an audio / video recording request is detected can be determined as a rotation of 0 degrees around a third specified direction.

[0084] Furthermore, the determination of the target angle and the determination of the angle change parameters using the target angle and the standard angle can be performed within a coordinate system. This coordinate system can be established based on the wearable device as a reference point, for example, by using the reference point as the origin. Alternatively, it can be a world coordinate system, allowing the angle change parameters to be transformed into the world coordinate system after being obtained in the coordinate system based on the wearable device as a reference point. The world coordinate system can be considered an absolute coordinate system.

[0085] For example, coordinate system transformation can be achieved using the following equation (2):

[0086]

[0087] Among them, w x w y and w z These are used to represent the values ​​of the x-axis, y-axis, and z-axis in the world coordinate system, respectively. θ and γ represent the angle values ​​in the coordinate system established with the wearable device as a reference point. as well as These are used to represent the angle values ​​in the world coordinate system. Mathematical analysis shows that when θ and γ are small, the transformation obtained through the above matrix operations yields equation (3):

[0088]

[0089] This allows us to obtain the angle variation parameters in the world coordinate system.

[0090] After acquiring the angle change parameters, the wearable device can transmit these parameters to the electronic device via a pre-established communication link. Optionally, since in practical applications, the target user's head typically only rotates in a specific direction, such as the z-axis in the world coordinate system, when the wearable device sends the angle change parameters to the electronic device, it can extract only the angle change parameters in that specific direction for transmission. This reduces the amount of data that needs to be transmitted, saves energy, and does not decrease the accuracy of the angle change parameters of the target user's head.

[0091] Step S250: Collect the voice emitted by the target user as a human voice audio signal.

[0092] The specific method for collecting the voice emitted by the target user as a human voice audio signal has been described in detail in the aforementioned embodiments and will not be repeated here.

[0093] It's easy to understand that subsequent adjustments to the human voice audio signal are made based on the angle change parameters of the target user's head. If the frequency of collecting these parameters is too low, the target user's head may have already rotated significantly, but the angle change parameters may not yet reflect this rotation, resulting in poor subsequent correction of the human voice audio signal. Therefore, in the embodiments provided in this application, the first frequency for collecting the angle change parameters of the target user's head using the wearable device and the second frequency for collecting the human voice emitted by the target user can be the same. For example, both the first and second frequencies can be 50Hz. This maximizes the correction effect on the human voice audio signal.

[0094] Step S260: Determine the position information of the target user in the video frame based on the angle change parameters.

[0095] Steps S250 and S260 have been described in detail in the foregoing embodiments and will not be repeated here.

[0096] Step S270: Determine the sound source location of the target sound source in the sound field where the electronic device is located based on the position coordinates.

[0097] Step S280: Determine the sound emission direction of the target sound source in the sound field of the electronic device based on the user's orientation.

[0098] Step S290: Determine the location distribution based on the sound source location and the sound direction.

[0099] The target sound source is the equivalent sound source for the target user. It should be noted that the target sound source can be a virtual sound source, i.e., a virtual equivalent sound source. In other words, to determine the location distribution, a virtual sound source can be created such that its location distribution within the sound field of the electronic device conforms to the specified location. The target sound source can be a concrete device that does not actually emit sound. However, in other embodiments, the target sound source can also be a real device that can emit sound, such as a loudspeaker or horn.

[0100] In some implementations, the location information may include location coordinates and the user's orientation. Specifically, the location coordinates obtained from the aforementioned location information can be used as the sound source location of the target sound source within the sound field of the electronic device. It is easily understood that the direction in which the target user's head is facing is the direction in which the target user emits sound; for example, when the target user speaks, the direction of the sound emitted by the speaking action is the direction in which the target user's head is facing. Therefore, the user's orientation in the location information can also be used as the sound emission direction of the target sound source within the sound field of the electronic device. This allows for further determination of the location distribution based on the sound source location and the sound emission direction.

[0101] The determination of the sound source location and direction can be made in a world coordinate system or in a coordinate system established with the wearable device as a reference point.

[0102] Step S2100: Generate correction parameters based on the location distribution.

[0103] Step S2110: Correct the human voice audio signal based on the correction parameters to obtain a corrected audio signal.

[0104] Correction parameters can be generated based on location distribution, thereby correcting the human voice audio signal. In some implementations, these correction parameters can be head-related transfer functions. These head-related transfer functions are sound effect localization algorithms that utilize techniques such as interaural time delay (ITD), interaural amplitude difference (IAD), and auricular frequency vibration to adjust the audio signal, creating a surround sound effect for the listener when the sound reaches them.

[0105] Specifically, a lookup table of impulse response parameters and position distributions can be obtained in advance. This lookup table may include at least one standard position distribution and the standard impulse response parameters corresponding to the standard position distribution. Therefore, after obtaining the position distribution, the target impulse parameter corresponding to that position distribution can be found in the lookup table as a correction parameter, which can then be used to correct the human voice audio signal.

[0106] In some implementations, the human voice audio signal can be a stereo audio signal, which may include a first sub-signal corresponding to a first channel and a second sub-signal corresponding to a second channel. For example, the first channel may be the left channel, and the second channel may be the right channel; therefore, the human voice audio signal may include a first sub-signal corresponding to the left channel and a second sub-signal corresponding to the right channel. Thus, when correcting the human voice audio signal using correction parameters, the first sub-signal and the second sub-signal can be adjusted separately.

[0107] Specifically, step S2110 may include steps S2111 to S2113.

[0108] Step S2111: Correct the first sub-signal based on the first correction parameter to obtain the first output signal.

[0109] Step S2112: Correct the second sub-signal based on the second correction parameter to obtain the second output signal.

[0110] Step S2113: Based on the first output signal and the second output signal, obtain the corrected audio signal.

[0111] To correct the first sub-signal and the second sub-signal separately, the correction parameters can be target impulse response parameters determined by the position distribution. These target impulse response parameters can include a first correction parameter and a second correction parameter. The first correction parameter corresponds to the first channel and can be used to correct the first sub-signal, while the second correction parameter corresponds to the second channel and can be used to correct the second sub-signal.

[0112] For example, the first and second correction parameters can be head-related impulse responses (HRIRs), which can be seen as the head-related transfer function obtained by convolution, and the head-related transfer function can be seen as the Fourier transform of the head-related impulse response.

[0113] The first and second correction parameters obtained from the head-related impulse response can be considered as a filter. Therefore, by correcting the first sub-signal using the first correction parameter, a filtered first output signal can be obtained; and by correcting the second sub-signal using the second correction parameter, a filtered second output signal can be obtained. Thus, based on the first and second output signals, the corrected audio signal is obtained. For example, the first and second output signals can be directly combined to obtain the corrected audio signal.

[0114] In some implementations, the first sub-signal and the second sub-signal in the human voice audio signal can be convolved with the correction parameters to obtain the first output signal and the second output signal, respectively. Specifically, when performing step S2111, step S2114 may be included: convolving the first sub-signal with the first correction parameter to obtain the first output signal. When performing step S2112, step S2115 may be included: convolving the second sub-signal with the second correction parameter to obtain the second output signal.

[0115] For an example, please refer to Figure 6 , Figure 6 A schematic diagram of the process for obtaining the first output signal and the second output signal provided in an embodiment of this application is shown. Figure 6 In the process, the first sub-signal 610 and the first correction parameter 620 are convolved by 630 to obtain the first output signal 640; similarly, the second sub-signal 650 and the second correction parameter 660 are convolved by 670 to obtain the second output signal 680.

[0116] It should be noted that the first sub-signal corresponding to the first channel, after being corrected, yields the first output signal, which generates the target signal in both the first and second channels. Similarly, the second sub-signal corresponding to the second channel, after being corrected, yields the second output signal, which also generates the target signal in both the first and second channels. For example, if the first channel is the left channel and the second channel is the right channel, and the first sub-signal is the left channel signal and the second sub-signal is the right channel signal, when correcting the left channel signal, it will generate the target signal in both the left and right channels; similarly, when correcting the right channel signal, it will generate the target signal in both the left and right channels.

[0117] Therefore, in some other embodiments, in order to improve the correction effect on the human voice audio signal and further improve the matching degree between the position distribution of the sound source corresponding to the corrected audio signal in the sound field of the electronic device and the position of the target user in the video picture, the first sub-signal can be corrected by the first correction parameter to obtain the output corresponding to the first channel and the second channel respectively, and then the second sub-signal can be corrected by the second correction parameter to obtain the output corresponding to the first channel and the second channel respectively, and then the corrected audio signal can be determined.

[0118] Specifically, the first output signal may include a first sub-output signal and a second sub-output signal. The first correction parameter may also include a first sub-correction parameter and a second sub-correction parameter. The first sub-output signal can be obtained by correcting the first sub-signal corresponding to the first channel using the first sub-correction parameter; and the second sub-output signal can be obtained by correcting the first sub-signal corresponding to the first channel using the second sub-correction parameter. For example, the first sub-output signal can be obtained by convolving the first sub-correction parameter with the first sub-signal; and the second sub-output signal can be obtained by convolving the second sub-correction parameter with the first sub-signal.

[0119] Here, the first sub-output signal corresponds to the output generated in the first channel after the first sub-signal has been corrected by the first correction parameter; similarly, the second sub-output signal corresponds to the output generated in the second channel after the first sub-signal has been corrected by the second correction parameter. For example, the first correction parameter can be represented by `hrir_data_ll`, the second correction parameter by `hrir_data_lr`, the first sub-signal by `dataL`, the first sub-output signal by `outLL`, and the second sub-output signal by `outLR`. Therefore, `outLL` can be obtained by convolving `dataL` with `hrir_data_ll`, and `outLR` can be obtained by convolving `dataL` with `hrir_data_lr`.

[0120] Similarly, the second output signal may also include a third sub-output signal and a fourth sub-output signal, and the second correction parameter may also include a third sub-correction parameter and a fourth sub-correction parameter. Correcting the second sub-signal corresponding to the second channel using the third sub-correction parameter yields the third sub-output signal; similarly, correcting the second sub-signal corresponding to the second channel using the fourth sub-correction parameter yields the fourth sub-output signal. For example, the third sub-output signal can be obtained by convolving the third sub-correction parameter with the second sub-signal; and the fourth sub-output signal can be obtained by convolving the fourth sub-correction parameter with the second sub-signal.

[0121] The third sub-output signal corresponds to the output of the second sub-signal in the first channel after being corrected by the third correction parameter; similarly, the fourth sub-output signal corresponds to the output of the second sub-signal in the second channel after being corrected by the fourth correction parameter. For example, the third correction parameter can be represented by `hrir_data_rl`, the fourth correction parameter by `hrir_data_rr`, the second sub-signal by `dataR`, the third sub-output signal by `outRL`, and the fourth sub-output signal by `outRR`. Therefore, `outRL` can be obtained by convolving `dataR` with `hrir_data_rl`, and `outRR` can be obtained by convolving `dataR` with `hrir_data_rr`.

[0122] Therefore, when obtaining the corrected audio signal based on the first output signal and the second output signal, the corrected audio signal can be determined comprehensively based on the first sub-output signal, the second sub-output signal, the third sub-output signal, and the fourth sub-output signal. Specifically, when performing step S2113, steps S2116 to S21111 may also be included.

[0123] Step S2116: Perform time-frequency transformation on the first sub-output signal, the second sub-output signal, the third sub-output signal, and the fourth sub-output signal respectively to obtain the first sub-frequency domain signal corresponding to the first sub-output signal, the second sub-frequency domain signal corresponding to the second sub-output signal, the third sub-frequency domain signal corresponding to the third sub-output signal, and the fourth sub-frequency domain signal corresponding to the fourth sub-output signal.

[0124] Step S2117: Superimpose the first sub-frequency domain signal with the third sub-frequency domain signal to obtain the first frequency domain output signal.

[0125] Step S2118: Superimpose the second sub-frequency domain signal with the fourth sub-frequency domain signal to obtain the second frequency domain output signal.

[0126] Step S2119: Perform inverse time-frequency transformation on the first frequency domain output signal to obtain the first corrected audio signal corresponding to the first channel.

[0127] Step S21110: Perform inverse time-frequency transformation on the second frequency domain output signal to obtain the second corrected audio signal corresponding to the second channel.

[0128] Step S21111: Obtain the corrected audio signal based on the first corrected audio signal and the second corrected audio signal.

[0129] It's easy to understand that the first, second, third, and fourth sub-output signals can all be time-domain signals, and performing operations like superposition on signals in the time domain is inconvenient. Therefore, we can transform all four sub-output signals to the frequency domain before processing. Specifically, we can perform time-frequency transformations on each of the first, second, third, and fourth sub-output signals to obtain the first sub-frequency domain signal corresponding to the first sub-output signal, the second sub-frequency domain signal corresponding to the second sub-output signal, the third sub-frequency domain signal corresponding to the third sub-output signal, and the fourth sub-frequency domain signal corresponding to the fourth sub-output signal. The time-frequency transformation can be a Fast Fourier Transform (FFT).

[0130] Furthermore, as described above, both the first and third sub-frequency domain signals correspond to the output signals of the first channel. Therefore, the first and third sub-frequency domain signals can be superimposed to obtain the first frequency domain output signal. Similarly, the second and fourth sub-frequency domain signals correspond to the output signals of the second channel, and thus, the second and fourth sub-frequency domain signals can be superimposed to obtain the second frequency domain output signal.

[0131] Since the corrected audio signal is generally a time-domain signal, the first frequency domain output signal can be subjected to an inverse time-frequency transform to obtain the first corrected audio signal corresponding to the first channel; and the second frequency domain output signal can be subjected to an inverse time-frequency transform to obtain the second corrected audio signal corresponding to the second channel. Thus, the corrected audio signal is obtained based on the first and second corrected audio signals. The inverse time-frequency transform can be an inverse fast Fourier transform (IFFT).

[0132] Since the first sub-signal corresponding to the first channel, after being corrected, yields the first output signal, this first output signal will generate the target signal in both the first and second channels; similarly, the second sub-signal corresponding to the second channel, after being corrected, yields the second output signal, which will also generate the target signal in both the first and second channels. The audio signal correction method provided in this application can correct the first and second sub-signals using first and second correction parameters respectively. Then, the resulting first, second, third, and fourth sub-output signals are converted to the frequency domain using time-frequency transformation for processing. Finally, the first and second frequency domain output signals are converted back to the time domain to determine the corrected audio signal. This further improves the matching degree between the position distribution of the sound source corresponding to the corrected audio signal in the sound field of the electronic device and the position of the target user in the video image.

[0133] Please see Figure 7 , Figure 7 This illustration shows a flowchart of an audio signal correction method provided in an embodiment of this application. This audio signal correction method can be applied to… Figure 1 In the electronic device 110 shown, the processor in the electronic device 110 can be used as the execution subject. The audio signal correction method specifically includes steps S310 to S370.

[0134] Step S310: In response to the audio and video recording request, collect the angle change parameters of the target user's head based on the wearable device.

[0135] Step S310 has been described in detail in the foregoing embodiments and will not be repeated here.

[0136] Step S320: Based on the audio signal acquisition component, acquire the voice emitted by the target user to obtain the first acquisition signal corresponding to the left channel of the stereo audio signal and the second acquisition signal corresponding to the right channel of the stereo audio signal.

[0137] Step S330: A six-channel audio signal is obtained by upmixing the first acquisition signal and the second acquisition signal, and is used as the human voice audio signal.

[0138] As described above, wearable devices can be equipped with audio signal acquisition components, allowing them to capture the voice of a target user. After acquiring the user's voice, the wearable device can send it to an electronic device. Furthermore, to enhance the spatial quality of the corrected voice audio signal and improve the user's listening experience, the electronic device can further upmix the user's voice to obtain a multi-channel audio signal, such as a six-channel audio signal. In other words, the user's voice can be upmixed into a six-channel audio signal, which can then be further modified.

[0139] The audio signal acquisition component can acquire the human voice emitted by the target user, which can be a stereo audio signal. Specifically, it can acquire the first acquisition signal corresponding to the first channel of the stereo audio signal and the second acquisition signal corresponding to the second channel of the stereo audio signal, where the first channel can be the left channel and the second channel can be the right channel.

[0140] Furthermore, the six-channel audio signal obtained by upmixing the first acquisition signal and the second acquisition signal as the human voice audio signal may specifically include steps S331 to S338.

[0141] Step S331: Determine a first part of the signal and a second part of the signal based on the first acquired signal and the second acquired signal, wherein the first part of the signal includes the target signal and the second part of the signal includes the ambient sound signal.

[0142] Step S332: Extract the signal corresponding to the left position from the first part of the signal and use it as the front left channel signal.

[0143] Step S333: Extract the signal corresponding to the right position in the first part of the signal as the front right channel signal.

[0144] Step S334: The average value of the first acquired signal and the second acquired signal is used as the center channel signal.

[0145] Step S335: Extract the signal corresponding to the left position from the second part of the signal as the surround left channel signal.

[0146] Step S336: Extract the signal corresponding to the right position from the second part of the signal as the surround right channel signal.

[0147] Step S337: The first acquisition signal and the second acquisition signal are superimposed and then low-pass filtered to obtain the bass channel signal.

[0148] Step S338: Obtain the human voice audio signal based on the front left channel signal, the front right channel signal, the center channel signal, the surround left channel signal, the surround right channel signal, and the bass channel signal.

[0149] The six-channel audio signal can specifically include the front left channel signal, the front right channel signal, the center channel signal, the surround left channel signal, the surround right channel signal, and the bass channel signal.

[0150] First, a first part of the signal and a second part of the signal can be determined based on the first and second acquisition signals. The first part of the signal includes a target signal, and the second part of the signal includes an ambient sound signal. The target signal can be determined based on the sounds emitted by the target user, such as sounds emitted by the target user's actions. For example, if the target user is speaking when the audio signal is acquired by the audio signal acquisition component, the target signal can include a human voice signal; if the target user is playing music on a musical instrument, the target signal can include a musical instrument sound signal; if the target user is typing on a keyboard, the target signal can also include keyboard typing sounds; if the target user is exercising, the target signal can also include the target user's breathing sounds. The ambient sound signal included in the second part of the signal can be determined based on sounds other than those emitted by the target user. For example, it can be white noise in the environment, noise from heavy equipment operating around the target user, or sounds from vehicles, etc.

[0151] Specifically, the first part of the signal and the second part of the signal can be determined by the following equations (4) and (5):

[0152]

[0153]

[0154] Where, x L (n) is used to characterize the first acquired signal, x R (n) is used to characterize the second acquired signal, s i (n) is used to characterize the first part of the signal. Used to characterize the weight value, a L (n) is used to characterize the portion corresponding to the left position in the second part of the signal, a R (n) is used to characterize the part corresponding to the right position in the second part of the signal.

[0155] Therefore, after acquiring the first and second acquisition signals, the first and second part signals can be determined based on equations (4) and (5) above. In some embodiments, the second part signal can be composed of the left-hand portion and the right-hand portion of the second part signal.

[0156] Furthermore, in a six-channel audio signal, the front channel signal is generally more important than the surround channel signal. The front channel signal includes the front left channel signal, the front right channel signal, and the center channel signal, while the surround channel signal includes the surround left channel signal and the surround right channel signal. Therefore, the front left channel signal and the front right channel signal can be determined using the first part of the signal. Specifically, the signal corresponding to the left position in the first part of the signal can be extracted as the front left channel signal; the signal corresponding to the right position in the first part of the signal can be extracted as the front right channel signal. For example, extracting the signal corresponding to the left position in the first part of the signal can be done by decorrelation of the signal corresponding to the right position in the first part of the signal, for example, through filtering, to obtain the front left channel signal. Similarly, extracting the signal corresponding to the right position in the first part of the signal can be done by decorrelation of the signal corresponding to the left position in the first part of the signal, for example, through filtering, to obtain the front right channel signal.

[0157] In some embodiments, the center channel signal can be determined directly from the first and second acquisition signals. For example, the average value of the first and second acquisition signals can be used as the center channel signal. In other embodiments, the signal corresponding to human voice in the first part of the signal can be extracted as the center channel signal.

[0158] Furthermore, since the second part of the signal can include ambient sound signals, and the surround channels in the six-channel signal can be primarily used to characterize ambient sound signals, the signal corresponding to the left position in the second part of the signal can be extracted as the surround left channel signal; and the signal corresponding to the right position in the second part of the signal can be extracted as the surround right channel signal. Similar to the extraction of the signal corresponding to the left position in the first part of the signal, the signal corresponding to the right position in the second part of the signal can be decorrelated, for example, through filtering, to obtain the surround left channel signal. Extracting the signal corresponding to the right position in the second part of the signal can also be achieved by decorrelating the signal corresponding to the left position in the second part of the signal, for example, through filtering, to obtain the surround right channel signal.

[0159] Furthermore, the bass channel signal in the six-channel audio signal can be used to characterize the lower frequency components of the front channel signal and the surround channel signal. Therefore, in some embodiments, the first and second acquired signals can be directly superimposed and then low-pass filtered to obtain the bass channel signal. For example, the low-pass filter threshold can be set to 125Hz, then the bass channel signal can include the signal portion corresponding to frequencies below 125Hz after superimposing the first and second acquired signals. It is easy to understand that the low-pass filter threshold can also be set to other values ​​as needed, such as 100Hz, 80Hz, etc. The specific value selection for the low-pass filter is not specifically limited in the embodiments of this application.

[0160] Therefore, a six-channel audio signal can be obtained based on the front left channel signal, the front right channel signal, the center channel signal, the surround left channel signal, the surround right channel signal, and the bass channel signal, that is, the human voice audio signal represented by the six-channel audio signal is obtained by upmixing.

[0161] Step S340: Determine the position information of the target user in the video frame based on the angle change parameters.

[0162] Step S350: Determine the location distribution of the target sound source in the sound field where the electronic device is located based on the location information, wherein the target sound source is the equivalent sound source of the target user.

[0163] Steps S340 and S350 have been described in detail in the foregoing embodiments and will not be repeated here.

[0164] Step S360: Based on the position distribution, the front left channel signal and the front right channel signal in the six-channel audio signal are corrected to obtain the corrected six-channel audio signal, wherein the front left channel signal and the front right channel signal are determined based on the target signal.

[0165] The six-channel audio signal is corrected by adjusting its position distribution to obtain a corrected six-channel audio signal. This correction can be applied to the front left and front right channels of the six-channel audio signal. Correction parameters can be generated through the position distribution; for example, the correction parameters can be target impulse response parameters. The front left and front right channels of the six-channel audio signal are then corrected based on these target impulse response parameters.

[0166] Similar to the correction of human voice audio signals using target impulse response parameters described in the previous embodiments, the target impulse response parameters may include a first correction parameter corresponding to the first channel and a second correction parameter corresponding to the second channel, where the first channel can be the left channel and the second channel can be the right channel. Thus, the first left channel can be corrected based on the first correction parameter to obtain a first output signal; the first right channel signal can be corrected based on the second correction parameter to obtain a second output signal. Then, based on the corrected first and second output signals, as well as the corrected center channel signal, surround left channel signal, surround right channel signal, and bass channel signal, the corrected six-channel audio signal is determined.

[0167] The specific method for correcting the front left channel signal using the first correction parameter to obtain the first output signal can be found in the previous embodiment regarding the correction of the first sub-signal based on the first correction parameter to obtain the first output signal; the specific method for correcting the front right channel signal using the second correction parameter to obtain the second output signal can be found in the previous embodiment regarding the correction of the second sub-signal based on the second correction parameter to obtain the second output signal, and will not be repeated here.

[0168] Step S370: Downmix the corrected six-channel audio signal to obtain a stereo audio signal, which is used as the corrected audio signal.

[0169] Because stereo audio signals have better compatibility when played back through a backend device, incompatibility may occur if six-channel audio signals are played back directly through the backend device. Therefore, to improve the compatibility of the backend device when playing back audio signals, after obtaining the corrected six-channel audio signal, it can be downmixed to obtain a stereo audio signal, which is then used as the corrected audio signal. Thus, the corrected audio signal played back by the backend device is a stereo audio signal, improving playback compatibility. The backend device can be any device capable of playing back audio signals, such as the electronic device in this embodiment, which can directly play back the corrected audio signal after generating it. Alternatively, it can be another electronic device; the electronic device in this embodiment can transmit the corrected audio signal to another electronic device after generating it, allowing the other electronic device to play back the corrected audio signal.

[0170] In some implementations, please refer to Figure 8 , Figure 8This diagram illustrates the process of obtaining a stereo audio signal through downmixing according to an embodiment of this application. The center channel signal and the bass channel signal can be added together to obtain a first intermediate signal; the front left channel signal and the front right channel signal can be added together to obtain a second intermediate signal; the front left channel signal and the front right channel signal can be subtracted to obtain a third intermediate signal; the surround left channel signal and the surround right channel signal can be added together to obtain a fourth intermediate signal; and the surround left channel signal and the surround right channel signal can be subtracted to obtain a fifth intermediate signal.

[0171] Furthermore, adding the first and second intermediate signals yields the sixth intermediate signal. Convolving the sixth intermediate signal with this signal yields the seventh intermediate signal. Convolving the third intermediate signal yields the tenth intermediate signal, the fourth intermediate signal yields the ninth intermediate signal, and the fifth intermediate signal yields the eleventh intermediate signal. Adding the seventh and ninth intermediate signals yields the eighth intermediate signal, and adding the tenth and eleventh intermediate signals yields the twelfth intermediate signal. Adding the eighth and twelfth intermediate signals yields the first sub-signal, and subtracting them yields the second sub-signal. The first sub-signal can then be used as the left channel signal of the stereo audio signal, and the second sub-signal can be used as the right channel signal.

[0172] Optionally, since the center channel signal may include a significant amount of vocal signal, and the first intermediate signal is obtained by adding the center channel signal and the bass channel signal, in order to highlight the vocal signal in the downmixed stereo audio signal, the loudness of the first intermediate signal can be increased, for example, by 3dB, before adding the first and second intermediate signals. Here, "including a significant amount of vocal signal" can be understood as the center channel signal including an audio signal with a significant number of vocal frequency components.

[0173] Optionally, the surround left channel signal and surround right channel signal can be directly passed through and then added to obtain the fourth intermediate signal; the surround left channel signal and surround right channel signal can be passed through and then subtracted to obtain the fifth intermediate signal. Alternatively, the surround left channel signal and surround right channel signal can be delayed, decorrelated, and then added to obtain the fourth intermediate signal; the surround left channel signal and surround right channel signal can be delayed, decorrelated, and then subtracted to obtain the fifth intermediate signal.

[0174] Optionally, to prevent the signal amplitude from being too large when the eighth intermediate signal is added to the twelfth intermediate signal to obtain the first sub-signal, and when the eighth intermediate signal is subtracted from the twelfth intermediate signal to obtain the second sub-signal, the eighth intermediate signal and the twelfth intermediate signal can be added together and then attenuated to obtain the first sub-signal; similarly, the eighth intermediate signal and the twelfth intermediate signal can be subtracted together and then attenuated to obtain the second sub-signal. For example, the attenuation can be 6dB.

[0175] Because directly correcting the stereo audio signal acquired by the wearable device results in a weak spatial sense in the corrected voice audio signal, the embodiment provided in this application can further upmix the voice emitted by the target user to obtain a six-channel audio signal. Then, based on the positional distribution, the front left and front right channel signals of the six-channel audio signal are corrected to obtain a corrected six-channel audio signal. This corrected six-channel audio signal is then downmixed to obtain a stereo audio signal, which serves as the corrected audio signal. This improves the spatial sense of the corrected audio signal. Furthermore, since the corrected six-channel audio signal is further downmixed to obtain a stereo audio signal after obtaining the corrected six-channel audio signal, the compatibility when the backend device plays back the corrected audio signal can be improved.

[0176] Please see Figure 9 , Figure 9 The diagram shows a structural block diagram of an audio signal correction device 900 provided in an embodiment of this application. The audio signal correction device 900 includes: a first acquisition unit 910, a second acquisition unit 920, a positioning unit 930, a position distribution determination unit 940, and a correction unit 950.

[0177] The first acquisition unit 910 is used to acquire the angle change parameters of the target user's head based on the wearable device in response to an audio and video recording request.

[0178] Optionally, the first acquisition unit 910 can also be used to acquire the linear velocity of the target user's head in a first specified direction based on the inertial measurement component; acquire the Coriolis force on the target user's head in a second specified direction based on the inertial measurement component; determine the target angle of the target user's head around a third specified direction based on the linear velocity and the Coriolis force, wherein the first specified direction, the second specified direction, and the third specified direction are perpendicular to each other; and determine the angle change parameter based on the difference between the target angle and the pre-acquired standard angle.

[0179] Optionally, the first acquisition unit 910 can also be used to determine the angular velocity of the target user's head orbiting in the third specified direction based on the linear velocity and the Coriolis force; and to integrate the angular velocity to obtain the target angle of the target user's head orbiting in the third specified direction.

[0180] The second acquisition unit 920 is used to acquire the voice emitted by the target user as a human voice audio signal. The first frequency based on the head angle change parameter acquired by the wearable device is the same as the second frequency of the acquired voice.

[0181] Optionally, the second acquisition unit 920 can also be used to acquire the voice emitted by the target user based on the audio signal acquisition component, to obtain a first acquisition signal corresponding to the left channel of the stereo audio signal and a second acquisition signal corresponding to the right channel of the stereo audio signal; and to obtain a six-channel audio signal based on the first acquisition signal and the second acquisition signal as the voice audio signal.

[0182] Optionally, the second acquisition unit 920 can also be used to determine a first part signal and a second part signal based on the first acquisition signal and the second acquisition signal, wherein the first part signal includes a target signal and the second part signal includes an ambient sound signal; extract the signal corresponding to the left position in the first part signal as the front left channel signal; extract the signal corresponding to the right position in the first part signal as the front right channel signal; take the average value of the first acquisition signal and the second acquisition signal as the center channel signal; extract the signal corresponding to the left position in the second part signal as the surround left channel signal; extract the signal corresponding to the right position in the second part signal as the surround right channel signal; superimpose the first acquisition signal and the second acquisition signal and perform low-pass filtering to obtain a bass channel signal; and obtain the human voice audio signal based on the front left channel signal, the front right channel signal, the center channel signal, the surround left channel signal, the surround right channel signal, and the bass channel signal.

[0183] The positioning unit 930 is used to determine the position information of the target user in the video frame based on the angle change parameters.

[0184] The location distribution determination unit 940 is used to determine the location distribution of the target sound source in the sound field where the electronic device is located based on the location information, wherein the target sound source is the equivalent sound source of the target user.

[0185] The correction unit 950 is used to correct the human voice audio signal based on the location distribution to obtain a corrected audio signal.

[0186] Optionally, the correction unit 950 can also be used to generate correction parameters based on the location distribution; and to correct the human voice audio signal based on the correction parameters to obtain the corrected audio signal.

[0187] Optionally, the correction unit 950 can also be used to correct the first sub-signal based on the first correction parameter to obtain a first output signal; correct the second sub-signal based on the second correction parameter to obtain a second output signal; and obtain the corrected audio signal based on the first output signal and the second output signal.

[0188] Optionally, the correction unit 950 can also be used to convolve the first sub-signal with the first correction parameter to obtain a first output signal; and to convolve the second sub-signal with the second correction parameter to obtain a second output signal.

[0189] Optionally, the correction unit 950 can also be used to perform time-frequency transformation on the first sub-output signal, the second sub-output signal, the third sub-output signal, and the fourth sub-output signal respectively to obtain a first sub-frequency domain signal corresponding to the first sub-output signal, a second sub-frequency domain signal corresponding to the second sub-output signal, a third sub-frequency domain signal corresponding to the third sub-output signal, and a fourth sub-frequency domain signal corresponding to the fourth sub-output signal; superimpose the first sub-frequency domain signal and the third sub-frequency domain signal to obtain a first frequency domain output signal; superimpose the second sub-frequency domain signal and the fourth sub-frequency domain signal to obtain a second frequency domain output signal; perform an inverse time-frequency transformation on the first frequency domain output signal to obtain a first corrected audio signal corresponding to the first channel; perform an inverse time-frequency transformation on the second frequency domain output signal to obtain a second corrected audio signal corresponding to the second channel; and obtain the corrected audio signal based on the first corrected audio signal and the second corrected audio signal. The time-frequency transformation includes a Fast Fourier Transform, and the inverse time-frequency transformation includes an Inverse Fast Fourier Transform.

[0190] Optionally, the correction unit 950 can also be used to look up the target impulse response parameter corresponding to the position distribution in a pre-acquired lookup table of impulse response parameters and position distributions, based on the position distribution, as the correction parameter, wherein the lookup table of impulse response parameters and position distributions includes at least one standard position distribution and a standard impulse response parameter corresponding to the standard position distribution.

[0191] Optionally, the correction unit 950 can also be used to correct the front left channel signal and the front right channel signal in the six-channel audio signal based on the position distribution to obtain a corrected six-channel audio signal, wherein the front left channel signal and the front right channel signal are determined based on the target signal; and the corrected six-channel audio signal is downmixed to obtain a stereo audio signal as the corrected audio signal.

[0192] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0193] In the several embodiments provided in this application, the coupling between the units can be electrical, mechanical, or other forms of coupling. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] Please see Figure 10 , Figure 10 This diagram illustrates a structural block diagram of an electronic device 110 provided in an embodiment of this application. The electronic device 110 may be a smartphone, laptop computer, desktop computer, tablet computer, etc. The electronic device 110 in this application may include one or more of the following components: a processor 111, a memory 112, and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more programs are configured to execute the methods described in the foregoing embodiments of the audio signal correction method.

[0195] Processor 111 may include one or more processing cores. Processor 111 connects to various parts within the electronic device 110 using various interfaces and lines, and performs various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets, or instruction sets stored in memory 112, and by calling data stored in memory 112. Optionally, processor 111 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 111 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and computer programs; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 111 and may be implemented using a separate communication chip. Specifically, the methods described in the foregoing embodiments can be executed by one or more processors 111.

[0196] In some implementations, memory 112 may include random access memory (RAM) or read-only memory (ROM). Memory 112 can be used to store instructions, programs, code, code sets, or instruction sets. Memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device 110 during use.

[0197] Optionally, the electronic device may also include an image acquisition module. Figure 10 (Not shown in the image), this image acquisition module can capture video footage. This image acquisition module can be a camera.

[0198] Please see Figure 11 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 1100 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0199] The computer-readable storage medium 1100 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has storage space for program code 1110 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 1110 may, for example, be compressed in a suitable form.

[0200] Please refer to Figure 12 The diagram illustrates a structural block diagram 1200 of a computer program product provided in an embodiment of this application. The computer program product 1200 includes a computer program / instructions 1210, which, when executed by a processor, implements the steps of the aforementioned method.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An audio signal correction method, characterized in that, Applied to electronic devices, the method includes: In response to an audio / video recording request, the inertial measurement unit configured on the wearable device worn by the target user collects the angle change parameters of the target user's head. The wearable device collects the voice emitted by the target user as a human voice audio signal. The position information of the target user in the video frame is determined based on the angle change parameters, wherein the position information includes the position coordinates of the target user in the video frame and the user's orientation; The location distribution of the target sound source in the sound field where the electronic device is located is determined based on the location information, wherein the target sound source is the equivalent sound source of the target user, and the location distribution includes the sound source position of the target sound source in the sound field determined based on the location coordinates, and the sound emission direction of the target sound source in the sound field determined based on the user's orientation; The human voice audio signal is corrected based on the location distribution to obtain a corrected audio signal.

2. The method according to claim 1, characterized in that, The step of correcting the human voice audio signal based on the location distribution to obtain a corrected audio signal includes: Correction parameters are generated based on the location distribution; The human voice audio signal is corrected based on the correction parameters to obtain the corrected audio signal.

3. The method according to claim 2, characterized in that, The human voice audio signal includes a first sub-signal corresponding to a first channel and a second sub-signal corresponding to a second channel. The correction parameters include target impulse response parameters, which include a first correction parameter corresponding to the first channel and a second correction parameter corresponding to the second channel. The step of correcting the human voice audio signal based on the correction parameters to obtain the corrected audio signal includes: The first sub-signal is corrected based on the first correction parameter to obtain the first output signal; The second sub-signal is corrected based on the second correction parameter to obtain the second output signal; The corrected audio signal is obtained based on the first output signal and the second output signal.

4. The method according to claim 3, characterized in that, The step of correcting the first sub-signal based on the first correction parameter to obtain the first output signal includes: The first sub-signal is convolved with the first correction parameter to obtain the first output signal; The step of correcting the second sub-signal based on the second correction parameter to obtain the second output signal includes: The second sub-signal is convolved with the second correction parameter to obtain the second output signal.

5. The method according to claim 3, characterized in that, The first output signal includes a first sub-output signal and a second sub-output signal, the second output signal includes a third sub-output signal and a fourth sub-output signal, and obtaining the corrected audio signal based on the first output signal and the second output signal includes: Time-frequency transformation is performed on the first sub-output signal, the second sub-output signal, the third sub-output signal and the fourth sub-output signal respectively to obtain the first sub-frequency domain signal corresponding to the first sub-output signal, the second sub-frequency domain signal corresponding to the second sub-output signal, the third sub-frequency domain signal corresponding to the third sub-output signal and the fourth sub-frequency domain signal corresponding to the fourth sub-output signal. The first sub-frequency domain signal is superimposed with the third sub-frequency domain signal to obtain the first frequency domain output signal; The second sub-frequency domain signal is superimposed with the fourth sub-frequency domain signal to obtain the second frequency domain output signal; The first frequency domain output signal is subjected to inverse time-frequency transformation to obtain the first corrected audio signal corresponding to the first channel; The second frequency domain output signal is subjected to inverse time-frequency transformation to obtain the second corrected audio signal corresponding to the second channel; The corrected audio signal is obtained based on the first corrected audio signal and the second corrected audio signal.

6. The method according to claim 2, characterized in that, The generation of correction parameters based on the location distribution includes: Based on the location distribution, a target impulse response parameter corresponding to the location distribution is found in a pre-acquired lookup table of impulse response parameters and location distributions, and used as the correction parameter. The lookup table of impulse response parameters and location distributions includes at least one standard location distribution and a standard impulse response parameter corresponding to the standard location distribution.

7. The method according to claim 1, characterized in that, Determining the location distribution of the target sound source within the sound field of the electronic device based on the location information includes: The location of the target sound source in the sound field where the electronic device is located is determined based on the location coordinates; The direction of sound emission from the target sound source in the sound field of the electronic device is determined based on the user's orientation; The location distribution is determined based on the location of the sound source and the direction of sound emission.

8. The method according to claim 1, characterized in that, The human voice audio signal includes a six-channel audio signal, which is obtained by upmixing the human voice emitted by the target user acquired by the audio signal acquisition component. The step of correcting the human voice audio signal based on the location distribution to obtain a corrected audio signal includes: Based on the positional distribution, the front left channel signal and the front right channel signal in the six-channel audio signal are corrected to obtain the corrected six-channel audio signal, wherein the front left channel signal and the front right channel signal are determined based on the target signal; The corrected six-channel audio signal is downmixed to obtain a stereo audio signal, which is then used as the corrected audio signal.

9. The method according to claim 1, characterized in that, The process of collecting the voice emitted by the target user as a human voice audio signal includes: Based on the audio signal acquisition component, the target user's voice is acquired to obtain the first acquisition signal corresponding to the left channel of the stereo audio signal and the second acquisition signal corresponding to the right channel of the stereo audio signal. A six-channel audio signal is obtained by mixing the first acquisition signal and the second acquisition signal, which serves as the human voice audio signal.

10. The method according to claim 9, characterized in that, The six-channel audio signal obtained by upmixing the first and second acquisition signals, as the human voice audio signal, includes: Based on the first acquired signal and the second acquired signal, a first part of the signal and a second part of the signal are determined, wherein the first part of the signal includes a target signal and the second part of the signal includes an ambient sound signal; Extract the signal corresponding to the left position from the first part of the signal as the front left channel signal; Extract the signal corresponding to the right position from the first part of the signal as the front right channel signal; The average value of the first and second acquired signals is used as the center channel signal; Extract the signal corresponding to the left position from the second part of the signal as the surround left channel signal; Extract the signal corresponding to the right position from the second part of the signal as the surround right channel signal; The first and second acquired signals are superimposed and then low-pass filtered to obtain the low-frequency channel signal. The human voice audio signal is obtained based on the front left channel signal, the front right channel signal, the center channel signal, the surround left channel signal, the surround right channel signal, and the bass channel signal.

11. The method according to claim 1, characterized in that, The first frequency at which the inertial measurement unit configured based on the wearable device worn by the target user collects the angle change parameters of the target user's head is the same as the second frequency at which the target user's voice is collected.

12. The method according to claim 1, characterized in that, The inertial measurement unit configured based on the wearable device worn by the target user collects the angle change parameters of the target user's head, including: The inertial measurement unit acquires the linear velocity of the target user's head in a first specified direction. The Coriolis force on the target user's head in a second specified direction is acquired based on the inertial measurement unit. The target angle at which the target user’s head orbits in a third specified direction is determined based on the linear velocity and the Coriolis force, wherein the first specified direction, the second specified direction and the third specified direction are perpendicular to each other. The angle change parameter is determined based on the difference between the target angle and the pre-acquired standard angle.

13. The method according to claim 12, characterized in that, Determining the target angle of the target user's head in the third specified direction based on the linear velocity and the Coriolis force includes: The angular velocity of the target user's head orbiting in the third specified direction is determined based on the linear velocity and the Coriolis force. Integrating the angular velocity yields the target angle at which the target user's head orbits in the third specified direction.

14. An audio signal correction device, characterized in that, Applied to electronic devices, the device includes: The first acquisition unit is used to acquire the angle change parameters of the target user's head based on the inertial measurement component configured on the wearable device worn by the target user in response to the audio and video recording request. The second acquisition unit is used to acquire the voice emitted by the target user through the wearable device as a voice audio signal; A positioning unit is used to determine the position information of the target user in the video frame based on the angle change parameters, wherein the position information includes the position coordinates of the target user in the video frame and the user's orientation; A location distribution determination unit is used to determine the location distribution of a target sound source in the sound field where the electronic device is located based on the location information, wherein the target sound source is an equivalent sound source of the target user, and the location distribution includes the sound source position of the target sound source in the sound field determined based on the location coordinates, and the sound emission direction of the target sound source in the sound field determined based on the user's orientation; The correction unit is used to correct the human voice audio signal based on the location distribution to obtain a corrected audio signal.

15. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-13.

17. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method described in any one of claims 1-13.

Images