Audio processing method and apparatus, device, medium, and program product
By processing video data frame by frame to identify sound sources and background audio, and using deep learning networks for spatial rendering, the problems of audio quality and operational complexity in VR videos are solved, and high-quality spatial audio is generated efficiently.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-04-09
AI Technical Summary
In existing technologies, the spatial audio generated for VR videos suffers from excessive noise and poor sound quality. Furthermore, it requires specialized equipment and synchronous processing, making the operation complex and difficult to generate high-quality spatial audio in batches.
By acquiring video data input by the user, the image is processed frame by frame to identify the sound source object and background audio. Features are extracted using a deep learning network, spatial rendering is performed in response to trigger events, and the target audio is generated, avoiding additional equipment and synchronization operations.
It enables the rapid and automated generation of high-quality spatial audio that matches VR videos, reduces operational complexity, and is suitable for a variety of terminal devices.
Smart Images
Figure CN2025093071_09042026_PF_FP_ABST
Abstract
Description
Audio processing methods, apparatus, equipment, media and program products
[0001] Cross-references to related applications
[0002] This disclosure claims priority to Chinese Patent Application No. 2024105649360, filed on May 8, 2024, entitled "Audio Processing Method, Apparatus, Device, Medium and Product", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to the field of spatial audio rendering technology, and in particular to an audio processing method, apparatus, device, medium, and program product. Background Technology
[0004] The essence of spatial audio is "computational audio," which means that spatial audio can simulate a sound source as a speaker device located in a fixed position in space. When the user turns their head or the device moves, the sound can change with the user's movement, allowing the user to experience an immersive surround sound experience.
[0005] With the rise of the Virtual Reality (VR) industry, spatial audio, as a crucial component of VR video, has become particularly important in its development and application within the VR field. Therefore, the ability to quickly generate audio suitable for VR videos is of paramount importance. Summary of the Invention
[0006] This disclosure provides an audio processing method, apparatus, device, medium, and program product that, when the video data is VR video, can automatically generate spatial audio matching the VR video based on the video data input by the user.
[0007] This disclosure provides an audio processing method, the method comprising:
[0008] Obtain the video data input by the user. The video data includes the first image of P frames; where P is a positive integer.
[0009] For the first image of the j-th frame in the first image of P frames, obtain the audio corresponding to the first image of the j-th frame. The audio includes the object audio of the sound source object and the background audio, where j is an integer greater than or equal to 1 and less than or equal to P.
[0010] In response to the received trigger event, the first object position of the sound source object is obtained, and the object audio of the sound source object is spatially rendered based on the first object position of the sound source object to obtain the spatially rendered object audio.
[0011] The target audio is generated based on the object audio and background audio after spatial rendering.
[0012] This disclosure also provides an audio processing apparatus, the audio processing apparatus comprising:
[0013] The acquisition module is configured to acquire video data input by the user, the video data including the first image of P frames; where P is a positive integer;
[0014] The acquisition module is also configured to acquire, for the j-th frame first image in the P-frame first image, the audio corresponding to the j-th frame first image, the audio including the object audio of the sound source object and the background audio, where j is an integer greater than or equal to 1 and less than or equal to P;
[0015] The acquisition module is also configured to, in response to a received trigger event, acquire the first object position of the sound source object, and perform spatial rendering of the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio.
[0016] The generation module is configured to generate target audio based on the spatially rendered object audio and background audio.
[0017] This disclosure also provides an electronic device, including: a memory configured to store computer program instructions; and a processor configured to read and execute the computer program instructions stored in the memory to perform the audio processing method as described above.
[0018] This disclosure also provides a computer storage medium storing computer program instructions, which, when executed by a processor, can implement the audio processing method described above.
[0019] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the audio processing method as described above.
[0020] In this embodiment, user-input video data, including P-frame first images, is acquired. For the j-th frame of the P-frame first images, the object audio and background audio of the sound source object corresponding to the j-th frame are acquired. Then, in response to a received trigger event, the first object position of the sound source object is acquired, and spatial rendering is performed on the object audio of the sound source object based on the first object position to obtain spatially rendered object audio. Finally, a target audio is generated based on the spatially rendered object audio and background audio. This allows for frame-by-frame processing of user-input video data, enabling rapid and automated generation of matching spatial audio without the need for additional audio acquisition equipment or audio-video synchronization, thus reducing operational complexity. In the case of VR video, spatial audio matching the VR video can be automatically generated based on the user-input VR video. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments of this disclosure will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 is a schematic flowchart of an audio processing method provided in an embodiment of this disclosure;
[0023] Figure 2 is a flowchart illustrating another audio processing method provided in an embodiment of this disclosure;
[0024] Figure 3 is a schematic diagram of the structure of a first deep learning network provided in an embodiment of this disclosure;
[0025] Figure 4 is a flowchart illustrating another audio processing method provided in an embodiment of this disclosure;
[0026] Figure 5 is a schematic diagram of a spherical coordinate system provided in an embodiment of this disclosure;
[0027] Figure 6 is a schematic diagram of the coordinate position of a sound source object in a two-dimensional coordinate system in the first image of the j-th frame, according to an embodiment of this disclosure.
[0028] Figure 7 is a schematic diagram of the structure of an audio processing device provided in an embodiment of this disclosure;
[0029] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0030] The features and exemplary embodiments of various aspects of this disclosure will now be described in detail. To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, will provide a further detailed description. It should be understood that the specific embodiments described herein are intended only to explain this disclosure and not to limit it. For those skilled in the art, this disclosure can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this disclosure by illustrating examples.
[0031] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0032] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.
[0033] The essence of spatial audio is "computational audio," which means that spatial audio can simulate a sound source as a speaker device located in a fixed position in space. When the user's head or the device rotates, the sound can change with the user's movement, allowing the user to experience an immersive surround sound experience.
[0034] With the rise of the VR industry, spatial audio, as a crucial component of VR video, has become particularly important in its development and application within the VR field. However, among related technologies, audio data obtained by capturing audio using the built-in recording function of panoramic video recording equipment often suffers from excessive noise, poor sound quality, and unsatisfactory results. Purchasing professional spatial audio capture equipment for recording audio would incur significant manpower, material, and financial costs, and would also require synchronized processing of video and audio, presenting a high technical barrier. In practical applications, there is also a technical solution that uses real-time rendering technology to configure spatial audio for panoramic videos. However, this solution is suitable for game scenarios, and in such scenarios, microphone capture equipment is required, limiting its application and preventing the batch generation of corresponding spatial audio for large amounts of panoramic content.
[0035] To address the aforementioned issues, this disclosure provides an audio processing method, apparatus, device, medium, and program product. It acquires user-input video data, including P-frame first images. For the j-th frame of the P-frame first images, it acquires object audio and background audio of a sound source object corresponding to the j-th frame. Furthermore, upon detecting a trigger event, it acquires positional change information of the sound source object in the j-th frame and, in response to the trigger event, performs spatial rendering of the object audio based on the positional change information. Based on the spatially rendered object audio and background audio, it generates target audio. This allows for the automatic generation of matching spatial audio based on user-input video data without requiring additional audio acquisition equipment. Additionally, since it processes images frame-by-frame in the video data, it eliminates the need for audio and video synchronization, reducing operational complexity. In the case of VR video data, this solution can automatically and flexibly generate spatial audio matching the VR video.
[0036] It should be noted that the audio processing method provided in this disclosure can be executed by an audio processing device or a control module within an audio processing device for executing the audio processing method. This disclosure uses an audio processing device executing the audio processing method as an example to illustrate the audio processing method provided in this disclosure.
[0037] The audio processing method provided in this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] Figure 1 is a schematic flowchart of an audio processing method provided in an embodiment of this disclosure.
[0039] As shown in Figure 1, the entity executing this audio processing method can be an audio processing device. Specifically, the method may include the following steps:
[0040] S110, acquire video data input by the user.
[0041] In some embodiments, the video data described above may include a P-frame first image; where P is a positive integer.
[0042] Additionally, it should be noted that the first image mentioned above can be a panoramic image, and correspondingly, the video data mentioned above can be panoramic video data, without further limitations.
[0043] S120, for the first image of the j-th frame in the first image of the P-frame, obtain the audio corresponding to the first image of the j-th frame.
[0044] In some embodiments, the audio mentioned above may include background audio and object audio of the sound source objects. The number of object audio files can be determined based on the number of sound source objects in the first image of the j-th frame; specifically, the number of object audio files may be less than or equal to the number of sound source objects. Furthermore, j can be an integer greater than or equal to 1 and less than or equal to P. Additionally, the background audio can be determined based on the scene category in the first image of the j-th frame, and is not specifically limited here.
[0045] Specifically, after acquiring the video data input by the user, the audio processing device can acquire the object audio and background audio of the sound source object corresponding to the first image of the j-th frame in the first image of the P-th frame, since the video data may include the first image of the P-th frame.
[0046] S130, in response to the received trigger event, obtain the first object position of the sound source object, and perform spatial rendering of the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio.
[0047] Specifically, after acquiring the audio corresponding to the first image of the j-th frame, if the audio processing device can receive a trigger event, it can then obtain the first object position of the sound source object in response to the received trigger event, and perform spatial rendering of the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio.
[0048] The aforementioned triggering events can be detected through triggering events in three terminal forms. Specifically, if the terminal corresponding to the audio processing device is a mobile phone, the user can adjust the field of view by dragging video data on the touch screen; if the terminal corresponding to the audio processing device is a TV, the user can adjust the field of view by using a large-screen VR virtual remote control; if the terminal corresponding to the audio processing device is a head-mounted display, the user can directly adjust the field of view by rotating their head.
[0049] It should also be noted that the aforementioned sound source object can be a single object or include multiple objects; no specific limitation is made here.
[0050] S140 generates target audio based on the spatially rendered object audio and background audio.
[0051] Specifically, after obtaining the spatially rendered object audio, the audio processing device can generate the target audio corresponding to the first image of the j-th frame based on the spatially rendered object audio and the background audio.
[0052] In this embodiment, user-input video data, including P-frame first images, is acquired. For the j-th frame of the P-frame first images, the object audio and background audio of the sound source object corresponding to the j-th frame are acquired. Then, in response to a received trigger event, the first object position of the sound source object is acquired, and spatial rendering is performed on the object audio of the sound source object based on the first object position to obtain spatially rendered object audio. Based on the spatially rendered object audio and background audio, a target audio is generated. This allows for frame-by-frame processing of user-input video data, enabling rapid and automated generation of matching spatial audio without the need for additional audio acquisition equipment or synchronization of audio and video, thus reducing operational complexity. In the case of VR video, this solution can automatically and flexibly generate spatial audio matching the VR video.
[0053] In one embodiment, when the audio includes object audio of the sound source object, as shown in FIG2, the above-mentioned S120 may specifically include the following steps:
[0054] S210, for the j-th frame of the first image in the P-frame, determine the sound source object in the j-th frame and the object label of the sound source object.
[0055] The aforementioned sound source object can be any object in the first image associated with sound. Correspondingly, the object label can be a waterfall, a television, a person, a car, or a dog, etc., without further limitation. It should also be noted that this embodiment does not specifically limit the number of sound source objects.
[0056] Specifically, after acquiring the video data input by the user, the audio processing device can determine the sound source object and the object label of the sound source object for the j-th frame first image in the P-frame first image included in the video data.
[0057] S220, the object audio that matches the object tag of the sound source object and is selected from the first preset audio library is determined as the object audio of the sound source object.
[0058] The first preset audio library can be an object audio library that is pre-set based on actual experience or circumstances.
[0059] Specifically, after determining the sound source object and the object tag of the sound source object, the audio processing device can filter the object audio that matches the object tag of the sound source object from the first preset audio library, and determine that the object audio is the object audio of the sound source object.
[0060] In this embodiment, when the audio includes the object audio of the sound source object, by determining the sound source object and the object tag of the sound source object in each frame image, the object audio that matches the sound source object in each frame image can be accurately determined from the first preset audio library.
[0061] Based on this, in one embodiment, the above-mentioned S210 may specifically include the following steps:
[0062] For the first image of the j-th frame in the first image of P-frame, the first deep learning network is used to process the first image of the j-th frame and extract the image features of the first image of the j-th frame;
[0063] Based on the image features of the first image in frame j, determine the sound source object in the first image and the object label of the sound source object.
[0064] Specifically, the audio processing device can process the j-th frame of the first image in the P-frame first image using a first deep learning network to extract the image features of the j-th frame first image, and then determine the sound source object in the first image and the object label of the sound source object based on the image features of the j-th frame first image.
[0065] The first deep learning network can be pre-configured based on practical experience or circumstances. For example, the first deep learning network may include a 50-layer structure, without further limitations. In one example, the first deep learning network mentioned above can be as shown in Figure 3. In one example, the first deep learning network can be executed in 6 stages, as follows:
[0066] The first stage first passes the input through a 7×7 convolutional layer, then through a BN layer, a ReLU activation function, and a 3×3 max pooling layer to obtain the output of the first stage;
[0067] The second stage consists of three Bottlenecks (corresponding to conv2 in Figure 3), each containing three convolutional layers: 1×1, 3×3, and 1×1. The first 1×1 convolution is used to reduce dimensionality, the second 3×3 convolution is used to further extract features, and the third 1×1 convolution is used to restore dimensionality.
[0068] The third, fourth, and fifth stages contain 4, 6, and 3 Bottlenecks, respectively, and each Bottleneck contains 3 convolutional layers.
[0069] Finally, the feature map output from the fifth stage of global average pooling is downsampled, and the final output is obtained through a fully connected layer and a softmax function.
[0070] Additionally, it should be noted that the first deep learning network mentioned above needs to be pre-trained using multiple training samples and multiple test samples to accurately identify various sound source objects. Specifically, the multiple training samples can be set to 1000 training samples, and the multiple test samples can include 60 test samples. The various sound source objects can be up to 180 different sound source objects, without further limitation.
[0071] In this embodiment, the image features of each frame can be accurately extracted through the first deep learning network, and then the sound source object and the object label of the sound source object in each frame can be accurately determined based on the extracted image features.
[0072] In one embodiment, when the audio includes background audio, as shown in FIG4, the aforementioned S120 may specifically include the following steps:
[0073] S410, for the first image of the j-th frame in the first image of the P-frame, determine the scene label of the first image of the j-th frame.
[0074] Scene tags can include, but are not limited to, tags used to represent scene categories such as classroom, road, garden, stream, forest, grassland, seaside, city, and cafe.
[0075] Specifically, after acquiring the video data input by the user, the audio processing device can determine the scene label of the first image of the jth frame in the first image of the P-frame included in the video data.
[0076] S420, Based on the scene label of the first image in the j-th frame, determine the background audio of the first image in the j-th frame.
[0077] Specifically, after determining the scene label of the first image in the j-th frame, the audio processing device can determine the background audio of the first image in the j-th frame based on the scene label.
[0078] In this embodiment, when the audio includes background audio, it is possible to accurately determine the background audio that best matches each frame of the image by determining the scene label in each frame.
[0079] Based on this, in one embodiment, the above-mentioned S410 may specifically include the following steps:
[0080] For the first image of the jth frame in the first image of P frames, the first image of the jth frame is divided according to a preset angle to obtain a preset number of second images corresponding to the first image of the jth frame. The preset number of second images are analyzed by the second deep learning network to obtain the scene category of each second image in the preset number of second images.
[0081] The scene label of the first image in frame j is determined based on the scene category of each second image in a preset number of second images.
[0082] The preset angle and preset number can be determined according to the actual situation. For example, if the first image is a panoramic image, the preset angle can be 60°, and the preset number can be 6, that is, a frame of the first image is divided into 6 second images, and each second image can cover the 60° panoramic view of the first image.
[0083] The second deep learning network mentioned above can employ k-Nearest Neighbor (KNN), Support Vector Machine (SVM), Back Propagation (BP) neural network, Convolutional Neural Networks (CNN), and transfer learning, among others.
[0084] In addition, scene categories can include, but are not limited to, classrooms, roads, gardens, streams, forests, grasslands, seasides, cities, cafes, etc.
[0085] Specifically, the audio processing device can divide the j-th frame of the first image in the P-frame first image according to a preset angle to obtain a preset number of second images corresponding to the j-th frame first image. Then, it can use a second deep learning network to parse the preset number of second images to obtain the scene category of each second image in the preset number of second images, and determine the scene label of the j-th frame first image based on the scene category of each second image in the preset number of second images.
[0086] In this embodiment, each frame of the j-th image can be divided, and then the multiple second images obtained through the division can be analyzed and identified to obtain the scene category of each second image in the multiple second images, so that the scene label of the first image of the j-th frame before division can be accurately determined based on the scene category of each second image in the multiple second images.
[0087] Based on this, in one embodiment, if the value of j is 1, the above-mentioned S420 may specifically include the following steps:
[0088] Background audio matching the scene tag of the first image in frame j is obtained from the second preset audio library;
[0089] The background audio that matches the scene label of the first image in frame j is determined as the background audio of the first image in frame j.
[0090] The second preset audio library can be a database for storing background audio that is pre-set based on actual experience or circumstances, and no specific limitation is made here.
[0091] Specifically, after the audio processing device determines the scene label of the first image in the j-th frame, if the value of j is 1, then the first image in the j-th frame can be determined to be the first frame image in the video data input by the user. At this time, the audio processing device can obtain the background audio that matches the scene label of the first image in the first frame from the second preset audio filter, and then determine the background audio that matches the scene label of the first image in the first frame as the background audio of the first image in the first frame.
[0092] In this embodiment, when the first image of the j-th frame is the first frame of video data, the background audio that matches the image in each frame can be accurately determined from the preset audio library.
[0093] In another embodiment, if j is greater than or equal to 2, the above-mentioned S420 may specifically include the following steps:
[0094] Determine whether the scene label of the first image in frame j is consistent with the scene label of the first image in frame (j-1).
[0095] If the scene label of the first image in frame j is inconsistent with the scene label of the first image in frame j-1, the background audio that matches the scene label of the first image in frame j will be selected from the second preset audio library and determined as the background audio of the first image in frame j.
[0096] If the scene label of the first image in frame j is consistent with the scene label of the first image in frame (j-1), then the background audio of the first image in frame (j-1) is determined to be the background audio of the first image in frame j.
[0097] The second preset audio library is a background audio library that is pre-set based on actual experience or circumstances, and no specific limitations are made here.
[0098] Specifically, when j is greater than or equal to 2, i.e., when the first image of frame j is not the first frame in the video data, the audio processing device can determine whether the scene label of the first image of frame j is consistent with the scene label of the first image of frame (j-1). If the scene label of the first image of frame j is inconsistent with the scene label of the first image of frame (j-1), the background audio that matches the scene label of the first image of frame j selected from the second preset audio library can be determined as the background audio of the first image of frame j. If the scene label of the first image of frame j is consistent with the scene label of the first image of frame (j-1), the background audio of the first image of frame (j-1) can be directly determined as the background audio of the first image of frame j.
[0099] In this embodiment, even if the first image in the j-th frame is not the first frame of the video data, the background audio that matches well in each frame can be accurately determined so that the target audio can be accurately generated subsequently.
[0100] In one embodiment, the step of obtaining the first object position of the sound source object in response to a received trigger event may specifically include the following steps:
[0101] In response to the triggering event, obtain posture change information and the position of the second object of the sound source object;
[0102] Based on the posture change information and the second object position of the sound source object, the first object position of the sound source object is determined.
[0103] The attitude change information may include the attitude change angle. This attitude change information can be captured by attitude sensing sensors such as gyroscopes and touch screens, and is used to update the coordinates of the sound source object in a spherical coordinate system. No further limitations are imposed here.
[0104] Furthermore, the second object position of the aforementioned sound source object can be understood as the object position before receiving the trigger event, that is, the object position before the position change. Correspondingly, the first object position of the aforementioned sound source object can be understood as the object position after receiving the trigger event, that is, the object position after the position change. It should also be noted that the object positions involved in the embodiments of this disclosure are all object positions in a spherical coordinate system.
[0105] Specifically, after receiving a trigger event, the audio processing device can, in response to the received trigger event, obtain posture change information and the second object position of the sound source object, and then determine the first object position of the sound source object based on the posture change information and the second object position of the sound source object.
[0106] In one example, the above attitude change information can be represented by the following formula (1): (Δθ, Δφ) (1)
[0107] If the second object position of the sound source object is p i (r,θ i ,φ i The first object position of the sound source object, determined by combining the attitude change information shown in formula (1), can be represented by the following formula (2): p i (r,θ i +Δθ,φ i +Δφ) (2)
[0108] In formula (2), i represents the sound source object, i = 1, 2, 3...
[0109] In this embodiment, the first object position of the sound source object can be accurately determined by the posture change information obtained in response to the received trigger event and the second object position of the sound source object, so that the object audio can be accurately spatially rendered subsequently.
[0110] In one embodiment, the step of obtaining the second object location of the sound source object may specifically include the following steps:
[0111] Obtain parameter information, which includes spatial location information corresponding to the video data, and device parameters of the image acquisition device used to acquire the video data;
[0112] Obtain the coordinate position of the sound source object in the two-dimensional coordinate system of the first image in frame j;
[0113] Map the coordinates to the spherical coordinate system to obtain the second object position of the sound source object.
[0114] The spherical coordinate system is constructed based on spatial location information and equipment parameters.
[0115] For example, the process of obtaining the position of the second object described above applies to all cases where the value of j is greater than or equal to 1; for example, the two-dimensional coordinate system can be constructed based on the first image of the j-th frame.
[0116] In some embodiments, the parameter information mentioned above may include spatial location information corresponding to the video data, which may include longitude coordinates, latitude coordinates, altitude, etc., without further limitation.
[0117] The aforementioned parameter information may also include device parameters of the image acquisition device used to acquire video data. This image acquisition device can be any electronic device with camera functionality, such as a camcorder. These device parameters may include the position, focal length, pixel resolution, and focal length of the image acquisition device.
[0118] Furthermore, the spherical coordinate system mentioned above is constructed based on the aforementioned spatial location information and device parameters. For example, this spherical coordinate system can be as shown in Figure 5.
[0119] Specifically, the audio processing device can acquire parameter information, which may include spatial location information corresponding to the video data, as well as device parameters of the image acquisition device used to acquire the video data. Additionally, the audio processing device can also acquire the coordinate position of the sound source object in the two-dimensional coordinate system where the first image in the j-th frame is located. Based on this, the audio processing device can map the coordinate position to a spherical coordinate system to obtain the second object position of the sound source object.
[0120] In one example, Figure 6 is a schematic diagram of the coordinate position of the sound source object in the two-dimensional coordinate system of the first image in the j-th frame. Based on this, the second object position of the sound source object can be calculated by the following formula, as shown below:
[0121] Where x i y i Let w and h be the coordinates of the i-th sound source object in the two-dimensional image, and let w and h be the width and height of the two-dimensional image.
[0122] In this embodiment, the second object position of the sound source object is accurately obtained by acquiring the coordinate position of the sound source object in the two-dimensional coordinate system where the first image of the j-th frame is located, and mapping the coordinate position to the spherical coordinate system.
[0123] In one embodiment, the step of spatially rendering the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio may specifically include the following steps:
[0124] Based on the preset correspondence between object position and sound intensity, the first object position of the sound source object is matched to obtain the target sound intensity of the sound source object.
[0125] Based on the target sound intensity of the sound source object, spatial rendering of the object audio is performed to obtain the spatially rendered object audio.
[0126] The preset correspondence between object position and sound intensity can be a pre-set correspondence based on actual experience or circumstances, and is not specifically limited here.
[0127] Specifically, after obtaining the first object position of the sound source object, the audio processing device can match the first object position of the sound source object based on the preset correspondence between the object position and the sound intensity to obtain the target sound intensity of the sound source object. Then, it can perform spatial rendering of the object audio of the sound source object based on the target sound intensity of the sound source object to obtain the spatially rendered object audio.
[0128] The rendering principle relies on the difference in sound intensity caused by distance when sound reaches the human ear, thus creating a sense of spatial orientation. Specifically, triggering an event causes a change in the position of a sound source object relative to a fixed reference point in space. This change in position is mapped to a corresponding difference in sound intensity. By adjusting the sound intensity difference of each sound source object relative to the reference point, spatial sound effects are achieved. When a sound source object i (i = 1, 2, 3, ...) moves away from the reference point, its sound intensity decreases; similarly, when a sound source object i (i = 1, 2, 3, ...) moves closer to the reference point, its sound intensity increases. The real-time sound intensity calculation method for sound source object i (i = 1, 2, 3, ...) is as follows:
[0129] The coordinate transformation of the reference point is as follows: The coordinates of the reference point in the spherical coordinate system are p0(r,θ0,φ0), which are transformed into the coordinates of the spatial rectangular coordinate system p0(x0,y0,z0), as shown below: x0=r×cos(θ0)cos(φ0) (5) y0=r×cos(θ0)sin(φ0) (6) z0=r×sin(θ0) (7)
[0130] Where r is the radius of the panoramic video sphere, and φ0 and θ0 are calculated by formulas (3) and (4).
[0131] The coordinate transformation of the sound source object, the coordinates of the sound source object in the spherical coordinate system at time k are: Similarly, by transforming it into three-dimensional rectangular coordinates, we can obtain... The details are as follows:
[0132] The distance between the sound source object i (i = 1, 2, 3...) and the reference point at time k can be calculated using the following formula:
[0133] The coordinate transformation of the sound source object is triggered by the event. The event occurs at time k+l, and the coordinates of the sound source object in the spherical coordinate system are given by formula (2). The corresponding coordinates of the sound source object i (i = 1, 2, 3...) in the spatial rectangular coordinate system are updated as follows:
[0134] The distance between the sound source object and the reference point at time k+l is shown below:
[0135] The sound intensity levels of the sound source object i (i = 1, 2, 3...) relative to the reference point at times k and k+l are respectively:
[0136] Among them, Li0 Let i be the initial distance between the sound source object i (i = 1, 2, 3...) and the reference point. As shown in formulas (11) and (15).
[0137] The sound intensity difference calculation, i.e., the sound intensity difference between the sound source object i (i = 1, 2, 3...) and the reference point at times k and k+l, is shown in the following formula:
[0138] Substituting formulas (16) and (17) into formula (18), we get:
[0139] The sound intensity of the sound source object is updated as follows:
[0140] Initially, the sound intensity of each sound source object i (i = 1, 2, 3...) relative to the reference point is configured by the developer. The sound intensity increases when the sound source is close to the reference point, and decreases when the sound source is far away from the reference point.
[0141] In this embodiment, the first object position of the sound source object can be matched based on the preset correspondence between the object position and the sound intensity to obtain the target sound intensity of the sound source object. Then, the object audio of the sound source object is spatially rendered based on the target sound intensity of the sound source object to accurately obtain the spatially rendered object audio.
[0142] In one embodiment, after spatial rendering of the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio, the audio processing method mentioned above may further include the following steps:
[0143] The object audio after spatial rendering is convolved using an objective function to obtain the object audio after convolution.
[0144] The objective function mentioned above is used to simulate the process of audio entering the human ear. In other words, the objective function can simulate the spatial structural characteristics generated by the spatial structure of the human ear.
[0145] Specifically, the audio processing device can use an objective function to perform convolution processing on the spatially rendered object audio to obtain the convolution-processed object audio.
[0146] In one example, the objective function mentioned above can be represented by the following formula:
[0147] Where f is the convolution function simulating the structure of the human ear, s is the audio stream of the sound source object rendered in the current frame, n is the current sampling time, N is the order of the convolution function, i.e. the filter order, and m is the delay. The process of sound entering the periosteum from the human ear can be described by the convolution function f.
[0148] Based on this, the steps mentioned above for generating target audio from object audio and background audio after spatial audio rendering can specifically include the following steps:
[0149] The target audio is generated based on the object audio and background audio after convolution processing.
[0150] Specifically, after obtaining the object audio after convolution processing, the audio processing device can generate target audio based on the object audio and background audio after convolution processing.
[0151] In this embodiment, a convolutional function can be used to process the spatially rendered object audio to obtain convolutionally processed object audio. Then, based on the convolutionally processed object audio and background audio, a target audio can be generated. Since the aforementioned objective function can simulate the process of audio entering the human ear, it allows users to experience more realistic and immersive three-dimensional spatial sound effects.
[0152] Based on the same inventive concept, this disclosure also provides an audio processing apparatus. The audio processing apparatus provided in this disclosure will be described in detail with reference to FIG7.
[0153] Figure 7 is a schematic diagram of the structure of an audio processing device provided in an embodiment of this disclosure.
[0154] As shown in Figure 7, the audio processing device 700 may include an acquisition module 710 and a generation module 720.
[0155] The acquisition module 710 is configured to acquire video data input by the user, the video data including the first image of P frames; where P is a positive integer;
[0156] The acquisition module 710 is also configured to acquire the audio corresponding to the first image of the jth frame in the first image of the P-frame, the audio including the object audio of the sound source object and the background audio, where j is an integer greater than or equal to 1 and less than or equal to P.
[0157] The acquisition module 710 is also configured to, in response to a received trigger event, acquire the first object position of the sound source object, and perform spatial rendering of the object audio of the sound source object based on the first object position of the sound source object, so as to obtain the spatially rendered object audio.
[0158] The generation module 720 is configured to generate target audio based on the spatially rendered object audio and background audio.
[0159] In one embodiment, the audio processing apparatus may further include a determining module configured to determine, for a first image j in a first image j of a first image p, a sound source object and an object label of the sound source object.
[0160] The determination module is also configured to identify object audio that matches the object tag of the sound source object and is selected from the first preset audio library as the object audio of the sound source object.
[0161] In one embodiment, the audio processing apparatus may further include a processing module configured to process the j-th frame first image in the P-frame first image using a first deep learning network to extract image features of the j-th frame first image.
[0162] The determination module is configured to determine the sound source object in the first image and the object label of the sound source object based on the image features of the first image in the j-th frame.
[0163] In one embodiment, the audio processing apparatus may further include a determining module configured to determine a scene label for a first image j in a first image j of a first image p;
[0164] The determination module is also configured to determine the background audio of the first image in the j-th frame based on the scene label of the first image in the j-th frame.
[0165] In one embodiment, the audio processing apparatus may further include a segmentation module configured to segment the j-th frame first image in the P-frame first image according to a preset angle to obtain a preset number of second images corresponding to the j-th frame first image, and to parse the preset number of second images using a second deep learning network to obtain the scene category of each second image in the preset number of second images.
[0166] The determination module is configured to determine the scene label of the first image in the j-th frame based on the scene category of each second image in a preset number of second images.
[0167] In one embodiment, if the value of j is 1, the audio processing device may further include a filtering module configured to filter background audio from a second preset audio library that matches the scene label of the first image in the j-th frame.
[0168] The determination module is configured to determine the background audio that matches the scene label of the first image in the j-th frame as the background audio of the first image in the j-th frame.
[0169] In one embodiment, if j is greater than or equal to 2, the audio processing device may further include a judgment module configured to judge whether the scene label of the first image in the j-th frame is consistent with the scene label of the first image in the (j-1)-th frame.
[0170] The determination module is configured to determine the background audio of the first image of the j-th frame as the background audio of the j-th frame if the scene label of the first image of the j-th frame is inconsistent with the scene label of the first image of the j-th frame.
[0171] The judgment module is configured to determine the background audio of the first image in the (j-1)th frame as the background audio of the first image in the jth frame if the scene label of the first image in the jth frame is consistent with the scene label of the first image in the (j-1)th frame.
[0172] In one embodiment, the audio processing apparatus may further include a determining module configured to, in response to a triggering event, acquire posture change information and a second object position of the sound source object;
[0173] The determination module is configured to determine the first object position of the sound source object based on the attitude change information and the second object position of the sound source object.
[0174] In one embodiment, the audio processing apparatus may further include a mapping module;
[0175] The acquisition module is configured to acquire parameter information, including spatial location information corresponding to the video data, and device parameters of the image acquisition device used to acquire the video data.
[0176] The acquisition module is configured to acquire the coordinate position of the sound source object in the two-dimensional coordinate system of the first image in the j-th frame;
[0177] The mapping module is configured to map coordinate positions to a spherical coordinate system to obtain the second object position of the sound source object; the spherical coordinate system is constructed based on spatial position information and device parameters.
[0178] In one embodiment, the audio processing apparatus further includes a matching module and a spatial rendering module.
[0179] The matching module is configured to match the first object position of the sound source object based on the preset correspondence between object position and sound intensity, so as to obtain the target sound intensity of the sound source object.
[0180] The spatial rendering module is configured to perform spatial rendering of the object audio of the sound source object based on the target sound intensity of the sound source object, and obtain the spatially rendered object audio.
[0181] In one embodiment, the audio processing module may further include a convolution processing module, configured to perform convolution processing on the spatially rendered object audio using an objective function to obtain the convolution-processed object audio, wherein the objective function is used to simulate the process of audio entering the human ear.
[0182] The generation module is configured to generate target audio based on the object audio after convolution processing and the background audio.
[0183] In this embodiment, user-input video data, including P-frame first images, is acquired. For the j-th frame of the P-frame first images, the object audio and background audio of the sound source object corresponding to the j-th frame are acquired. Then, in response to a received trigger event, the first object position of the sound source object is acquired, and spatial rendering is performed on the object audio of the sound source object based on the first object position to obtain spatially rendered object audio. Finally, a target audio is generated based on the spatially rendered object audio and background audio. This allows for frame-by-frame processing of user-input video data, enabling rapid and automated generation of matching spatial audio without the need for additional audio acquisition equipment or synchronization of audio and video, thus reducing operational complexity.
[0184] Each module in the audio processing apparatus provided in this disclosure can implement the method steps of the embodiments shown in FIG1, FIG2 or FIG4, and achieve the corresponding technical effects. For the sake of brevity, it will not be described in detail here.
[0185] Figure 8 shows a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this disclosure.
[0186] An electronic device may include a processor 801 and a memory 802 storing computer program instructions.
[0187] Specifically, the processor 801 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.
[0188] Memory 802 may include a mass storage device configured to store data or instructions. For example, and not limitingly, memory 802 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 802 may include removable or non-removable (or fixed) media. Where appropriate, memory 802 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 802 is a non-volatile solid-state memory.
[0189] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0190] The processor 801 implements any of the audio processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 802.
[0191] In one example, the electronic device may also include a communication interface 803 and a bus 810. As shown in Figure 8, the processor 801, memory 802, and communication interface 803 are connected via the bus 810 and communicate with each other.
[0192] The communication interface 803 is mainly configured to enable communication between modules, devices, units and / or equipment in the embodiments of this disclosure.
[0193] Bus 810 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 810 may include one or more buses. Although specific buses are described and illustrated in embodiments of this disclosure, this disclosure contemplates any suitable bus or interconnect.
[0194] Furthermore, in conjunction with the audio processing method in the above embodiments, this disclosure can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the audio processing method provided in this disclosure.
[0195] This disclosure also provides a computer program product, which includes a computer program that is executed by a processor to implement the audio processing method provided in this disclosure.
[0196] It should be clarified that this disclosure is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this disclosure is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this disclosure.
[0197] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this disclosure are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0198] It should also be noted that the exemplary embodiments mentioned in this disclosure describe methods or systems based on a series of steps or apparatus. However, this disclosure is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0199] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable audio processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable audio processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0200] The above are merely specific embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the protection scope of this disclosure.
Claims
1. An audio processing method, the method comprising: Acquire video data input by the user, the video data including P frames of the first image; where P is a positive integer; For the first image of the j-th frame in the first image of the P-frame, obtain the audio corresponding to the first image of the j-th frame. The audio includes the object audio of the sound source object and the background audio, where j is an integer greater than or equal to 1 and less than or equal to P. In response to the received trigger event, the first object position of the sound source object is obtained, and the object audio of the sound source object is spatially rendered based on the first object position of the sound source object to obtain the spatially rendered object audio. The target audio is generated based on the object audio rendered in the space and the background audio.
2. The method according to claim 1, wherein, The step of obtaining the audio corresponding to the j-th frame of the first image in the P-frame first image includes: For the j-th frame of the first image in the P-frame, determine the sound source object in the j-th frame and the object label of the sound source object; The object audio that matches the object tag of the sound source object and is selected from the first preset audio library is determined as the object audio of the sound source object.
3. The method according to claim 2, wherein, The step of determining the sound source object and the object label of the sound source object in the j-th frame of the first image of the P-frame includes: For the first image of the j-th frame in the first image of the P-frame, the first image of the j-th frame is processed using a first deep learning network to extract the image features of the first image of the j-th frame; Based on the image features of the first image in the j-th frame, determine the sound source object in the first image and the object label of the sound source object.
4. The method according to claim 1, wherein, The step of obtaining the audio corresponding to the j-th frame of the first image in the P-frame first image includes: For the first image of the j-th frame in the first image of the P-frame, determine the scene label of the first image of the j-th frame; Based on the scene label of the first image in the j-th frame, determine the background audio of the first image in the j-th frame.
5. The method according to claim 4, wherein, Determining the scene label of the j-th frame first image in the P-frame first image includes: For the j-th frame first image in the P-frame first image, the j-th frame first image is divided according to a preset angle to obtain a preset number of second images corresponding to the j-th frame first image. The preset number of second images are then analyzed using a second deep learning network to obtain the scene category of each second image in the preset number of second images. Based on the scene category of each second image in the preset number of second images, the scene label of the first image in the j-th frame is determined.
6. The method according to claim 4 or 5, wherein, If the value of j is 1, determining the background audio of the first image in the j-th frame based on the scene label of the first image in the j-th frame includes: Background audio matching the scene tag of the first image in the j-th frame is obtained from the second preset audio library; The background audio that matches the scene label of the first image in the j-th frame is determined as the background audio of the first image in the j-th frame.
7. The method according to claim 4 or 5, wherein, If j is greater than or equal to 2, determining the background audio of the first image in the j-th frame based on the scene label of the first image in the j-th frame includes: Determine whether the scene label of the first image in the j-th frame is consistent with the scene label of the first image in the (j-1)-th frame; If the scene label of the first image in frame j is inconsistent with the scene label of the first image in frame (j-1), the background audio that matches the scene label of the first image in frame j, selected from the second preset audio library, will be determined as the background audio of the first image in frame j. If the scene label of the first image in frame j is consistent with the scene label of the first image in frame (j-1), then the background audio of the first image in frame (j-1) is determined to be the background audio of the first image in frame j.
8. The method according to claim 1, wherein, The step of obtaining the first object position of the sound source object in response to the received trigger event includes: In response to the triggering event, acquire posture change information and the second object position of the sound source object; Based on the posture change information and the second object position of the sound source object, the first object position of the sound source object is determined.
9. The method according to claim 8, wherein, Obtaining the second object position of the sound source object includes: Obtain parameter information; wherein, the parameter information includes spatial location information corresponding to the video data, and device parameters of the image acquisition device used to acquire the video data; Obtain the coordinate position of the sound source object in the two-dimensional coordinate system of the first image in the j-th frame; The coordinate position is mapped to a spherical coordinate system to obtain the second object position of the sound source object; wherein the spherical coordinate system is constructed based on the spatial position information and the device parameters.
10. The method according to claim 1, wherein, The step of spatially rendering the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio includes: Based on the preset correspondence between object position and sound intensity, the first object position of the sound source object is matched to obtain the target sound intensity of the sound source object; Based on the target sound intensity of the sound source object, spatial rendering is performed on the object audio of the sound source object to obtain the spatially rendered object audio.
11. The method according to claim 1 or 10, wherein, After spatially rendering the object audio of the sound source object based on the first object position of the sound source object to obtain the spatially rendered object audio, the method further includes: The object audio rendered in the space is convolved using an objective function to obtain the object audio after convolution. The objective function is used to simulate the process of audio entering the human ear. The process of generating target audio based on the spatially rendered object audio and the background audio includes: The target audio is generated based on the object audio after convolution and the background audio.
12. An audio processing apparatus, the audio processing apparatus comprising: The acquisition module is configured to acquire video data input by the user, the video data including P-frame first images; where P is a positive integer; The acquisition module is further configured to acquire audio corresponding to the j-th frame first image in the P-frame first image, wherein the audio includes object audio of the sound source object and background audio, and j is an integer greater than or equal to 1 and less than or equal to P. The acquisition module is further configured to, in response to a received trigger event, acquire the first object position of the sound source object, and perform spatial rendering of the object audio of the sound source object based on the first object position of the sound source object, to obtain the spatially rendered object audio. The generation module is configured to generate target audio based on the spatially rendered object audio and the background audio.
13. The audio processing apparatus according to claim 12, wherein, The audio processing device further includes a determining module configured to determine, for the j-th frame of the first image in the P-frame, a sound source object and an object label of the sound source object. The determining module is further configured to determine the object audio that matches the object tag of the sound source object and is selected from the first preset audio library as the object audio of the sound source object.
14. The audio processing apparatus according to claim 13, wherein, The audio processing device further includes a processing module configured to process the j-th frame first image in the P-frame first image using a first deep learning network and extract the image features of the j-th frame first image. The determining module is configured to determine the sound source object in the first image and the object label of the sound source object based on the image features of the first image in the j-th frame.
15. The audio processing apparatus according to claim 12, wherein, The audio processing device further includes a determining module configured to determine a scene label for the j-th frame of the first image in the P-frame first image; and to determine the background audio of the j-th frame based on the scene label of the first image.
16. The audio processing apparatus according to claim 15, wherein, The audio processing device further includes a segmentation module, configured to segment the j-th frame first image in the P-frame first image according to a preset angle to obtain a preset number of second images corresponding to the j-th frame first image, and to parse the preset number of second images using a second deep learning network to obtain the scene category of each second image in the preset number of second images; The determining module is configured to determine the scene label of the first image in the j-th frame based on the scene category of each of the preset number of second images.
17. The audio processing apparatus according to claim 15 or 16, wherein, If the value of j is 1, the audio processing device further includes a filtering module, which is configured to filter background audio that matches the scene label of the first image of the j-th frame from the second preset audio library; The determining module is configured to determine the background audio that matches the scene label of the first image in the j-th frame as the background audio of the first image in the j-th frame.
18. The audio processing apparatus according to claim 15 or 16, wherein, If j is greater than or equal to 2, the audio processing device further includes a judgment module, configured to judge whether the scene label of the first image in the j-th frame is consistent with the scene label of the first image in the (j-1)-th frame; The determining module is configured to, when the scene label of the first image in the j-th frame is inconsistent with the scene label of the first image in the (j-1)-th frame, determine the background audio of the first image in the j-th frame as the background audio of the first image in the j-th frame by selecting the background audio that matches the scene label of the first image in the second preset audio library. The judgment module is configured to determine the background audio of the first image in the (j-1)th frame as the background audio of the first image in the jth frame if the scene label of the first image in the jth frame is consistent with the scene label of the first image in the (j-1)th frame.
19. The audio processing apparatus according to claim 12, wherein, The audio processing device further includes a determination module; The acquisition module is configured to acquire posture change information and the second object position of the sound source object in response to the triggering event; The determining module is configured to determine the first object position of the sound source object based on the posture change information and the second object position of the sound source object.
20. The audio processing apparatus according to claim 19, wherein, The audio processing device also includes a mapping module; The acquisition module is configured to acquire parameter information; wherein, the parameter information includes spatial location information corresponding to the video data, and device parameters of the image acquisition device used to acquire the video data; The acquisition module is further configured to acquire the coordinate position of the sound source object in the two-dimensional coordinate system of the first image in the j-th frame; The mapping module is configured to map the coordinate position to a spherical coordinate system to obtain the second object position of the sound source object; wherein the spherical coordinate system is constructed based on the spatial position information and the device parameters.
21. The audio processing apparatus according to claim 12, wherein, The audio processing device also includes a matching module and a spatial rendering module; The matching module is configured to match the first object position of the sound source object based on a preset correspondence between object position and sound intensity, so as to obtain the target sound intensity of the sound source object. The spatial rendering module is configured to perform spatial rendering of the object audio of the sound source object based on the target sound intensity of the sound source object, so as to obtain the spatially rendered object audio.
22. The audio processing apparatus according to claim 12 or 21, wherein, The audio processing module further includes a convolution processing module, which is configured to perform convolution processing on the spatially rendered object audio using an objective function to obtain the convolution-processed object audio. The objective function is used to simulate the process of audio entering the human ear. The generation module is configured to generate the target audio based on the object audio and background audio after the convolution process.
23. An electronic device, the electronic device comprising: A processor and a memory storing computer program instructions; wherein the processor reads and executes the computer program instructions to implement the audio processing method as described in any one of claims 1 to 11.
24. A computer storage medium storing computer program instructions that, when executed by a processor, implement the audio processing method as described in any one of claims 1 to 11.
25. A computer program product comprising a computer program that, when executed by a processor, implements the audio processing method according to any one of claims 1 to 11.