Audio Processing Method, Apparatus, Display Device, and Computer-Readable Storage Medium

Abstract

This application discloses an audio processing method, apparatus, display device, and computer-readable storage medium. The method comprises: obtaining the camera angle corresponding to the current playback screen of a target display device; determining the main channel corresponding to the current playback screen based on the camera angle; and determining the target audio signal corresponding to each of multiple channels based on the original audio signal corresponding to the main channel and the current playback screen. The method of this application can be used to determine the target audio signal corresponding to each channel in combination with the camera angle corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to an audio processing method, apparatus, display device and computer-readable storage medium. Background Art

[0002] Currently, mainstream devices used to display images, such as televisions, are 2D. The camera's perspective is the same as the human eye's. Therefore, the optimal viewing experience for viewers should be based on the camera's viewing angle. However, existing devices suffer from inaccurate positioning of audio and video. For example, if someone stands on the edge of a cliff and watches the sound of a waterfall, the viewing angle is from above, while the audio signal is output from the left, right, or surround channels. Summary of the Invention

[0003] The embodiments of the present application provide an audio processing method, apparatus, display device, and computer-readable storage medium, which can determine the target audio signal corresponding to each channel in combination with the camera perspective corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0004] The technical solutions adopted by the present invention to solve the problem are as follows:

[0005] In a first aspect, the present application provides an audio processing method, which is applied to a target display device, the target display device including multiple channels, and the audio processing method includes:

[0006] Get the camera angle corresponding to the current playback screen of the target display device;

[0007] Based on the camera's viewing angle, determine the main audio channel corresponding to the current playback image;

[0008] Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0009] In some embodiments of the present application, determining the main sound channel corresponding to the current playback image based on the camera perspective includes:

[0010] If the camera perspective is an upward perspective, determine that the main channel corresponding to the current playback image is the sky channel;

[0011] If the camera perspective is a horizontal perspective, determine that the main channels corresponding to the current playback image are the left and right channels;

[0012] If the camera perspective is a bird's-eye view, determine that the main channel corresponding to the current playback image is the ground channel.

[0013] In some embodiments of the present application, determining a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture includes:

[0014] Determine, based on the main channel, a channel gain value corresponding to each of the multiple channels;

[0015] The target audio signal corresponding to each channel is determined based on the channel gain value and the original audio signal corresponding to the current playing picture.

[0016] In some embodiments of the present application, determining a channel gain value corresponding to each of the multiple channels based on the main channel includes:

[0017] Determine a channel gain value corresponding to a main channel among the multiple channels as a first gain value;

[0018] The channel gain values corresponding to the other channels except the main channel among the multiple channels are determined as second gain values; wherein the first gain value is greater than the second gain value.

[0019] In some embodiments of the present application, the multiple sound channels include a sky channel, a ground channel, and left and right channels. The target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, wherein the first speaker, the second speaker, and the third speaker are respectively connected to the central processor;

[0020] Among them, the target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

[0021] In some embodiments of the present application, the first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers;

[0022] When the first speaker and the second speaker are built-in speakers, after determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, the method includes:

[0023] Performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively;

[0024] Outputting the filtered target audio signal corresponding to the sky channel through the first speaker;

[0025] Outputting the filtered target audio signal corresponding to the ground channel through the second speaker;

[0026] The target audio signals corresponding to the left and right channels are output through the third speaker.

[0027] In some embodiments of the present application, obtaining the camera angle corresponding to the current playback image of the target display device includes:

[0028] Obtaining a first image and a second image corresponding to a current playback screen of a target display device;

[0029] Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to a currently played image;

[0030] Based on the target camera external parameters, determine the camera angle corresponding to the current playback image of the target display device.

[0031] In some embodiments of the present application, determining the target camera extrinsic parameter of the target camera corresponding to the currently played image based on the first image and the second image includes:

[0032] Performing pixel matching on the first image and the second image to obtain a plurality of second target pixels corresponding to a plurality of first target pixels in the first image in the second image;

[0033] Determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback image;

[0034] Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel coordinates corresponding to each second target pixel point;

[0035] Determining a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates and the reprojected pixel point coordinates corresponding to the second target pixel point;

[0036] The initial camera intrinsic parameters and the initial camera extrinsic parameters are updated based on the reprojection error, and the step of determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera intrinsic parameters of the target camera corresponding to the current playback image is continued until the reprojection error satisfies the first condition;

[0037] The updated initial camera external parameters are determined as the target camera external parameters of the target camera corresponding to the current playback image.

[0038] In a second aspect, an embodiment of the present invention further provides an audio processing device, which is applied to a target display device, the target display device including multiple channels, and the audio processing device including:

[0039] An information acquisition module is used to obtain the camera angle of view corresponding to the current playback screen of the target display device;

[0040] A first determining module is used to determine the main channel corresponding to the current playback picture based on the camera perspective;

[0041] The second determining module is configured to determine a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture.

[0042] In a third aspect, the present application further provides a display device, comprising:

[0043] one or more processors;

[0044] Memory; and

[0045] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement any one of the audio processing methods of the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is loaded by a processor to execute the steps in any one of the audio processing methods in the first aspect.

[0047] The beneficial effects of the present invention are as follows: the main channel corresponding to the current playback screen is determined based on the camera angle corresponding to the current playback screen of the target display device, the target audio signal corresponding to each channel of multiple channels is determined based on the original audio signal corresponding to the main channel and the current playback screen, and the target audio signal corresponding to each channel can be determined in combination with the camera angle corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a schematic diagram of a scene of a target display device provided by an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a scenario of a speaker installation position of a target display device provided by an embodiment of the present invention;

[0051] Figure 3 is a flowchart of an audio processing method provided by an embodiment of the present invention;

[0052] Figure 4This is a flow chart of a specific embodiment of obtaining the camera angle of view corresponding to the current playback image provided by an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of a specific embodiment of determining a target audio signal provided by an embodiment of the present invention;

[0054] Figure 6 is a principle block diagram of an audio processing device provided by an embodiment of the present invention;

[0055] Figure 7 It is a schematic structural diagram of an embodiment of a display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0057] In the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," etc. may explicitly or implicitly include one or more of the described features.

[0058] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0059] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, the corresponding data exist for the computer device to process. The details will not be repeated here.

[0060] The embodiments of the present application provide an audio processing method, apparatus, display device, and computer-readable storage medium, which are described in detail below.

[0061] The audio processing method provided in the embodiment of the present application is applied to a target display device, which can be any device with a display function, such as a TV, a laptop, a tablet computer, a projector, etc. Figure 1 As shown, Figure 1 The following is a schematic diagram of a scene of a target display device provided in an embodiment of the present application. The target display device may include a central processing unit (CPU) 110 and multiple channels. Channels refer to independent audio signals collected or played back at different spatial locations during recording or playback. The CPU 110 is primarily used to obtain the camera perspective corresponding to the current playback screen of the target display device; based on the camera perspective, determine the main channel corresponding to the current playback screen; and based on the main channel and the original audio signal corresponding to the current playback screen, determine the target audio signal corresponding to each of the multiple channels. The target audio signal corresponding to each channel can be determined in combination with the camera perspective corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0062] Further, continue to refer to Figure 1 As shown, the multiple channels include a sky channel, a ground channel, and left and right channels. The target display device also includes a first speaker 120, a second speaker 130, and a third speaker 140. The first speaker 120, the second speaker 130, and the third speaker 140 are respectively connected to the central processor 110. The audio signal corresponding to the sky channel is output through the first speaker 120, the audio signal corresponding to the ground channel is output through the second speaker 130, and the audio signal corresponding to the left and right channels is output through the third speaker 140.

[0063] Optionally, continue with reference to Figure 1 As shown, the multiple sound channels also include a surround sound channel, and the target display device also includes a fourth speaker 150. The fourth speaker 150 is connected to the central processor 110, and the audio signal corresponding to the surround sound channel is output through the fourth speaker 150. For example, referring to Figure 2 As shown, the target display device includes multiple speakers, which are respectively arranged above the user, below the user, in front of the user and behind the user. The multiple speakers can output audio signals corresponding to the four channels of sky, ground, left and right, and surround, thereby generating a 360° spatial sound field in the horizontal and vertical directions.

[0064] In some embodiments, the target display device may also include a memory 160, which is connected to the central processing unit 110, and the memory 160 is used to store data, such as camera information, such as camera viewing angle, initial camera external parameters, initial camera internal parameters, etc., such as audio information, such as original audio signals, target audio signals, etc.

[0065] In addition, continue to refer to Figure 1 As shown, the target display device also includes a microphone 170, which is connected to the central processing unit 110. The target display device can receive the user's voice control instructions through the microphone 170 and perform corresponding control operations based on the voice control instructions. For example, if the user sends a voice control instruction "turn off" to the target display device, the target display device will perform the turn-off operation after receiving the control instruction through the microphone 170.

[0066] It should be noted that Figure 1 The scene diagram of the target display device shown is only an example. The target display device and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the target display device and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0067] First, an audio processing method is provided in an embodiment of the present application. The executor of the audio processing method is an audio processing device, and the audio processing device is applied to a target display device. The target display device includes multiple channels. The audio processing method includes: obtaining the camera perspective corresponding to the current playback screen of the target display device; based on the camera perspective, determining the main channel corresponding to the current playback screen; based on the original audio signal corresponding to the main channel and the current playback screen, determining the target audio signal corresponding to each channel in the multiple channels.

[0068] like Figure 3 FIG. 2 is a flow chart of an embodiment of an audio processing method in an embodiment of the present application. The audio processing method may include the following steps S201 to S203, which are specifically as follows:

[0069] Step S201: Obtain the camera angle corresponding to the current playback image of the target display device.

[0070] In the embodiment of the present application, the camera angle corresponding to the currently playing screen refers to the relative position and angle between the target camera used to shoot the currently playing screen and the object being photographed in the currently playing screen. The camera angle corresponding to the currently playing screen can be any one of a level angle, a top-down angle, and an upward angle. The level angle refers to a perspective in which the target camera and the object being photographed in the currently playing screen are approximately at the same height, the top-down angle refers to a perspective in which the target camera is above the object being photographed in the currently playing screen, and the upward angle refers to a perspective in which the target camera is below the object being photographed in the currently playing screen.

[0071] Optionally, the camera angle corresponding to the current playback screen can be determined by the target display device based on the current playback screen. The camera angle corresponding to the current playback screen can also be obtained from other computer devices through the network, Bluetooth, infrared, etc., which is not limited in this embodiment.

[0072] In some embodiments, the step of obtaining the camera perspective corresponding to the current playback screen of the target display device specifically includes: obtaining the current playback screen of the target display device; performing target recognition on the current playback screen to obtain the target recognition result of the current playback screen; and determining the camera perspective corresponding to the current playback screen based on the target recognition result.

[0073] In an embodiment of the present application, the step of performing target recognition on the current playback image and obtaining the target recognition result of the current playback image specifically includes: inputting the current playback image into a target recognition model, and outputting the target recognition result of the current playback image through the target recognition model. The target recognition model is a pre-trained neural network model for performing target recognition on the current playback image.

[0074] Furthermore, when determining the camera perspective corresponding to the current playback image based on the target recognition result, the camera perspective corresponding to the current playback image can be determined based on the target recognition result and the first association information. The first association information is used to characterize the correspondence between the recognition result and the camera perspective. For example, when the target recognition result is an airplane, the camera perspective corresponding to the current playback image is determined to be an upward-looking perspective; when the target recognition result is a waterfall, the camera perspective corresponding to the current playback image is determined to be a downward-looking perspective.

[0075] In other embodiments, referring to Figure 4 As shown, the above-mentioned acquisition of the camera angle corresponding to the current playback image of the target display device may include steps S301 to S303, which are specifically as follows:

[0076] Step S301: Acquire a first image and a second image corresponding to a current playback screen of a target display device.

[0077] In the embodiment of the present application, the currently playing image includes multiple continuous images. The first image and the second image can be any two continuous images from the multiple images, or the first image and the second image can be any two non-continuous images from the multiple images, which is not limited in this embodiment. Optionally, the first image and the second image can be any two continuous images from the multiple images, which can improve the accuracy of the determined camera angle of view.

[0078] Step S302: Determine the target camera extrinsic parameters of the target camera corresponding to the current playback image based on the first image and the second image.

[0079] In an embodiment of the present application, the target camera external parameters are the camera external parameters corresponding to the target camera used to shoot the currently played screen, which are obtained by performing camera calibration based on the first image and the second image. Camera calibration refers to the process of determining the internal and external parameters of the camera. The camera internal parameters include the focal length and distortion coefficient of the camera, etc., and the camera external parameters include the position and posture of the camera.

[0080] In some embodiments, the step of determining the target camera extrinsic parameters of the target camera corresponding to the current playback screen based on the first image and the second image specifically includes: performing pixel matching on the first image and the second image to obtain multiple second target pixels corresponding to multiple first target pixels in the first image in the second image; determining the normalized spatial point coordinates corresponding to each of the multiple first target pixels based on the initial camera intrinsic parameters of the target camera corresponding to the current playback screen; reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel coordinates corresponding to each second target pixel; determining the reprojection error corresponding to each second target pixel based on the original pixel coordinates and the reprojected pixel coordinates corresponding to the second target pixel; updating the initial camera intrinsic parameters and the initial camera extrinsic parameters based on the reprojection error, and continuing to perform the step of determining the normalized spatial point coordinates corresponding to each of the multiple first target pixels based on the initial camera intrinsic parameters of the target camera corresponding to the current playback screen until the reprojection error meets the first condition; and determining the updated initial camera extrinsic parameters as the target camera extrinsic parameters of the target camera corresponding to the current playback screen.

[0081] Optionally, the process of determining the normalized spatial point coordinates corresponding to each first target pixel point can be expressed as: , , , Indicates the The normalized spatial point coordinates corresponding to the first target pixel, Indicates the The original pixel coordinates corresponding to the first target pixel point, Indicates the initial camera internal parameters, Represents the depth value of the target camera, Indicates the conversion parameter between the depth value and the actual distance. Optionally, when the target camera is not a depth camera, .

[0082] In some embodiments, the process of determining the reprojected pixel coordinates is as follows: , represents the scale factor, Indicates the The normalized spatial point coordinates corresponding to the first target pixel, Indicates the target camera The projection matrix, and Indicates the target camera The initial camera external parameters, Indicates the target camera The rotation matrix of Indicates the target camera The translation vector of Indicates the The reprojected pixel coordinates of the second target pixel corresponding to the first target pixel.

[0083] Furthermore, the process of determining the reprojection error can be expressed as: ,in, Indicates the The reprojected pixel coordinates of the second target pixel corresponding to the first target pixel, Indicates the The original pixel coordinates of the second target pixel points corresponding to the first target pixel points.

[0084] In some embodiments, the reprojection error satisfies the first condition when the reprojection errors corresponding to all second target pixel points are less than the first threshold value. The reprojection error satisfies the first condition when the sum of the squares of the reprojection errors corresponding to all second target pixel points is less than the second threshold value. The reprojection error satisfies the first condition when the sum of the squares of the reprojection errors corresponding to all second target pixel points obtained twice before and after is less than the third threshold value. This embodiment does not limit this.

[0085] Step S303: Determine the camera viewing angle corresponding to the current playback image of the target display device based on the target camera external parameters.

[0086] In an embodiment of the present application, the target camera external parameters include the rotation matrix and translation matrix of the target camera. By analyzing the rotation matrix in the target camera external parameters, the orientation of the target camera can be determined, thereby determining whether the camera perspective corresponding to the current playback screen of the target display device is looking straight ahead, looking down, or looking up.

[0087] Step S202: Determine the main sound channel corresponding to the current playback image based on the camera viewing angle.

[0088] In the embodiment of the present application, the primary sound channel is the sound channel among the multiple sound channels of the target display device that matches the camera perspective corresponding to the currently playing screen. The primary sound channel can be any one or more of the multiple sound channels. For example, when the camera perspective is an upward-looking perspective, the primary sound channels corresponding to the currently playing screen are the sky channel and the left and right sound channels. When the camera perspective is a level-looking perspective, the primary sound channels corresponding to the currently playing screen are the left and right sound channels. When the camera perspective is a downward-looking perspective, the primary sound channels corresponding to the currently playing screen are the ground channel and the active sound channel.

[0089] For another example, when the camera perspective is an upward perspective, the main channel corresponding to the current playback picture is the sky channel. When the camera perspective is a horizontal perspective, the main channels corresponding to the current playback picture are the left and right channels. When the camera perspective is a downward perspective, the main channel corresponding to the current playback picture is the ground channel.

[0090] In some embodiments, the multiple sound channels include left and right sound channels, a sky sound channel, and a ground sound channel. The main sound channel is a sound channel among the multiple sound channels that matches the camera perspective corresponding to the current playback screen. Optionally, the above-mentioned step of determining the main sound channel corresponding to the current playback screen based on the camera perspective specifically includes: if the camera perspective is an upward perspective, determining that the main sound channel corresponding to the current playback screen is the sky sound channel; if the camera perspective is a horizontal perspective, determining that the main sound channels corresponding to the current playback screen are the left and right sound channels; if the camera perspective is a downward perspective, determining that the main sound channels corresponding to the current playback screen are the ground sound channel. This embodiment determines the main sound channel corresponding to the current playback screen based on the camera perspective, and can adjust the audio in combination with the camera perspective corresponding to the current playback screen to improve the accuracy of the sound and image position of the current playback screen.

[0091] Step S203: Determine a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback image.

[0092] In this embodiment of the present application, the original audio signal is the original audio signal corresponding to the currently playing image. For example, for a playback image of a person standing on the edge of a cliff watching a waterfall, the original audio signal is the impact sound of the water; for a playback image of an airplane flying overhead, the original audio signal is the sound of the airplane flying overhead. This embodiment determines the target audio signal corresponding to each of the multiple channels based on the original audio signals corresponding to the main channel and the currently playing image. This allows the sound of the currently playing image to be output by the channel that matches the currently playing image, thereby improving the accuracy of the sound and image positioning of the currently playing image.

[0093] In some embodiments, reference Figure 5 As shown, the above step S203, based on the original audio signal corresponding to the main channel and the current playback picture, determines the target audio signal corresponding to each of the multiple channels, which may include steps S401 to S402, as follows:

[0094] S401: Determine a channel gain value corresponding to each of a plurality of channels based on a main channel.

[0095] In an embodiment of the present application, the step of determining, based on the main channel, a channel gain value corresponding to each of the multiple channels specifically includes: determining the channel gain value corresponding to the main channel among the multiple channels as a first gain value; and determining the channel gain values corresponding to the other channels among the multiple channels except the main channel as a second gain value; wherein the first gain value is greater than the second gain value.

[0096] For example, the first gain value is 1 and the second gain value is 0.7. If the main channel is the ground channel, the channel gain value corresponding to the ground channel is set to 1, and the channel gain values corresponding to the other channels except the ground channel in the multiple channels are set to 0.7. In this way, the sound of the currently playing picture can be mainly output by its main channel, thereby improving the accuracy of the sound and image position of the currently playing picture.

[0097] S402: Determine a target audio signal corresponding to each channel based on the channel gain value and the original audio signal corresponding to the current playing picture.

[0098] In some embodiments, the step of determining the target audio signal corresponding to each channel based on the channel gain value and the original audio signal corresponding to the current playback screen specifically includes: multiplying the channel gain value corresponding to each channel and the original audio signal corresponding to the current playback screen to obtain the target audio signal corresponding to each channel.

[0099] For example, when the multiple channels of the target display device include a sky channel, a ground channel, and left and right channels, and the sky channel is the main channel, the process of determining the target audio signal corresponding to each channel can be expressed as follows: , , , Indicates the target audio signal corresponding to the sky channel, Indicates the target audio signal corresponding to the left and right channels, Represents the target audio signal corresponding to the ground channel, represents the original audio signal, represents the first gain value, Indicates the second gain value.

[0100] In some embodiments, the multiple audio channels include an overhead channel, a floor channel, and left and right channels. The target display device includes a central processing unit (CPU), a first speaker, a second speaker, and a third speaker, wherein the first speaker, the second speaker, and the third speaker are respectively connected to the CPU. The target audio signal corresponding to the overhead channel is output through the first speaker, the target audio signal corresponding to the floor channel is output through the second speaker, and the target audio signal corresponding to the left and right channels is output through the third speaker.

[0101] In some embodiments, the first speaker and the second speaker are external speakers. After determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the currently played screen, the above-mentioned method includes: outputting the target audio signal corresponding to the sky channel through the first speaker; outputting the target audio signal corresponding to the ground channel through the second speaker; and outputting the target audio signal corresponding to the left and right channels through the third speaker.

[0102] In other embodiments, the first speaker and the second speaker are built-in speakers. The above-mentioned determination of the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the currently played screen includes: filtering the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; outputting the filtered target audio signal corresponding to the sky channel through the first speaker; outputting the filtered target audio signal corresponding to the ground channel through the second speaker; and outputting the target audio signals corresponding to the left and right channels through the third speaker.

[0103] In an embodiment of the present application, the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel can be filtered separately using a preset spatial filter. The spatial filter refers to a digital filter generated using head-related transfer function (HRTF) technology. When the first speaker and the second speaker are built-in speakers, the target display device needs to rely on reflection to form a corresponding sound field. This embodiment filters the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel separately, which can improve the quality of the target audio signals corresponding to the sky channel and the ground channel, and thus improve the sound field effect corresponding to the current playback image.

[0104] In order to better implement the audio processing method in the embodiment of the present application, based on the audio processing method, the embodiment of the present application also provides an audio processing device, such as Figure 6 As shown, the audio processing device 600 includes:

[0105] The information acquisition module 610 is used to obtain the camera angle corresponding to the current playback image of the target display device;

[0106] A first determining module 620 is configured to determine the primary sound channel corresponding to the current playback image based on the camera viewing angle;

[0107] The second determining module 630 is configured to determine a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture.

[0108] In an embodiment of the present application, the main channel corresponding to the current playback screen is determined based on the camera perspective corresponding to the current playback screen of the target display device, and the target audio signal corresponding to each channel of the multiple channels is determined based on the original audio signal corresponding to the main channel and the current playback screen. The target audio signal corresponding to each channel can be determined in combination with the camera perspective corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0109] In some embodiments of the present application, the first determining module 620 determines the primary sound channel corresponding to the current playback image based on the camera viewing angle, including:

[0110] If the camera perspective is an upward perspective, determine that the main channel corresponding to the current playback image is the sky channel;

[0111] If the camera perspective is a horizontal perspective, determine that the main channels corresponding to the current playback image are the left and right channels;

[0112] If the camera perspective is a bird's-eye view, determine that the main channel corresponding to the current playback image is the ground channel.

[0113] In some embodiments of the present application, the second determination module 630 determines the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, including:

[0114] Determine, based on the main channel, a channel gain value corresponding to each of the multiple channels;

[0115] The target audio signal corresponding to each channel is determined based on the channel gain value and the original audio signal corresponding to the current playing picture.

[0116] In some embodiments of the present application, the second determining module 630 determines the channel gain value corresponding to each of the multiple channels based on the main channel, including:

[0117] Determine a channel gain value corresponding to a main channel among the multiple channels as a first gain value;

[0118] The channel gain values corresponding to the other channels except the main channel among the multiple channels are determined as second gain values; wherein the first gain value is greater than the second gain value.

[0119] In some embodiments of the present application, the multiple sound channels include a sky channel, a ground channel, and left and right channels. The target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, wherein the first speaker, the second speaker, and the third speaker are respectively connected to the central processor;

[0120] Among them, the target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

[0121] In some embodiments of the present application, the first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers. When the first speaker and the second speaker are built-in speakers, after determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, the second determination module 630 is further configured to:

[0122] Performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively;

[0123] Outputting the filtered target audio signal corresponding to the sky channel through the first speaker;

[0124] Outputting the filtered target audio signal corresponding to the ground channel through the second speaker;

[0125] The target audio signals corresponding to the left and right channels are output through the third speaker.

[0126] In some embodiments of the present application, the information acquisition module 610 acquires the camera angle corresponding to the current playback image of the target display device, including:

[0127] Obtaining a first image and a second image corresponding to a current playback screen of a target display device;

[0128] Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to a currently played image;

[0129] Based on the target camera external parameters, determine the camera angle corresponding to the current playback image of the target display device.

[0130] In some embodiments of the present application, the information acquisition module 610 determines the target camera extrinsic parameters of the target camera corresponding to the current playback image based on the first image and the second image, including:

[0131] Performing pixel matching on the first image and the second image to obtain a plurality of second target pixels corresponding to a plurality of first target pixels in the first image in the second image;

[0132] Determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback image;

[0133] Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel coordinates corresponding to each second target pixel point;

[0134] Determining a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates and the reprojected pixel point coordinates corresponding to the second target pixel point;

[0135] The initial camera intrinsic parameters and the initial camera extrinsic parameters are updated based on the reprojection error, and the step of determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera intrinsic parameters of the target camera corresponding to the current playback image is continued until the reprojection error satisfies the first condition;

[0136] The updated initial camera external parameters are determined as the target camera external parameters of the target camera corresponding to the current playback image.

[0137] The present application also provides a display device that integrates any of the audio processing devices provided in the present application. The display device includes:

[0138] one or more processors;

[0139] Memory; and

[0140] One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the audio processing method in any of the above audio processing method embodiments.

[0141] The embodiment of the present application also provides a display device, which integrates any of the audio processing devices provided in the embodiment of the present application. Figure 7 , which shows a schematic structural diagram of a display device involved in an embodiment of the present application, specifically:

[0142] The display device may include one or more processing core processors 801, one or more computer-readable storage media memories 802, a power supply 803, an input unit 804, and other components. Those skilled in the art will appreciate that Figure 7 The display device structure shown in the figure does not constitute a limitation on the display device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0143] The processor 801 is the control center of the display device. It uses various interfaces and lines to connect the various parts of the entire display device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it performs various functions of the display device and processes data, thereby monitoring the display device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly handles wireless communications. It is understood that the above-mentioned modem processor may not be integrated into the processor 801.

[0144] Memory 802 can be used to store software programs and modules. Processor 801 executes various functional applications and data processing by running the software programs and modules stored in memory 802. Memory 802 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the display device. Furthermore, memory 802 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 802 may also include a memory controller to provide processor 801 with access to memory 802.

[0145] The display device also includes a power supply 803 for supplying power to various components. Preferably, the power supply 803 can be logically connected to the processor 801 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 803 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0146] The display device may further include an input unit 804, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0147] Although not shown, the display device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the display device will load the executable files corresponding to one or more application processes into the memory 802 according to the following instructions, and the processor 801 will run the application stored in the memory 802 to implement various functions as follows:

[0148] Get the camera angle corresponding to the current playback screen of the target display device;

[0149] Based on the camera's viewing angle, determine the main audio channel corresponding to the current playback image;

[0150] Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0151] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0152] To this end, embodiments of the present application provide a computer-readable storage medium, which may include a read-only memory (ROM), random access memory (RAM), a disk, or an optical disk. A computer program is stored on the computer-readable storage medium, which is loaded by a processor to execute the steps of any of the audio processing methods provided in embodiments of the present application. For example, the computer program loaded by the processor may execute the following steps:

[0153] Get the camera angle corresponding to the current playback screen of the target display device;

[0154] Based on the camera's viewing angle, determine the main audio channel corresponding to the current playback image;

[0155] Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0156] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above and will not be repeated here.

[0157] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to implement as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments and will not be repeated here.

[0158] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0159] The above is a detailed introduction to an audio processing method, device, display device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio processing method, characterized in that: Applied to a target display device, the target display device includes multiple channels, and the audio processing method includes: Obtaining a camera angle corresponding to a current playback screen of the target display device; the camera angle refers to the relative position and angle between a target camera used to capture the current playback screen and an object captured in the current playback screen; the camera angle corresponding to the current playback screen includes any one of a level viewing angle, a downward viewing angle, and an upward viewing angle; Determining the main sound channel corresponding to the current playback image based on the camera viewing angle; Determining a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture; The determining, based on the camera viewing angle, the main sound channel corresponding to the current playback image includes: If the camera viewing angle is an upward viewing angle, determining that the main sound channel corresponding to the current playback image is the sky channel; If the camera viewing angle is a level viewing angle, determining that the main sound channels corresponding to the current playback image are left and right sound channels; If the camera viewing angle is a bird's-eye view, it is determined that the main sound channel corresponding to the current playback image is a ground sound channel.

2. The audio processing method according to claim 1, wherein: The determining, based on the original audio signal corresponding to the main channel and the current playback picture, a target audio signal corresponding to each of the multiple channels includes: Determining, based on the main channel, a channel gain value corresponding to each of the multiple channels; The target audio signals corresponding to the respective channels are determined based on the channel gain values and the original audio signals corresponding to the currently played picture.

3. The audio processing method according to claim 2, characterized in that The determining, based on the main channel, a channel gain value corresponding to each of the multiple channels includes: Determine a channel gain value corresponding to the main channel among the multiple channels as a first gain value; The channel gain values corresponding to the other channels among the multiple channels except the main channel are determined as second gain values; wherein the first gain value is greater than the second gain value.

4. The audio processing method according to any one of claims 1 to 3, characterized in that: The multiple sound channels include an overhead sound channel, a ground sound channel, and left and right sound channels. The target display device includes: a central processing unit, a first speaker, a second speaker, and a third speaker, wherein the first speaker, the second speaker, and the third speaker are respectively connected to the central processing unit; The target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

5. The audio processing method according to claim 4, characterized in that: The first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers; When the first speaker and the second speaker are built-in speakers, after determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, the method includes: performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; Outputting the filtered target audio signal corresponding to the sky channel through the first speaker; Outputting the filtered target audio signal corresponding to the ground channel through the second speaker; The target audio signals corresponding to the left and right channels are output through the third speaker.

6. The audio processing method according to claim 1, wherein: The obtaining of the camera angle corresponding to the current playback image of the target display device includes: Acquire a first image and a second image corresponding to a current playback screen of the target display device; Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to the currently playing picture; Based on the target camera external parameter, a camera viewing angle corresponding to the current playback image of the target display device is determined.

7. The audio processing method according to claim 6, characterized in that: The determining, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to the current playback image includes: Performing pixel matching on the first image and the second image to obtain a plurality of second target pixels in the second image corresponding to a plurality of first target pixels in the first image; Determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the currently played image; Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain reprojected pixel coordinates corresponding to each second target pixel point; Determining a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates and the reprojected pixel point coordinates corresponding to the second target pixel point; The initial camera intrinsic parameter and the initial camera extrinsic parameter are updated based on the reprojection error, and the step of determining the normalized spatial point coordinates corresponding to each of the plurality of first target pixel points based on the initial camera intrinsic parameter of the target camera corresponding to the current playback image is continued until the reprojection error satisfies a first condition; The updated initial camera extrinsic parameter is determined as the target camera extrinsic parameter of the target camera corresponding to the current playback image.

8. An audio processing device, characterized in that: Applied to a target display device, the target display device includes multiple channels, and the audio processing device includes: An information acquisition module is configured to acquire a camera viewing angle corresponding to a current playback image of the target display device; the camera viewing angle refers to a relative position and angle between a target camera used to capture the current playback image and an object captured in the current playback image; the camera viewing angle corresponding to the current playback image includes any one of a level viewing angle, a downward viewing angle, and an upward viewing angle; A first determining module, configured to determine a primary sound channel corresponding to the current playback image based on the camera viewing angle; a second determining module, configured to determine a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture; The first determining module determines the main sound channel corresponding to the current playback picture based on the camera viewing angle, including: If the camera viewing angle is an upward viewing angle, determining that the main sound channel corresponding to the current playback image is the sky channel; If the camera viewing angle is a level viewing angle, determining that the main sound channels corresponding to the current playback image are left and right sound channels; If the camera viewing angle is a bird's-eye view, it is determined that the main sound channel corresponding to the current playback image is a ground sound channel.

9. A display device, characterized in that: The display device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the audio processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the audio processing method according to any one of claims 1 to 7.

Images