Audio processing method and device, display equipment and computer readable storage medium

Abstract

The invention discloses an audio processing method and apparatus, a display device and a computer readable storage medium. The method comprises the steps of obtaining a camera visual angle corresponding to a current playing picture of a target display device; determining a main sound channel corresponding to the current playing picture based on the camera visual angle; and determining a target audio signal corresponding to each sound channel in the plurality of sound channels based on the main sound channel and the original audio signal corresponding to the current playing picture. Through adoption of the method, the target audio signal corresponding to each sound channel can be determined in combination with the camera view angle corresponding to the current playing picture, and the accuracy of the sound image position of the current playing picture is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to an audio processing method, device, display equipment and computer-readable storage medium. Background Art

[0002] At present, mainstream terminal devices for displaying images, such as TVs, are all 2D. The camera perspective of the video is the perspective of the human eye, so the best experience for the audience should be to see the image and hear the sound from the camera's perspective. However, the picture scenes of existing terminal devices have the problem of inaccurate sound and image positions. For example, when a person stands on the edge of a cliff and watches the sound of a waterfall, the picture perspective is looking down, but the audio signal is output from the left, right or surround channels. Summary of the invention

[0003] The embodiments of the present application provide an audio processing method, apparatus, display device, and computer-readable storage medium, which can determine the target audio signal corresponding to each channel in combination with the camera perspective corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0004] The technical solution adopted by the present invention to solve the problem is as follows: In a first aspect, the present application provides an audio processing method, which is applied to a target display device, the target display device includes multiple channels, and the audio processing method includes: Get the camera angle corresponding to the current playback screen of the target display device; Based on the camera angle, determine the main channel corresponding to the current playback image; Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0005] In some implementation schemes of the present application, determining the main channel corresponding to the current playback picture based on the camera viewing angle includes: If the camera perspective is an upward perspective, determine that the main channel corresponding to the current playback image is the sky channel; If the camera perspective is a horizontal perspective, determine that the main channels corresponding to the current playback image are left and right channels; If the camera perspective is a bird's-eye view, determine that the main channel corresponding to the current playback image is the ground channel.

[0006] In some embodiments of the present application, based on the original audio signal corresponding to the main channel and the current playback picture, determining the target audio signal corresponding to each of the multiple channels includes: Based on the main channel, determine the channel gain value corresponding to each channel in the multiple channels; Based on the channel gain value and the original audio signal corresponding to the current playing picture, the target audio signal corresponding to each channel is determined.

[0007] In some embodiments of the present application, based on the main channel, determining the channel gain value corresponding to each channel in the multiple channels includes: Determine a channel gain value corresponding to a main channel among the multiple channels as a first gain value; The channel gain values ​​corresponding to the other channels except the main channel among the multiple channels are determined as second gain values; wherein the first gain value is greater than the second gain value.

[0008] In some embodiments of the present application, the plurality of channels include a sky channel, a ground channel, and left and right channels, and the target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, and the first speaker, the second speaker, and the third speaker are respectively connected to the central processor; Among them, the target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

[0009] In some embodiments of the present application, the first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers; When the first speaker and the second speaker are built-in speakers, after determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture, the method includes: Performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; Outputting the filtered target audio signal corresponding to the sky channel through the first speaker; Outputting the target audio signal corresponding to the ground channel after filtering through the second speaker; The target audio signals corresponding to the left and right channels are outputted through the third speaker.

[0010] In some implementation schemes of the present application, obtaining the camera viewing angle corresponding to the current playback screen of the target display device includes: Acquire a first image and a second image corresponding to a current playback screen of a target display device; Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to a currently played image; Based on the target camera external parameters, determine the camera viewing angle corresponding to the current playback screen of the target display device.

[0011] In some implementation schemes of the present application, determining the target camera external parameter of the target camera corresponding to the current playback screen based on the first image and the second image includes: Perform pixel matching on the first image and the second image to obtain a plurality of second target pixel points in the second image corresponding to a plurality of first target pixel points in the first image; Determine the normalized spatial point coordinates corresponding to each first target pixel point among a plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback screen; Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel point coordinates corresponding to each second target pixel point; Determine a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates and the reprojected pixel point coordinates corresponding to the second target pixel point; The initial camera internal parameters and the initial camera external parameters are updated based on the reprojection error, and the step of determining the normalized spatial point coordinates corresponding to each first target pixel point in a plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback screen is continued until the reprojection error satisfies the first condition; The updated initial camera extrinsic parameters are determined as the target camera extrinsic parameters of the target camera corresponding to the current playback screen.

[0012] In a second aspect, an embodiment of the present invention further provides an audio processing device, which is applied to a target display device, the target display device includes multiple channels, and the audio processing device includes: An information acquisition module is used to obtain the camera viewing angle corresponding to the current playback screen of the target display device; A first determination module, used to determine the main channel corresponding to the current playback picture based on the camera viewing angle; The second determination module is used to determine the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture.

[0013] In a third aspect, the present application further provides a display device, the display device comprising: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement any one of the audio processing methods in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the audio processing method of any one of the first aspects.

[0015] The beneficial effects of the present invention are as follows: based on the camera viewing angle corresponding to the current playback screen of the target display device, the main channel corresponding to the current playback screen is determined; based on the original audio signal corresponding to the main channel and the current playback screen, the target audio signal corresponding to each channel in the multiple channels is determined; the target audio signal corresponding to each channel can be determined in combination with the camera viewing angle corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a schematic diagram of a scene of a target display device provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a scene of a speaker installation position of a target display device provided by an embodiment of the present invention; Figure 3 is a flowchart of an audio processing method provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of a specific embodiment of the process of obtaining the camera viewing angle corresponding to the current playback picture provided by an embodiment of the present invention; Figure 5 is a schematic diagram of a specific embodiment flow of determining a target audio signal provided by an embodiment of the present invention; Figure 6 is a principle block diagram of an audio processing device provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of the structure of a display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0019] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the features.

[0020] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0021] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0022] The embodiments of the present application provide an audio processing method, apparatus, display device, and computer-readable storage medium, which are described in detail below.

[0023] The audio processing method provided in the embodiment of the present application is applied to a target display device, which can be any device with a display function, such as a TV, a laptop, a tablet computer, a projector, etc. Figure 1 As shown, Figure 1A scene diagram of a target display device provided in an embodiment of the present application, the target display device may include a central processing unit (CPU) 110 and multiple channels, where channels refer to independent audio signals collected or played back at different spatial locations when recording or playing sounds. The CPU 110 is mainly used to obtain the camera perspective corresponding to the current playback screen of the target display device; based on the camera perspective, determine the main channel corresponding to the current playback screen; based on the main channel and the original audio signal corresponding to the current playback screen, determine the target audio signal corresponding to each channel in the multiple channels, and can determine the target audio signal corresponding to each channel in combination with the camera perspective corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0024] Further, continue to refer to Figure 1 As shown, the multiple channels include a sky channel, a ground channel, and left and right channels. The target display device also includes a first speaker 120, a second speaker 130, and a third speaker 140. The first speaker 120, the second speaker 130, and the third speaker 140 are respectively connected to the central processor 110. The audio signal corresponding to the sky channel is output through the first speaker 120, the audio signal corresponding to the ground channel is output through the second speaker 130, and the audio signal corresponding to the left and right channels is output through the third speaker 140.

[0025] Optionally, continue to refer to Figure 1 As shown, the multiple channels also include a surround channel, and the target display device also includes a fourth speaker 150, which is connected to the central processor 110, and the audio signal corresponding to the surround channel is output through the fourth speaker 150. Figure 2 As shown, the target display device includes multiple speakers, which are respectively arranged above the user, below the user, in front of the user and behind the user. The multiple speakers can output audio signals corresponding to four channels of sky, ground, left and right, and surround, thereby generating a 360° spatial sound field in the horizontal and vertical directions.

[0026] In some embodiments, the target display device may also include a memory 160, which is connected to the central processor 110, and the memory 160 is used to store data, such as camera information, such as camera viewing angle, initial camera external parameters, initial camera internal parameters, etc., such as audio information, such as original audio signals, target audio signals, etc.

[0027] In addition, continue to refer to Figure 1As shown, the target display device also includes a microphone 170, which is connected to the central processor 110. The target display device can obtain the user's voice control instruction through the microphone 170 and perform corresponding control operations based on the voice control instruction. For example, the user sends a "close" voice control instruction to the target display device, and the target display device performs the close operation after receiving the control instruction through the microphone 170.

[0028] It should be noted that Figure 1 The scene diagram of the target display device shown is merely an example. The target display device and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the target display device and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0029] First, an audio processing method is provided in an embodiment of the present application. The executor of the audio processing method is an audio processing device. The audio processing device is applied to a target display device. The target display device includes multiple channels. The audio processing method includes: obtaining the camera perspective corresponding to the current playback screen of the target display device; based on the camera perspective, determining the main channel corresponding to the current playback screen; based on the original audio signal corresponding to the main channel and the current playback screen, determining the target audio signal corresponding to each channel in the multiple channels.

[0030] like Figure 3 FIG. 2 is a flow chart of an embodiment of an audio processing method in an embodiment of the present application. The audio processing method may include the following steps S201 to S203, which are as follows: Step S201: Obtain the camera viewing angle corresponding to the current playback image of the target display device.

[0031] In the embodiment of the present application, the camera angle corresponding to the current playback screen refers to the relative position and angle between the target camera used to shoot the current playback screen and the object being photographed in the current playback screen. The camera angle corresponding to the current playback screen can be any one of a horizontal angle, a downward angle, and an upward angle. Among them, the horizontal angle refers to the angle of view where the target camera and the object being photographed in the current playback screen are roughly at the same height, the downward angle refers to the angle of view where the target camera is higher than the object being photographed in the current playback screen, and the upward angle refers to the angle of view where the target camera is lower than the object being photographed in the current playback screen.

[0032] Optionally, the camera viewing angle corresponding to the current playback screen can be determined by the target display device based on the current playback screen. The camera viewing angle corresponding to the current playback screen can also be obtained from other computer devices through the network, Bluetooth, infrared, etc., which is not limited in this embodiment.

[0033] In some embodiments, the step of obtaining the camera viewing angle corresponding to the current playback screen of the target display device specifically includes: obtaining the current playback screen of the target display device; performing target recognition on the current playback screen to obtain a target recognition result of the current playback screen; and determining the camera viewing angle corresponding to the current playback screen based on the target recognition result.

[0034] In the embodiment of the present application, the step of performing target recognition on the current playing picture and obtaining the target recognition result of the current playing picture specifically includes: inputting the current playing picture into the target recognition model, and outputting the target recognition result of the current playing picture through the target recognition model. The target recognition model is a pre-trained neural network model for performing target recognition on the current playing picture.

[0035] Furthermore, when determining the camera perspective corresponding to the current playback screen based on the target recognition result, the camera perspective corresponding to the current playback screen can be determined based on the target recognition result and the first association information. The first association information is used to characterize the corresponding relationship between the recognition result and the camera perspective. For example, when the target recognition result is an airplane, the camera perspective corresponding to the current playback screen is determined to be an upward perspective. When the target recognition result is a waterfall, the camera perspective corresponding to the current playback screen is determined to be a downward perspective.

[0036] In other embodiments, referring to Figure 4 As shown, the above-mentioned acquisition of the camera angle corresponding to the current playback screen of the target display device may include steps S301 to S303, which are specifically as follows: Step S301: Acquire a first image and a second image corresponding to a current playback screen of a target display device.

[0037] In the embodiment of the present application, the current playback screen includes multiple continuous images, and the first image and the second image can be any two continuous images in the multiple images, or the first image and the second image can be any two discontinuous images in the multiple images, which is not limited in the present embodiment. Optionally, the first image and the second image can be any two continuous images in the multiple images, which can improve the accuracy of the determined camera viewing angle.

[0038] Step S302: Determine the target camera external parameters of the target camera corresponding to the current playback screen based on the first image and the second image.

[0039] In the embodiment of the present application, the target camera external parameters are the camera external parameters corresponding to the target camera used to shoot the currently played screen obtained by performing camera calibration based on the first image and the second image. Camera calibration refers to the process of determining the internal and external parameters of the camera. The camera internal parameters include the focal length and distortion coefficient of the camera, etc., and the camera external parameters include the position and posture of the camera.

[0040] In some embodiments, based on the first image and the second image, the step of determining the target camera extrinsic parameters of the target camera corresponding to the current playback screen specifically includes: performing pixel matching on the first image and the second image to obtain multiple second target pixel points corresponding to multiple first target pixel points in the first image in the second image; determining the normalized spatial point coordinates corresponding to each of the multiple first target pixel points based on the initial camera intrinsic parameters of the target camera corresponding to the current playback screen; reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel coordinates corresponding to each second target pixel point; determining the reprojection error corresponding to each second target pixel point based on the original pixel coordinates and the reprojected pixel coordinates corresponding to the second target pixel point; updating the initial camera intrinsic parameters and the initial camera extrinsic parameters based on the reprojection error, and continuing to perform the step of determining the normalized spatial point coordinates corresponding to each of the multiple first target pixel points based on the initial camera intrinsic parameters of the target camera corresponding to the current playback screen until the reprojection error meets the first condition; determining the updated initial camera extrinsic parameters as the target camera extrinsic parameters of the target camera corresponding to the current playback screen.

[0041] Optionally, the process of determining the normalized spatial point coordinates corresponding to each first target pixel point can be expressed as: , , , Indicates The normalized spatial point coordinates corresponding to the first target pixel, Indicates The original pixel coordinates corresponding to the first target pixel point, Indicates the initial camera internal parameters, Represents the depth value of the target camera, Indicates the conversion parameter between the depth value and the actual distance. Optionally, when the target camera is not a depth camera, .

[0042] In some embodiments, the process of determining the coordinates of the reprojected pixel point is: , represents the scale factor, Indicates The normalized spatial point coordinates corresponding to the first target pixel, Indicates the target camera The projection matrix, and Indicates the target camera The initial camera external parameters, Indicates the target camera The rotation matrix of Indicates the target camera The translation vector of Indicates The reprojected pixel point coordinates of the second target pixel point corresponding to the first target pixel point.

[0043] Furthermore, the determination process of the reprojection error can be expressed as: ,in, Indicates The reprojected pixel coordinates of the second target pixel corresponding to the first target pixel, Indicates The original pixel coordinates of the second target pixel points corresponding to the first target pixel points.

[0044] In some embodiments, the reprojection error satisfies the first condition when the reprojection errors corresponding to all second target pixel points are less than the first threshold value. The reprojection error satisfies the first condition when the sum of the squares of the reprojection errors corresponding to all second target pixel points is less than the second threshold value. The reprojection error satisfies the first condition when the sum of the squares of the reprojection errors corresponding to all second target pixel points obtained twice before and after is less than the third threshold value. This embodiment does not limit this.

[0045] Step S303: Determine the camera viewing angle corresponding to the current playback image of the target display device based on the target camera external parameters.

[0046] In an embodiment of the present application, the target camera external parameters include the rotation matrix and translation matrix of the target camera. By analyzing the rotation matrix in the target camera external parameters, the orientation of the target camera can be determined, thereby determining whether the camera perspective corresponding to the current playback screen of the target display device is looking straight ahead, looking down, or looking up.

[0047] Step S202: Determine the main sound channel corresponding to the current playback image based on the camera viewing angle.

[0048] In the embodiment of the present application, the main channel is a channel among the multiple channels of the target display device that matches the camera angle corresponding to the current playback screen, and the main channel can be any one or more of the multiple channels. For example, when the camera angle is an upward angle, the main channels corresponding to the current playback screen are the sky channel and the left and right channels; when the camera angle is a horizontal angle, the main channels corresponding to the current playback screen are the left and right channels; when the camera angle is a downward angle, the main channels corresponding to the current playback screen are the ground channel and the action channel.

[0049] For another example, when the camera perspective is an upward perspective, the main channel corresponding to the current playback picture is the sky channel. When the camera perspective is a horizontal perspective, the main channels corresponding to the current playback picture are the left and right channels. When the camera perspective is a downward perspective, the main channel corresponding to the current playback picture is the ground channel.

[0050] In some embodiments, the multiple channels include left and right channels, a sky channel, and a ground channel, and the main channel is a channel among the multiple channels that matches the camera viewing angle corresponding to the current playback screen. Optionally, the above step of determining the main channel corresponding to the current playback screen based on the camera viewing angle specifically includes: if the camera viewing angle is an upward viewing angle, determining that the main channel corresponding to the current playback screen is the sky channel; if the camera viewing angle is a horizontal viewing angle, determining that the main channel corresponding to the current playback screen is the left and right channels; if the camera viewing angle is a downward viewing angle, determining that the main channel corresponding to the current playback screen is the ground channel. This embodiment determines the main channel corresponding to the current playback screen based on the camera viewing angle, and can adjust the audio in combination with the camera viewing angle corresponding to the current playback screen to improve the accuracy of the sound and image position of the current playback screen.

[0051] Step S203: Determine a target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture.

[0052] In the embodiment of the present application, the original audio signal is the original audio signal corresponding to the current playback picture. For example, for a playback picture of a person standing on the edge of a cliff and watching a waterfall, the original audio signal is the impact sound of water flow. For a playback picture of an airplane flying over a person's head, the original audio signal is the sound of the airplane flying over the head. This embodiment determines the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, so that the sound of the current playback picture can be output by the channel matching the current playback picture, thereby improving the accuracy of the sound and image position of the current playback picture.

[0053] In some embodiments, reference Figure 5As shown, the above step S203 determines the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, which may include steps S401 to S402, which are specifically as follows: S401: Determine a channel gain value corresponding to each of a plurality of channels based on a main channel.

[0054] In an embodiment of the present application, based on the main channel, the step of determining a channel gain value corresponding to each channel among the multiple channels specifically includes: determining a channel gain value corresponding to the main channel among the multiple channels as a first gain value; determining channel gain values ​​corresponding to other channels among the multiple channels except the main channel as a second gain value; wherein the first gain value is greater than the second gain value.

[0055] For example, the first gain value is 1 and the second gain value is 0.7. If the main channel is the ground channel, the channel gain value corresponding to the ground channel is set to 1, and the channel gain values ​​corresponding to the other channels except the ground channel among the multiple channels are set to 0.7. In this way, the sound of the current playback picture can be mainly output by its main channel, thereby improving the accuracy of the sound and image position of the current playback picture.

[0056] S402: Determine a target audio signal corresponding to each channel based on the channel gain value and the original audio signal corresponding to the current playing picture.

[0057] In some embodiments, based on the channel gain value and the original audio signal corresponding to the current playback screen, the step of determining the target audio signal corresponding to each channel specifically includes: multiplying the channel gain value corresponding to each channel and the original audio signal corresponding to the current playback screen to obtain the target audio signal corresponding to each channel.

[0058] For example, when the multiple channels of the target display device include a sky channel, a ground channel, and left and right channels, and the sky channel is the main channel, the process of determining the target audio signal corresponding to each channel can be expressed as: , , , Indicates the target audio signal corresponding to the sky channel, Indicates the target audio signal corresponding to the left and right channels. Represents the target audio signal corresponding to the ground channel, represents the original audio signal, represents the first gain value, Indicates the second gain value.

[0059] In some embodiments, the plurality of channels include a sky channel, a ground channel, and left and right channels, and the target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, and the first speaker, the second speaker, and the third speaker are respectively connected to the central processor. The target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signal corresponding to the left and right channels is output through the third speaker.

[0060] In some embodiments, the first speaker and the second speaker are external speakers. After determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the currently played picture, the above-mentioned method includes: outputting the target audio signal corresponding to the sky channel through the first speaker; outputting the target audio signal corresponding to the ground channel through the second speaker; and outputting the target audio signal corresponding to the left and right channels through the third speaker.

[0061] In other embodiments, the first speaker and the second speaker are built-in speakers, and the above-mentioned determination of the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the currently played picture includes: filtering the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; outputting the filtered target audio signal corresponding to the sky channel through the first speaker; outputting the filtered target audio signal corresponding to the ground channel through the second speaker; and outputting the target audio signals corresponding to the left and right channels through the third speaker.

[0062] In an embodiment of the present application, the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel can be filtered separately by a preset spatial filter, and the spatial filter refers to a digital filter generated using the head-related transfer function (HRTF) technology. When the first speaker and the second speaker are built-in speakers, the target display device needs to form a corresponding sound field by reflection. In this embodiment, the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel are filtered separately, which can improve the quality of the target audio signals corresponding to the sky channel and the ground channel, and then improve the sound field effect corresponding to the current playback picture.

[0063] In order to better implement the audio processing method in the embodiment of the present application, based on the audio processing method, an audio processing device is also provided in the embodiment of the present application, such as Figure 6 As shown, the audio processing device 600 includes: The information acquisition module 610 is used to acquire the camera angle corresponding to the current playback screen of the target display device; A first determination module 620, configured to determine a main sound channel corresponding to a current playback image based on a camera viewing angle; The second determination module 630 is used to determine the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture.

[0064] In an embodiment of the present application, the main channel corresponding to the current playback screen is determined based on the camera viewing angle corresponding to the current playback screen of the target display device, and the target audio signal corresponding to each channel of the multiple channels is determined based on the original audio signal corresponding to the main channel and the current playback screen. The target audio signal corresponding to each channel can be determined in combination with the camera viewing angle corresponding to the current playback screen, thereby improving the accuracy of the sound and image position of the current playback screen.

[0065] In some embodiments of the present application, the first determination module 620 determines the main channel corresponding to the current playback picture based on the camera viewing angle, including: If the camera perspective is an upward perspective, determine that the main channel corresponding to the current playback image is the sky channel; If the camera perspective is a horizontal perspective, determine that the main channels corresponding to the current playback image are left and right channels; If the camera perspective is a bird's-eye view, determine that the main channel corresponding to the current playback image is the ground channel.

[0066] In some embodiments of the present application, the second determination module 630 determines the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, including: Based on the main channel, determine the channel gain value corresponding to each channel in the multiple channels; Based on the channel gain value and the original audio signal corresponding to the current playing picture, the target audio signal corresponding to each channel is determined.

[0067] In some embodiments of the present application, the second determination module 630 determines the channel gain value corresponding to each channel in the multiple channels based on the main channel, including: Determine a channel gain value corresponding to a main channel among the multiple channels as a first gain value; The channel gain values ​​corresponding to the other channels except the main channel among the multiple channels are determined as second gain values; wherein the first gain value is greater than the second gain value.

[0068] In some embodiments of the present application, the plurality of channels include a sky channel, a ground channel, and left and right channels, and the target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, and the first speaker, the second speaker, and the third speaker are respectively connected to the central processor; Among them, the target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

[0069] In some embodiments of the present application, the first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers. When the first speaker and the second speaker are built-in speakers, after the second determination module 630 determines the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playback picture, the second determination module 630 is further used to: Performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; Outputting the filtered target audio signal corresponding to the sky channel through the first speaker; Outputting the target audio signal corresponding to the ground channel after filtering through the second speaker; The target audio signals corresponding to the left and right channels are outputted through the third speaker.

[0070] In some embodiments of the present application, the information acquisition module 610 acquires the camera angle corresponding to the current playback screen of the target display device, including: Acquire a first image and a second image corresponding to a current playback screen of a target display device; Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to a currently played image; Based on the target camera external parameters, determine the camera viewing angle corresponding to the current playback screen of the target display device.

[0071] In some embodiments of the present application, the information acquisition module 610 determines the target camera external parameters of the target camera corresponding to the current playback screen based on the first image and the second image, including: Perform pixel matching on the first image and the second image to obtain a plurality of second target pixel points in the second image corresponding to a plurality of first target pixel points in the first image; Determine the normalized spatial point coordinates corresponding to each first target pixel point among a plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback screen; Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain the reprojected pixel point coordinates corresponding to each second target pixel point; Determine a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates and the reprojected pixel point coordinates corresponding to the second target pixel point; The initial camera internal parameters and the initial camera external parameters are updated based on the reprojection error, and the step of determining the normalized spatial point coordinates corresponding to each first target pixel point in a plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback screen is continued until the reprojection error satisfies the first condition; The updated initial camera extrinsic parameters are determined as the target camera extrinsic parameters of the target camera corresponding to the current playback screen.

[0072] The embodiment of the present application further provides a display device, which integrates any one of the audio processing devices provided in the embodiment of the present application, and the display device includes: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the audio processing method in any of the above audio processing method embodiments.

[0073] The present application also provides a display device that integrates any of the audio processing devices provided in the present application. Figure 7 As shown, it shows a schematic diagram of the structure of the display device involved in the embodiment of the present application, specifically: The display device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will appreciate that Figure 7 The display device structure shown in the figure does not constitute a limitation on the display device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 801 is the control center of the display device. It uses various interfaces and lines to connect various parts of the entire display device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it executes various functions of the display device and processes data, thereby monitoring the display device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0074] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the display device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0075] The display device also includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 803 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0076] The display device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0077] Although not shown, the display device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 801 in the display device will load the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application programs stored in the memory 802, thereby realizing various functions, as follows: Get the camera angle corresponding to the current playback screen of the target display device; Based on the camera angle, determine the main channel corresponding to the current playback image; Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0078] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0079] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any audio processing method provided in the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps: Get the camera angle corresponding to the current playback screen of the target display device; Based on the camera angle, determine the main channel corresponding to the current playback image; Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0081] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0082] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0083] The above is a detailed introduction to an audio processing method, device, display device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio processing method, characterized in that: Applied to a target display device, the target display device includes multiple channels, and the audio processing method includes: Obtain the camera viewing angle corresponding to the current playback screen of the target display device; Based on the camera viewing angle, determining a main sound channel corresponding to the current playback picture; Based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels is determined.

2. The audio processing method according to claim 1, characterized in that: The determining, based on the camera viewing angle, the main sound channel corresponding to the current playback picture includes: If the camera viewing angle is an upward viewing angle, determining that the main sound channel corresponding to the current playback picture is a sky sound channel; If the camera viewing angle is a horizontal viewing angle, determining that the main channels corresponding to the current playback picture are left and right channels; If the camera viewing angle is a bird's-eye view, it is determined that the main sound channel corresponding to the current playback image is a ground sound channel.

3. The audio processing method according to claim 1, characterized in that: The determining, based on the original audio signal corresponding to the main channel and the current playing picture, a target audio signal corresponding to each of the multiple channels includes: Based on the main channel, determining a channel gain value corresponding to each channel in the multiple channels; Based on the channel gain values ​​and the original audio signal corresponding to the current playing picture, the target audio signal corresponding to each channel is determined.

4. The audio processing method according to claim 3, characterized in that: The determining, based on the main channel, a channel gain value corresponding to each of the multiple channels includes: Determine a channel gain value corresponding to the main channel among the multiple channels as a first gain value; The channel gain values ​​corresponding to the other channels among the multiple channels except the main channel are determined as second gain values; wherein the first gain value is greater than the second gain value.

5. The audio processing method according to any one of claims 1 to 4, characterized in that: The multiple sound channels include a sky sound channel, a ground sound channel, and left and right sound channels. The target display device includes: a central processor, a first speaker, a second speaker, and a third speaker, wherein the first speaker, the second speaker, and the third speaker are respectively connected to the central processor; Among them, the target audio signal corresponding to the sky channel is output through the first speaker, the target audio signal corresponding to the ground channel is output through the second speaker, and the target audio signals corresponding to the left and right channels are output through the third speaker.

6. The audio processing method according to claim 5, characterized in that: The first speaker and the second speaker are external speakers, or the first speaker and the second speaker are built-in speakers; When the first speaker and the second speaker are built-in speakers, after determining the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture, the method includes: Performing filtering processing on the target audio signal corresponding to the sky channel and the target audio signal corresponding to the ground channel respectively; Outputting, through the first speaker, the target audio signal corresponding to the sky channel after filtering; Outputting the target audio signal corresponding to the ground channel after filtering through the second speaker; The target audio signals corresponding to the left and right channels are outputted through the third speaker.

7. The audio processing method according to claim 1, characterized in that: The obtaining of the camera angle of view corresponding to the current playback screen of the target display device includes: Acquire a first image and a second image corresponding to a current playback screen of the target display device; Determine, based on the first image and the second image, a target camera extrinsic parameter of a target camera corresponding to the current playback screen; Based on the target camera external parameters, a camera viewing angle corresponding to the current playback picture of the target display device is determined.

8. The audio processing method according to claim 7, characterized in that: The determining, based on the first image and the second image, a target camera external parameter of a target camera corresponding to the current playback screen includes: Performing pixel matching on the first image and the second image to obtain a plurality of second target pixel points in the second image corresponding to a plurality of first target pixel points in the first image; Determine the normalized spatial point coordinates corresponding to each first target pixel point among a plurality of first target pixel points based on the initial camera internal parameters of the target camera corresponding to the current playback picture; Reprojecting the normalized spatial point coordinates onto the second image based on the initial camera extrinsic parameters corresponding to the target camera to obtain reprojected pixel point coordinates corresponding to each second target pixel point; Determine a reprojection error corresponding to each second target pixel point based on the original pixel point coordinates corresponding to the second target pixel point and the reprojection pixel point coordinates; Based on the reprojection error, the initial camera intrinsic parameter and the initial camera extrinsic parameter are updated, and the step of determining the normalized spatial point coordinates corresponding to each first target pixel point in a plurality of first target pixel points based on the initial camera intrinsic parameter of the target camera corresponding to the current playback screen is continued until the reprojection error satisfies the first condition; The updated initial camera external parameters are determined as the target camera external parameters of the target camera corresponding to the current playback screen.

9. An audio processing device, characterized in that: Applied to a target display device, the target display device includes multiple channels, and the audio processing device includes: An information acquisition module, used to acquire the camera viewing angle corresponding to the current playback screen of the target display device; A first determination module, used to determine the main channel corresponding to the current playback picture based on the camera viewing angle; The second determination module is used to determine the target audio signal corresponding to each of the multiple channels based on the original audio signal corresponding to the main channel and the current playing picture.

10. A display device, characterized in that: The display device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the audio processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the audio processing method according to any one of claims 1 to 8.

Images