Display device and audio processing method

By separating the audio data with vocals and adjusting the weight, and enhancing the sound effects with background audio, the problem of poor sound orientation is solved, and the three-dimensional sense of audio processing and user experience are improved.

CN114615534BActive Publication Date: 2024-11-15HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210102847.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-27
Publication Date
2024-11-15
Estimated Expiration
2042-01-27

AI Technical Summary

Technical Problem

The prior art has poor sense of sound orientation in audio processing, especially in stereo scenes, which makes it difficult to effectively improve the user's viewing experience.

Method used

By separating the acquired audio data, merging the vocal data, adjusting the vocal weight of the audio output interface according to the lip motion detection results in the image, combining the background audio data for sound enhancement processing, and outputting high-quality audio data.

Benefits of technology

It significantly improves the three-dimensionality of the sound, improves the user's viewing experience, and makes the audio processing more realistic and vivid.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114615534B_ABST
    Figure CN114615534B_ABST
Patent Text Reader

Abstract

The present application relates to a display device and an audio processing method, which are applied to the field of audio processing technology. The display device includes: a controller, which is configured to: perform vocal separation on first channel audio data and second channel audio data respectively to obtain first channel first vocal audio data, first channel first background audio data, second channel first vocal audio data and second channel first background audio data; merge the first channel first vocal audio data and the second channel first vocal audio data to obtain target vocal audio data; obtain image data, and if lip movement coordinates are detected, determine the vocal weight corresponding to each audio output interface according to the lip movement coordinates and the coordinates of each audio output interface; merge the product of the target vocal audio data and the vocal weight, and the first channel first background audio data and / or the second channel first background audio data, and perform sound enhancement processing to obtain and output audio data to improve the stereo effect of the audio.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a display device and an audio processing method. Background Art

[0002] With the improvement of chip AI (artificial intelligence) computing power, machine learning has been widely used in the fields of image and sound, and has also been combined in many scenarios. In related technologies, audio and video synchronization can be achieved through machine learning. However, this method has a poor sense of direction of sound. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a display device, an audio processing method, a storage medium and a program product.

[0004] According to a first aspect of the present application, there is provided a display device, comprising: a controller and a plurality of audio output interfaces;

[0005] The controller is configured to: perform vocal separation on the acquired first channel audio data and second channel audio data respectively, to obtain first channel first vocal audio data and first channel first background audio data, and second channel first vocal audio data and second channel first background audio data;

[0006] Merging the first human voice audio data of the first channel and the first human voice audio data of the second channel to obtain target human voice audio data;

[0007] Acquire image data at the time when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if lip movement coordinates on the display device screen are detected, determine the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces;

[0008] For each audio output interface, determining, according to the coordinates of the audio output interface, the first background audio data of the first channel and / or the first background audio data of the second channel corresponding to the audio output interface;

[0009] The product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface are combined, and sound effect enhancement processing is performed to obtain audio data corresponding to the audio output interface;

[0010] For each audio output interface, it is configured to: output audio data corresponding to the audio output interface.

[0011] In some embodiments, the controller is further configured to: perform gain processing on the first channel first human voice audio data and the second channel first human voice audio data according to the first gain, respectively, to obtain the first channel second human voice audio data and the second channel second human voice audio data;

[0012] Performing gain processing on the first background audio data of the first channel and the first background audio data of the second channel according to a second gain, respectively, to obtain the second background audio data of the first channel and the second background audio data of the second channel; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device;

[0013] The controller is configured to:

[0014] Merging the first channel second human voice audio data and the second channel second human voice audio data to obtain target human voice audio data; and

[0015] For each audio output interface, determining, according to the coordinates of the audio output interface, the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface;

[0016] The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface.

[0017] In some embodiments, the controller is further configured to: if no lip movement coordinates are detected, for each audio output interface, determine the vocal weight corresponding to the audio output interface respectively according to the ratio of the energy of the first human voice audio data of the first channel to the energy of the first human voice audio data of the second channel, and the coordinates of the audio output interface.

[0018] In some embodiments, the screen includes: a left area, a middle area, and a right area;

[0019] The controller is configured to: if the coordinates of the audio output interface correspond to the left area, determine that the audio output interface corresponds to the first background audio data of the first channel;

[0020] If the coordinates of the audio output interface correspond to the right area, determining that the audio output interface corresponds to the first background audio data of the second channel;

[0021] If the coordinates of the audio output interface correspond to the middle area, it is determined that the audio output interface corresponds to the first background audio data of the first channel and the first background audio data of the second channel.

[0022] In some embodiments, the screen includes: a middle area and a non-middle area;

[0023] The controller is configured to: if the lip movement coordinates are located in the non-middle area, determine the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces;

[0024] If the lip movement coordinates are located in the middle area, the human voice weights respectively corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction.

[0025] In some embodiments, the controller is configured to: for each audio output interface, determine the area corresponding to the audio output interface on the screen according to the coordinates of the audio output interface;

[0026] If the lip movement coordinates are located in the area corresponding to the audio output interface, determining that the human voice weight corresponding to the audio output interface is a first value;

[0027] If the lip movement coordinates are outside the area corresponding to the audio output interface, it is determined that the human voice weight corresponding to the audio output interface is a second value, and the second value is smaller than the first value.

[0028] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding value, and each preset sound effect mode has a corresponding value;

[0029] The sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of the plurality of preset sound clarity control modes, and the target sound effect mode is one of the plurality of preset sound effect modes;

[0030] The controller is configured to determine a first gain and a second gain according to a first value corresponding to the target sound clarity control mode and / or a second value corresponding to the target sound effect mode.

[0031] According to a second aspect of the present application, there is provided an audio processing method, which is applied to a display device, and the method comprises:

[0032] Separating the acquired first channel audio data and second channel audio data into human voices, respectively, to obtain first channel first human voice audio data and first channel first background audio data, and second channel first human voice audio data and second channel first background audio data;

[0033] Merging the first human voice audio data of the first channel and the first human voice audio data of the second channel to obtain target human voice audio data;

[0034] Acquire image data at the time when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if lip movement coordinates on the display device screen are detected, determine the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces of the display device;

[0035] For each audio output interface, determining, according to the coordinates of the audio output interface, the first background audio data of the first channel and / or the first background audio data of the second channel corresponding to the audio output interface;

[0036] The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface, and the audio data is output through the audio output interface.

[0037] In some embodiments, the method further comprises:

[0038] Performing gain processing on the first channel first human voice audio data and the second channel first human voice audio data according to a first gain, respectively, to obtain first channel second human voice audio data and second channel second human voice audio data;

[0039] Performing gain processing on the first background audio data of the first channel and the first background audio data of the second channel according to a second gain, respectively, to obtain the second background audio data of the first channel and the second background audio data of the second channel; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device;

[0040] The step of merging the first channel first human voice audio data and the second channel first human voice audio data to obtain target human voice audio data includes:

[0041] Merging the first channel second human voice audio data and the second channel second human voice audio data to obtain target human voice audio data;

[0042] The step of determining, for each audio output interface, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel, comprises:

[0043] For each audio output interface, determining, according to the coordinates of the audio output interface, the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface;

[0044] The method of merging the product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface, and performing sound effect enhancement processing to obtain the audio data corresponding to the audio output interface includes:

[0045] The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface.

[0046] In some embodiments, the method further comprises:

[0047] If the lip movement coordinates are not detected, for each audio output interface, the vocal weight corresponding to the audio output interface is determined according to the ratio of the energy of the first vocal audio data of the first channel to the energy of the first vocal audio data of the second channel, and the coordinates of the audio output interface.

[0048] In some embodiments, the screen includes: a left area, a middle area, and a right area;

[0049] The step of determining, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel includes:

[0050] If the coordinates of the audio output interface correspond to the left area, determining that the audio output interface corresponds to the first background audio data of the first channel;

[0051] If the coordinates of the audio output interface correspond to the right area, determining that the audio output interface corresponds to the first background audio data of the second channel;

[0052] If the coordinates of the audio output interface correspond to the middle area, it is determined that the audio output interface corresponds to the first background audio data of the first channel and the first background audio data of the second channel.

[0053] In some embodiments, the screen includes: a middle area and a non-middle area;

[0054] The step of determining the human voice weights respectively corresponding to the multiple audio output interfaces according to the lip movement coordinates and the coordinates of the multiple audio output interfaces of the display device includes:

[0055] If the lip movement coordinates are located in the non-middle area, determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces;

[0056] If the lip movement coordinates are located in the middle area, the human voice weights respectively corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction.

[0057] In some embodiments, for each audio output interface, the area corresponding to the audio output interface on the screen is determined according to the coordinates of the audio output interface;

[0058] If the lip movement coordinates are located in the area corresponding to the audio output interface, determining that the human voice weight corresponding to the audio output interface is a first value;

[0059] If the lip movement coordinates are outside the area corresponding to the audio output interface, it is determined that the human voice weight corresponding to the audio output interface is a second value, and the second value is smaller than the first value.

[0060] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding value, and each preset sound effect mode has a corresponding value;

[0061] The sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of the plurality of preset sound clarity control modes, and the target sound effect mode is one of the plurality of preset sound effect modes;

[0062] The method further comprises:

[0063] The first gain and the second gain are determined according to the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode.

[0064] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the audio processing method described in the second aspect is implemented.

[0065] According to a fourth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the audio processing method described in the second aspect.

[0066] Compared with the related art, the technical solutions provided by some embodiments of the present application have the following advantages:

[0067] The audio processing method of the embodiment of the present application, in a stereo scene, after the first channel audio data and the second channel audio data are respectively separated into human voices, the separated first channel first human voice audio data and the second channel first human voice audio data can be first merged to obtain target human voice audio data, and the target human voice audio data is used as the human voice audio to be output. Then, according to the position of the person speaking in the image, the human voice weight corresponding to each audio output interface is adjusted, that is, the weight corresponding to the output human voice audio, and according to the position of the audio output interface, the weight of the background audio output by each audio output interface is adjusted, so as to enhance the stereoscopic sense of the sound and improve the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0069] In order to more clearly illustrate the technical solutions in some embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0070] Figure 1 A schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application;

[0071] Figure 2 is a hardware configuration block diagram of a display device 200 according to one or more embodiments of the present application;

[0072] Figure 3 is a hardware configuration block diagram of a control device 100 according to one or more embodiments of the present application;

[0073] Figure 4 A schematic diagram of software configuration in a display device 200 according to one or more embodiments of the present application;

[0074] Figure 5 A schematic diagram showing an icon control interface of an application in a display device 200 according to one or more embodiments of the present application;

[0075] Fig. 6A A schematic diagram of a system architecture of an audio processing method in some embodiments of the present application;

[0076] Figure 6B A schematic diagram of an audio processing method in some embodiments of the present application;

[0077] Figure 7 A schematic diagram for sound separation;

[0078] Figure 8 A schematic diagram of an audio processing method in some embodiments of the present application;

[0079] Fig. 9A A schematic diagram of the distribution angle of standard recording studio or home audio speakers;

[0080] Fig. 9B A schematic diagram of the angles of the TV speakers;

[0081] Fig. 9C A schematic diagram for changing the energy distribution relationship of a TV speaker;

[0082] Fig.10 A schematic diagram of the function f(x) in some embodiments of the present application;

[0083] Fig.11A A schematic diagram of a system architecture of an audio processing method in some embodiments of the present application;

[0084] Fig. 11B A schematic diagram of an audio processing method in some embodiments of the present application;

[0085] Fig.12 A schematic diagram of an audio processing method in some embodiments of the present application;

[0086] Fig.13A A schematic diagram of a system architecture of an audio processing method in some embodiments of the present application;

[0087] Fig. 13B A schematic diagram of an audio processing method in some embodiments of the present application;

[0088] Fig.14 A schematic diagram of the speaker distribution;

[0089] Fig.15A A schematic diagram of a system architecture of an audio processing method in some embodiments of the present application;

[0090] Fig. 15B A schematic diagram of an audio processing method in some embodiments of the present application;

[0091] Fig.16A schematic diagram of performing time domain transformation on original vocal audio data in some embodiments of the present application;

[0092] Fig.17 A schematic diagram of frequency domain transformation of original vocal audio data in some embodiments of the present application.

[0093] Fig.18 A flow chart of an audio processing method in some embodiments of the present application;

[0094] Fig.19 A flow chart of an audio processing method in some embodiments of the present application;

[0095] Fig. 20 A flow chart of an audio processing method in some embodiments of the present application;

[0096] Fig.21 This is a flow chart of an audio processing method in some embodiments of the present application. DETAILED DESCRIPTION

[0097] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0098] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0099] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application, such as Figure 1As shown, the user can operate the display device 200 through the mobile terminal 300 and the control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user instructions through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices can also be used to control the display device 200.

[0100] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and realize connection communication through the network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 also communicates data with the server 400 through various communication methods. The display device 200 can be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide an intelligent network TV function that provides a computer support function.

[0101] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play the role of an interactive intermediary between the user and the display device 200. The communication interface 130 is used to communicate with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button or an alternative module.

[0102] Figure 3 FIG. 2 shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment. Figure 3The display device 200 shown includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, an external memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0103] In some embodiments, the controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the external memory. The controller 250 controls the overall operation of the display device 200. The user can input a user command through the graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through the sensor to receive the user input command.

[0104] In some embodiments, "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on a display screen of an electronic device, wherein a control may include at least one of icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0105] Figure 4 FIG. 1 is a schematic diagram of software configuration in a display device 200 according to one or more embodiments of the present application. Figure 4As shown in FIG. 1 , the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer), the application framework layer (Application Framework layer), the Android runtime and system library layer (System runtime library layer), and the kernel layer. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0106] Figure 5 FIG. 1 is a schematic diagram showing an icon control interface of an application in a display device 200 according to one or more embodiments of the present application. Figure 5 As shown in , the application layer includes at least one application that can display corresponding icon controls in the display, such as: live TV application icon control, video on demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV application can provide live TV through different signal sources. Video on demand application can provide videos from different storage sources. Unlike live TV application, video on demand provides video display from certain storage sources. Media center application can provide applications for playing various multimedia content. Application center can provide storage for various applications.

[0107] The implementation of this application in Android system is as follows Fig. 6A As shown, the Android system mainly includes an application layer, a middleware and a core layer. The implementation logic can be in the middleware, and the middleware includes: an audio decoder, a sound separation module, a gain control module, a sound enhancement module and an audio output interface. The audio decoder is used to perform audio decoding processing on a signal source input through a broadcast signal, a network, a USB or an HDMI, etc., to obtain audio data. The sound separation module is used to perform sound separation on the decoded audio data. For example, the human voice audio and the background audio can be separated by a human voice separation method. The gain control module can obtain the user's sound control mode for the display device, and perform different gain processing on the human voice audio and the background audio respectively to enhance the human voice audio or the background audio. The merging module is used to merge the gain-processed human voice audio and the background audio to obtain merged audio data, and the sound enhancement module is used to perform sound enhancement processing on the merged audio data to obtain the target audio data. The audio output interface is used to output the target audio data.

[0108] It should be noted that the above implementation logic can be implemented in the middleware as well as in the core layer. Alternatively, it can be implemented in the middleware and the core layer. For example, the audio decoder and the sound separation module can be implemented in the middleware, and the modules after the sound separation module can be implemented in the core layer.

[0109] With the above Fig. 6A Correspondingly, Figure 6B A schematic diagram of an audio processing method in some embodiments of the present application. After the audio decoder decodes the acquired sound signal, the first audio data can be obtained. The sound separation module can realize the sound separation of the first audio data through the AI ​​(artificial intelligence) technology and the pre-trained neural network model to obtain the first target audio data and the first background audio data. For example, the human voice can be separated by the human voice separation model, and the human voice is the first target audio data. The car sound can be separated by the pre-trained car sound separation model, and the car sound is the first target audio data. The first background audio data is the audio data other than the first target audio data. The gain control module can obtain the first gain and the second gain according to the sound control mode, and the values ​​of the first gain and the second gain are not equal. According to the first gain, the first target audio data is gain-processed to obtain the second target audio data, and the first background audio data is gain-processed according to the second gain to obtain the second background audio data. After the second target audio data and the second background audio data are merged and the sound effect enhancement process is performed, the second audio data is obtained and output. The present application enhances the first target audio data or the first background audio data by performing non-proportional gain processing on the first target audio data and the first background audio data, thereby improving the effect of sound enhancement.

[0110] The following first introduces the display devices of some embodiments of the present application.

[0111] In some embodiments, the display device 200 may be a terminal device with a display function, such as a television, a smart phone, a computer, a learning machine, etc. The display device 200 includes:

[0112] The controller 250 is configured to perform sound separation on the acquired first audio data to obtain first target audio data and first background audio data.

[0113] The first audio data refers to audio data containing at least two mixed sounds. For example, the first audio data may include human voice and background music. The human voice is separated by a pre-trained human voice separation model, and other sounds except the human voice are background sounds. In this case, the first target audio data is the human voice, and the first background audio data is the background sound.

[0114] See also Figure 7 , Figure 7 This is a schematic diagram of sound separation. The sounds in normal life and in film and television works are mixed together by various sounds, such as Figure 7 In the figure, sound signal 1 is the sound of a musical instrument, and sound signal 2 is the sound of a person singing. A mixed sound signal is a sound signal in which the sound of a musical instrument and the sound of a person singing are mixed together during recording, audio and video production. Traditional sound effect algorithms based on fixed logic operations cannot separate the two sounds in a mixed sound signal, but with the help of AI technology, sound separation can be achieved, and audio 1 similar to a musical instrument and audio 2 similar to a human voice can be obtained.

[0115] Alternatively, the first audio data includes a variety of mixed sounds such as human voices, car sounds, gunshots, and background music. The human voice can be separated by a human voice separation model, the car sound can be separated by a pre-trained car sound separation model, and the gunshot can be separated by a pre-trained gunshot separation model. The other sounds in the first audio data except the separated human voices, car sounds, and gunshots are taken as background sounds. At this time, the first target audio data may include human voices, car sounds, and gunshots, and the first background audio data is the background sound.

[0116] The user can select a sound control mode according to his or her own preferences, and the first gain and the second gain can be determined according to the sound control mode. The controller 250 is configured to: perform gain processing on the first target audio data according to the first gain to obtain the second target audio data; perform gain processing on the first background audio data according to the second gain to obtain the second background audio data. That is, the first target audio data and the first background audio data are subjected to gain processing of different sizes to enhance the first target audio data or the first background audio data. Afterwards, the second target audio data and the second background audio data are merged, and sound enhancement processing is performed to obtain the second audio data.

[0117] It is understandable that if the first gain and the second gain are both 0 dB, then the signal after merging the second target audio data and the second background audio data is highly similar to the signal before the sound separation. The merged signal is subjected to sound enhancement processing by a sound enhancement algorithm to obtain the second audio data. The sound enhancement algorithm includes but is not limited to AGC (automatic gain control), DRC (Dynamic range compression), EQ (equalizer), virtual surround, etc.

[0118] The audio output interface 270 is configured to output second audio data.

[0119] In some embodiments, the controller 250 is configured to: determine the type of sound enhancement mode corresponding to the first audio data according to the sound control mode corresponding to the display device; the type of sound enhancement mode refers to the type of sound that the user wants to enhance, and determine the first gain and the second gain corresponding to the type of sound enhancement mode according to the sound control mode corresponding to the display device. Different types of sound enhancement modes have different corresponding first gains and second gains.

[0120] In some embodiments, according to the sound control mode, the type of sound enhancement mode corresponding to the first audio data can be determined first, and the type of sound enhancement mode indicates the type of sound that the user wants to enhance. Different types of sound enhancement modes may also have different methods for determining the first gain and the second gain. Therefore, after determining the type of sound enhancement mode, the first gain and the second gain corresponding to the type of sound enhancement mode can be determined according to the sound control mode. For example, the type of sound enhancement mode may include a sound enhancement mode and a background enhancement mode. The sound enhancement mode indicates that the user wants to enhance the first target audio data, and the background enhancement mode indicates that the user wants to enhance the first background audio data.

[0121] In some embodiments, the controller 250 is configured as follows: if the type of the sound effect enhancement mode corresponding to the first audio data is the sound enhancement mode, that is, the first target audio data is enhanced, and the first gain is greater than the second gain. If the type of the sound effect enhancement mode corresponding to the first audio data is the background enhancement mode, that is, the first background audio data is enhanced, and the first gain is less than the second gain.

[0122] Assuming that the first gain is G1 and the second gain is G2, if the user wants to enhance the first target audio data, the first target audio data can be enhanced without changing the first background audio data, that is, G1 can be a value greater than 0dB and G2 is equal to 0dB. If the user wants to enhance the first background audio data, the first target audio data can be unchanged and the first background audio data can be enhanced, that is, G1 is equal to 0dB and G2 is a value greater than 0dB.

[0123] In some embodiments, in order to ensure that there is no positive gain that causes audio signal distortion, the range of G1 and G2 can be [-911BB, 0dB]. If the type of the sound enhancement mode corresponding to the first audio data is the sound enhancement mode, the first gain is set to 0dB, and the second gain is determined according to the sound control mode, wherein the second gain is less than 0dB. In this way, without changing the first target audio data, the purpose of enhancing the first target audio data is achieved by weakening the first background audio data. If the type of the sound enhancement mode corresponding to the first audio data is the background enhancement mode, the first gain is determined according to the sound control mode, and the second gain is set to 0dB, wherein the first gain is less than 0dB. In this way, without changing the first background audio data, the purpose of enhancing the first background audio data is achieved by weakening the first target audio data.

[0124] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes. The user can adjust the clarity of the human voice according to their needs and preferences, and select a target sound clarity control mode from multiple preset sound clarity control modes, each preset sound clarity control mode has a corresponding numerical value. For example, multiple preset sound clarity control modes are divided into multiple different levels, and each level corresponds to a different numerical value. The user can also select a target sound effect mode from a variety of preset sound effect modes (such as standard mode, music mode, movie mode, etc.), and each preset sound effect mode has a corresponding numerical value.

[0125] Among them, the preset sound clarity control mode indicates the sound clarity of the display device, which may include multiple different levels. If the value corresponding to the preset sound clarity control mode is M1, the user can adjust the sound clarity through the menu. In order to simplify the calculation, the menu adjustment value can be normalized to a value within [0,1], that is, M1 is greater than or equal to 0 and less than or equal to 1. Assuming that 0.5 represents the default value of the display device when it leaves the factory, a value greater than 0.5 indicates a higher sound clarity, and a value less than 0.5 indicates a lower sound clarity.

[0126] The preset sound effect mode indicates the sound effect mode of the display device, which may include standard sound effects, music sound effects, movie sound effects, and news sound effects, etc. If the value corresponding to the preset sound effect mode is M2, M2 may also be a normalized value, assuming that the value of M2 in the standard mode is 0.5, the value of M2 in the music mode is 0.6, the value of M2 in the movie mode is 0.7, and the value of M2 in the news mode is 0.8.

[0127] The sound control modes corresponding to the display device include: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of a plurality of preset sound clarity control modes, and the target sound effect mode is one of a plurality of preset sound effect modes. The controller 250 is configured to: determine the type of the sound effect enhancement mode corresponding to the first audio data according to the first numerical value corresponding to the target sound clarity control mode and / or the second numerical value corresponding to the target sound effect mode. That is, a numerical value can be obtained according to the first numerical value and / or the second numerical value, and the type of the sound effect enhancement mode can be determined according to the numerical value. Furthermore, according to the first numerical value and / or the second numerical value, the first gain and the second gain corresponding to the type of the sound effect enhancement mode are determined.

[0128] In some embodiments, a third value can be obtained according to the first value and the second value, and the type of the sound enhancement mode is determined based on the third value. Assuming a normalized scenario, when the third value can be 1, it means that the first target audio data and the first background audio data are not enhanced. When the third value is greater than 1, it means that the first target audio data is enhanced, and when the third value is less than 1, it means that the first background audio data is enhanced. In some embodiments, the third value T can be expressed as the following formula:

[0129] T=(2×M1)×(2×M2) (1)

[0130] It can be understood that in the standard mode, the values ​​of M1 and M2 are different, and the expression of the third value T may also be different.

[0131] For example, when the user has not adjusted the sound control mode of the display device, the first value corresponding to the target sound clarity control mode is 0.5, and the second value corresponding to the target sound effect mode is also 0.5. At this time, T is equal to 1, and the first gain G1 and the second gain G2 can both be 0dB, that is, no gain processing is performed on the first target audio data and the first background audio data.

[0132] If the user adjusts the sound control mode of the display device, assuming that the first value corresponding to the target sound clarity control mode is 0.7, and the second value corresponding to the target sound effect mode is 0.8. At this time, the value of T is greater than 1, that is, the first target audio data is enhanced. As mentioned above, G1 and G2 are both numbers not greater than 0dB, so G1 can be set to 0 and G2 can be set to a value less than 0. In some embodiments, G2 can be expressed as the following formula:

[0133]

[0134] Of course, the method for determining G2 is not limited to this. For example, the formula (2) can be simply transformed.

[0135] On the contrary, if the user adjusts the sound control mode of the display device, the value of T is less than 1, indicating that the first background audio data is enhanced. At this time, G2 can be set to 0 and G1 can be set to a value less than 0. In some embodiments, G1 can be expressed as the following formula:

[0136]

[0137] Of course, the method for determining G1 is not limited to this. For example, the formula (3) can be simply transformed.

[0138] See also Figure 8 , Figure 8 A schematic diagram of an audio processing method in some embodiments of the present application. In a stereo display device, after the audio decoder performs decoding, the audio data of the left and right channels are independently processed for human voice separation, gain processing and sound effect enhancement, and then sent to the corresponding speakers.

[0139] Since most of the speakers of display devices are located at the bottom of the display device and emit sound downwards, and because the distance between the two speakers is relatively close (usually around 0.3 to 0.8 meters), and the viewing distance of people is generally around 2.0 to 2.5 meters, the angle is only 8° to 14°. The azimuth resolution limit of people is about 5°, that is, the distance between the two speakers of the display device is relatively close to the azimuth resolution limit of people. When creating a general stereo sound source (standard recording studio), the angle between the left and right channels is 60°. See Fig. 9A , Fig. 9A This is a diagram of the distribution angle of standard recording studio or home audio speakers. It can be seen that the angle between the left and right channels is 60°. When creating a sound source, the sound usually does not exist in only one channel, but in both channels at the same time. When the creator wants to show that the sound is on the left, the sound on the left will be louder than the right. On the contrary, when the creator wants to show that the sound is on the right, the sound on the right will be louder than the left.

[0140] However, such creations are made at an angle of 60°, see Fig. 9B , Fig. 9B This is a diagram of the angle of the TV speakers. At this angle, the virtual sound image of all sound elements is reduced, which is different from the creator's intention based on the 60° speaker. When the angle of the two speakers is reduced to 8° to 14°, if the left and right channel ratio is still the same as before, the sound image obtained by the audience will become very blurred, and it will be difficult to hear the direction of the sound.

[0141] In order to improve the sense of position, the signal ratio of the sound in the left and right speakers can be changed while the physical conditions such as the speakers remain unchanged. For example, if the energy distribution ratio of a sound in the source is 7:3 in the left and right channels, the sense of position of the sound field can be enhanced by changing the energy distribution ratio to 8:2 or 9:1. Fig. 9C , Fig. 9C A schematic diagram of changing the energy distribution relationship of the TV speakers. It can be seen that after changing the energy distribution relationship, the car is closer to the left speaker in the audience's subjective hearing.

[0142] Normally, the energy of the background music used to create atmosphere in movies and TV dramas is basically the same in the left and right channels, or the signal is exactly the same, but the typical sounds used to express the sense of direction are allocated to different channels. The typical sounds include but are not limited to human voices, gunshots, car sounds, airplane sounds, etc. If the energy of the left and right channels is still calculated according to the above method, and then the energy ratio of the two channels is simply changed, the center of the background music with the sound image in the center will also be changed, so this method is not advisable.

[0143] In some embodiments, the first audio data includes at least one third target audio data belonging to a preset sound type (for example, a sound type that expresses a sense of direction), and the third target audio data includes but is not limited to human voices, gunshots, car sounds, airplane sounds, etc.

[0144] In order to solve the above problem, the controller 250 is further configured to separate at least one third target audio data and third background audio data from the first audio data.

[0145] As mentioned above, the first audio data refers to audio data containing at least two mixed sounds, and human voices, gunshots, car sounds, etc. can be separated from the first audio data through trained and different neural network models. The third target audio data is a type of audio data. The first audio data can include one or more third target audio data, and the audio data in the first audio data other than the third target audio data is the third background audio data. For example, when the first audio data includes human voices and car sounds, the first audio data includes two third target audio data, namely human voices and car sounds, and the sounds other than human voices and car sounds are background sounds. For each type of third target audio data, the following process can be performed.

[0146] Since the third target audio data is used to express a sense of orientation, the third target audio data includes audio data of at least two different channels (e.g., a first channel and a second channel). In some embodiments, the first channel and the second channel may be a left channel and a right channel, respectively. For example, the third target audio data includes audio data of two channels, i.e., the first channel initial target audio data and the second channel initial target audio data. The first channel initial target audio data and the second channel initial target audio data may be left channel audio data and right channel audio data, respectively. For another example, the following first channel initial background audio data and the second channel initial background audio data may be left channel initial background audio data and right channel initial background audio data, respectively.

[0147] It can be understood that the energies of the first channel initial target audio data and the second channel initial target audio data in the third target audio data are different. Therefore, the first energy value of the first channel initial target audio data and the second energy value of the second channel initial target audio data of a single third target audio data can be obtained, and the third gain corresponding to the first channel initial target audio data and the fourth gain corresponding to the second channel initial target audio data can be determined based on the first energy value and the second energy value.

[0148] The initial target audio data of the first channel is gain processed according to the third gain to obtain the first channel first gain audio data, that is, the first channel audio data after gain processing; the initial target audio data of the second channel is gain processed according to the fourth gain to obtain the second channel first gain audio data, that is, the second channel audio data after gain processing; wherein the third gain and the fourth gain are determined according to the first energy value and the second energy value. In this way, the initial target audio data of the first channel is gain processed according to the third gain and the initial target audio data of the second channel is gain processed according to the fourth gain, respectively, so that the sense of direction of the third target audio data can be further improved. At the same time, the center of the third background audio data can be unchanged.

[0149] For example, if the first energy value of the initial target audio data of the first channel is greater than the second energy value of the initial target audio data of the second channel, the third gain may be greater than the fourth gain, for example, the third gain may be set to a value greater than 0dB, and the fourth gain may be set to 0dB, that is, no gain processing is performed on the initial target audio data of the second channel. If the first energy value is equal to the second energy value, it means that the energies of the two are equal, the third gain is equal to the fourth gain, or no processing is performed. If the first energy value is less than the second energy value, the third gain may be less than the fourth gain, for example, the third gain is set to 0dB, that is, no gain processing is performed on the initial target audio data of the first channel, and the fourth gain is set to a value greater than 0dB.

[0150] In some embodiments, in order to ensure that there is no positive gain causing audio signal distortion, if the first energy value is greater than the second energy value, the third gain can be set to 0dB, and the fourth gain can be determined based on the first energy value and the second energy value, wherein the fourth gain is less than 0dB. The initial target audio data of the first channel is gain processed according to the third gain to obtain the first channel first gain audio data; the initial target audio data of the second channel is gain processed according to the fourth gain to obtain the second channel first gain audio data.

[0151] If the first energy value is less than the second energy value, a third gain may be determined according to the first energy value and the second energy value, and the third gain is less than 0 dB, and the fourth gain is set to 0 dB. Gain processing is performed on the initial target audio data of the first channel according to the third gain to obtain first channel first gain audio data; and gain processing is performed on the initial target audio data of the second channel according to the fourth gain to obtain second channel first gain audio data.

[0152] Finally, the first channel first gain audio data and the first channel initial background audio data of the third background audio data are merged, and sound effect enhancement processing is performed to obtain the first channel first enhanced audio data; the second channel first gain audio data and the second channel initial background audio data of the third background audio data are merged, and sound effect enhancement processing is performed to obtain the second channel first enhanced audio data.

[0153] By acquiring the first energy value of the first channel initial target audio data and the second energy value of the second channel initial target audio data of the third target audio data, the energy relationship between the first channel initial target audio data and the second channel initial target audio data can be analyzed. According to the energy relationship, different gain processing is performed on the first channel initial target audio data and the second channel initial target audio data, so that the audio data of the channel with high energy is made stronger, so as to better enhance the sense of direction of the sound and improve the effect of sound enhancement.

[0154] It should be noted that, in the case where the third target audio data includes audio data of more channels, the processing process is similar to this, and will not be repeated here.

[0155] The audio output interface 270 includes: a first output interface and a second output interface; the first output interface is configured to: output the first enhanced audio data of the first channel; the second output interface is configured to: output the first enhanced audio data of the second channel.

[0156] In some embodiments, the sound control mode, the first energy value, and the second energy value can also be considered simultaneously to perform gain processing on the third target audio data and the third background audio data. The controller 250 is also configured to: determine the fifth gain and the sixth gain corresponding to the single third target audio data according to the sound control mode, the first energy value, and the second energy value corresponding to the display device. The fifth gain and the sixth gain are the gains corresponding to the first channel initial target audio data and the second channel initial target audio data of the third target audio data, respectively. The fifth gain and the sixth gain can be different.

[0157] The seventh gain is determined according to the sound control mode corresponding to the display device; wherein the seventh gain refers to the gain corresponding to the third background audio data. Since the center of the third background audio data is not changed, the seventh gain is used to perform gain processing on the initial background audio data of the first channel and the initial background audio data of the second channel, that is, the same gain processing is performed on the initial background audio data of the first channel and the initial background audio data of the second channel.

[0158] Afterwards, the initial target audio data of the first channel is gain processed according to the fifth gain to obtain the second gain audio data of the first channel, i.e., the audio data of the first channel after gain processing. The initial target audio data of the second channel is gain processed according to the sixth gain to obtain the second gain audio data of the second channel, i.e., the audio data of the second channel after gain processing; the initial background audio data of the first channel and the initial background audio data of the second channel are gain processed according to the seventh gain respectively to obtain the first channel gain background audio data (i.e., the background audio data of the first channel after gain processing) and the second channel gain background audio data (i.e., the background audio data of the second channel after gain processing).

[0159] It should be noted that the first channel second gain audio data and the aforementioned first channel first gain audio data are both first channel audio data after gain processing is performed on the first channel initial target audio data, and the difference is that the corresponding gain values ​​during the gain processing are different. Similarly, the second channel second gain audio data and the aforementioned second channel first gain audio data are both second channel audio data after gain processing is performed on the second channel initial target audio data, and the difference is that the corresponding gain values ​​during the gain processing are different.

[0160] The audio output interface 270 includes: a first output interface and a second output interface; the first output interface is configured to: output the second enhanced audio data of the first channel; the second output interface is configured to: output the second enhanced audio data of the second channel.

[0161] In some embodiments, the controller 250 is configured to: determine the type of sound effect enhancement mode corresponding to the first audio data according to the sound control mode corresponding to the display device; determine the energy magnitude relationship between the left and right channels according to the first energy value of the initial target audio data of the first channel and the second energy value of the initial target audio data of the second channel. Determine the fifth gain and the sixth gain corresponding to the type of sound effect enhancement mode and the energy magnitude relationship between the left and right channels according to the sound control mode corresponding to the display device, the first energy value and the second energy value; determine the seventh gain corresponding to the type of sound effect enhancement mode and the energy magnitude relationship between the left and right channels according to the sound control mode corresponding to the display device.

[0162] Different types of sound enhancement modes result in different gain processing methods for the third target audio data and the third background audio data. Different energy size relationships between the left and right channels result in different gain processing methods for the first channel initial target audio data and the second channel initial target audio data. The type of sound enhancement mode is used to determine whether to enhance the third target audio data or the third background audio data, and the energy size relationship between the left and right channels is used to determine whether to enhance the first channel initial target audio data or the second channel initial target audio data. Therefore, different types of sound enhancement modes and energy size relationships between the left and right channels correspond to different fifth gains, sixth gains, and seventh gains.

[0163] For example, if the type of the sound enhancement mode is the sound enhancement mode, the fifth gain and the sixth gain are both greater than the seventh gain, and if the first energy is greater than the second energy, the fifth gain is greater than the sixth gain. If the first energy is equal to the second energy, the fifth gain may be equal to the sixth gain. If the first energy is less than the second energy, the fifth gain is less than the sixth gain.

[0164] If the type of the sound effect enhancement mode is the background enhancement mode, the fifth gain and the sixth gain are both less than the seventh gain, and if the first energy is greater than the second energy, the fifth gain is greater than the sixth gain. If the first energy is equal to the second energy, the fifth gain may be equal to the sixth gain. If the first energy is less than the second energy, the fifth gain is less than the sixth gain.

[0165] In some embodiments, in the sound enhancement mode, the third value T may be greater than 1. Assuming that the first energy value is P L , the second energy value is P R , if P L Greater than P R , at this time, the fifth gain can be equal to 0dB, and the sixth gain and the seventh gain are both less than 0dB. 1L =0dB, the sixth gain can be expressed as the following formula:

[0166]

[0167] The seventh gain can be expressed by the following formula:

[0168]

[0169] If the third value T is greater than 1, P L is less than or equal to P R , at this time, the sixth gain is equal to 0 dB, and the fifth and seventh gains are both less than 0 dB. For example, the fifth gain can be expressed by the following formula:

[0170]

[0171] The sixth gain G 1R = 0 dB, and the seventh gain can be expressed by the following formula:

[0172]

[0173] If the third value T is less than or equal to 1, P L is greater than P R , at this time, the fifth and sixth gains are both less than 0, and the seventh gain is equal to 0 dB. For example, the fifth gain can be expressed by the following formula:

[0174] G 1L = 20×logT (8)

[0175] The sixth gain can be expressed by the following formula:

[0176]

[0177] The seventh gain G2 = 0 dB.

[0178] If the third value T is less than or equal to 1, P L is less than or equal to P R , at this time, the fifth and sixth gains are both less than 0 dB, and the seventh gain is equal to 0 dB. For example, the fifth gain can be expressed by the following formula:

[0179]

[0180] The sixth gain can be expressed by the following formula:

[0181] G 1R = 20×log T(11)

[0182] The seventh gain G2 = 0 dB.

[0183] Among them, when x is between (0.5, 1), f(x) > x; when x is between (0, 0.5), f(x) < x; when x is equal to 0.5, f(x) = 0.5. See Fig.10 , Fig.10 This is a schematic diagram of the function f(x) in some embodiments of the present application. It can be seen that the trend of f(x) changing with x satisfies the above relationship. It should be noted that the trend of f(x) changing with x is not limited to this, for example, it can be an exponential type, a parabola type, or a combination of multiple forms, as long as the above relationship is satisfied.

[0184] It should be noted that the determination method of the fifth gain, the sixth gain and the seventh gain is not limited thereto, for example, a simple deformation of the above formula may be used, etc. Moreover, the fifth gain, the sixth gain and the seventh gain may also be greater than or equal to 0 dB.

[0185] The controller 250 is configured to: merge the first channel second gain audio data and the first channel gain background audio data, and perform sound effect enhancement processing to obtain and output the first channel second enhanced audio data; merge the second channel second gain audio data and the second channel gain background audio data, and perform sound effect enhancement processing to obtain and output the second channel second enhanced audio data.

[0186] The present application can also simultaneously consider the control mode and the energy relationship between the initial target audio data of the first channel and the initial target audio data of the second channel to determine the gain values ​​corresponding to the initial target audio data of the first channel and the initial target audio data of the second channel, thereby further improving the effect of sound enhancement.

[0187] As mentioned above, the sound separation algorithm usually uses artificial intelligence technology. After the sound is processed by artificial intelligence, it is processed by sound enhancement, which may cause the sound processing to take a long time, so that the speaker output time is later than the image, that is, the sound and picture are out of sync. In order to solve this problem, the present application also provides a solution.

[0188] The implementation of this solution in Android system can be as follows Fig.11AAs shown, the Android system mainly includes an application layer, a middleware and a core layer. The implementation logic can be in the middleware, and the middleware includes: an audio decoder, a sound separation module, a sound enhancement module, a gain control module, a delay module and an audio output interface. The audio decoder is used to perform audio decoding processing on a signal source input through a broadcast signal, a network, a USB or an HDMI, etc., to obtain audio data. The sound separation module is used to perform sound separation on the decoded audio data, for example, the human voice audio can be separated by a human voice separation method. The sound enhancement module is used to perform sound enhancement processing on the decoded audio data, and the gain control module can obtain the user's sound control mode for the display device, and perform different gain processing on the separated audio and the audio after sound enhancement. Since the time consumed by sound separation and sound enhancement is usually different, the delay module can perform delay processing on the two audio data after gain processing. The merging module is used to merge the two audios after gain processing to obtain merged audio data. The audio output interface is used to output the merged audio data.

[0189] It should be noted that the above implementation logic can be implemented in the middleware as well as in the core layer. Alternatively, it can be implemented in the middleware and the core layer. For example, the audio decoder and the sound separation module can be implemented in the middleware, and the remaining modules can be implemented in the core layer.

[0190] With the above Fig.11A Correspondingly, Fig. 11B A schematic diagram of an audio processing method in some embodiments of the present application. After the audio decoder decodes the acquired sound signal, the first audio data can be obtained. The sound separation module can use AI technology to achieve sound separation of the first audio data through a pre-trained neural network model to obtain the first target audio data. The first target audio data can be a human voice, a car sound, etc. At the same time, the first audio data can be subjected to sound enhancement processing to obtain the second audio data. The gain control module can obtain the first gain and the second gain according to the sound control mode, and the values ​​of the first gain and the second gain are not equal. According to the first gain, the first target audio data is gain-processed to obtain the second target audio data, and according to the second gain, the second audio data is gain-processed to obtain the third audio data. According to the time consumed by the sound separation module and the time consumed by the sound enhancement module, it is determined to delay the second target audio data, or to delay the third target audio data. Afterwards, the second target audio data and the third audio data are merged.

[0191] It can be seen that through sound separation, only one sound, namely the first target audio data, can be separated without separating the background sound, thereby reducing the time consumed by sound separation. In addition, by processing sound separation and sound enhancement in parallel instead of in series, the time consumed by the entire audio processing flow can be further shortened, thereby improving the effect of audio-visual synchronization.

[0192] Based on this, some embodiments of the present application further provide a display device 200 including:

[0193] The controller 250 may also be configured to perform sound separation and sound effect enhancement processing on the acquired first audio data to obtain first target audio data and second audio data.

[0194] The first audio data refers to audio data containing at least two mixed sounds. For example, the first audio data may include human voices and background music. The first target audio data usually refers to the audio data that the user wants to enhance, which may be human voices or other sounds, for example, suitable for scenes such as watching movies and TV series, listening to music, etc. The human voice can be separated by a pre-trained human voice separation model. In this case, the first target audio data is the human voice. Alternatively, the first audio data includes a variety of mixed sounds such as human voices, car sounds, gunshots, and background music. The car sound can be separated by a pre-trained car sound separation model. In this case, the first target audio data is the car sound. The above-mentioned sound separation process can only separate one sound (the first target audio data). Compared with separating multiple sounds, the time consumed by the separation process can be reduced.

[0195] The present application can also perform sound enhancement processing on the first audio data. In order to reduce the total duration of audio processing, the sound enhancement processing process and the sound separation processing process can be processed in parallel instead of in series, which can further shorten the duration of the entire audio processing flow, thereby improving the effect of audio-visual synchronization. Among them, the sound enhancement algorithm includes but is not limited to automatic gain control, dynamic range planning, equalizer, virtual surround, etc.

[0196] The first target audio data is gain processed according to the first gain to obtain the second target audio data; the second audio data is gain processed according to the second gain to obtain the third audio data, wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device. The first target audio data and the second audio data are gain processed respectively by different gains to improve the overall effect of the sound enhancement.

[0197] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding numerical value, and each preset sound effect mode has a corresponding numerical value. The user can adjust the sound control mode of the display device according to his or her needs and preferences. After the display device obtains the sound control mode set by the user, the sound control mode corresponding to the display device includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of the multiple preset sound clarity control modes, and the target sound effect mode is one of the multiple preset sound effect modes. Therefore, according to the first numerical value corresponding to the target sound clarity control mode and / or the second numerical value corresponding to the target sound effect mode, the first gain and the second gain are determined, wherein the first gain may be greater than the second gain.

[0198] As mentioned above, the first target audio data generally refers to the audio data that the user wants to enhance. Therefore, when the types of sound enhancement modes include sound enhancement mode and background enhancement mode, it can be applicable to the scene of sound enhancement mode. Assuming a normalized scenario, a third value is obtained according to the first value and the second value, and when the third value is greater than 1, the first target audio data is enhanced. In some embodiments, the third value T can be expressed as: (2×M1)×(2×M2). It can be understood that the values ​​of M1 and M2 are different in the standard mode, and the expression of the third value T may also be different.

[0199] In some embodiments, in order to ensure that there is no positive gain causing audio signal distortion, the first gain and the second gain may be less than or equal to 0dB. For example, the first gain may be set to 0dB; the second gain may be determined based on the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode, so that the second gain is less than 0dB. It should be noted that the method for determining the first gain and the second gain may refer to the description in the aforementioned embodiment, and will not be repeated here.

[0200] Since the process of sound separation and the process of sound effect enhancement processing for the first audio data can be processed in parallel, and the time consumed for sound separation and the time consumed for sound effect enhancement processing for the first audio data are usually different, therefore, if the second target audio data and the third audio data are directly merged, the sound signals cannot overlap, resulting in an echo problem.

[0201] In order to solve this problem, the second target audio data or the third audio data can be delayed to synchronize the second target audio data with the third audio data, and the second target audio data and the third audio data can be combined to obtain the fourth audio data. In this way, it can avoid the problem that the sound signals cannot overlap and cause echoes.

[0202] The audio output interface 270 is configured to output fourth audio data.

[0203] In some embodiments, the controller 250 is configured to: obtain the first duration consumed during sound separation and the second duration consumed during sound enhancement processing; and perform delay processing on the second target audio data or the third audio data according to the first duration and the second duration. In other words, the duration consumed by the sound separation and the sound enhancement processing can be directly counted. If the duration consumed by the sound separation is short, the second target audio data can be delayed; if the duration consumed by the sound enhancement processing is short, the third audio data can be delayed, and finally the second target audio data and the third audio data are synchronized.

[0204] When the computing unit for running sound separation and sound enhancement is dedicated or the system resources are sufficient, the first duration and the second duration can be calculated based on measurements to obtain one or several sets of fixed values. However, in actual scenarios, the sound separation algorithm is usually not dedicated to the chip of the display device, but is used together with the image AI algorithm using the APU (Accelerated Processing Unit) or GPU (graphics processing unit). As a result, the computing time for sound separation is often not a fixed value, but has a certain degree of volatility. The actual measurement shows that the volatility is between ±20ms. Fig. 6A The system architecture shown in the figure, although this fluctuation will affect the synchronization of audio and video, but the range of audio and video delay that people can tolerate is usually ±30ms. Therefore, this fluctuation is acceptable. Fig.11A In the system architecture shown, the same sound is processed in two links and then merged. If the error of the same sound exceeds ±5ms, it will cause obvious sound quality problems, so precise alignment is required.

[0205] because Fig.11A In the system architecture shown, there is a situation where the same sound is processed in two links, so there is a certain correlation between the first target audio data and the second audio data. In some embodiments, the controller 250 is configured to: determine the time difference between the first target audio data and the second audio data according to the correlation between the first target audio data and the second audio data; and perform delay processing on the second target audio data or the third audio data according to the time difference.

[0206] In some cases, if the time consumed by the sound separation and sound enhancement processing cannot be directly counted or the statistics are inaccurate, the correlation between the first target audio data and the second audio data can also be analyzed. According to the correlation, the time difference between the first target audio data and the second audio data is determined, and then the delay processing is performed.

[0207] In some embodiments, the correlation between the first target audio data and the second audio data can be compared by a time domain window function. The controller 250 is configured to: obtain a first audio segment of the first target audio data within a time period t, and the first audio segment can be an audio segment of any duration t; obtain a second audio segment of the second audio data within the time period t (i.e., the same time as the first audio segment), as well as a plurality of third audio segments before the second audio segment and a plurality of fourth audio segments after the second audio segment; wherein the durations corresponding to the third audio segment and the fourth audio segment are both equal to the duration of the time period t.

[0208] Determine the correlations between the first audio segment and the second audio segment, the third audio segment and the fourth audio segment, respectively, and determine the audio segment with the highest correlation; and determine the time difference between the audio segment with the highest correlation and the first audio segment as the time difference between the first target audio data and the second audio data.

[0209] That is, a segment is cut from the first target audio data, recorded as w, and at the same time, multiple segments of the second audio data in the same time period are cut using the same window, recorded as w(x), and the convolution values ​​of w and all the data in w(x) are calculated one by one to obtain the correlation data of w and w(x). The time difference between w(x) and w with the highest correlation is determined as the time difference between the first target audio data and the second audio data.

[0210] Alternatively, a segment may be captured from the second audio data, and multiple segments of the first target audio data within the same time period may be captured using the same window, and correlation calculation may be performed in the same manner as described above to determine the time difference between the first target audio data and the second audio data.

[0211] It should be noted that the window width is closely related to the delay calculation accuracy. The window width is t, and the calculation accuracy is also t. However, the smaller t is, the larger the corresponding amount of calculation will be. In addition, if the data within t is calculated point by point, the amount of calculation will also be relatively large. The method of alternate point calculation can be used to reduce the amount of calculation by half. The specific accuracy can be selected according to the computing power of the processor.

[0212] In a normal stereo TV, the left and right channels are separated and played back independently. Figure 8The method shown in the system architecture is to perform gain processing on the two audio data obtained after separation through the first gain and the second gain respectively, and then merge them, and then send them to the corresponding speakers after sound enhancement processing. Although this architecture is simple, the audio data of the left and right channels need to be calculated by the sound separation algorithm, and the sound separation algorithm usually uses the same physical operation processor and is superimposed in time, so it requires a high AI processing capability of the chip. It can be seen that how to reduce the application of sound separation determines whether this solution can be applied in more display devices.

[0213] See also Fig.12 , Fig.12 Schematic diagram of an audio processing method in some embodiments of the present application. Fig.12 As shown, the left channel audio data and the right channel audio data output by the audio decoder are not only subjected to sound enhancement and gain processing respectively, but are also combined into one signal for sound separation, and the separated first target audio data is subjected to gain processing. The sound signals of the two links are then delayed, and the sound signals in the sound separation link are finally superimposed on the left and right channels in the sound enhancement link. In this way, the amount of sound separation calculations can be reduced by half, making the implementation feasibility of this solution higher.

[0214] In some embodiments, the first audio data includes first channel initial audio data and second channel initial audio data. That is, the first audio data may include audio data of two channels, for example, the first channel initial audio data and the second channel initial audio data may be left channel audio data and right channel audio data contained in the first audio data.

[0215] The controller 250 is configured to perform sound effect enhancement processing on the first channel initial audio data and the second channel initial audio data respectively to obtain first channel sound effect enhanced audio data (i.e., the first channel audio data after sound effect enhancement) and second channel sound effect enhanced audio data (i.e., the second channel audio data after sound effect enhancement).

[0216] It should be noted that, for the sound separation process, the first audio data (i.e., the audio data after the initial audio data of the first channel and the initial audio data of the second channel are merged) can be directly separated to obtain the first target audio data, so as to reduce the computational complexity of the sound separation by half.

[0217] The first target audio data can be gain processed according to the first gain to obtain the second target audio data; the first channel sound effect enhanced audio data and the second channel sound effect enhanced audio data can be gain processed according to the second gain to obtain the first channel target audio data and the second channel target audio data.

[0218] Delay processing is performed on the second target audio data or the first channel target audio data to synchronize the second target audio data with the first channel target audio data; and delay processing is performed on the second target audio data or the second channel target audio data to synchronize the second target audio data with the second channel target audio data.

[0219] Similarly, the duration consumed by sound separation and the duration consumed by sound enhancement processing are usually different. Therefore, delay processing can be performed first and then merged. In some embodiments of the present application, the first duration consumed by sound separation, the second duration consumed by sound enhancement processing of the initial audio data of the first channel, and the third duration consumed by sound enhancement processing of the initial audio data of the second channel can also be counted. According to the first duration and the second duration, the second target audio data or the first channel target audio data is delayed; according to the first duration and the third duration, the second target audio data or the second channel target audio data is delayed.

[0220] Alternatively, it is also possible to determine the correlation between the first target audio data and the first channel sound effect enhanced audio data, and perform delay processing on the second target audio data or the first channel target audio data based on the correlation; determine the correlation between the first target audio data and the second channel sound effect enhanced audio data, and perform delay processing on the second target audio data or the second channel target audio data based on the correlation.

[0221] It is understandable that the second duration consumed by performing sound enhancement processing on the initial audio data of the first channel and the third duration consumed by performing sound enhancement processing on the initial audio data of the second channel are usually equal, or the difference is small and can be ignored. Therefore, in order to reduce the amount of calculation, it is also possible to only count the duration consumed by one of the sound enhancement processes. Alternatively, it is sufficient to determine the correlation between the first target audio data and the first channel sound enhancement audio data (second channel sound enhancement audio data).

[0222] Then, the second target audio data is merged with the first channel target audio data and the second channel target audio data respectively to obtain first channel merged audio data and second channel merged audio data;

[0223] The audio output interface 270 includes: a first output interface and a second output interface; the first output interface is configured to: output the first channel merged audio data; the second output interface is configured to: output the second channel merged audio data.

[0224] As mentioned above, sound separation can be achieved through artificial intelligence technology. In the case where the first audio data includes the first channel initial audio data and the second channel initial audio data, if the first channel initial audio data and the second channel initial audio data are respectively subjected to sound separation and sound effect enhancement processing, the sound separation will consume a large amount of computation. Therefore, the processing power of the chip in the display device is required to be high. In order to solve this problem, the first channel initial audio data and the second channel initial audio data can be merged, that is, the first audio data can be directly sound separated, and the first target audio data obtained by separation is gain processed to obtain the second target audio data. The second target audio data is merged with the first channel target audio data and the second channel target audio data respectively. In this way, the amount of computation for sound separation can be reduced by half, so that the present solution can be implemented even when the processing power of the chip is not very high, thereby improving the applicability of the present solution.

[0225] With the improvement of chip AI computing power, machine learning has been widely used in the fields of image and sound, and even combined in many scenarios. This application also provides a solution to improve the stereo effect of sound. The implementation in the Android system can be as follows Fig.13A As shown, the Android system mainly includes an application layer, a middleware and a core layer. The implementation logic can be in the middleware, and the middleware may include: an audio decoder, a human voice separation module, a gain control module, an image decoder, a sound distribution module, a merging module, a sound effect enhancement module and an audio output interface. The audio decoder is used to perform audio decoding processing on a signal source input through a broadcast signal, a network, a USB or an HDMI, etc., to obtain audio data. The human voice separation module is used to perform human voice separation on the decoded left channel audio data and the right channel audio data, respectively, to obtain left channel human voice audio data and left channel background audio data, as well as right channel human voice audio data and right channel background audio data. The sound distribution module is used to perform lip movement detection on the image decoded and output by the image decoder to determine the weight of the human voice audio output by each audio output interface and the weight of the background audio. The merging module is used to merge the human voice audio and the background audio according to the weight of the human voice audio and the weight of the background audio to obtain the merged audio data. The sound effect enhancement module is used to perform sound effect enhancement processing on the merged audio data to obtain the audio data after sound effect enhancement. The audio output interface is used to output the audio data after sound effect enhancement.

[0226] It should be noted that the above implementation logic can be implemented in the middleware as well as in the core layer. Alternatively, it can be implemented in the middleware and the core layer. For example, the audio decoder and the human voice separation module can be implemented in the middleware, and other modules can be implemented in the core layer.

[0227] With the above Fig.13A Correspondingly, Fig. 13BA schematic diagram of an audio processing method in some embodiments of the present application. The audio decoder can decode and output left channel audio data and right channel audio data, and can perform vocal separation on the left channel audio data and the right channel audio data, respectively, to obtain left channel vocal audio data and left channel background audio data, as well as right channel vocal audio data and right channel background audio data. For example, the vocal separation of the left channel audio data and the vocal separation of the right channel audio data can be achieved through AI technology and a pre-trained neural network model. The left channel vocal audio data and the right channel vocal audio data are merged to obtain the target vocal audio data.

[0228] At the same time, the image decoder can decode and obtain the image at the moment when the left channel audio data and the right channel audio data are located, and perform lip movement detection on the image. According to the lip movement detection results, the weight of the target human voice audio data in each audio output interface is determined. In addition, the weight of the left channel background audio data and the right channel background audio data output by the audio output interface can be determined according to the coordinates of the audio output interface. Afterwards, according to the weight of the target human voice audio data in each audio output interface, the audio output interface outputs the weight of the left channel background audio data and the right channel background audio data, and merges the human voice audio and the background audio. Finally, the merged audio is processed for sound enhancement and output.

[0229] It can be seen that for stereo display devices, after separating the left channel audio data and the right channel audio data, the separated left channel audio data and the right channel audio data can be merged first. Then, according to the position of the person speaking in the image, the human voice weight corresponding to each audio output interface is adjusted, that is, the weight corresponding to the output human voice audio, and according to the position of the audio output interface, the weight of the background audio output of each audio output interface is adjusted, so as to enhance the stereoscopic sense of the sound and improve the user's viewing experience.

[0230] In some embodiments of the present application, the display device 200 includes: a controller 250 and a plurality of audio output interfaces 270;

[0231] The controller 250 is configured to perform vocal separation on the acquired first channel audio data and second channel audio data respectively to obtain first channel first vocal audio data and first channel first background audio data, and second channel first vocal audio data and second channel first background audio data.

[0232] The first channel audio data and the second channel audio data are audio data of two different channels acquired at the same time, and the first channel audio data and the second channel audio data can make the sound more stereoscopic. For example, the first channel audio data and the second channel audio data can be left channel audio data and right channel audio data, respectively.

[0233] For the first channel audio data, first channel first human voice audio data and first channel first background audio data can be obtained by voice separation (such as artificial intelligence technology). The first channel first human voice audio data refers to the human voice in the first channel audio data, and the number of the first channel first human voice audio data can be multiple, that is, the human voices of multiple people can be extracted. The audio data other than the first channel first human voice audio data is the first channel first background audio data. Similarly, the second channel audio data can be subjected to voice separation to obtain the second channel first human voice audio data and the second channel first background audio data.

[0234] The first vocal audio data of the first channel and the first vocal audio data of the second channel are combined to obtain target vocal audio data.

[0235] In some embodiments of the present application, the separated first channel first voice audio data and second channel first voice audio data are not directly assigned to the first channel and the second channel to be merged with the background audio, but the first channel first voice audio data and the second channel first voice audio data are directly merged to obtain the target voice audio data. Then, according to the position of the person speaking in the image, the output of the target voice audio data on each audio output interface is allocated.

[0236] It should be noted that if the human voice audio includes multiple characters, for each character, the first human voice audio data of the first channel and the first human voice audio data of the second channel corresponding to the character are merged to obtain the target human voice audio data of the character. The method for allocating the target human voice audio data of each character is similar, and the target human voice audio data of one character is used as an example for explanation.

[0237] The controller 250 is configured to: obtain image data at the moment when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if the lip movement coordinates in the display device screen are detected, determine the vocal weight corresponding to the audio output interface based on the lip movement coordinates and the coordinates of the single audio output interface.

[0238] In the display device, in addition to the audio decoder decoding to obtain audio data, the image decoder can also decode to obtain the corresponding image data. In the case of audio and video synchronization, the image data corresponding to the audio can be obtained at the same time. Here, the image data at the moment of the first channel audio data and the second channel audio data can be obtained.

[0239] When human voice audio is extracted through voice separation, the image data usually has a corresponding person image. Therefore, lip movement detection can be performed on the image data to obtain lip movement coordinates, that is, the position coordinates of the person's lips. For example, artificial intelligence technology can be used to detect whether there is lip information in the image data and whether there is lip movement. If there are lips that are moving, the lip movement coordinates can be detected.

[0240] The lip movement coordinates indicate the position of the person in the image speaking on the screen, and the coordinates of the multiple audio output interfaces indicate the position of the output audio. It can be understood that the closer the lip movement coordinates are to the audio output interface, the greater the human voice weight corresponding to the audio output interface. The greater the human voice weight, the greater the energy of the human voice audio output by the audio output interface.

[0241] In some embodiments, the controller 250 is configured to: for each audio output interface, determine the area corresponding to the audio output interface on the screen based on the coordinates of the audio output interface; if the lip movement coordinates are within the area corresponding to the audio output interface, determine that the vocal weight corresponding to the audio output interface is a first value; if the lip movement coordinates are outside the area corresponding to the audio output interface, determine that the vocal weight corresponding to the audio output interface is a second value, and the second value is less than the first value.

[0242] In some embodiments of the present application, the screen may be pre-divided into corresponding areas for each audio output interface according to the coordinates of each audio output interface. It is understandable that the closer the lip movement coordinates are to the area corresponding to the audio output interface, the greater the voice weight corresponding to the audio output interface.

[0243] For example, the screen is divided into a left area and a right area, and the lower left and lower right of the screen each contain a speaker. The lip movement coordinates can be the position coordinates (x, y) of the actual pixel point. If the row resolution of the video being played is L and the column resolution is C, then the lip movement coordinates can be normalized to the following formula:

[0244] x'=x÷C,y'=y÷L (12)

[0245] If x' is less than 0.5, it means that the lip movement coordinates are in the left area, and if x' is greater than 0.5, it means that the lip movement coordinates are in the right area.

[0246] If the lip movement coordinates are in the left area of ​​the screen, then the vocal weight corresponding to the speaker at the bottom left of the screen and the vocal weight corresponding to the speaker at the bottom right of the screen can be set to 1 and 0 respectively, that is, the target vocal audio data is output through the speaker at the bottom left of the screen, and the speaker at the bottom right of the screen does not output the target vocal audio data. Alternatively, the vocal weight corresponding to the speaker at the bottom left of the screen and the vocal weight corresponding to the speaker at the bottom right of the screen can be set to 0.8 and 0.2 respectively, which can be determined by referring to the specific position of the lip movement coordinates in the left area. The closer the lip movement coordinates are to the left side of the left area, the greater the difference between the vocal weight corresponding to the speaker at the bottom left of the screen and the vocal weight corresponding to the speaker at the bottom right of the screen; the closer the lip movement coordinates are to the right side of the left area, that is, the closer to the middle of the screen, the smaller the difference between the vocal weight corresponding to the speaker at the bottom left of the screen and the vocal weight corresponding to the speaker at the bottom right of the screen.

[0247] See also Fig.14 , Fig.14 Figure 1 is a schematic diagram of speaker distribution. It can be seen that the display device includes four speakers, which are located at the lower left, lower right, upper left and upper right of the screen. The corresponding areas of the four speakers on the screen are as follows: Fig.14 As shown, they are the lower left area, lower right area, upper left area and upper right area of ​​the screen respectively. The lip movement coordinates are located in the upper left area, and the vocal weights corresponding to the four speakers at the lower left, lower right, upper left and upper right can be: 0, 0, 1 and 0 respectively. Alternatively, the vocal weights corresponding to the four speakers at the lower left, lower right, upper left and upper right can also be 0.2, 0, 0.8 and 0, etc., so that the final effect is located at the upper left of the screen in a subjective sense of hearing.

[0248] In some embodiments, the screen includes: a middle area and a non-middle area. The controller 250 is configured to: if the lip movement coordinates are located in the non-middle area, determine the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces. That is, the human voice weights corresponding to the multiple audio output interfaces respectively can be determined according to the above method.

[0249] If the lip movement coordinates are located in the middle area, the human voice weights corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction. That is, when the lip movement coordinates are located in the middle area of ​​the screen, the human voice weights corresponding to the various audio output interfaces can be flexibly configured according to the volume, direction and position relationship of the audio output interfaces, so that the final effect is preferably located in the center of the screen with a subjective sense of hearing.

[0250] For example, for Fig.14In the speakers shown, the speaker below the screen faces downward, and the speaker above the screen faces upward. Based on this orientation, the louder the volume of the speaker, the smaller the vocal gain corresponding to the speaker, and the smaller the volume of the speaker, the larger the vocal gain corresponding to the speaker. In this way, the subjective listening experience can be located in the middle of the screen. Alternatively, if the volume of the four speakers is the same, the vocal gains corresponding to the four speakers can be the same.

[0251] If multiple speakers are unevenly distributed around the screen, and the orientation of each speaker is not directly downward or upward, the vocal weight can be determined based on the specific position relationship, orientation, and volume of the multiple speakers, so that the subjective listening experience is located in the middle of the screen. It is understandable that the vocal weight corresponding to each speaker can include a variety of different situations.

[0252] The controller 250 is configured to: determine, according to the coordinates of the audio output interface, whether the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel.

[0253] For background audio data, since it is not related to human voice, it is possible to directly determine whether the audio output interface outputs the first channel first background audio data, the second channel first background audio data, or the first channel first background audio data and the second channel first background audio data based on the coordinates of the audio output interface.

[0254] In some embodiments, the screen includes: a left area and a right area. If the coordinates of the audio output interface correspond to the left area, it is determined that the audio output interface corresponds to the first channel initial background audio data; if the coordinates of the audio output interface correspond to the right area, it is determined that the audio output interface corresponds to the second channel initial background audio data. If the lower left and lower right of the screen each contain a speaker, corresponding to the left area and the right area respectively, the speaker at the lower left of the screen can output the first channel initial background audio data, and the speaker at the lower right of the screen can output the second channel initial background audio data.

[0255] In some embodiments, the screen includes: a left area, a middle area and a right area; the controller 250 is configured to: if the coordinates of the audio output interface correspond to the left area, determine that the audio output interface corresponds to the first channel first background audio data; if the coordinates of the audio output interface correspond to the right area, determine that the audio output interface corresponds to the second channel first background audio data; if the coordinates of the audio output interface correspond to the middle area, determine that the audio output interface corresponds to the first channel first background audio data and the second channel first background audio data.

[0256] For example, the lower left, lower center, and lower right of the screen each include a speaker, corresponding to the left area, middle area, and right area, respectively. The speaker at the lower left of the screen can output the first channel first background audio data, the speaker at the lower center of the screen can simultaneously output the first channel first background audio data and the second channel first background audio data, and the speaker at the lower right of the screen can output the second channel first background audio data.

[0257] The controller 250 is configured to: merge the product of the target vocal audio data and the vocal weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface, and perform sound enhancement processing to obtain the audio data corresponding to the audio output interface.

[0258] After determining the vocal audio (i.e., the product of the target vocal audio data and the vocal weight corresponding to the audio output interface) and background audio (i.e., the first background audio data of the first channel and / or the first background audio data of the second channel) corresponding to each audio output interface, the vocal audio and the background audio can be merged and sound enhancement processing can be performed to obtain the audio data corresponding to the audio output interface.

[0259] The single audio output interface 270 is configured to output audio data corresponding to the audio output interface.

[0260] In some embodiments, after the left channel audio data and the right channel audio data are respectively subjected to vocal separation, different gain processing may be performed on the vocal audio and the background audio to highlight and enhance the vocal audio or the background audio.

[0261] The controller 250 is also configured to: perform gain processing on the first channel first vocal audio data and the second channel first vocal audio data according to the first gain, respectively, to obtain the first channel second vocal audio data and the second channel second vocal audio data; perform gain processing on the first channel first background audio data and the second channel first background audio data according to the second gain, respectively, to obtain the first channel second background audio data and the second channel second background audio data; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device.

[0262] It should be noted that the first channel first vocal audio data and the second channel first vocal audio data both belong to vocal audio and can correspond to the same first gain, and the first channel first background audio data and the second channel first background audio data both belong to background audio and can correspond to the same second gain.

[0263] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding numerical value, and each preset sound effect mode has a corresponding numerical value; the sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of the multiple preset sound clarity control modes, and the target sound effect mode is one of the multiple preset sound effect modes; the controller 250 is configured to: determine the first gain and the second gain according to the first numerical value corresponding to the target sound clarity control mode and / or the second numerical value corresponding to the target sound effect mode.

[0264] It can be seen that the user can control the sound control mode of the display device according to his or her own preferences. Furthermore, the controller 250 can determine how to perform gain processing on the first channel first vocal audio data and the second channel first vocal audio data, and how to perform gain processing on the first channel first background audio data and the second channel first background audio data according to the sound control mode.

[0265] It should be noted that the method for determining the first gain and the second gain is the same as the method for determining the first gain and the second gain in the aforementioned embodiment. For details, please refer to the description in the aforementioned embodiment, which will not be repeated here.

[0266] The controller 250 is configured to: merge the second human voice audio data of the first channel and the second human voice audio data of the second channel to obtain target human voice audio data; and for each audio output interface, determine the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface according to the coordinates of the audio output interface; merge the product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface, and perform sound enhancement processing to obtain audio data corresponding to the audio output interface.

[0267] In some embodiments, the image data does not contain a person, or even if the image data contains a person, the person's lips are not displayed, for example, only the person's side face or the person's back is displayed. Or, even if the person's lips are displayed, the person's lips are not moving, and the lip movement coordinates cannot be detected at this time. The controller 250 is also configured to: if the lip movement coordinates are not detected, for each audio output interface, the human voice weight corresponding to each audio output interface can be directly determined based on the ratio of the energy of the first human voice audio data of the first channel to the energy of the first human voice audio data of the second channel, as well as the coordinates of the audio output interface.

[0268] For example, if the lower left and lower right sides of the screen each contain a speaker, and the ratio of the energy of the left-channel vocal audio data to the energy of the right-channel vocal audio data is greater than 1, the vocal weight corresponding to the speaker at the lower left side of the screen may be greater than the vocal weight corresponding to the speaker at the lower right side of the screen. If the ratio of the energy of the left-channel vocal audio data to the energy of the right-channel vocal audio data is 0.6:0.4, then the vocal weight corresponding to the speaker at the lower left side of the screen may be 0.6, and the vocal weight corresponding to the speaker at the lower right side of the screen may be 0.4. Alternatively, in order to further enhance the sense of directionality of the sound, the vocal weight corresponding to the speaker at the lower left side of the screen may be 0.7, and the vocal weight corresponding to the speaker at the lower right side of the screen may be 0.3.

[0269] At present, the karaoke function of a TV is usually completed in a singing APP. Singing APPs have rich functions and better user experience, but the media resources of singing APPs are relatively limited. For example, the original singer A of a song is a male singer, and the cover singer B is a female singer. When a female user C wants to sing this song, the singing APP may only record the accompaniment video of the original singer A, but there is no accompaniment video of singer B, resulting in the inability to find a suitable accompaniment. Or, although the method of subtracting two channels is adopted to eliminate the vocals in stereo songs. However, this method sometimes loses the bass in the song, and the resulting accompaniment sound is relatively weak, without the sense of singing accompaniment, and the user experience is poor.

[0270] Therefore, some embodiments of the present application also provide a technical solution, that is, by using vocal separation technology to remove the vocals in the song being played, so that users can find their favorite songs without relying on singing apps, such as playing familiar songs through online music players, or playing audio and video content that they paid for through TV. Then turn on the vocal elimination function to remove the original vocals in the audio, so as to achieve singing without being restricted by media resources. At the same time, according to the energy of the singing vocals collected by the microphone, the original vocals can be added to the accompaniment in whole or in part to avoid affecting the singing experience due to the singer's low singing level.

[0271] The implementation of this technical solution in the Android system can be as follows Fig.15AAs shown, the Android system mainly includes an application layer, a middleware and a core layer. The implementation logic can be in the middleware, and the middleware includes: an audio decoder, a vocal separation module, an audio input interface, an original singer volume control module, a merging module, a sound effect enhancement module, a gain control module, a delay module and an audio output interface. The audio decoder is used to perform audio decoding processing on a signal source input through a broadcast signal, a network, a USB or an HDMI, etc., to obtain audio data. The vocal separation module is used to perform vocal separation on the decoded audio data, and separate the original vocal audio and the accompaniment audio. The audio input interface is used to receive the singing audio input by the user, and the original singer volume control module determines the size of the original singer audio merged into the accompaniment audio, that is, the target vocal audio, according to the singing audio data and the separated original singer vocal audio. The merging module is used to merge the accompaniment audio, the singing audio and the target vocal audio to obtain the merged audio data. The sound effect enhancement module is used to perform sound effect enhancement processing on the merged audio data, and the audio output interface is used to output the audio data after the sound effect enhancement processing.

[0272] It should be noted that the above implementation logic can be implemented in the middleware as well as in the core layer. Alternatively, it can be implemented in the middleware and the core layer. For example, the audio decoder and the human voice separation module can be implemented in the middleware, and other modules can be implemented in the core layer.

[0273] With the above Fig.15A Correspondingly, Fig. 15B A schematic diagram of the audio processing method in some embodiments of the present application. After the audio decoder decodes and obtains the song audio data, the original vocal audio data and the accompaniment audio data are obtained by voice separation. At the same time, the microphone can collect the singing vocal audio data input by the user, and the target vocal audio data can be determined according to the original vocal audio data and the singing vocal audio data, that is, the size of the original vocal audio data merged into the accompaniment audio data. The singing vocal audio data, the target vocal audio data and the accompaniment audio data are merged, and the sound effect enhancement processing is performed before output.

[0274] Some embodiments of the present application further provide a display device 200, including:

[0275] The controller 250 is configured to: obtain song audio data, separate the vocals from the song audio data, and obtain original singer's vocal audio data and accompaniment audio data.

[0276] The song audio data can be any song, including songs included in the singing APP and songs not included in the singing APP. By performing vocal separation on the song audio data, for example, the original vocal audio data and the accompaniment audio data can be separated by artificial intelligence technology. It can be seen that for any song, the corresponding accompaniment audio data can be separated.

[0277] The controller 250 is also configured to: determine the original singer's gain based on the energy of the original singer's vocal audio data in each time period and the energy of the singing vocal audio data collected in the time period; and perform gain processing on the original singer's vocal audio data in the time period based on the original singer's gain to obtain the target vocal audio data.

[0278] During the singing process, the user can sing the song through the audio input interface (such as a microphone). At this time, the singing vocal audio data can be collected, and the user may have problems such as out of tune and poor pitch when singing. In addition, the vocal separation is calculated in real time by the main chip of the display device, and there may be problems such as unclean vocal separation or individual noise introduced during separation. In order to solve this problem, when the user is not singing or in a gap in singing, the original vocal audio separated from the vocals can be fully or partially merged into the accompaniment to set off the atmosphere of the singing scene, and when it is detected that the user is singing, the original vocal audio can be reduced or muted through the volume control of the original vocal audio, and the user's singing sound is played mainly.

[0279] Since each song corresponds to a relatively long time period, when processing audio data, the audio data can be processed according to a preset time period. That is, the audio data of each time period is processed in chronological order. The time period can be 0.8 seconds, 1 second, etc.

[0280] For each time period, the original singer's gain can be obtained based on the energy of the original singer's vocal audio data and the energy of the singing vocal audio data. The original singer's vocal audio data is gain processed using the original singer's gain to obtain the target vocal audio data, that is, the audio data merged into the accompaniment audio data.

[0281] In some embodiments, the original singer's gain is less than or equal to a preset gain threshold. For example, the preset gain threshold may be 0.1dB, 0dB, -0.1dB, etc. When the preset gain threshold is equal to 0dB, the original singer's gain is less than or equal to 0dB. When the original singer's gain is equal to 0dB, it means that all of the original singer's vocal audio data is merged into the accompaniment audio data; when the original singer's gain is less than 0dB, it means that part of the original singer's vocal audio data is merged into the accompaniment audio data. When the preset gain threshold is less than 0dB, the original singer's gain is also less than 0dB, indicating that part of the original singer's vocal audio data is merged into the accompaniment audio data. When the preset gain threshold is greater than 0dB, it means that the original singer's vocal audio data can be merged into the accompaniment audio data after enhancement processing.

[0282] In some embodiments, the controller 250 is configured as follows: if the energy of the singing vocal audio data is less than a preset energy threshold, which is a relatively small energy value, it can be considered that the user is not singing, and the original singer gain can be set to the preset gain threshold, for example, the original singer gain is set to 0dB, that is, the original singer audio data is directly used as the target vocal audio data. If the energy of the singing vocal audio data is greater than or equal to the preset energy threshold, it can be considered that the user has started singing, and the original singer gain is determined based on the energy ratio between the energy of the singing vocal audio data and the energy of the original singer audio data, so that the original singer gain is less than the preset gain threshold, that is, the energy of the original singer audio data can be reduced and used as the target vocal audio data.

[0283] In some embodiments, in order to ensure that the sound merged into the accompaniment audio data is relatively stable, rather than fluctuating with the volume of the singing vocal audio data, the energy ratio between the energy of the singing vocal audio data and the energy of the original singing vocal audio data and the corresponding relationship between the energy of the original singing vocal gain and the original singing gain can be pre-established. For example, when the energy ratio is within a certain energy ratio range, the original singing gain can correspond to the same value. For example, if the energy ratio is less than or equal to 0.25, it means that the energy of the singing vocal audio data is small, w = 0dB, and the original singing vocal audio data can be completely merged into the accompaniment audio data; if 0.25 < energy ratio < 0.75, it means that the energy of the singing vocal audio data is moderate, w = -6dB, and the original singing vocal audio data can be partially merged into the accompaniment audio data; if the energy ratio is greater than or equal to 0.75, it means that the energy of the singing vocal audio data is large, and the original singing vocal audio data can be completely turned off, and only the singing vocal audio data can be played.

[0284] The controller 250 is configured to merge the accompaniment audio data, the target vocal audio data and the singing vocal audio data within the time period and perform sound effect enhancement processing to obtain the target audio data. The present application further merges the target vocal audio data on the basis of merging the accompaniment audio data and the singing vocal audio data. The target vocal audio data refers to all of the original singing vocal audio data, or part of the original singing vocal audio data, so the target audio data finally outputted is richer and has better effect.

[0285] The audio output interface 270 is configured to output target audio data.

[0286] In some embodiments of the present application, for any song, the accompaniment audio data can be obtained by separating the human voice, so that the user can sing without being restricted by media resources. In addition, it can be determined whether to add the original vocal audio data to the accompaniment audio data, or to partially add the original vocal audio data, according to the user's singing level, so as to enhance the user's singing experience.

[0287] In some embodiments, the controller 250 is further configured to: obtain the original singer gain corresponding to the previous time period. If the original singer gain corresponding to the current time period is the same as the original singer gain corresponding to the previous time period, it means that the energy ratio between the energy of the singing vocal audio data corresponding to the previous time period and the energy of the original singing vocal audio data is smaller than the energy ratio corresponding to the current time period, for example, it is in the same energy ratio range, indicating that the user's singing is relatively stable and the user is very familiar with the sung songs. The time period can be extended to reduce the processing frequency of the above process until the extended time period is less than the first time threshold (for example, it can be 2 seconds, etc.). That is, reduce the processing frequency of the above process, instead of frequently merging the target vocal audio data obtained based on the original singing vocal audio data into the accompaniment audio data during the singing interval. Of course, the time period cannot be extended indefinitely to avoid the time period being too long and affecting the final singing effect.

[0288] If the original singer's gain corresponding to the current time period is different from the original singer's gain corresponding to the previous time period, it means that the volume changed when the user was singing and was not in sync with the original singer. The user may not be able to sing or sing inaccurately. At this time, shorten the time period, that is, quickly call out the target audio data, and merge the target audio data into the accompaniment audio data until the shortened time period is greater than the second time threshold (for example, it can be 0.25 seconds, etc.), where the first time threshold is greater than the second time threshold.

[0289] The above audio processing process can improve the effect of accompaniment when singing, compared with the method of simply subtracting the left and right channel audio data to eliminate the original vocal audio data. However, in professional singing apps, in addition to the music library with left and right channel audio data subtracted, there are also many professional accompaniment music libraries. The accompaniment music library is not obtained by eliminating the original vocal audio data by subtracting the left and right channel audio data, but by recording the accompaniment audio data in a separate track when recording music. For many songs, in addition to the accompaniment, there are also some professional accompaniment vocals. In some embodiments of the present application, all vocals can be identified and eliminated. Although the effect of a separate music accompaniment track can be approximated, the accompaniment left behind lacks a sense of atmosphere because the harmony of the accompaniment vocals is also eliminated. In addition, vocal separation is to strip out the signal belonging to the vocal feature in the original audio signal, but the sound of the vocals and the instrument will overlap in the frequency domain, and when separating the vocals, the instrument sound that overlaps with the vocals will also be stripped out together.

[0290] In order to solve this problem, the separated original vocal audio data can be transformed to obtain accompaniment audio data, and then the accompaniment audio data can be merged into the accompaniment at a certain ratio to make up for the hollow feeling of the accompaniment. The ratio is related to the energy of the vocal audio data. Specifically, when the energy of the vocal audio data increases, the ratio also increases, and when the singing voice decreases, the ratio also decreases.

[0291] In some embodiments, in order to avoid the problem of eliminating the harmony of professional backing singers when separating the vocals, the controller 250 is further configured to generate first accompaniment audio data based on the original vocal audio data in each time period.

[0292] As mentioned above, if the energy of the singing vocal audio data is less than the preset energy threshold, it means that the user is not singing or the singing voice is very small, and the original singing vocal audio data can be merged into the accompaniment audio data. In this case, it is not necessary to generate the first accompaniment audio data. Therefore, in some embodiments, when the energy of the singing vocal audio data is greater than or equal to the preset energy threshold, the first accompaniment audio data is generated according to the original singing vocal audio data in each time period.

[0293] In some embodiments, the original vocal audio data can be transformed in the time domain to generate the first accompaniment audio data. The controller 250 is configured to: obtain a plurality of different delays and the gain corresponding to each delay; for each delay, perform delay processing on the original vocal audio data in each time period according to the delay to obtain the first delayed audio data; perform gain processing on the delayed audio data according to the gain corresponding to the delay to obtain the second delayed audio data; and merge the plurality of second delayed audio data to obtain the first accompaniment audio data.

[0294] See also Fig.16 , Fig.16 A schematic diagram of time domain transformation of original vocal audio data in some embodiments of the present application.

[0295] Get multiple different delays and the gain corresponding to each delay. Multiple different delays and the gain corresponding to each delay can be pre-set. Multiple different delays can be equally spaced. The longer the delay, the smaller the gain. Therefore, the gain corresponding to multiple different delays gradually decreases. For example, T1 is 10ms, T2 is 20ms, T3 is 30ms..., gain 1 is 0dB, gain 2 is -6dB, gain 3 is -10dB...

[0296] For each delay, the original vocal audio data in each time period can be delayed according to the delay to obtain the first delayed audio data. And the delayed audio data can be gain processed according to the gain corresponding to the delay to obtain the second delayed audio data. For example, for T1, the original vocal audio data can be delayed according to 10ms to obtain the first delayed audio data, and the first delayed audio data can be gain processed according to 0dB to obtain the second delayed audio data. For T2, T3..., the same processing can be performed to obtain the corresponding second delayed audio data.

[0297] Afterwards, the plurality of second delayed audio data are merged to obtain the first accompaniment audio data.

[0298] In this way, after different delays and different gains are added together, a reverberation effect similar to that in a room or stadium can be formed. That is, the original singer's voice sounds like many people singing together, turning the original singer's voice into music with a chorus feel.

[0299] In some embodiments, the original vocal audio data can also be transformed in the frequency domain to generate the first accompaniment audio data. The controller 250 is configured to: determine the sound zone to which the original vocal audio data belongs; perform pitch-up or pitch-down processing on the original vocal audio data according to the sound zone to obtain the first accompaniment audio data. In this way, accompaniment can be formed, and the accompaniment is not in the same tone as the original. For example, for professional performances, there are professional accompaniment teams, and the voices they sing are not in the same part as the original singer, for example, they may be 3 degrees higher or 3 degrees lower than the original singer.

[0300] See also Fig.17 , Fig.17 A schematic diagram of frequency domain transformation of the original vocal audio data in some embodiments of the present application. Through fundamental frequency analysis, the sound range to which the original vocal audio data belongs can be determined. Among them, fundamental frequency analysis is to perform FFT (Fast Fourier Transform) on the human voice and find the first peak, and the peak frequency is the fundamental frequency. The singer's pitch can be known based on the fundamental frequency. For example, the frequency of middle C, i.e. "do", is 261.6Hz. Based on the calculated tone of the current sound, the frequency corresponding to a few degrees of increase or decrease can be calculated.

[0301] It should be noted that there is a certain difference in the pitch of different pitches, and they can be calculated separately. For example, for the piano key spectrum, the algorithm principle of raising or lowering by 3 degrees can be explained in detail according to the piano keyboard. If the pitch range of the current original vocal audio data is middle C, that is, C4, raising by 3 degrees is white keyboard E4, and there are 4 semitones in the middle, that is, the current sound is pitched up. If the current original vocal audio data is B3, the pitch is raised by 3 degrees to D4, a total of 3 semitones, that is, the frequency is raised. multiple.

[0302] In some embodiments of the present application, the original vocal audio data can also be processed with an up-pitch or down-pitch according to the singing habits of general singers. Specifically, for non-professional singers, there are usually problems that the bass is not low enough and the treble is not high enough. Therefore, in some embodiments, in order to solve the problem that the bass is not low enough and the treble is not high enough when non-professional singers sing. The controller 250 is configured as follows: if the sound range is the bass range, the original vocal audio data is down-pitch processed to obtain the first accompaniment audio data; if the sound range is the treble range, the original vocal audio data is up-pitch processed to obtain the first accompaniment audio data; if the sound range is the middle range, the original vocal audio data is up-pitch processed and down-pitch processed to obtain the first vocal audio data and the second vocal audio data respectively; the first vocal audio data and the second vocal audio data are used as the first accompaniment audio data.

[0303] Specifically, when the original vocal audio data is lower than a certain low pitch, the pitch-down operation is started, and when the original vocal audio data is higher than a certain high pitch, the pitch-up operation is started. For example, when it is higher than C5, the pitch-up operation is started, that is, the gain of the pitch-down operation is controlled to be minimum, that is, silent, and the gain of the pitch-up operation is controlled to be 0dB, that is, the first accompaniment audio number generated includes the audio data after the pitch-up operation. On the contrary, when it is lower than C4, the pitch-down operation is started, the gain of the pitch-down operation is controlled to be 0dB, and the gain of the pitch-up operation is controlled to be minimum, that is, silent, that is, the first accompaniment audio number generated includes the audio data after the pitch-down operation. When it is between C4 and C5, the gains of the pitch-up operation and the pitch-down operation can be -6dB, that is, the first accompaniment audio number generated includes both the audio data after the tone operation and the audio data after the pitch-down operation.

[0304] It should be noted that if the first accompaniment audio data is merged into the accompaniment audio data according to the energy of the original vocal audio data, the original accompaniment style and timbre may be affected. The purpose of the accompaniment is to enrich and beautify the singing voice when the singing voice exists. Therefore, the energy of the accompaniment audio data finally merged into the accompaniment audio data can be less than the energy of the singing vocal audio data. For example, it is 12dB smaller than the singing vocal audio data.

[0305] Therefore, after generating the first accompaniment audio data, the controller 250 is configured to: determine the accompaniment gain based on the energy of the singing vocal audio data collected within the time period; wherein the accompaniment gain is positively correlated with the energy of the singing vocal audio data collected within the time period; perform gain processing on the first accompaniment audio data through the accompaniment gain to obtain the second accompaniment audio data; wherein the energy of the second accompaniment audio data is less than the energy of the singing vocal audio data.

[0306] It is understandable that the greater the energy of the vocal audio data, the greater the energy of the accompaniment audio data eventually merged into the accompaniment audio data. Therefore, the accompaniment gain is positively correlated with the energy of the vocal audio data collected during the time period. Assuming that the energy of the vocal audio data is E, the accompaniment gain m can be calculated according to the following formula: m = E-12. In this way, the energy of the second accompaniment audio data obtained by the accompaniment gain is less than the energy of the vocal audio data. Of course, the calculation method of the accompaniment gain is not limited to this, and the above formula can be simply deformed to calculate the accompaniment gain.

[0307] The controller 250 is configured to merge the accompaniment audio data, the second accompaniment audio data, the target vocal audio data and the singing vocal audio data within a time period and perform sound effect enhancement processing to obtain the target audio data.

[0308] In this way, on the basis of the accompaniment audio data, the singing vocal audio data and the target vocal audio data, the second accompaniment audio data is further added, so as to avoid the problem of poor accompaniment effect caused by stripping the accompaniment audio data in the song during the vocal separation process, thereby improving the overall effect of the accompaniment and ultimately enhancing the user's singing experience.

[0309] Corresponding to the above display device embodiment, the present application also provides an audio processing method. It can be understood that, Figure 18 to Figure 21 The steps involved in the actual implementation may include more steps, or fewer steps, and the order of these steps may also be different, so as to implement the audio processing method provided in the embodiment of the present invention.

[0310] See also Fig.18 , Fig.18 A flowchart of an audio processing method in some embodiments of the present application may include the following steps:

[0311] Step S1810: performing sound separation on the acquired first audio data to obtain first target audio data and first background audio data.

[0312] Step S1820, performing gain processing on the first target audio data according to the first gain to obtain second target audio data, and performing gain processing on the first background audio data according to the second gain to obtain second background audio data. The first gain and the second gain are determined according to the sound control mode corresponding to the display device.

[0313] Step S1830: merge the second target audio data and the second background audio data, perform sound enhancement processing, and obtain and output second audio data.

[0314] In the above audio processing method, after separating the first target audio data and the first background audio data from the first audio data, the first target audio data can be gain processed according to the first gain to obtain the second target audio data; and the first background audio data can be gain processed according to the second gain to obtain the second background audio data. The second target audio data and the second background audio data are merged, and sound enhancement processing is performed to obtain and output the second audio data. Since the first gain and the second gain are determined according to the sound control mode corresponding to the display device, the first target audio data and the first background audio data can be combined with the current viewing needs of the user. By performing non-proportional gain processing on the first target audio data and the first background audio data and then merging them, the first target audio data or the first background audio data can be enhanced according to the viewing needs of the user, thereby improving the effect of sound enhancement.

[0315] In some embodiments, the audio processing method further includes:

[0316] Determining, according to the sound control mode, a type of sound effect enhancement mode corresponding to the first audio data;

[0317] According to the sound control mode, a first gain and a second gain corresponding to the type of the sound effect enhancement mode are determined.

[0318] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding value, and each preset sound effect mode has a corresponding value;

[0319] The sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of a plurality of preset sound clarity control modes, and the target sound effect mode is one of a plurality of preset sound effect modes;

[0320] Determining the type of the sound enhancement mode corresponding to the first audio data according to the sound control mode includes:

[0321] Determining the type of the sound effect enhancement mode corresponding to the first audio data according to the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode;

[0322] Determining, according to the sound control mode, a first gain and a second gain corresponding to the type of the sound effect enhancement mode, comprising:

[0323] According to the first value and / or the second value, a first gain and a second gain corresponding to the type of the sound enhancement mode are determined.

[0324] In some embodiments, determining the first gain and the second gain corresponding to the type of the sound effect enhancement mode according to the sound control mode includes:

[0325] If the type of the sound effect enhancement mode corresponding to the first audio data is a sound enhancement mode, the first gain is greater than the second gain;

[0326] If the type of the sound enhancement mode corresponding to the first audio data is the background enhancement mode, the first gain is less than the second gain.

[0327] In some embodiments, the first audio data includes at least one third target audio data belonging to a preset sound type;

[0328] The above audio processing method also includes:

[0329] separating at least one third target audio data and third background audio data from the first audio data;

[0330] Acquire a first energy value of initial target audio data of a first channel and a second energy value of initial target audio data of a second channel of a single third target audio data;

[0331] Performing gain processing on the initial target audio data of the first channel according to the third gain to obtain first-channel first-gain audio data; performing gain processing on the initial target audio data of the second channel according to the fourth gain to obtain second-channel first-gain audio data; wherein the third gain and the fourth gain are determined according to the first energy value and the second energy value;

[0332] Merging the first channel first gain audio data and the first channel initial background audio data of the third background audio data, and performing sound effect enhancement processing to obtain and output the first channel first enhanced audio data;

[0333] The first gain audio data of the second channel and the initial background audio data of the second channel of the third background audio data are combined, and sound effect enhancement processing is performed to obtain and output the first enhanced audio data of the second channel.

[0334] In some embodiments, the audio processing method further includes:

[0335] Determine a fifth gain and a sixth gain corresponding to a single third target audio data according to the sound control mode, the first energy value, and the second energy value;

[0336] determining a seventh gain according to a sound control mode;

[0337] Performing gain processing on the initial target audio data of the first channel according to the fifth gain to obtain the second gain audio data of the first channel; performing gain processing on the initial target audio data of the second channel according to the sixth gain to obtain the second gain audio data of the second channel;

[0338] Performing gain processing on the initial background audio data of the first channel and the initial background audio data of the second channel respectively according to the seventh gain to obtain the first channel gain background audio data and the second channel gain background audio data;

[0339] The first channel second gain audio data and the first channel gain background audio data are combined, and sound effect enhancement processing is performed to obtain and output the first channel second enhanced audio data;

[0340] The second channel second gain audio data and the second channel gain background audio data are merged, and sound effect enhancement processing is performed to obtain and output the second channel second enhanced audio data.

[0341] In some embodiments, determining a fifth gain and a sixth gain corresponding to a single third target audio data according to the sound control mode, the first energy value, and the second energy value includes:

[0342] Determining, according to the sound control mode, a type of sound effect enhancement mode corresponding to the first audio data;

[0343] Determine the energy magnitude relationship between the left and right channels according to a first energy value of the initial target audio data of the first channel and a second energy value of the initial target audio data of the second channel;

[0344] Determine, according to the sound control mode, the first energy value and the second energy value, a fifth gain and a sixth gain corresponding to the type of the sound enhancement mode and the energy magnitude relationship between the left and right channels;

[0345] According to the sound control mode, the seventh gain is determined, including:

[0346] According to the sound control mode, a seventh gain corresponding to the type of the sound enhancement mode and the energy size relationship between the left and right channels is determined.

[0347] See also Fig.19 , Fig.19 A flowchart of an audio processing method in some embodiments of the present application may include the following steps:

[0348] Step S1910: Perform sound separation and sound enhancement processing on the acquired first audio data to obtain first target audio data and second audio data.

[0349] Step S1920, gain processing is performed on the first target audio data according to the first gain to obtain second target audio data, and gain processing is performed on the second audio data according to the second gain to obtain third audio data, wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device.

[0350] Step S1930: Delay processing is performed on the second target audio data or the third audio data to synchronize the second target audio data with the third audio data.

[0351] Step S1940: merge the second target audio data and the third audio data to obtain and output fourth audio data.

[0352] In the audio processing method of some embodiments of the present application, since the sound separation algorithm only separates the target sound and does not separate the background sound, the time consumed by the sound separation algorithm can be reduced by half. In addition, sound separation and sound enhancement can be processed in parallel instead of serially, which can further shorten the time consumed by the entire audio processing flow, thereby improving the effect of audio-visual synchronization. In addition, the second target audio data or the third audio data is delayed. For example, the delay processing can be performed in the link with less computing time in the sound enhancement link and the sound separation link, so that the second target audio data and the third audio data are synchronized and then merged to avoid the echo problem, thereby improving the audio-visual synchronization effect while not reducing the effect of sound enhancement.

[0353] In some embodiments, delay processing is performed on the second target audio data or the third audio data, including:

[0354] Obtaining a first duration consumed during sound separation and a second duration consumed during sound enhancement processing;

[0355] Delay processing is performed on the second target audio data or the third audio data according to the first duration and the second duration.

[0356] In some embodiments, delay processing is performed on the second target audio data or the third audio data, including:

[0357] determining a time difference between the first target audio data and the second audio data according to a correlation between the first target audio data and the second audio data;

[0358] According to the time difference, delay processing is performed on the second target audio data or the third audio data.

[0359] In some embodiments, determining the time difference between the first target audio data and the second audio data according to the correlation between the first target audio data and the second audio data includes:

[0360] Acquire a first audio segment of the first target audio data within a time period t;

[0361] Acquire a second audio segment of the second audio data within a time period t, as well as a plurality of third audio segments before the second audio segment and a plurality of fourth audio segments after the second audio segment; wherein the durations corresponding to the third audio segment and the fourth audio segment are both equal to the duration of the time period t;

[0362] Determine the correlation between the first audio segment and the second audio segment, the third audio segment and the fourth audio segment, and determine the audio segment with the highest correlation;

[0363] The time difference between the audio segment with the highest correlation and the first audio segment is determined as the time difference between the first target audio data and the second audio data.

[0364] In some embodiments, the first audio data includes first channel initial audio data and second channel initial audio data;

[0365] Performing sound enhancement processing on the first audio data to obtain second audio data includes:

[0366] Performing sound effect enhancement processing on the initial audio data of the first channel and the initial audio data of the second channel respectively, to obtain first channel sound effect enhanced audio data and second channel sound effect enhanced audio data;

[0367] Performing gain processing on the second audio data according to the second gain to obtain third audio data includes:

[0368] Performing gain processing on the first channel sound effect enhanced audio data and the second channel sound effect enhanced audio data according to the second gain, respectively, to obtain the first channel target audio data and the second channel target audio data;

[0369] Delay processing is performed on the second target audio data or the third audio data to synchronize the second target audio data with the third audio data, including:

[0370] Delay processing is performed on the second target audio data or the first channel target audio data so that the second target audio data is synchronized with the first channel target audio data; and delay processing is performed on the second target audio data or the second channel target audio data so that the second target audio data is synchronized with the second channel target audio data;

[0371] The second target audio data and the third audio data are combined to obtain fourth audio data, including:

[0372] The second target audio data is respectively merged with the first channel target audio data and the second channel target audio data to obtain first channel merged audio data and second channel merged audio data.

[0373] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding value, and each preset sound effect mode has a corresponding value;

[0374] The sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of a plurality of preset sound clarity control modes, and the target sound effect mode is one of a plurality of preset sound effect modes; the above audio processing method also includes:

[0375] The first gain and the second gain are determined according to a first value corresponding to the target sound clarity control mode and / or a second value corresponding to the target sound effect mode, wherein the first gain is greater than the second gain.

[0376] In some embodiments, determining the first gain and the second gain according to the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode includes:

[0377] Set the first gain to 0dB;

[0378] The second gain is determined according to the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode, so that the second gain is less than 0dB.

[0379] See also Fig. 20 , Fig. 20 Another flowchart of the audio processing method in some embodiments of the present application, applied to a display device, may include the following steps:

[0380] Step S2010, performing vocal separation on the acquired first channel audio data and second channel audio data respectively to obtain first channel first vocal audio data and first channel first background audio data, and second channel first vocal audio data and second channel first background audio data.

[0381] Step S2020: merge the first channel first vocal audio data and the second channel first vocal audio data to obtain target vocal audio data.

[0382] Step S2030, obtain the image data at the moment when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if the lip movement coordinates on the display device screen are detected, determine the vocal weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces of the display device.

[0383] Step S2040: for each audio output interface, determine, according to the coordinates of the audio output interface, whether the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel.

[0384] Step S2050, merge the product of the target vocal audio data and the vocal weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface, and perform sound enhancement processing to obtain the audio data corresponding to the audio output interface, and output the audio data through the audio output interface.

[0385] In the audio processing method of some embodiments of the present application, in a stereo scene, after the first channel audio data and the second channel audio data are respectively separated into human voices, the separated first channel first human voice audio data and the second channel first human voice audio data can be merged to obtain target human voice audio data, and the target human voice audio data is used as the human voice audio to be output. Then, according to the position of the person speaking in the image, the human voice weight corresponding to each audio output interface is adjusted, that is, the weight corresponding to the output human voice audio, and according to the position of the audio output interface, the weight of the background audio output by each audio output interface is adjusted, so as to enhance the stereoscopic sense of the sound and improve the user's viewing experience.

[0386] In some embodiments, the audio processing method further includes:

[0387] Performing gain processing on the first channel first human voice audio data and the second channel first human voice audio data according to the first gain, respectively, to obtain the first channel second human voice audio data and the second channel second human voice audio data;

[0388] Performing gain processing on the first background audio data of the first channel and the first background audio data of the second channel according to the second gain, respectively, to obtain the second background audio data of the first channel and the second background audio data of the second channel; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device;

[0389] The first vocal audio data of the first channel and the first vocal audio data of the second channel are combined to obtain target vocal audio data, including:

[0390] Merging the second human voice audio data of the first channel and the second human voice audio data of the second channel to obtain target human voice audio data;

[0391] For each audio output interface, determining, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel, includes:

[0392] For each audio output interface, determining, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first channel second background audio data and / or the second channel second background audio data;

[0393] The product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface are combined, and sound effect enhancement processing is performed to obtain audio data corresponding to the audio output interface, including:

[0394] The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface.

[0395] In some embodiments, the above-mentioned sound effect processing method further includes:

[0396] If the lip movement coordinates are not detected, for each audio output interface, the vocal weight corresponding to each audio output interface is determined based on the ratio of the energy of the first vocal audio data of the first channel to the energy of the first vocal audio data of the second channel, and the coordinates of the audio output interface.

[0397] In some embodiments, the screen includes: a left region, a middle region, and a right region;

[0398] Determining, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel, comprises:

[0399] If the coordinates of the audio output interface correspond to the left area, determining that the audio output interface corresponds to the first background audio data of the first channel;

[0400] If the coordinates of the audio output interface correspond to the right area, determining that the audio output interface corresponds to the first background audio data of the second channel;

[0401] If the coordinates of the audio output interface correspond to the middle area, it is determined that the audio output interface corresponds to the first background audio data of the first channel and the first background audio data of the second channel.

[0402] In some embodiments, the screen includes: a middle region and a non-middle region;

[0403] Determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces of the display device, including:

[0404] If the lip movement coordinates are located in a non-middle area, determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces;

[0405] If the lip movement coordinates are located in the middle area, the human voice weights corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction.

[0406] In some embodiments, for each audio output interface, the area corresponding to the audio output interface on the screen is determined according to the coordinates of the audio output interface;

[0407] If the lip movement coordinates are located in the area corresponding to the audio output interface, determining that the human voice weight corresponding to the audio output interface is a first value;

[0408] If the lip movement coordinates are outside the area corresponding to the audio output interface, it is determined that the human voice weight corresponding to the audio output interface is a second value, and the second value is smaller than the first value.

[0409] In some embodiments, the display device corresponds to multiple preset sound clarity control modes and / or multiple preset sound effect modes; each preset sound clarity control mode has a corresponding value, and each preset sound effect mode has a corresponding value;

[0410] The sound control mode includes: a target sound clarity control mode and / or a target sound effect mode; wherein the target sound clarity control mode is one of a plurality of preset sound clarity control modes, and the target sound effect mode is one of a plurality of preset sound effect modes;

[0411] The above audio processing method also includes:

[0412] The first gain and the second gain are determined according to the first value corresponding to the target sound clarity control mode and / or the second value corresponding to the target sound effect mode.

[0413] Some embodiments of the present application also provide an audio processing method, which can achieve singing without media resource restrictions through vocal separation. At the same time, according to the energy of the singing voice collected by the microphone, the original singing voice can be fully or partially added to the accompaniment to avoid affecting the singing experience due to the low singing level of the singer.

[0414] See also Fig.21 , Fig.21 Another flowchart of the audio processing method in some embodiments of the present application, applied to a display device, may include the following steps:

[0415] Step S2110, obtain song audio data, separate the vocals from the song audio data, and obtain original singer's vocal audio data and accompaniment audio data.

[0416] Step S2120, determines the original singer's gain based on the energy of the original singer's vocal audio data in each time period and the energy of the singing vocal audio data collected in the time period, and performs gain processing on the original singer's vocal audio data in the time period based on the original singer's gain to obtain the target vocal audio data.

[0417] Step S2130, merging the accompaniment audio data, target vocal audio data and singing vocal audio data in each time period and performing sound effect enhancement processing to obtain and output the target audio data.

[0418] The sound effect processing method of some embodiments of the present application can obtain the original vocal audio data and accompaniment audio data by voice separation for song audio data. In this way, for any song, even the song not included in the singing APP can be sung by this method. In addition, according to the energy of the singing vocal audio data and the energy of the original singing vocal audio data collected in real time, the original sound gain is determined, and the original singing vocal audio data is gain-processed according to the original sound gain to obtain the target vocal audio data. Since the original singing gain is determined according to the energy of the singing vocal audio data and the energy of the original singing vocal audio data, the target vocal audio data is merged into the accompaniment audio data, that is, according to the user's singing situation, the original singing vocal audio data is merged into the accompaniment audio data, for example, all the original singing vocal audio data are merged into the accompaniment audio data, or, part of the original singing vocal audio data is merged into the accompaniment audio data, so as to improve the accompaniment effect when the user sings and improve the user experience.

[0419] In some embodiments, the original song gain is less than or equal to a preset gain threshold.

[0420] In some embodiments, determining the original singer gain according to the energy of the original singer's vocal audio data in each time period and the energy of the singing vocal audio data collected in the time period includes:

[0421] If the energy of the singing voice audio data is less than the preset energy threshold, the original singer gain is set to the preset gain threshold;

[0422] If the energy of the singing vocal audio data is greater than or equal to a preset energy threshold, the original singer's gain is determined based on the energy ratio between the energy of the singing vocal audio data and the energy of the original singer's vocal audio data, so that the original singer's gain is less than the preset gain threshold.

[0423] In some embodiments, the above-mentioned sound effect processing method further includes:

[0424] Obtaining the original singer gain corresponding to the previous time period, if the original singer gain corresponding to the current time period is the same as the original singer gain corresponding to the previous time period, extending the time period until the extended time period is less than the first time threshold;

[0425] If the original singer gain corresponding to the current time period is different from the original singer gain corresponding to the previous time period, shorten the time period until the shortened time period is greater than the second time threshold, wherein the first time threshold is greater than the second time threshold.

[0426] In some embodiments, the above-mentioned sound effect processing method further includes:

[0427] Generate first accompaniment audio data according to the original vocal audio data in each time period;

[0428] Determine the accompaniment gain according to the energy of the vocal audio data collected during the time period; wherein the accompaniment gain is positively correlated with the energy of the vocal audio data collected during the time period;

[0429] Performing gain processing on the first accompaniment audio data through the accompaniment gain to obtain second accompaniment audio data; wherein the energy of the second accompaniment audio data is less than the energy of the singing human voice audio data;

[0430] The accompaniment audio data, the target vocal audio data and the singing vocal audio data within the time period are merged, and sound effect enhancement processing is performed to obtain the target audio data, specifically including:

[0431] The accompaniment audio data, the second accompaniment audio data, the target vocal audio data and the singing vocal audio data within a time period are merged, and sound effect enhancement processing is performed to obtain the target audio data.

[0432] In some embodiments, generating first accompaniment audio data according to the original vocal audio data in each time period includes:

[0433] Get multiple different delays and the gain corresponding to each delay;

[0434] For each delay, delay processing is performed on the original vocal audio data in each time period according to the delay to obtain first delayed audio data;

[0435] Performing gain processing on the delayed audio data according to the gain corresponding to the delay to obtain second delayed audio data;

[0436] The plurality of second delayed audio data are combined to obtain the first accompaniment audio data.

[0437] In some embodiments, generating first accompaniment audio data according to the original vocal audio data in each time period includes:

[0438] Determine the audio range to which the original vocal audio data belongs;

[0439] The original vocal audio data is pitched up or down according to the pitch range to obtain the first accompaniment audio data.

[0440] In some embodiments, the original vocal audio data is pitched up or pitched down according to the pitch range, including:

[0441] If the vocal range is the bass range, the original vocal audio data is down-tuned to obtain the first accompaniment audio data;

[0442] If the pitch range is a high pitch range, the original vocal audio data is pitched up to obtain first accompaniment audio data;

[0443] If the vocal range is the middle range, the original vocal audio data is subjected to pitch-up processing and pitch-down processing to obtain first vocal audio data and second vocal audio data respectively;

[0444] The first vocal audio data and the second vocal audio data are used as the first accompaniment audio data.

[0445] The specific details of each step in the above method have been described in detail in the corresponding display device, so they will not be repeated here.

[0446] Some embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned audio processing method and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0447] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0448] The present application provides a computer program product, including: when the computer program product is run on a computer, the computer is enabled to implement the above-mentioned audio processing method.

[0449] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0450] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A display device, characterized in that: include: controller and multiple audio output interfaces; The controller is configured to: perform vocal separation on the acquired first channel audio data and second channel audio data respectively, to obtain first channel first vocal audio data and first channel first background audio data, and second channel first vocal audio data and second channel first background audio data; Merging the first human voice audio data of the first channel and the first human voice audio data of the second channel to obtain target human voice audio data; Acquire image data at the time when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if lip movement coordinates on the display device screen are detected, determine the human voice weight corresponding to the audio output interface according to the lip movement coordinates and the coordinates of a single one of the multiple audio output interfaces; Determine, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel; The product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface are combined, and sound effect enhancement processing is performed to obtain audio data corresponding to the audio output interface; A single audio output interface is configured to output audio data corresponding to the audio output interface.

2. The display device according to claim 1, characterized in that The controller is further configured to: perform gain processing on the first channel first human voice audio data and the second channel first human voice audio data according to a first gain, respectively, to obtain the first channel second human voice audio data and the second channel second human voice audio data; Performing gain processing on the first background audio data of the first channel and the first background audio data of the second channel according to a second gain, respectively, to obtain the second background audio data of the first channel and the second background audio data of the second channel; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device; The controller is configured to: Merging the first channel second human voice audio data and the second channel second human voice audio data to obtain target human voice audio data; and For each audio output interface, determining, according to the coordinates of the audio output interface, the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface; The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface.

3. The display device according to claim 1, characterized in that The controller is also configured to: if no lip movement coordinates are detected, for each audio output interface, determine the vocal weight corresponding to the audio output interface respectively according to the ratio of the energy of the first vocal audio data of the first channel to the energy of the first vocal audio data of the second channel, and the coordinates of the audio output interface.

4. The display device according to claim 1, characterized in that The screen includes: a left area, a middle area and a right area; the controller is configured to: If the coordinates of the audio output interface correspond to the left area, determining that the audio output interface corresponds to the first background audio data of the first channel; If the coordinates of the audio output interface correspond to the right area, determining that the audio output interface corresponds to the first background audio data of the second channel; If the coordinates of the audio output interface correspond to the middle area, it is determined that the audio output interface corresponds to the first background audio data of the first channel and the first background audio data of the second channel.

5. The display device according to claim 1, characterized in that The screen includes: a middle area and a non-middle area; and the controller is configured to: If the lip movement coordinates are located in the non-middle area, determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces; If the lip movement coordinates are located in the middle area, the human voice weights respectively corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction.

6. The display device according to claim 1, characterized in that The controller is configured to: for each audio output interface, determine the area corresponding to the audio output interface on the screen according to the coordinates of the audio output interface; If the lip movement coordinates are located in the area corresponding to the audio output interface, determining that the human voice weight corresponding to the audio output interface is a first value; If the lip movement coordinates are outside the area corresponding to the audio output interface, it is determined that the human voice weight corresponding to the audio output interface is a second value, and the second value is smaller than the first value.

7. An audio processing method, characterized in that: Applied to a display device, the method comprises: Separating the acquired first channel audio data and second channel audio data into human voices, respectively, to obtain first channel first human voice audio data and first channel first background audio data, and second channel first human voice audio data and second channel first background audio data; Merging the first human voice audio data of the first channel and the first human voice audio data of the second channel to obtain target human voice audio data; Acquire image data at the time when the first channel audio data and the second channel audio data are located, perform lip movement detection on the image data, and if lip movement coordinates on the display device screen are detected, determine the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces of the display device; For each audio output interface, determining, according to the coordinates of the audio output interface, the first background audio data of the first channel and / or the first background audio data of the second channel corresponding to the audio output interface; The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface, and the audio data is output through the audio output interface.

8. The method according to claim 7, characterized in that The method further comprises: Performing gain processing on the first channel first human voice audio data and the second channel first human voice audio data according to a first gain, respectively, to obtain first channel second human voice audio data and second channel second human voice audio data; Performing gain processing on the first background audio data of the first channel and the first background audio data of the second channel according to a second gain, respectively, to obtain the second background audio data of the first channel and the second background audio data of the second channel; wherein the first gain and the second gain are determined according to the sound control mode corresponding to the display device; The step of merging the first channel first human voice audio data and the second channel first human voice audio data to obtain target human voice audio data includes: Merging the first channel second human voice audio data and the second channel second human voice audio data to obtain target human voice audio data; The step of determining, for each audio output interface, according to the coordinates of the audio output interface, that the audio output interface corresponds to the first background audio data of the first channel and / or the first background audio data of the second channel, comprises: For each audio output interface, determining, according to the coordinates of the audio output interface, the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface; The method of merging the product of the target human voice audio data and the human voice weight corresponding to the audio output interface, and the first channel first background audio data and / or the second channel first background audio data corresponding to the audio output interface, and performing sound effect enhancement processing to obtain the audio data corresponding to the audio output interface includes: The product of the target vocal audio data and the vocal weight corresponding to the audio output interface, as well as the first channel second background audio data and / or the second channel second background audio data corresponding to the audio output interface are merged, and sound enhancement processing is performed to obtain the audio data corresponding to the audio output interface.

9. The method according to claim 7, characterized in that: The method further comprises: If the lip movement coordinates are not detected, for each audio output interface, the vocal weight corresponding to the audio output interface is determined according to the ratio of the energy of the first vocal audio data of the first channel to the energy of the first vocal audio data of the second channel, and the coordinates of the audio output interface.

10. The method according to claim 7, characterized in that The screen includes: a middle area and a non-middle area; Determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces, including: If the lip movement coordinates are located in the non-middle area, determining the human voice weights corresponding to the multiple audio output interfaces respectively according to the lip movement coordinates and the coordinates of the multiple audio output interfaces; If the lip movement coordinates are located in the middle area, the human voice weights respectively corresponding to the multiple audio output interfaces are determined according to the coordinates of the multiple audio output interfaces and the attribute information of the multiple audio output interfaces, wherein the attribute information includes volume and / or direction.