Target sound optimization method, system and device and medium

Abstract

The invention discloses a target sound optimization method, system and device and a medium, and the method comprises the steps: obtaining a real-time video image; determining a current film and television scene based on the real-time video image; based on the current film and television scene, determining a target optimization strategy, the target optimization strategy comprising a gain parameter of a target frequency band; and carrying out optimization processing on the audio signal based on the target optimization strategy. According to the method, the type of the played program can be intelligently identified, voice enhancement is carried out in real time in the voice-based program, and the tone quality of the program is not damaged in the non-voice-based program.

Description

Technical Field

[0001] The present invention relates to the technical field of audio output of smart television systems, and in particular to a target sound optimization method and system, device and medium. Background Art

[0002] For the elderly or other TV users with hearing loss, when watching TV programs dominated by human voices, they may not be able to hear the human voices in the programs clearly, which may lead to difficulties in understanding the meaning. It is necessary to develop a set of audio processing methods to enhance the human voice frequency band of the audio when playing TV programs dominated by human voices, so as to improve the comprehensibility of the human voice content. The existing smart TV system does not have the function of intelligently identifying the type of TV program currently being played and selectively enhancing the human voice. If the loudness of the human voice frequency band is simply increased for all programs, there will be a problem of sound quality degradation when playing other non-human voice programs.

[0003] Therefore, it is hoped to propose a target sound optimization method and system, device and medium, which can intelligently identify the type of program being played, and achieve real-time vocal enhancement in programs dominated by human voices, while not damaging the sound quality of programs not dominated by human voices. Summary of the invention

[0004] The present specification discloses a target sound optimization method, which includes: acquiring a real-time video image; determining a current film and television scene based on the real-time video image; determining a target optimization strategy based on the current film and television scene, wherein the target optimization strategy includes a gain parameter of a target frequency band; and optimizing an audio signal based on the target optimization strategy.

[0005] In some embodiments, determining the current film and television scene based on the real-time video image includes:

[0006] Based on the real-time video image, the current film and television scene is determined through a scene recognition model, and the scene recognition model is a machine learning model.

[0007] In some embodiments, the current film and television scene includes: a human voice film and television program scene and a common film and television program scene.

[0008] In some embodiments, the target optimization strategy includes: in response to the current film and television scene being a vocal film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0.1~10db; in response to the current film and television scene being an ordinary film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0db.

[0009] The present invention also discloses a target sound optimization system, comprising an AI scene recognition module, an audio strategy module, and a DSP sound effect processing module; the AI ​​scene recognition module is configured to acquire real-time video images, and determine the current film and television scene based on the real-time video images; the audio strategy module is configured to determine a target optimization strategy based on the current film and television scene, and the target optimization strategy includes a gain parameter of a target frequency band; the DSP sound effect processing module is configured to optimize the audio signal based on the target optimization strategy.

[0010] In some embodiments, the AI ​​scene recognition module is further configured to: determine the current film and television scene based on the real-time video image through a scene recognition model, and the scene recognition model is a machine learning model.

[0011] In some embodiments, the current film and television scene includes: a human voice film and television program scene and a common film and television program scene.

[0012] In some embodiments, the target optimization strategy includes: in response to the current film and television scene being a vocal film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0.1~10db; in response to the current film and television scene being an ordinary film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0db; the audio strategy module is further configured to send the target optimization strategy to the DSP sound effect processing module through a DSP interface function.

[0013] The present invention also discloses a target sound optimization device, comprising a processor, wherein the processor is used to execute the target sound optimization method.

[0014] The present invention also discloses a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the target sound optimization method as described above.

[0015] Beneficial effects: The target sound optimization method and device of the present invention solves the problem that the existing smart TV system cannot intelligently identify the type of film and television programs and selectively enhance the human voice by combining AI intelligent scene recognition technology and real-time audio processing technology; it realizes the function of AI intelligently identifying the type of film and television programs currently being played and enhancing the human voice of film and television programs dominated by human voices in real time. At the same time, the three modules of this system cooperate with each other to achieve AI intelligent recognition of the type of film and television programs currently being played, and real-time enhancement of the human voice of film and television programs dominated by human voices, without affecting the sound effects of non-human voice film and television programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0017] Figure 1 is a schematic diagram of the module structure of a target sound optimization device according to some embodiments of this specification;

[0018] Figure 2 is an exemplary flow chart of a target sound optimization method according to some embodiments of this specification;

[0019] Figure 3 is an exemplary schematic diagram of a scene recognition model according to some embodiments of this specification. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0021] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0022] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0023] When the elderly and people with hearing loss watch TV programs, the existing smart TV systems do not have the function of real-time recognition of the types of programs dominated by human voices and intelligent voice enhancement. The existing practices either do not process the program, resulting in the problem of difficulty in understanding voices when such people watch movies, or strengthen the human voice fixedly, resulting in the destruction of the sound quality of programs that are not dominated by human voices.

[0024] Therefore, this specification provides a target sound optimization method and system, which solves the problem that the existing smart TV system cannot intelligently enhance the human voice of the program by combining AI intelligent scene recognition and real-time audio processing technology. It realizes AI intelligent recognition of the type of program being played, and real-time enhancement of the human voice of programs dominated by human voice without affecting the functions of non-human voice programs.

[0025] Figure 1 It is a schematic diagram of the module structure of the target sound optimization system shown in some embodiments of this specification.

[0026] like Figure 1 As shown, the target sound optimization system 100 (hereinafter referred to as system 100 ) can include an AI scene recognition module 110 , an audio strategy module 120 , and a DSP sound effect processing module 130 .

[0027] The AI ​​scene recognition module is configured to acquire real-time video images and determine the current film and television scene based on the real-time video images; the audio strategy module is configured to determine the target optimization strategy based on the current film and television scene, and the target optimization strategy includes the gain parameters of the target frequency band; the DSP sound effect processing module is configured to optimize the audio signal based on the target optimization strategy.

[0028] In some embodiments, the AI ​​scene recognition module is further configured to: determine the current film and television scene based on the real-time video image through a scene recognition model, and the scene recognition model is a machine learning model.

[0029] In some embodiments, in the smart TV system, the AI ​​scene recognition module runs in the system background, recognizes the video image in real time, and classifies the film and television program. When the AI ​​scene recognition module recognizes the type of the current film and television program, it sends the current film and television program type to the audio strategy module.

[0030] In some embodiments, the current film and television scene includes: a human voice film and television program scene and a common film and television program scene.

[0031] In some embodiments, the audio strategy module runs in the system background, receives the current film and television program type in real time, and sends DSP sound effect parameters to the DSP sound effect processing module.

[0032] In some embodiments, the target optimization strategy includes: in response to the current film and television scene being a vocal film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0.1~10db; in response to the current film and television scene being an ordinary film and television program scene, the target frequency band is 300Hz~3400Hz; the gain parameter is 0db; the audio strategy module is further configured to send the target optimization strategy to the DSP sound effect processing module through a DSP interface function.

[0033] For details about the above embodiments, please refer to Figure 2-Figure 3 And related instructions.

[0034] In some embodiments of this specification, the AI ​​scene recognition module realizes the recognition of human voice film and television programs, the audio strategy module realizes the switching of real-time human voice enhancement EQ parameters, and the DSP sound effect processing module realizes the human voice enhancement EQ sound effect processing of audio data. The three modules cooperate with each other to achieve AI intelligent recognition of the type of film and television programs currently being played, and real-time human voice enhancement of film and television programs dominated by human voices without affecting the sound effects of non-human voice film and television programs.

[0035] It should be noted that the above description of the target sound optimization system and its platform is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the various platforms or form a subsystem to connect with other platforms without deviating from this principle. In some embodiments, Figure 1 The AI ​​scene recognition module 110, audio strategy module 120, and DSP sound effect processing module 130 disclosed in the specification can be different platforms in a system, or a platform can realize the functions of two or more platforms mentioned above. For example, each platform can share a storage database, or each platform can have its own storage database. Such variations are within the scope of protection of this specification.

[0036] Figure 2 FIG. 1 is an exemplary flow chart of a target sound optimization method according to some embodiments of this specification. Figure 2 As shown, the process 200 includes the following steps 210 to 240. In some embodiments, the process 200 may be executed by the system 100.

[0037] Step 210: Acquire real-time video images.

[0038] Real-time video images refer to the images of the film and television programs currently being played on smart TVs and other playback devices. The AI ​​scene recognition module can obtain real-time video images by taking screenshots or taking photos.

[0039] Step 220: Determine the current film and television scene based on the real-time video image.

[0040] In some embodiments, the AI ​​scene recognition module can determine the current film and television scene based on the real-time video image through a scene recognition model, where the scene recognition model is a machine learning model. For more information about the scene recognition model, see Figure 3 The corresponding content.

[0041] In some embodiments, the AI ​​scene recognition module runs in the system background, recognizes video images in real time, and classifies the film and television programs.

[0042] In some embodiments, the current film and television scene includes: a human voice film and television program scene and a common film and television program scene. For example, a human voice film and television program scene corresponds to a human voice film and television program ProgramV: including variety shows, sports, news, education, shopping, live broadcasts, etc. The common film and television program scene corresponds to a common film and television program ProgramN: including non-human voice film and television programs.

[0043] Step 230: Determine a target optimization strategy based on the current film and television scene.

[0044] The target optimization strategy includes a gain parameter of a target frequency band. In some embodiments, the target optimization strategy includes: in response to the current film and television scene being a human voice film and television program scene, the target frequency band is 300Hz to 3400Hz; the gain parameter is 0.1 to 10db; in response to the current film and television scene being an ordinary film and television program scene, the target frequency band is 300Hz to 3400Hz; the gain parameter is 0db.

[0045] Human speech (also called voice), that is, the sound of human speech, usually has a frequency range of 300Hz to 3400Hz. This case takes the enhancement of human speech as an example. In practice, the sound of other specific frequency bands can also be enhanced in combination with the processing idea of ​​this case. As an example only, in order to enhance the human voice frequency band, a certain amplitude of loudness gain processing can be performed on the frequency within the range of 300Hz to 3400Hz.

[0046] In some embodiments, the audio strategy module may send the target optimization strategy to the DSP sound effect processing module through a DSP interface function.

[0047] For example, the target optimization strategy includes the audio strategy PolicyVE for film and television programs with human voice; and the audio strategy PolicyN for ordinary film and television programs; wherein, the audio strategy PolicyVE for film and television programs with human voice includes a set of human voice audio EQ parameters EQVE, and the gain of the frequency band of 300Hz to 3400Hz is 0.1 to 10db; the audio strategy PolicyN for ordinary film and television programs includes a set of ordinary audio EQ parameters EQN, and the gain of the frequency band of 300Hz to 3400Hz is 0db.

[0048] When the audio policy module receives that the current film and television program type is a human voice film and television program ProgramV, the audio policy module switches to the human voice film and television program audio policy PolicyVE, and sends the EQ parameter EQVE to the DSP sound effect processing module through the DSP interface function.

[0049] When the audio policy module receives that the current film and television program type is ordinary film and television program ProgramN, the audio policy module switches to ordinary film and television program audio policy PolicyN, and sends EQ parameters EQN to the DSP sound effect processing module through the DSP interface function.

[0050] Step 240: Optimize the audio signal based on the target optimization strategy.

[0051] The DSP sound effect processing module processes the system audio signal according to the received sound effect parameters.

[0052] For example, when the EQ parameter EQVE is received, the gain for the frequency band of 300 Hz to 3400 Hz is 0.1 to 10 db. When the EQ parameter EQN is received, the gain for the frequency band of 300 Hz to 3400 Hz is 0 db.

[0053] The target sound optimization method of the present application can solve the problem that the existing smart TV system cannot intelligently enhance the human voice of the program by combining AI intelligent scene recognition and real-time audio processing technology, realize AI intelligent recognition of the type of program being played, and enhance the human voice of programs dominated by human voice in real time without affecting the functions of non-human voice programs.

[0054] It should be noted that the above description of the process 200 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process 200 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0055] The method of the present invention is further described below by taking a specific playback instance scenario as an example:

[0056] Scenario 1: The user plays variety shows (such as variety shows, sports, news, education, shopping, live broadcasts, etc.); the AI ​​scene recognition module recognizes that the current program is a human voice program ProgramV, and sends the current program type to the audio policy module. The audio policy module receives that the current program type is a human voice program, switches to the human voice program audio policy PolicyVE, and sends the current EQ parameter EQVE to the DSP sound effect processing module; the DSP sound effect processing module receives that the current EQ parameter is EQVE, and the gain for the 300Hz~3400Hz frequency band is 1db.

[0057] Scenario 2: The user plays a news-related TV program. The AI ​​scene recognition module identifies the current program as a human voice TV program ProgramV, and sends the current program type to the audio policy module. The audio policy module receives that the current program type is a human voice TV program ProgramV, switches to the human voice TV program audio policy PolicyVE, and sends the current EQ parameter EQVE to the DSP sound effect processing module.

[0058] Scenario three: The user plays ordinary film and television programs (not variety shows, sports, news, education, shopping, live broadcasts, etc.). The AI ​​scene recognition module recognizes that the current program is an ordinary film and television program ProgramN, and sends the current program type to the audio policy module. The audio policy module receives that the current program type is an ordinary film and television program ProgramN, switches to the ordinary film and television program audio policy PolicyN, and sends the current EQ parameter EQN to the DSP sound effect processing module; the DSP sound effect processing module receives that the current EQ parameter is EQN, and the gain of the 300Hz~3400Hz frequency band is 0db

[0059] Figure 3 is an exemplary schematic diagram of a scene recognition model according to some embodiments of this specification.

[0060] In some embodiments, Figure 3 As shown, the AI ​​scene recognition module can determine the current film and television scene 303 through the real-time video image 301 through the scene recognition model 302.

[0061] The scene recognition model is used to determine the model of the current film and television scene. In some embodiments, the scene recognition model is a machine learning model. For example, at least one of an LSTM (Long-Short Term Memory) model, a recurrent neural network (RNN) model, or other models.

[0062] In some embodiments, the scene recognition model can be trained by a large number of training samples and training labels corresponding to the training samples. For example, the AI ​​scene recognition module can perform multiple rounds of iterations based on a large number of training samples and training labels, and when the preset conditions are met, the iteration is terminated to obtain a trained scene recognition model. The preset conditions may be the convergence of the loss function, the number of iterations reaching a threshold, etc.

[0063] In some embodiments, the AI ​​scene recognition module can select multiple training samples, input one or more samples into the prediction model, and obtain model prediction outputs corresponding to the multiple training samples; based on the model prediction outputs corresponding to the one or more samples and the training labels, substitute them into the formula of a predefined loss function to calculate the value of the loss function; based on the value of the loss function, reversely iterate and update the model parameters in the initial scene recognition model through gradient descent or other methods.

[0064] Each group of training samples in the training samples may include sample video images. The training samples may be obtained through historical movie viewing data. The training labels corresponding to the training samples are actual film and television scenes corresponding to each group of training samples. The training labels may be obtained through manual labeling or automatic labeling.

[0065] In some embodiments of the present specification, actual film and television scenes are determined through scene recognition models, which can improve the efficiency and accuracy of scene determination and reduce misjudgments, thereby making subsequent determined strategies more accurate.

[0066] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0067] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "some embodiments" mentioned twice or more in different places in this specification do not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0068] In addition, the order of processing elements and sequences, the use of alphanumeric characters, or the use of other names described in this specification are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some of the invention embodiments currently considered useful through various examples, it should be understood that such details are only for illustrative purposes and are not limited to the disclosed embodiments. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0069] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. In fact, the features of the embodiment are less than all the features of the single embodiment disclosed above.

[0070] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0071] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0072] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A target sound optimization method, characterized in that: The method comprises: Acquire real-time video images; Based on the real-time video image, determining the current film and television scene; Based on the current film and television scene, determining a target optimization strategy, the target optimization strategy including a gain parameter of a target frequency band; The audio signal is optimized based on the target optimization strategy.

2. The method according to claim 1, characterized in that: The determining the current film and television scene based on the real-time video image includes: Based on the real-time video image, the current film and television scene is determined through a scene recognition model, and the scene recognition model is a machine learning model.

3. The method according to claim 1, characterized in that: The current film and television scene includes: a human voice film and television program scene and a common film and television program scene.

4. The method according to claim 3, characterized in that: The target optimization strategy includes: In response to the current film and television scene being a human voice film and television program scene, the target frequency band is 300 Hz to 3400 Hz; the gain parameter is 0.1 to 10 db; In response to the current film and television scene being a common film and television program scene, the target frequency band is 300 Hz to 3400 Hz; and the gain parameter is 0 db.

5. A target sound optimization system, characterized in that: Including AI scene recognition module, audio strategy module, and DSP sound effect processing module; The AI ​​scene recognition module is configured to obtain a real-time video image and determine a current film and television scene based on the real-time video image; The audio strategy module is configured to determine a target optimization strategy based on the current film and television scene, wherein the target optimization strategy includes a gain parameter of a target frequency band; The DSP sound effect processing module is configured to optimize the audio signal based on the target optimization strategy.

6. The system according to claim 5, characterized in that: The AI ​​scene recognition module is further configured to: Based on the real-time video image, the current film and television scene is determined through a scene recognition model, and the scene recognition model is a machine learning model.

7. The system according to claim 5, characterized in that: The current film and television scene includes: a human voice film and television program scene and a common film and television program scene.

8. The system according to claim 7, characterized in that: The target optimization strategy includes: In response to the current film and television scene being a human voice film and television program scene, the target frequency band is 300 Hz to 3400 Hz; the gain parameter is 0.1 to 10 db; In response to the current film and television scene being a common film and television program scene, the target frequency band is 300 Hz to 3400 Hz; the gain parameter is 0 db; The audio strategy module is further configured to send the target optimization strategy to the DSP sound effect processing module through a DSP interface function.

9. A target sound optimization device, comprising a processor, characterized in that: The processor is used to execute the target sound optimization method as described in any one of claims 1-4.

10. A computer-readable storage medium storing computer instructions, characterized in that: After the computer reads the computer instructions in the storage medium, the computer executes the target sound optimization method as described in any one of claims 1 to 4.

Images