Audio and video processing method and device, storage medium and terminal

By identifying the audio scene type of audio and video data and applying corresponding echo cancellation strategies, the problem of echo interference in audio and video interaction is solved, and the audio and video quality and user experience are improved.

CN118918915APending Publication Date: 2024-11-08GUANGZHOU KINDLINK INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410975287.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In audio and video interaction scenarios, due to the reflection propagation characteristics of sound waves, the remote audio signal is played through the local speaker and echo interference causes the audio data collected by the microphone to contain echo noise, affecting the audio and video quality and user experience.

Method used

By collecting audio and video data in the current scene, identifying the audio scene type corresponding to the audio and video data and determining the corresponding echo cancellation strategy, using a targeted echo processing algorithm to process the audio and video data, and transmitting it to the target playback terminal to eliminate echoes.

Benefits of technology

It improves the accuracy and stability of echo cancellation, improves the terminal's audio and video echo cancellation capabilities in diverse scenarios, and ensures high-quality transmission of audio and video data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918915A_ABST
    Figure CN118918915A_ABST
Patent Text Reader

Abstract

The invention discloses an audio and video processing method and device, a storage medium and a terminal, and is applied to the technical field of computers. Collecting audio and video data in the current scene; identifying an audio scene type corresponding to the audio and video data, and determining an echo cancellation strategy corresponding to the audio scene type; and performing echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmitting the processed audio and video data to the target playing terminal, so that the target playing terminal plays the processed audio and video data. In the embodiment of the invention, if the scene information of the audio and video data determines the corresponding audio processing requirement to a certain extent, the echo processing strategy corresponding to the scene type is adaptively used according to the specific scene type to which the audio and video data belongs, so that each audio and video can obtain a targeted echo processing scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio and video processing method, device, storage medium and terminal. Background Art

[0002] At present, audio and video interaction functions have been widely used in various scenarios such as online meetings and educational classrooms. In audio and video interaction scenarios, the terminals of the two interacting parties play the audio and video of the remote terminal users through speakers and displays, while also collecting the audio and video of the local terminal users through microphones and cameras.

[0003] However, due to the reflection propagation characteristics of sound waves, after the remote audio signal is played through the local speaker, it will be reflected multiple times in the space and then collected by the local microphone. Therefore, the audio data collected by the local microphone contains not only the audio data of the local user, but also the echo data that causes interference. If the audio and video collected by the local end are not echo-cancelled, then when the local audio and video are played remotely, these echo noises will hinder the remote user from receiving and understanding important information in the audio and video, resulting in a poor user interaction experience. Summary of the invention

[0004] The present application provides an audio and video processing method, device, storage medium and terminal, which can solve the technical problem of poor audio and video quality caused by poor echo processing effect on audio and video in related technologies.

[0005] In a first aspect, an embodiment of the present application provides an audio and video processing method, the method comprising:

[0006] Collect audio and video data in the current scene;

[0007] Identifying the audio scene type corresponding to the audio and video data, and determining the echo cancellation strategy corresponding to the audio scene type;

[0008] The audio and video data are subjected to echo cancellation processing according to the echo cancellation strategy, and the processed audio and video data are transmitted to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0009] The beneficial effects brought about by the technical solution of the first aspect include at least: the scene information of the audio and video data determines its corresponding audio processing requirements to a certain extent. Therefore, according to the specific scene type to which the audio and video data belongs, the echo processing strategy corresponding to the scene type is adaptively used, so that each audio and video can obtain a targeted echo processing solution, thereby improving the accuracy and stability of echo cancellation, and improving the terminal's ability to perform echo cancellation on audio and video in diversified scenarios.

[0010] In some possible implementations, the above-mentioned identification of the audio scene type corresponding to the above-mentioned audio and video data includes: if there is at least one available scene classification rule, judging whether there is a target scene classification rule hit by the above-mentioned audio and video data based on the current features of the above-mentioned audio and video data, the above-mentioned scene classification rule specifying the correspondence between at least one feature of the audio and video data and at least one audio scene type; if the above-mentioned target scene classification rule exists, determining the audio scene type corresponding to the above-mentioned audio and video data based on the above-mentioned target scene classification rule.

[0011] Through the technical solutions in the above possible implementation methods, it is possible to determine whether the current audio and video data matches the scene classification rules based on the defined scene classification rules. If it matches, the audio scene type corresponding to the audio and video data is directly determined according to the scene classification rules. In this way, some audio and video data with specific characteristics can be determined as specific scene types according to needs, realizing a more flexible audio and video processing solution and enhancing practicality in actual scenarios.

[0012] In some possible implementations, the above-mentioned available scene classification rules are predefined scene classification rules; or, the above-mentioned available scene classification rules are predefined scene classification rules after the user makes an enabling operation based on the rule-based enabling control.

[0013] Through the technical solutions in the above possible implementations, the available rules may be in two situations: one is that they can be used directly after deployment, or the user can control the use of the rules through controls after the rules are deployed. That is, the user can determine whether to use the rules by operating the controls based on their own needs, so that the scene classification of audio and video data is more personalized and meets user needs.

[0014] In some possible implementations, the features of the audio and video data include at least one of speaker switch state features, microphone switch state features, picture features, and environment features.

[0015] Through the technical solutions in the above possible implementation methods, it is possible to specify in the scene classification rules what scene a video is in when it has what kind of features, thereby realizing the flexible application of the scene classification rules to various audio and video data.

[0016] In some possible implementations, the above-mentioned identifying the audio scene type corresponding to the above-mentioned audio and video data includes: inputting the above-mentioned audio and video data into a scene analysis big model, and determining the audio scene type corresponding to the above-mentioned audio and video data output by the above-mentioned scene analysis big model.

[0017] Through the technical solutions in the above possible implementation methods, it is also possible to use a pre-trained large model to efficiently and accurately analyze audio and video data, thereby identifying the scene type of the audio and video data.

[0018] In some possible implementations, the echo cancellation strategy specifies at least one echo cancellation algorithm preconfigured for the audio processing requirements corresponding to the audio scene type.

[0019] Through the technical solutions in the above possible implementation methods, the echo cancellation strategy corresponding to each audio scene type is pre-configured according to the audio processing requirements in the scene, and the strategy includes at least one targeted echo processing algorithm, which makes it possible for the echo cancellation strategy corresponding to each audio scene type to accurately solve the audio echo problem in the scene.

[0020] In some possible implementations, the method further includes: when it is detected that the current features of the audio and video data have changed, re-execute the step of identifying the audio scene type corresponding to the audio and video data.

[0021] Through the technical solutions in the above possible implementation methods, the echo cancellation strategy can be flexibly adjusted according to the current characteristics of the real-time audio and video data to ensure the high quality of the audio and video data transmitted to the target playback terminal.

[0022] In a second aspect, an embodiment of the present application provides an audio and video processing device, the device comprising:

[0023] Data collection module, used to collect audio and video data in the current scene;

[0024] A scene analysis module, used to identify the audio scene type corresponding to the audio and video data, and determine the echo cancellation strategy corresponding to the audio scene type;

[0025] The echo cancellation module is used to perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0026] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method.

[0027] In a fourth aspect, an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is suitable for being loaded by the processor and executing the steps of the method.

[0028] In a fifth aspect, a computer program product is provided, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 An exemplary system architecture diagram of an audio and video processing method provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of an audio and video processing method provided in an embodiment of the present application;

[0032] Figure 3 A flowchart of an audio and video processing method provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the logical steps of an audio and video processing method provided in an embodiment of the present application;

[0034] Figure 5 A structural block diagram of an audio and video processing device provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0037] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. And in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0038] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0039] In the audio and video interaction scenario, since the local end will collect the audio and video of the local user and play the audio and video transmitted from the remote end at the same time, the local microphone is likely to collect the remote audio data played by the speaker into the local audio and video, forming an echo for the local audio and video. These echo data from the remote end are mixed into the original audio data of the local end, which not only weakens the accuracy of the original local audio information, but also seriously affects the smoothness of communication and interaction between multiple users. Especially in educational scenarios, clear audio and video transmission is crucial to ensure the quality and efficiency of remote teaching interaction.

[0040] Therefore, in order to solve the echo problem, the collected audio and video are generally processed by echo cancellation to remove the echo interference in the audio and video and ensure the quality of the audio and video. The common echo cancellation method is generally to capture the audio and video signals in the classroom through microphones and cameras, and then use the preset conventional echo cancellation algorithm to process the audio and video signals to effectively identify and eliminate the echo components caused by environmental reflections, equipment interference and other factors.

[0041] However, conventional echo processing methods face some problems in actual application scenarios. Since a fixed universal echo cancellation algorithm is often used to process all collected audio and video, this can improve the quality of audio and video to a certain extent, but it often cannot accurately match the needs of each specific scenario. For example, in educational application scenarios, there may be different types of subdivided scenarios such as multiple people speaking and single person speaking. The differences in these scenarios will cause changes in the characteristics of audio and video, so a single algorithm is difficult to adapt to complex and changeable specific scenarios.

[0042] Therefore, an embodiment of the present application provides an audio and video processing method to solve the above-mentioned technical problem of poor audio and video quality caused by poor echo processing effect on audio and video.

[0043] See also Figure 1 , Figure 1 An exemplary system architecture diagram of an audio and video processing method provided in an embodiment of the present application.

[0044] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0045] The terminal 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the terminal 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various electronic devices, including but not limited to multimedia recording and broadcasting hosts, smart phones, tablet computers, laptop portable computers, and desktop computers. When the terminal 101 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0046] In an embodiment of the present application, terminal 101 first collects audio and video data in the current scene; then, terminal 101 identifies the audio scene type corresponding to the audio and video data, and determines the echo cancellation strategy corresponding to the audio scene type; after determining the echo cancellation strategy corresponding to the audio and video data, terminal 101 can perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0047] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0048] Alternatively, the system architecture may not include the server 103. In other words, the server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification may be applied to a system structure that only includes the terminal 101, and the embodiments of this application do not limit this.

[0049] It should be understood that Figure 1 The number of terminals, networks and servers in the figure is only for illustration and any number of terminals, networks and servers may be used according to implementation requirements.

[0050] See also Figure 2 , Figure 2 A flowchart of an audio and video processing method provided in an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes audio and video processing, or a processor in the terminal that executes the audio and video processing method, or an audio and video processing service in the terminal that executes the audio and video processing method. For the convenience of description, the specific execution process of the audio and video processing method is introduced below by taking the execution subject as the processor in the terminal as an example.

[0051] like Figure 2 As shown, the audio and video processing method may at least include:

[0052] S202: Collect audio and video data in the current scene.

[0053] Optionally, when users use audio and video to interact, the local terminal usually collects audio and video data in the current scene, and then transmits it to the remote target terminal, so that the target terminal plays the audio and video data collected locally. At the same time, the remote terminal also collects audio and video data in its scene in real time and transmits it to the local terminal. The local terminal also plays the audio and video data received from the remote terminal, thereby allowing multiple users to interact only based on audio and video.

[0054] Specifically, when collecting audio and video data, microphones and cameras pre-installed in the environment can be used to collect audio and video data. The number, location, and specific parameters of microphones and cameras can be determined according to the collection requirements in the actual scenario. For example, in scenarios such as classrooms and large conference rooms, it is considered that there may be multiple people in the room at the same time for multi-person discussions, unison reading, etc., and when a single person speaks, it is easy to cause unclear sound collection due to the large space. For such scenarios, multiple high-performance microphones and high-definition cameras can be deployed in the room to accurately capture the audio and video information in the room. All microphones and cameras are managed uniformly to achieve real-time collection, encoding, compression, transmission, and storage of audio and video data. In order to facilitate user operation, a software end for controlling audio and video recording can also be provided to users, and a convenient operation interface can be provided to users on the software end. Operators can control the opening and operation of each audio and video acquisition device according to their own needs through the software end to improve efficiency and convenience.

[0055] S204: Identify the audio scene type corresponding to the audio and video data, and determine the echo cancellation strategy corresponding to the audio scene type.

[0056] Optionally, in the general echo cancellation process, a fixed echo cancellation algorithm is often used to process all audio and video data. However, in teaching, conference and other scenarios, there may be many situations including but not limited to single-person speech, two-person conversation, and multi-person reading. In these different situations, the sound processing requirements for audio and video are also different. Therefore, the general echo cancellation algorithm lacks sufficient flexibility in the case of changing audio scenes, and cannot flexibly adjust the audio and video processing strategy according to the processing requirements in the actual scene, which limits the depth and breadth of educational technology applications.

[0057] Optionally, in order to ensure the quality of audio and video data in various scenarios, a suitable echo processing algorithm can be used for different audio and video data according to the scene information to which the audio and video data belongs. Then in an embodiment of the present application, for the collected audio and video data, it is first necessary to identify the audio scene type corresponding to the audio and video data, and then determine the echo elimination algorithm suitable for the audio and video data according to the audio scene type. It should be noted that the audio scene type identified for the audio and video data is based on the audio characteristics in the audio and video data, such as a single sound source, multiple sound sources, multiple people speaking in unison, media sound source playback, etc., that is, the audio scene type can explain what kind of echo processing method the audio and video data requires.

[0058] Furthermore, the scene information of the audio and video data determines its corresponding audio processing requirements to a certain extent. Therefore, in order to quickly determine the echo cancellation algorithm corresponding to the audio and video data, different echo cancellation strategies can be pre-configured for various possible scene types. For example, in a scene where a single person is speaking, considering that the speaker's voice needs to be clear and distinct, and other echo noises should be eliminated, an echo cancellation algorithm optimized for a single sound source can be used; in a scene where two people are speaking, it is necessary to highlight the voice of the local user in the local audio and video data and remove the echo of the remote user. Therefore, the double-talk suppression technology and the multi-channel adaptive filter can be combined to process the audio and video data.

[0059] Specifically, the echo cancellation strategy may specify at least one echo cancellation algorithm pre-configured for the audio processing requirements corresponding to the audio scene type. In a feasible implementation, the echo cancellation strategy corresponding to an audio scene type may include multiple echo cancellation algorithms. In this case, the echo cancellation strategy will also specify the number of times and order in which each echo cancellation algorithm is used, that is, the specific processing method of each echo cancellation algorithm on audio and video data. It can be understood that an echo cancellation algorithm can be simultaneously applied to echo cancellation strategies corresponding to multiple audio scene types. The use of each echo cancellation algorithm in a specific echo cancellation strategy can be deployed according to the needs of the actual application scenario, and the embodiments of the present application do not specifically limit this.

[0060] Then, in the embodiment of the present application, after the audio scene type of the audio and video data is identified, the echo cancellation strategy corresponding to the audio scene type can be further determined. In this way, through the subsequent determined echo processing strategy, each audio and video can obtain a targeted echo processing solution, which improves the accuracy and stability of echo cancellation and improves the ability of the terminal to perform echo cancellation on audio and video in diverse scenarios.

[0061] S206: Perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0062] Optionally, in combination with the above-described process, after the echo cancellation strategy corresponding to the audio and video data is determined, the audio data is subjected to echo cancellation processing according to the echo cancellation strategy to obtain the audio and video data after the echo is removed.

[0063] Furthermore, the processed audio and video data is sent to the remote target playback terminal that is interacting with the local terminal, so that the target playback terminal plays the processed audio and video data. Considering that when there are many interacting parties, it is necessary to transmit and receive data between multiple terminals, and the distance between the terminals of each party is usually far, and the positions of the interacting parties are not fixed, in order to enable users of all parties to interact conveniently, the transmission process of audio and video data can be completed through a cloud server. Cloud servers generally have high computing power and strong transmission capabilities, and are very suitable as intermediate nodes to transmit data between multiple terminals. Therefore, using cloud servers to transmit data can ensure that efficient interactive services are provided to users.

[0064] In an embodiment of the present application, an audio and video processing method is provided, which collects audio and video data in the current scene; identifies the audio scene type corresponding to the audio and video data, and determines the echo cancellation strategy corresponding to the audio scene type; performs echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmits the processed audio and video data to the target playback terminal, so that the target playback terminal plays the processed audio and video data. The scene information of the audio and video data determines its corresponding audio processing requirements to a certain extent. Then, according to the specific scene type to which the audio and video data belongs, the echo processing strategy corresponding to the scene type is adaptively used, so that each audio and video can obtain a targeted echo processing solution, which improves the accuracy and stability of echo cancellation and improves the terminal's ability to perform echo cancellation on audio and video in diverse scenes.

[0065] See also Figure 3 , Figure 3 A flowchart of an audio and video processing method provided in an embodiment of the present application.

[0066] like Figure 3 As shown, the audio and video processing method may at least include:

[0067] S302: Collect audio and video data in the current scene.

[0068] Regarding step S302, please refer to the detailed description in step S202, which will not be repeated here.

[0069] S304: If there is at least one available scene classification rule, determine whether there is a target scene classification rule that the audio and video data hits based on the current features of the audio and video data. The scene classification rule specifies the correspondence between at least one feature of the audio and video data and at least one audio scene type.

[0070] Optionally, see Figure 4 , Figure 4 A logical step diagram of an audio and video processing method provided in an embodiment of the present application. Figure 4As shown, firstly, the audio and video data of the current scene are collected through the camera and the microphone. After the audio and video data are collected, the audio scene type of the audio and video data is further identified.

[0071] Optionally, when establishing audio and video interaction between users, they often control the opening and closing of devices such as speakers, microphones, and cameras according to their needs during the interaction process. For example, when the local user does not need to speak, he usually chooses to directly turn off the local microphone, when the remote user does not need to speak, the local end can also actively turn off the microphone of the remote channel, and when the local user does not need the picture, he will choose to turn off the local camera, etc. Therefore, the speaker switch state characteristics, microphone switch state characteristics, picture characteristics, environmental characteristics and other feature information of the audio and video data can also explain the scene information of the audio and video data to a certain extent, that is, the characteristics of the audio and video data can be used to identify the audio scene type of the audio and video data.

[0072] Specifically, in order to determine whether audio and video data belongs to a certain audio scene type based on its features, the correspondence between various features of audio and video data and audio scene types can be predefined, that is, some scene classification rules are predefined, and the scene classification rules are used to define the correspondence between at least one feature of audio and video data and at least one audio scene type. Figure 4 When there is at least one preset available scene classification rule, it can be determined whether there is a target scene classification rule that the audio and video data hits based on the current features of the audio and video data. If it hits, the corresponding audio scene type is determined according to the target scene classification rule. If it does not hit, it is necessary to continue to use other methods to determine the audio scene type.

[0073] For example, if there is a scene classification rule that stipulates that when only the local microphone of the audio and video data is turned on, the audio scene type is determined to be a single-person speech scene; then when multiple users in an actual application interact, if it is identified that only the local microphone is turned on and all other remote microphones are turned off, this scene classification rule is hit. According to this scene classification rule, the audio scene type corresponding to the audio and video data can be directly determined as a single-person speech scene.

[0074] Furthermore, the correspondence between the characteristics of video data and the audio scene type in the scene classification rules is not unique and fixed, but can be set by the user according to his needs in the actual application scenario, so that the user can customize the rules, thereby processing various audio and video data more flexibly and personalized. There are also multiple ways to customize the rules. One is to pre-define by the user, including defining the correspondence between the specific characteristics of audio and video data and the specific audio scene type in the rules, so that some audio and video data with specific characteristics can be determined as specific scene types according to needs, realizing more flexible audio and video processing solutions, and enhancing practicality in actual scenes. Or a variety of rules can be preset in advance by the technician, and the rule activation control of each rule can be provided to the user in the software end, so that when the user needs to enable a certain rule, the corresponding rule can be activated by making an activation operation based on the rule activation control. In this way, the user can determine whether to use the rules based on his own needs by operating the controls, so that the scene classification of audio and video data is more personalized and meets user needs.

[0075] S306: If there is a target scene classification rule, determine the audio scene type corresponding to the audio and video data according to the target scene classification rule, and determine the echo cancellation strategy corresponding to the audio scene type.

[0076] Optionally, from the above introduction, it can be known that if the features of the audio and video data hit the target scene classification rule, it means that it currently fully meets the feature conditions of the audio scene type of the target scene classification rule, then the audio scene type corresponding to the audio and video data can be directly determined according to the target scene classification rule. Figure 4 , and further determine the echo cancellation strategy corresponding to the audio scene type, and use the echo cancellation strategy to process the audio and video data to eliminate the echo noise in the audio and video data and improve the quality of the audio and video.

[0077] S308: Input the audio and video data into the scene analysis big model, determine the audio scene type corresponding to the audio and video data output by the scene analysis big model, and determine the echo cancellation strategy corresponding to the audio scene type.

[0078] Alternatively, if there is no available scene classification rule, or the current audio and video data does not match the target scene classification rule, then other methods need to be used to determine the audio scene type. Figure 4 In a feasible implementation, the audio and video data can be input into the scene analysis model to determine the audio scene type corresponding to the audio and video data output by the scene analysis model.

[0079] Specifically, the scene analysis model is a model that is trained to convergence based on sample audio and video data. The sample audio and video data corresponds to a standard scene label that characterizes the correct scene type. After the sample audio and video data is input into the model, the model outputs the predicted scene label of the sample audio and video data. Then the model calculates the loss value based on the standard scene label and the predicted scene label according to the preset loss function formula, adjusts the parameters in the model according to the loss value, and continues to use the model with adjusted parameters for the next round of training until convergence to obtain a scene analysis model that can be deployed in the application scenario. In this process, the model can learn the characteristic patterns of different audio scenes, so that it has the ability to classify new audio data into scenes.

[0080] Optionally, when audio and video data is input into the scene analysis model, the audio and video data is first encoded to obtain embedded feature representations that can be received and analyzed by the model. These embedded feature representations can capture key information such as the spectrum, energy distribution, pitch, sound source, etc. of the audio signal in the audio and video data, providing a basis for subsequent scene classification. After the embedded feature representations of the audio and video data are input into the scene analysis model, the scene analysis model can perform scene recognition and output the audio scene type corresponding to the audio and video data, such as a single-person speaking scene, a multi-person reading scene, etc. Please refer to Figure 4 After determining the audio scene type, the corresponding echo cancellation strategy is further determined, and the echo cancellation strategy is used to process the audio and video data to eliminate the echo noise in the audio and video data and improve the quality of the audio and video.

[0081] S310: Perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0082] Regarding step S310, please refer to the detailed description in step S206, which will not be repeated here.

[0083] S312: When it is detected that the current features of the audio and video data have changed, the step of identifying the audio scene type corresponding to the audio and video data is re-executed.

[0084] Optionally, the collection of audio and video data is ongoing. During this process, the user may change the recording method of the audio and video, such as turning on the microphone, turning on the camera, starting to show courseware, etc. Then, as these audio and video features change, the current scene corresponding to the audio and video data may also change, for example, it may change from a scene of a single person speaking to a scene of a two-person conversation. That is, when it is detected that the current features of the audio and video data have changed, it means that the audio scene type may also have changed at this time, and the audio scene type corresponding to the audio and video data can be re-identified, and then the echo cancellation strategy suitable for the current scene is determined based on the newly identified audio scene type, and the audio and video data is echo-cancelled according to the new echo cancellation strategy. In this way, the echo cancellation strategy is flexibly adjusted according to the current features of the real-time audio and video data, ensuring the high quality of the audio and video data transmitted to the target playback terminal.

[0085] In an embodiment of the present application, an audio and video processing method is provided, which can determine whether the current audio and video data hits the scene classification rule based on the defined scene classification rule. If it hits, the audio scene type corresponding to the audio and video data is directly determined according to the scene classification rule, so that some audio and video data with specific characteristics can be determined as specific scene types according to the needs, so as to achieve a more flexible audio and video processing scheme and enhance the practicality in the actual scene. Among them, the available rules may be in two situations: one is that they can be used directly after deployment, or the user can control the use of the rules through the control after the rule is deployed. The user can determine whether to use the rules by operating the control based on his own needs, so that the scene classification of the audio and video data is more personalized and meets the user's needs. In addition to identifying the scene of audio and video according to the rules, the scene type of audio and video data can also be efficiently and accurately identified through a pre-trained large model. After identifying the audio scene type, the corresponding echo cancellation strategy can be determined. Each echo cancellation strategy is pre-configured according to the audio processing requirements in the scene, and the strategy includes at least one targeted echo processing algorithm, which enables the echo cancellation strategy corresponding to each audio scene type to accurately solve the audio echo problem in the scene. In addition to the above, when the current characteristics of the above audio and video data are monitored to change, the corresponding echo cancellation strategy can be flexibly adjusted according to the current characteristics of the real-time audio and video data to ensure the high quality of the audio and video data transmitted to the target playback terminal.

[0086] See also Figure 5 , Figure 5 This is a structural block diagram of an audio and video processing device provided in an embodiment of the present application. Figure 5 As shown, the audio and video processing device 500 includes:

[0087] The data collection module 510 is used to collect audio and video data in the current scene;

[0088] A scene analysis module 520, for identifying an audio scene type corresponding to the audio and video data, and determining an echo cancellation strategy corresponding to the audio scene type;

[0089] The echo cancellation module 530 is used to perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to the target playback terminal so that the target playback terminal plays the processed audio and video data.

[0090] Optionally, the scene analysis module 520 is also used to determine whether there is a target scene classification rule that the audio and video data hits based on the current features of the audio and video data if there is at least one available scene classification rule, and the scene classification rule specifies the correspondence between at least one feature of the audio and video data and at least one audio scene type; if there is a target scene classification rule, the audio scene type corresponding to the audio and video data is determined according to the target scene classification rule.

[0091] Optionally, the available scene classification rule is a predefined scene classification rule; or, the available scene classification rule is a scene classification rule that is predefined and enabled by a user based on a rule-based enabling control.

[0092] Optionally, the characteristics of the audio and video data include at least one of speaker switch state characteristics, microphone switch state characteristics, picture characteristics, and environment characteristics.

[0093] Optionally, the scene analysis module 520 is further used to input the audio and video data into the scene analysis big model, and determine the audio scene type corresponding to the audio and video data output by the scene analysis big model.

[0094] Optionally, the echo cancellation strategy specifies at least one echo cancellation algorithm preconfigured for an audio processing requirement corresponding to an audio scene type.

[0095] Optionally, the audio and video processing device 500 further includes: a real-time monitoring module, which is used to re-execute the step of identifying the audio scene type corresponding to the audio and video data when it is monitored that the current characteristics of the audio and video data have changed.

[0096] In an embodiment of the present application, an audio and video processing device is provided, wherein a data acquisition module is used to collect audio and video data in the current scene; a scene analysis module is used to identify the audio scene type corresponding to the audio and video data, and determine the echo cancellation strategy corresponding to the audio scene type; an echo cancellation module is used to perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to a target playback terminal, so that the target playback terminal plays the processed audio and video data. The scene information of the audio and video data determines its corresponding audio processing requirements to a certain extent. Then, according to the specific scene type to which the audio and video data belongs, the echo processing strategy corresponding to the scene type is adaptively used, so that each audio and video can obtain a targeted echo processing solution, which improves the accuracy and stability of echo cancellation and improves the terminal's ability to perform echo cancellation on audio and video in a variety of scenes.

[0097] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0098] See also Figure 6 , Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 6 As shown, the terminal 600 may include: at least one terminal processor 601 , at least one network interface 604 , a user interface 603 , a memory 605 , and at least one communication bus 602 .

[0099] The communication bus 602 is used to realize the connection and communication between these components.

[0100] The user interface 603 may include a display screen (Display) and a camera (Camera), and the optional user interface 603 may also include a standard wired interface and a wireless interface.

[0101] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0102] Among them, the terminal processor 601 may include one or more processing cores. The terminal processor 601 uses various interfaces and lines to connect various parts in the entire terminal 600, and executes various functions and processes data of the terminal 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Optionally, the terminal processor 601 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The terminal processor 601 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the terminal processor 601, and it can be implemented separately through a chip.

[0103] Among them, the memory 605 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 605 includes a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 605 may also be optionally at least one storage device located away from the aforementioned terminal processor 601. As Figure 6 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an audio and video processing program.

[0104] exist Figure 6In the terminal 600 shown, the user interface 603 is mainly used to provide an input interface for the user and obtain the data input by the user; and the terminal processor 601 can be used to call the audio and video processing program stored in the memory 605, and specifically perform the following operations:

[0105] Collect audio and video data in the current scene;

[0106] Identify the audio scene type corresponding to the audio and video data, and determine the echo cancellation strategy corresponding to the audio scene type;

[0107] The audio and video data are subjected to echo cancellation processing according to the echo cancellation strategy, and the processed audio and video data are transmitted to the target playback terminal, so that the target playback terminal plays the processed audio and video data.

[0108] In some embodiments, when the terminal processor 601 executes the identification of the audio scene type corresponding to the audio and video data, it specifically performs the following steps: if there is at least one available scene classification rule, it is determined whether there is a target scene classification rule hit by the audio and video data based on the current features of the audio and video data, and the scene classification rule specifies the correspondence between at least one feature of the audio and video data and at least one audio scene type; if there is a target scene classification rule, the audio scene type corresponding to the audio and video data is determined based on the target scene classification rule.

[0109] In some embodiments, the available scene classification rules are predefined scene classification rules; or, the available scene classification rules are predefined scene classification rules after the user makes an enabling operation based on the rule enabling control.

[0110] In some embodiments, the characteristics of the audio and video data include at least one of speaker switch state characteristics, microphone switch state characteristics, picture characteristics, and environment characteristics.

[0111] In some embodiments, when the terminal processor 601 identifies the audio scene type corresponding to the audio and video data, it specifically performs the following steps: input the audio and video data into the scene analysis model, and determine the audio scene type corresponding to the audio and video data output by the scene analysis model.

[0112] In some embodiments, the echo cancellation policy specifies at least one echo cancellation algorithm preconfigured for audio processing requirements corresponding to the audio scene type.

[0113] In some embodiments, the terminal processor 601 further specifically performs the following steps: when it is detected that the current features of the audio and video data have changed, re-execute the step of identifying the audio scene type corresponding to the audio and video data.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0115] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above process or function according to the embodiment of this specification is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted by the above computer-readable storage medium. The above computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state drives (SSD)), etc.

[0117] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] The above is a description of an audio and video processing method, device, storage medium and terminal provided in the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio and video processing method, characterized in that: The method comprises: Collect audio and video data in the current scene; Identifying an audio scene type corresponding to the audio and video data, and determining an echo cancellation strategy corresponding to the audio scene type; The audio and video data are subjected to echo cancellation processing according to the echo cancellation strategy, and the processed audio and video data are transmitted to a target playback terminal, so that the target playback terminal plays the processed audio and video data.

2. The method according to claim 1, characterized in that The identifying the audio scene type corresponding to the audio and video data includes: If there is at least one available scene classification rule, determining whether there is a target scene classification rule hit by the audio and video data according to the current features of the audio and video data, wherein the scene classification rule specifies a correspondence between at least one feature of the audio and video data and at least one audio scene type; If the target scene classification rule exists, the audio scene type corresponding to the audio and video data is determined according to the target scene classification rule.

3. The method according to claim 2, characterized in that The available scene classification rules are predefined scene classification rules; Alternatively, the available scene classification rule is a scene classification rule that is predefined and is enabled by a user based on a rule-based enabling control.

4. The method according to claim 2, characterized in that: The characteristics of the audio and video data include at least one of a speaker switch state characteristic, a microphone switch state characteristic, a picture characteristic, and an environment characteristic.

5. The method according to claim 1, characterized in that The identifying the audio scene type corresponding to the audio and video data includes: The audio and video data are input into a scene analysis model, and the audio scene type corresponding to the audio and video data output by the scene analysis model is determined.

6. The method according to claim 1, characterized in that The echo cancellation strategy specifies at least one echo cancellation algorithm preconfigured for the audio processing requirements corresponding to the audio scene type.

7. The method according to claim 1, characterized in that The method further comprises: When it is monitored that the current feature of the audio and video data changes, the step of identifying the audio scene type corresponding to the audio and video data is re-executed.

8. An audio and video processing device, characterized in that: The device comprises: Data collection module, used to collect audio and video data in the current scene; A scene analysis module, used to identify the audio scene type corresponding to the audio and video data, and determine the echo cancellation strategy corresponding to the audio scene type; The echo cancellation module is used to perform echo cancellation processing on the audio and video data according to the echo cancellation strategy, and transmit the processed audio and video data to a target playback terminal so that the target playback terminal plays the processed audio and video data.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 7.

10. A terminal, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

Citation Information

Cited By

  • Audio echo cancellation method, module and related equipment

    CN121600946A

  • Audio echo cancellation method, module and related device

    CN121600946B