Video processing method, device and computer readable storage medium

Abstract

Embodiments of the present application provide a video processing method, device and computer readable storage medium, wherein the method comprises: performing image recognition on a to-be-processed video to obtain an image recognition result of a target object in the to-be-processed video; performing audio information extraction on the to-be-processed video to obtain to-be-processed audio; performing feature information extraction on the to-be-processed audio according to the image recognition result to obtain audio features; performing noise reduction processing on the to-be-processed audio through the audio features to obtain noise-reduced audio; and fusing the noise-reduced audio into the to-be-processed video to obtain a processed video.

Description

Technical Field

[0001] This application relates to the field of audio and video processing technology, and includes, but is not limited to, a video processing method, apparatus, and computer-readable storage medium. Background Technology

[0002] In today's globalized world, online meetings are playing an increasingly important role in people's work, life, and studies. Especially during special periods, when some work, study, and life are conducted online, effectively removing background noise during meetings is an important aspect of ensuring meeting quality and strengthening effective communication.

[0003] Currently, related technologies use both hardware and software noise reduction to denoise audio in online meetings. However, these technologies only address the noise from the sound perspective, and their effectiveness decreases or even fails when the environment in which the speaker is located changes. Therefore, providing a noise reduction method that can adapt to environmental changes is an urgent problem to be solved. Summary of the Invention

[0004] In view of the problems in related technologies, this application provides a video processing method, apparatus and computer-readable storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video processing method, including:

[0007] Perform image recognition on the video to be processed to obtain the image recognition result of the target object in the video to be processed;

[0008] The audio information of the video to be processed is extracted to obtain the audio to be processed;

[0009] Based on the image recognition results, feature information is extracted from the audio to be processed to obtain audio features;

[0010] The audio to be processed is denoised using the audio features to obtain the denoised audio.

[0011] The noise-reduced audio is then fused into the video to be processed to obtain the processed video.

[0012] This application provides a video processing apparatus, including:

[0013] The recognition module is used to perform image recognition on the video to be processed, and obtain the image recognition result of the target object in the video to be processed.

[0014] An audio information extraction module is used to extract audio information from the video to be processed, thereby obtaining the audio to be processed.

[0015] The feature information extraction module is used to extract feature information from the audio to be processed based on the image recognition result, so as to obtain audio features;

[0016] The noise reduction processing module is used to perform noise reduction processing on the audio to be processed based on the audio features, so as to obtain the noise-reduced audio;

[0017] The fusion module is used to fuse the noise-reduced audio into the video to be processed, thereby obtaining the processed video.

[0018] This application provides a video processing device, including:

[0019] The memory is used to store executable instructions; the processor is used to implement the video processing method described above when executing the executable instructions stored in the memory.

[0020] This application provides a computer-readable storage medium storing executable instructions, which are used to cause a processor to execute the executable instructions to implement the above-described video processing method.

[0021] The video processing method, apparatus, and computer-readable storage medium provided in this application embodiment perform image recognition and audio information extraction on the video to be processed, respectively obtaining the image recognition result of the target object and the audio to be processed. Based on the image recognition result of the target object, the audio features of the video to be processed are obtained. Noise reduction is performed on the audio corresponding to the target object based on the audio features, and the noise-reduced audio is fused into the video to be processed, thereby achieving video processing. Thus, the video processing method provided in this application embodiment performs noise reduction on the audio to be processed based on the video image and audio features corresponding to the target object. This allows the application embodiment to accurately reduce noise in different environments by combining the image and audio of the target object, improving the noise reduction effect. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of the video processing method provided in the embodiments of this application;

[0023] Figure 2 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0027] Figure 6 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0028] Figure 7 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0029] Figure 8 This is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the composition structure of the video processing apparatus provided in the embodiments of this application;

[0031] Figure 10 This is a schematic diagram of the composition structure of the video processing device provided in the embodiments of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0034] In related technologies, background noise removal during meetings is mainly achieved through two methods: collecting audio information from specific spatial directions (hardware noise reduction) and collecting existing noise sample audio information (software noise reduction). Hardware noise reduction primarily uses devices such as external sound cards or condenser microphone arrays to collect sound from specific directions and distances, filtering out noise from other directions and distances to achieve noise reduction. Software noise reduction, on the other hand, involves pre-collecting ambient noise samples and using software to analyze the sample noise to obtain characteristic information such as frequency, amplitude, or magnitude. At the start of the actual meeting, software filtering or active noise suppression is used to filter out or cancel background noise, retaining only valid speech to achieve noise reduction.

[0035] However, both hardware and software noise reduction technologies have their own drawbacks and cannot be effectively applied in many scenarios. For example, hardware noise reduction solutions, whether high-performance external sound cards or condenser microphone arrays, are relatively expensive and inconvenient to carry, making them unsuitable for home and transportation use; hardware noise reduction cannot effectively distinguish noise sources that are in the same direction and at the same distance as meeting participants; if meeting participants are far from the device, the hardware's ability to capture noise will also decrease; at the same time, the relevant hardware equipment requires prior installation and debugging, which is a high barrier to entry for ordinary users.

[0036] In software-based noise reduction solutions, the denoising is based on pre-recorded noise samples. Therefore, software noise reduction can only filter and suppress noise with similar characteristic values ​​to the noise samples, requiring the background noise to remain constant. If the characteristic values ​​of the background noise change, the effect will decrease significantly. It is also ineffective against sudden noises, such as the sound of doors opening and closing, the sound of people talking, and short-lived noises like car horns.

[0037] Because related technologies only consider the sound itself when establishing noise reduction models, these models, once determined, cannot adapt to changing circumstances. When external conditions change and no longer conform to the model settings, the noise reduction effect decreases or even fails. This application provides a video processing method that performs image recognition and audio information extraction on the video to be processed, obtaining the image recognition result of the target object and the audio to be processed, respectively. Based on the image recognition result of the target object, the audio features of the video to be processed are obtained. Noise reduction is then applied to the audio corresponding to the target object based on these audio features, and the denoised audio is fused back into the video to be processed, thus achieving video processing. In this way, the video processing method provided by this application reduces noise in the audio based on the image and audio features corresponding to the target object. This allows the application to combine the image and audio of the target object in different environments, accurately reducing noise in the audio corresponding to the target object and enhancing the noise reduction effect.

[0038] Figure 1 This is a schematic diagram illustrating an application scenario of the video processing method provided in the embodiments of this application, such as... Figure 1As shown, the video processing system for implementing the video processing method includes a terminal 10, a network 20, and a server 30. The server 30 obtains the video processing request sent by the terminal 10 through the network 20. The server 30 performs image recognition and audio information extraction on the video to be processed, obtains the image recognition result of the target object and the audio to be processed, obtains the audio features of the target object based on the image recognition result, performs noise reduction processing on the audio to be processed based on the audio features, obtains the noise-reduced audio, merges the noise-reduced audio into the video to be processed, obtains the noise-reduced video, and sends the noise-reduced video to the terminal 10 through the network 20. The terminal 10 visualizes the noise-reduced video, and the user can directly display the received noise-reduced video on the current interface 10-1.

[0039] The video processing method provided in this application will be described below with reference to exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 2 , Figure 2 This is a flowchart illustrating the video processing method provided in the embodiments of this application, which will be combined with... Figure 2 The steps shown are explained.

[0040] Step S201: Perform image recognition on the video to be processed to obtain the image recognition result of the target object in the video to be processed.

[0041] In some embodiments, the video to be processed refers to a video from which audio noise reduction is required. The video processing method provided in this application can be applied to video conferencing, where the video to be processed can be the video of each participant. The video processing method provided in this application can also be applied to scenarios where students are taking online classes, where the video to be processed is the online class video of each student.

[0042] Here, the target object is the person making the sound in the video, which can be a person or an object. For example, the target object could be the participants in a meeting, a smart device making a sound during the meeting, or an object that makes a sound due to environmental factors during the meeting.

[0043] In some embodiments, image recognition of the video to be processed can be performed on each frame of the video, and the image recognition result can be the identification of the target user's facial features when speaking. For example, expressions during normal speech, expressions when encountering an unexpected situation, expressions of surprise, or expressions of anger; the image recognition result can also be the time point of the target user's speech.

[0044] Step S202: Extract audio information from the video to be processed to obtain the audio to be processed.

[0045] In some embodiments, extracting audio information from a video to be processed means separating the audio stream of the video to be processed, which is the audio to be processed.

[0046] Step S203: Based on the image recognition result, extract feature information from the audio to be processed to obtain audio features.

[0047] In some embodiments, audio features refer to the audio features of the sound-producing objects in the audio to be processed. In an audio to be processed, there may be multiple objects making sounds. Extracting feature information based on image recognition results can mean determining the sound audio corresponding to each sound-producing object based on the sound time of each sound-producing object obtained from image recognition, extracting feature information for each sound audio, and obtaining audio feature information such as the pitch, tone, and timbre of the sound of each object. The obtained audio feature information is then determined as the audio features of each sound-producing object.

[0048] Step S204: Using the audio features, perform noise reduction processing on the audio to be processed to obtain the noise-reduced audio.

[0049] In some embodiments, noise reduction of the audio to be processed refers to suppressing or filtering noise in the audio to be processed, or amplifying the effective audio in the audio to be processed, thereby achieving noise reduction. For example, embodiments of this application can divide the audio to be processed into effective audio and noise based on audio characteristics. For effective audio, methods such as sound amplification or sampling enhancement can be used to increase the clarity and volume of the effective audio. For the noise portion, methods such as filtering or inverse suppression can be used to suppress or remove the noise, thereby obtaining the noise after noise reduction.

[0050] Step S205: The noise-reduced audio is fused into the video to be processed to obtain the processed video.

[0051] In some embodiments, the denoised audio and the video to be processed can be fused together at each time point, starting from the playback time of the video to be processed, according to the playback order of the video to be processed, to obtain the denoised video.

[0052] The video processing method provided in this application embodiment reduces noise in the audio to be processed based on the image and audio features corresponding to the target object. This allows the embodiment to accurately reduce noise in the audio to be processed by combining the image and audio of the target object in different environments, thereby improving the noise reduction effect.

[0053] In some embodiments, image recognition of the video to be processed can yield the speaking time and speaking characteristics of the target object. Figure 3 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 3As shown, in some embodiments, step S201 can be implemented through the following steps:

[0054] Step S301: Perform image recognition on each video frame of the video to be processed to determine the object feature information of the target object in each video frame.

[0055] In some embodiments, the object feature information of the target object may refer to facial change information of the target object's face and the corresponding time points. For example, it may refer to whether the target object's face shows surprise or anger, and record the time points corresponding to the surprise or anger expressions. In this embodiment, after acquiring the video to be processed, image recognition is performed on each video frame of the target object in the video to be processed to obtain all object feature information of the target object in the video to be processed.

[0056] Step S302: Determine the audio processing information of the audio to be processed corresponding to the target object based on the object feature information.

[0057] In some embodiments, the target object may be a target person, and the object feature information may correspond to the facial change information of the target person. Step S302 can be implemented through the following steps:

[0058] Step S3021: Based on the facial change information, determine the speech start time, speech end time, and facial features of the target object.

[0059] In some embodiments, the mouth movement information of the target person can be determined based on the facial change information of the target person. When the mouth movement information of the target person continues for a period of time, it can be determined that the target person is speaking. At this time, the starting time of the mouth movement information of the target person can be determined as the start time of the target person's speech, and the ending time of the mouth movement information of the target person can be determined as the end time of the target person's speech.

[0060] In some embodiments, facial features of the target person can also be determined based on facial change information. Facial features can be some characteristic expressions corresponding to the target person, such as angry expressions, joyful expressions, or surprised expressions.

[0061] Step S3022: Determine the speech start time, speech end time, and facial features as the audio processing information of the target object.

[0062] Step S303: Determine the audio processing information as the image recognition result of the target object.

[0063] This application embodiment uses image recognition to determine the audio processing information corresponding to the target user in the video to be processed. Based on the audio processing information, the audio segment corresponding to the target object can be determined, so that the audio segment corresponding to the target object can be accurately denoised in the subsequent process, thereby improving the accuracy of the denoising target.

[0064] In some embodiments, the audio features corresponding to the target object can be obtained based on the image recognition results. Figure 4 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, in some embodiments, step S203 can be implemented through the following steps:

[0065] Step S401: Determine the audio start time and audio end time corresponding to the target object in the audio to be processed based on the speech start time and speech end time of the target object.

[0066] In some embodiments, since the audio to be processed is separated from the video to be processed, the audio to be processed and the video to be processed have the same duration, and at the same time, the picture in the video to be processed corresponds to the sound in the audio to be processed. Therefore, the start time and end time of the target object's speech can be determined as the start time and end time of the audio corresponding to the target object in the audio to be processed.

[0067] Step S402: Determine the audio segment located between the start time and end time of the audio in the audio to be processed as the target audio.

[0068] Here, the target audio refers to the audio corresponding to the target object when it makes a sound.

[0069] Step S403: Extract feature information from the target audio to obtain the audio features of the target audio.

[0070] It should be noted that the audio features of the target audio refer to the frequency, amplitude, intensity, loudness, timbre, pitch, and feature vectors of the target audio after sampling and digitization.

[0071] In some embodiments, feature extraction of target audio can be performed by first extracting a multidimensional feature vector for each target audio segment, and then extracting feature information from that feature vector. There are many methods for extracting multidimensional feature vectors from target audio, such as perceptual linear prediction, linear predictive coding, or mel-frequency cepstral coefficients.

[0072] In some embodiments, after obtaining the multidimensional feature vector, the frequency of the target audio can be determined by the spectral envelope method or the cepstral method, or other audio features of the target audio can be obtained by other methods.

[0073] This application embodiment extracts the audio features of the target audio by performing feature information extraction on the target audio corresponding to the target object, which can accurately locate the effective audio of the target object and the noise other than the effective audio in the target audio.

[0074] In some embodiments, the audio to be processed may contain abnormal sounds such as exclamations. It is necessary to determine whether the abnormal sound is valid audio emitted by the target object or noise caused by the environment. Figure 5 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 5 As shown, in some embodiments, the embodiments of this application may also include the following steps:

[0075] Step S501: Obtain the abnormal audio in the audio to be processed and the abnormal audio time point corresponding to the abnormal audio.

[0076] Here, abnormal audio can be a sudden sound within the audio to be processed. For example, abnormal audio could be the sound of a stool suddenly falling over, the sound of a door opening or closing, the sound of someone talking, the sound of a car horn outside, or the startled cry of the target object. When abnormal audio is acquired, the corresponding time point of the abnormal audio will also be acquired. The abnormal audio time point can be the moment the abnormal audio occurs or the duration of the abnormal audio.

[0077] Step S502: Obtain the facial features of the target person at the abnormal audio time point.

[0078] In some embodiments, when an abnormal audio time point is obtained, the abnormal facial features of the target person at that abnormal audio time point can be determined in the video to be processed based on the abnormal audio time point. The abnormal facial features can be the mouth features of the target person or the face features of the target person. For example, it can be determined whether the target person at the abnormal audio time point has made a sound, or whether their face has facial movements such as anger or surprise.

[0079] Step S503: When the facial features of the abnormal point match any facial information in the preset facial information database, the abnormal audio is determined to be valid audio; wherein, the facial information database includes facial information of the target person in different vocal states.

[0080] In some embodiments, a facial information database can be pre-set for the target person, which includes at least all facial information of the target person in different vocal states. For example, the facial information database includes at least the facial expressions of the target person when crying, surprised, exclaiming, and angry.

[0081] In this embodiment, when abnormal facial features are obtained at an abnormal audio time point, these abnormal facial features are matched with all facial information in a preset facial information database. If the abnormal facial features match any facial information in the preset facial information database, it is determined that the abnormal audio was emitted by the target person, and the abnormal audio is considered valid audio. Here, determining whether the abnormal facial features match facial information in the facial information database can be done through facial feature similarity. If the similarity between the abnormal facial features and any facial information exceeds a preset similarity threshold, it can be considered that the abnormal facial features match the facial information.

[0082] Step S504: When the facial features of the abnormal point do not match any facial information in the facial information database, the abnormal audio is determined to be noise.

[0083] In some embodiments, if an abnormal facial feature does not match any facial information in the facial information database, it is considered that the abnormal facial feature did not originate from the target person, and the abnormal audio is determined to be noise.

[0084] The embodiments of this application analyze sudden audio in the audio to be processed, eliminate sudden noise in the audio to be processed, and improve the noise reduction effect.

[0085] In some embodiments, a pre-trained audio denoising model can be used to denoise the audio to be processed. This audio denoising model can be trained using a preset audio / video model. Figure 6 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 6 As shown, in some embodiments, the audio noise reduction model can be obtained through the following steps:

[0086] Step S601: Collect audio and video samples of the target object.

[0087] In some embodiments, the audio and video samples can be collected from the target object in a noise-free environment. That is, the audio samples contain only the audio of the target object and do not contain sudden audio or other noise, and the video samples contain only the image information of the target object.

[0088] Step S602: Extract feature information from the audio sample to determine the audio sample features of the target object.

[0089] In some embodiments, the method for feature extraction of audio samples can be the same as the method for feature information extraction in step S403, or other feasible feature information extraction methods can be used to extract features from audio samples, thereby obtaining the audio sample features of the target object. Audio sample features can also include features such as sample frequency, sample amplitude, sample sound intensity, sample loudness, sample timbre, sample pitch, and the sample feature vector after audio sampling and digitization.

[0090] Step S603: Identify the video sample to determine the video sample features of the target object.

[0091] In some embodiments, video sample features may be facial features of the target object, i.e., all facial features corresponding to whether the target object is speaking or not. Video sample features can be determined by performing image recognition on each frame of the video sample.

[0092] Step S604: Train the preset audio and video model using the audio sample features and the video sample features to obtain the audio noise reduction model.

[0093] In some embodiments, the preset audio and video model may or may not be a noise reduction model. The preset audio and video model is trained using audio sample features and video sample features to train the preset audio and video model into an audio noise reduction model corresponding to the target object.

[0094] In some embodiments, the audio noise reduction model includes at least a feature matching module and an audio separation module. Figure 7 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 7 As shown, in some embodiments, step S204 can be implemented through the following steps:

[0095] Step S701: Using a pre-trained audio denoising model, the target audio is decomposed into effective audio and noise based on the audio features.

[0096] In some embodiments, decomposing the target audio into valid audio and noise using a trained audio denoising model can be achieved through the following steps:

[0097] Step S7011: Through the feature matching module, the audio features of the target audio are matched with the audio sample features of the target object to determine the similarity between the audio features and the audio sample features.

[0098] In some embodiments, the similarity between audio features and audio sample features can be obtained through the feature matching module in the audio noise reduction model. The similarity includes at least the similarity between the audio features and the audio sample features in terms of features such as frequency, amplitude, or timbre.

[0099] Step S7012: Through the audio decomposition module, the audio segments in the target audio whose similarity reaches a preset similarity threshold are determined as the valid audio, and the other audio segments in the target audio besides the valid audio are determined as noise.

[0100] In some embodiments, the preset similarity threshold can be preset based on the vocal characteristics of the target object. If the audio characteristics of the target object are similar in each vocalization, it indicates that the vocal characteristics of the target object are relatively fixed and there is no significant difference. In this case, the value of the preset similarity threshold can be set relatively high. For example, the value of the preset similarity threshold can be 90%. When the similarity between the audio characteristics of an audio segment and the audio sample characteristics is greater than 90%, the audio segment is determined as the valid audio, and other audio segments in the target audio other than the valid audio are determined as noise.

[0101] In some embodiments, if the audio features of the target object differ significantly in each vocalization, it indicates that the vocal features of the target object are not fixed and there are significant differences. In this case, the preset similarity threshold value can be set relatively low, for example, the preset similarity threshold value can be 70%.

[0102] Step S702: Process the target audio using at least one of the following methods: enhance the effective audio and suppress the noise to obtain the noise-reduced audio.

[0103] In some embodiments, after determining the effective audio and noise in the target audio, the noise can be suppressed or filtered by tools such as an external sound card or noise reduction software, while the effective audio is enhanced to increase its clarity and volume, resulting in a noise-reduced video.

[0104] In this embodiment, the target audio corresponding to the target object is divided into effective audio and noise. The noise is accurately reduced and the effective audio is enhanced, thereby improving the noise reduction effect.

[0105] The following describes an exemplary application of this application in a real-world application scenario.

[0106] This application's embodiments are primarily used in scenarios where audio and video coexist and the user's face is unobstructed, a typical scenario being students taking online classes at home. This application's embodiments use facial video data analysis as an anchor point to determine the timing of the user's speech in the time domain, and use user audio samples as a basis to separate effective audio data in the frequency domain. The combination of these two methods completes the filtering and elimination of noise.

[0107] Figure 8 This is a flowchart illustrating the video processing method provided in an embodiment of this application, as shown below. Figure 8 As shown, in some embodiments, the embodiments of this application can be implemented through the following steps:

[0108] Step S801: Record user facial information.

[0109] Step S802: Perform facial recognition on the user's facial information.

[0110] Step S803: Detect whether the user has bound audio features.

[0111] Here, if it is detected that the user has not bound an audio feature, step S804 is executed; if it is detected that the user has bound an audio feature, step S807 is executed.

[0112] In some embodiments, the audio features of the user's (i.e. the target object's) avatar image video data and the user's audio sample data are associated to ensure that the current speaker is the user himself / herself.

[0113] Step S804: Collect audio samples from the user.

[0114] Step S805: Analyze the audio samples and extract their features to establish a noise reduction model.

[0115] In some embodiments, the user's (i.e. the target object's) own audio data is collected, and audio learning and training analysis are performed through local or cloud software to obtain audio comparison features (i.e. audio sample features) and establish a noise reduction model (i.e. audio noise reduction model).

[0116] Step S806: Bind the noise reduction model to the user's facial information.

[0117] In some embodiments, after obtaining the user's facial information, the facial information can be associated with the collected audio information in order to determine whether the audio information is spoken by the user (including single-person and multi-person environments) to prevent the noise reduction model from misjudging.

[0118] Step S807: Enter the meeting.

[0119] Step S808: Recognize the current user's lip movements.

[0120] Step S809: Determine whether the user has spoken.

[0121] In some embodiments, if it is determined that the audio (i.e. the audio to be processed) is spoken by a user (i.e. the target object), step S810 is executed; if it is determined that the audio is not spoken by a user, step S811 is executed.

[0122] Step S810: Collect the user's real-time voice.

[0123] In some embodiments, during a meeting, the start and end times of a user's speech can be determined by recognizing the user's facial movements.

[0124] Step S811: Treat the currently collected speech as noise.

[0125] Step S812: Separate user speech and background noise according to the model, and determine whether there is sudden noise based on user facial expression recognition.

[0126] In some embodiments, the audio collected during the speaking time period is analyzed in the frequency domain using a noise reduction model. The audio information (i.e., audio sample features) during the non-speaking time period is compared, and the part with similar features is extracted from the audio collected during the speaking time period based on the valid audio sample (i.e., audio sample). This part is defined as valid data, and the rest is defined as noise.

[0127] Step S813: Noise suppression and user voice enhancement.

[0128] In some embodiments, noise can be removed by filtering or reverse suppression, and effective audio can be enhanced by sound amplification or sampling enhancement to increase the clarity and volume of the effective audio. The audio is then uploaded to the server for multi-channel mixing and distributed to each conference terminal.

[0129] Step S814: Compare the changes in audio during and outside the speaking period to improve the noise reduction model.

[0130] In some embodiments, step S812 can be continued by improving or refining the noise reduction model.

[0131] In some embodiments, the user's real-time voice information can be separated by comparing the differences between the user's speaking and non-speaking audio, and its audio feature values ​​can be collected to modify and improve the existing noise reduction model, thereby improving the noise reduction performance of the entire system.

[0132] Step S815: Transmit the processed audio to each conference terminal.

[0133] In some embodiments, sudden audio information generated during a user's speech can also be determined based on the user's facial features to determine whether the sound was made by the user. For example, expressions such as surprise or anger can be associated with relevant audio to determine whether the sudden audio is noise.

[0134] In this embodiment of the application, during a meeting, the user's face is first recognized, and then the start and end times of speaking are determined based on facial changes, such as mouth movements. Compared to non-speaking periods, the majority of the new audio information added during speaking periods is user speech, which can be separated from user speech in noisy backgrounds by comparison. New feature values ​​are analyzed and the noise reduction model is improved to enable the model to adapt and continuously improve the noise reduction effect.

[0135] The noise filtering model in this embodiment is constructed based on the user's own voice information, rather than background noise. Facial recognition distinguishes between speaking and non-speaking periods, and comparing audio changes between the two periods allows for the separation of the user's real-time voice, which is then used to continuously improve and optimize the noise model. The noise reduction model provided in this embodiment is an active noise reduction model, which can eliminate not only continuous noise but also sudden and changing noise.

[0136] This application combines valid user audio information samples and user facial image information to separate background noise. It can not only handle continuous background noise but also effectively suppress sudden and changing background noise. Furthermore, because it can accurately determine the user's speaking time period, by comparing the differences in audio information across different time periods, the noise reduction process possesses self-learning and self-optimization capabilities.

[0137] Figure 9 This is a schematic diagram of the composition structure of the video processing device provided in the embodiments of this application, such as... Figure 9 As shown, the video processing apparatus 900 includes:

[0138] The recognition module 901 is used to perform image recognition on the video to be processed, and obtain the image recognition result of the target object in the video to be processed; the audio information extraction module 902 is used to extract audio information from the video to be processed, and obtain the audio to be processed; the feature information extraction module 903 is used to extract feature information from the audio to be processed based on the image recognition result, and obtain audio features; the noise reduction processing module 904 is used to perform noise reduction processing on the audio to be processed through the audio features, and obtain the noise-reduced audio; the fusion module 905 is used to fuse the noise-reduced audio into the video to be processed, and obtain the processed video.

[0139] In some embodiments, the audio information extraction module 902 is further configured to perform image recognition on each video frame of the video to be processed, determine the object feature information of the target object in each video frame; determine the audio processing information of the audio to be processed corresponding to the target object based on the object feature information; and determine the audio processing information as the image recognition result of the target object.

[0140] In some embodiments, the target object is a target person, and the object feature information is the facial change information of the target person; the audio information extraction module 902 is further configured to determine the speech start time, speech end time, and facial features of the target object based on the facial change information; and to determine the speech start time, speech end time, and facial features as the audio processing information of the target object.

[0141] In some embodiments, the feature information extraction module 903 is further configured to determine the audio start time and audio end time corresponding to the target object in the audio to be processed based on the speech start time and speech end time of the target object; determine the audio segment in the audio to be processed located between the audio start time and the audio end time as the target audio; and perform feature information extraction on the target audio to obtain the audio features of the target audio.

[0142] In some embodiments, the video processing apparatus further includes: a first acquisition module, configured to acquire abnormal audio in the audio to be processed and the abnormal audio time point corresponding to the abnormal audio; a second acquisition module, configured to acquire abnormal facial features of the target person at the abnormal audio time point; a first matching module, configured to determine that the abnormal audio is valid audio when the abnormal facial features match any facial information in a preset facial information database; wherein the facial information database includes facial information of the target person in different vocal states; and a second matching module, configured to determine that the abnormal audio is noise when the abnormal facial features do not match any facial information in the facial information database.

[0143] In some embodiments, the noise reduction processing module 904 is further configured to decompose the target audio into effective audio and noise based on the audio features using a pre-trained audio noise reduction model; and to process the target audio using at least one of the following processing methods: enhancing the effective audio and suppressing the noise to obtain the noise-reduced audio.

[0144] In some embodiments, the video processing apparatus further includes: an acquisition module for acquiring audio samples and video samples of the target object; a feature extraction module for extracting feature information from the audio samples to determine the audio sample features of the target object; an identification module for identifying the video samples to determine the video sample features of the target object; and a training module for training a preset audio-video model using the audio sample features and the video sample features to obtain the audio noise reduction model.

[0145] In some embodiments, the audio noise reduction model includes at least: a feature matching module and an audio separation module; the noise reduction processing module 904 is further configured to, through the feature matching module, perform feature matching between the audio features of the target audio and the audio sample features of the target object, and determine the similarity between the audio features and the audio sample features; through the audio decomposition module, determine the audio segments in the target audio whose similarity reaches a preset similarity threshold as the effective audio, and determine the other audio segments in the target audio besides the effective audio as the noise.

[0146] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0147] It should be noted that, in the embodiments of this application, if the above-described video processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0148] Correspondingly, embodiments of this application provide a video processing device. Figure 10 This is a schematic diagram of the composition structure of the video processing device provided in the embodiments of this application, such as... Figure 10As shown, the video processing device 100 includes at least a processor 101 and a computer-readable storage medium 102 configured to store executable instructions, wherein the processor 101 generally controls the overall operation of the video processing device. The computer-readable storage medium 102 is configured to store instructions and applications executable by the processor 101, and may also cache data to be processed or processed by various modules in the processor 101 and the video processing device 100, and may be implemented using flash memory or random access memory (RAM).

[0149] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 2 The method shown.

[0150] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0151] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0152] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0153] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0154] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0156] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video processing method, characterized in that, include: Perform image recognition on the video to be processed to obtain the image recognition result of the target object in the video to be processed; The target object is the target person; The audio information of the video to be processed is extracted to obtain the audio to be processed; Based on the image recognition results, feature information is extracted from the audio to be processed to obtain audio features; Using the audio features, at least one of the noise and valid audio in the audio to be processed is denoised to obtain the denoised audio. The noise-reduced audio is then fused into the video to be processed to obtain the processed video; The video processing method further includes: Obtain the abnormal audio in the audio to be processed and the time point of the abnormal audio corresponding to the abnormal audio; Obtain the facial features of the target person at the abnormal audio time point; When the facial features of the abnormal point match any facial information in the preset facial information database, the abnormal audio is determined to be valid audio; wherein, the facial information database includes facial information of the target person in different vocal states; When the facial features of the abnormal point do not match any facial information in the facial information database, the abnormal audio is determined to be noise.

2. The method according to claim 1, characterized in that, The step of performing image recognition on the video to be processed to obtain the image recognition result of the target object in the video to be processed includes: Image recognition is performed on each video frame of the video to be processed to determine the object feature information of the target object in each video frame; Based on the object feature information, determine the audio processing information of the audio to be processed corresponding to the target object; The audio processing information is determined as the image recognition result of the target object.

3. The method according to claim 2, characterized in that, The object feature information is the facial change information of the target person; The step of determining the audio processing information of the audio to be processed corresponding to the target object based on the object feature information includes: Based on the facial change information, the start time of the speech, the end time of the speech, and the facial features of the target object are determined; The speech start time, speech end time, and facial features are determined as the audio processing information of the target object.

4. The method according to claim 3, characterized in that, The step of extracting feature information from the audio to be processed based on the image recognition result to obtain audio features includes: Based on the speech start time and speech end time of the target object, determine the audio start time and audio end time corresponding to the target object in the audio to be processed; The audio segment located between the start time and the end time of the audio in the audio to be processed is identified as the target audio; Feature information is extracted from the target audio to obtain the audio features of the target audio.

5. The method according to claim 4, characterized in that, The step of performing noise reduction processing on the audio to be processed using the audio features to obtain the noise-reduced audio includes: Using a pre-trained audio denoising model, the target audio is decomposed into effective audio and noise based on the audio features. The target audio is processed using at least one of the following methods: enhancing the effective audio and suppressing the noise to obtain the noise-reduced audio.

6. The method according to claim 5, characterized in that, The method further includes: Collect audio and video samples of the target object; Feature information is extracted from the audio samples to determine the audio sample features of the target object; The video samples are identified to determine the video sample features of the target object; The audio noise reduction model is obtained by training a preset audio and video model using the audio sample features and the video sample features.

7. The method according to claim 5, characterized in that, The audio noise reduction model includes at least: a feature matching module and an audio separation module; The step of using a pre-trained audio denoising model to decompose the target audio into effective audio and noise based on the audio features includes: The feature matching module performs feature matching between the audio features of the target audio and the audio sample features of the target object to determine the similarity between the audio features and the audio sample features. The audio decomposition module identifies audio segments in the target audio whose similarity reaches a preset similarity threshold as valid audio segments, and identifies other audio segments in the target audio besides the valid audio segments as noise.

8. A video processing apparatus, characterized in that, include: The recognition module is used to perform image recognition on the video to be processed, and obtain the image recognition result of the target object in the video to be processed. The target object is the target person; An audio information extraction module is used to extract audio information from the video to be processed, thereby obtaining the audio to be processed. The feature information extraction module is used to extract feature information from the audio to be processed based on the image recognition result, so as to obtain audio features; The noise reduction processing module is used to perform noise reduction processing on at least one of the noise and effective audio in the audio to be processed based on the audio features, so as to obtain the noise-reduced audio; The fusion module is used to fuse the noise-reduced audio into the video to be processed, thereby obtaining the processed video; The video processing device further includes: The first acquisition module is used to acquire the abnormal audio in the audio to be processed and the abnormal audio time point corresponding to the abnormal audio. The second acquisition module is used to acquire the abnormal facial features of the target person at the abnormal audio time point; The first matching module is used to determine that the abnormal audio is valid audio when the facial features of the abnormal point match any facial information in the preset facial information database; wherein, the facial information database includes facial information of the target person in different vocal states; The second matching module is used to determine that the abnormal audio is noise when the facial features of the abnormal point do not match any facial information in the facial information database.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the video processing method according to any one of claims 1 to 7.

Images