A voice detection method, device, equipment and readable storage medium

Abstract

The embodiment of the present application discloses a speech detection method, apparatus, device and readable storage medium, wherein the method comprises: obtaining an image data set, determining a first data frame corresponding to the starting speech in the audio data associated with the image data set based on the image data set, the image data set comprising image data corresponding to a second data frame, the second data frame being the first data frame or any data frame after the first data frame; obtaining image features of the image data corresponding to the second data frame, and audio features of the audio data corresponding to the second data frame; determining a speech detection result corresponding to the second data frame based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, the speech detection result being used to indicate the probability of speech existing in the second data frame. The embodiment of the present application can improve the accuracy of speech detection.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing, and in particular to a voice detection method, apparatus, device and readable storage medium. Background Art

[0002] As electronic devices become more intelligent, they provide more and more functions, such as live video and video recording. These functions require cameras to collect video data containing voice. In order to obtain clearer voice effects and apply them to corresponding business scenarios such as noise estimation, it is necessary to detect the probability of voice in each data frame in the video data, so as to estimate the noise of the video data.

[0003] The current speech detection method generally detects the speech in the entire video data and determines the probability of the presence of speech in each data frame. Due to the presence of noise in many scenarios, the accuracy of speech detection is low. Summary of the invention

[0004] The embodiments of the present application provide a speech detection method, apparatus, device and readable storage medium, which can improve the accuracy of speech detection.

[0005] In a first aspect, the present application provides a speech detection method, comprising:

[0006] Acquire an image data set, and determine, based on the image data set, a first data frame corresponding to an initial speech in audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame;

[0007] Acquire image features of the image data corresponding to the second data frame, and audio features of the audio data corresponding to the second data frame;

[0008] Based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, a speech detection result corresponding to the second data frame is determined, where the speech detection result is used to indicate the probability that speech exists in the second data frame.

[0009] In a second aspect, the present application provides a speech detection device, characterized in that the device comprises:

[0010] a start speech determination unit, configured to obtain an image data set, and determine, based on the image data set, a first data frame corresponding to the start speech in the audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame;

[0011] a feature acquisition unit, configured to acquire image features of the image data corresponding to the second data frame, and audio features of the audio data corresponding to the second data frame;

[0012] A speech detection unit is used to determine a speech detection result corresponding to the second data frame based on an image feature corresponding to the second data frame and an audio feature corresponding to the second data frame, wherein the speech detection result is used to indicate a probability that speech exists in the second data frame.

[0013] In conjunction with the second aspect, in a possible implementation manner, the image data set includes image data corresponding to a plurality of data frames; and the initial speech determination unit is specifically configured to:

[0014] respectively extracting image features of image data corresponding to each data frame in the multiple data frames;

[0015] Performing image detection on the image features of the image data corresponding to each data frame to obtain an image detection result corresponding to each data frame, wherein the image detection result corresponding to each data frame is used to indicate the probability that speech exists in each data frame;

[0016] At least one data frame to be selected in which the probability of the existence of speech is greater than a speech threshold is obtained, and a data frame including the earliest time in the at least one data frame to be selected is determined as a first data frame corresponding to the initial speech in the audio data.

[0017] In conjunction with the second aspect, in a possible implementation manner, the speech detection unit is specifically configured to:

[0018] Performing image detection on the image features corresponding to the second data frame to obtain an image detection result;

[0019] Performing audio detection on the audio feature corresponding to the second data frame to obtain an audio detection result;

[0020] Based on the image detection result and the audio detection result, a speech detection result corresponding to the second data frame is determined.

[0021] In conjunction with the second aspect, in a possible implementation manner, the speech detection unit is specifically configured to:

[0022] Identify a target scene associated with the image data set, and obtain a first weight corresponding to the image and a second weight corresponding to the audio in the target scene;

[0023] Based on the first weight, the image detection result, the second weight and the audio detection result, a speech detection result corresponding to the second data frame is determined.

[0024] In conjunction with the second aspect, in a possible implementation manner, the image detection result corresponding to the second data frame indicates that the probability that speech exists in the second data frame is less than or equal to a speech threshold; and the speech detection unit is specifically configured to:

[0025] Respectively obtain image detection results of the first N data frames of the second data frame and image detection results of the last M data frames of the second data frame, where M and N are both natural numbers;

[0026] If the image detection results of the first N data frames indicate that the probability of the presence of speech is greater than the speech threshold, and the image detection results of the last M data frames indicate that the probability of the presence of speech is greater than the speech threshold, then obtain the first speech probability mean between the image detection results corresponding to the second data frame, the image detection results of the first N data frames, and the image detection results of the last M data frames, and determine the first speech probability mean as the image detection result corresponding to the second data frame.

[0027] In conjunction with the second aspect, in a possible implementation manner, the speech detection device further includes a network training unit, configured to:

[0028] Get sample image sets and sample image labels;

[0029] Using the initial image detection network to extract features from each sample image in the sample image set, to obtain sample image features of each sample image;

[0030] Performing image detection on each sample image feature to obtain a sample image detection result of each sample image;

[0031] The initial image detection network is trained based on the sample image detection result of each sample image and the sample image label to obtain a target image detection network;

[0032] The target image detection network is used to perform image detection on the image features corresponding to the second data frame to obtain an image detection result.

[0033] In conjunction with the second aspect, in a possible implementation manner, the network training unit is further used to:

[0034] Get sample audio sets and sample audio labels;

[0035] Using the initial audio detection network to extract features from each sample audio in the sample audio set, to obtain sample audio features of each sample audio;

[0036] Performing audio detection on each sample audio feature to obtain a sample audio detection result of each sample audio;

[0037] The initial audio detection network is trained based on the sample audio detection result of each sample audio and the sample audio label to obtain a target audio detection network;

[0038] The target audio detection network is used to perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result.

[0039] In conjunction with the second aspect, in a possible implementation manner, the speech detection unit is specifically configured to:

[0040] Performing feature fusion on the image feature corresponding to the second data frame and the audio feature corresponding to the second data frame to obtain a fusion feature corresponding to the second data frame;

[0041] Perform speech detection on the fused feature to obtain a speech detection result corresponding to the second data frame.

[0042] In conjunction with the second aspect, in a possible implementation manner, the network training unit is further used to:

[0043] Acquire a sample feature set and a sample voice label set, wherein the sample feature set includes sample audio features and sample image features corresponding to a plurality of data frames, and the sample voice label set includes sample voice labels corresponding to the plurality of data frames;

[0044] Using the initial speech detection network to perform feature fusion on the sample image features and sample audio features corresponding to each data frame, to obtain the sample fusion features corresponding to each data frame;

[0045] Perform speech detection on the sample fusion features corresponding to each data frame to obtain the sample speech detection result corresponding to each data frame;

[0046] Based on the sample speech detection result corresponding to each data frame and the sample speech label set, the initial speech detection network is trained to obtain a target speech detection network;

[0047] The target speech detection network is used to perform speech detection on the fused feature to obtain a speech detection result corresponding to the second data frame.

[0048] In a third aspect, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0049] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program code, and the processor is used to call the computer program code so that a computer device including the processor executes the voice detection method.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor performs the above-mentioned speech detection method.

[0051] In a fifth aspect, the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the speech detection method provided in various optional ways in the first aspect of the present application.

[0052] In an embodiment of the present application, by acquiring image data corresponding to a plurality of data frames, the first data frame corresponding to the starting speech in the audio data associated with the image data can be determined based on the image data, so that the features of the image number and the features of the audio data of the first data frame corresponding to the starting speech and any subsequent data frame can be obtained, and the speech detection result corresponding to any data frame is determined in combination with the features of the image data and the features of the audio data, and the probability of the presence of speech in any data frame is determined, and then which data frames in the entire audio data have speech, which data frames do not have speech, and which data frames have speech that is noise. Since the image data is used to determine the data frame corresponding to the starting speech, there is no need to perform audio detection on the audio data before the data frame corresponding to the starting speech, which can reduce the amount of data processing. Furthermore, since the speech in the audio data is further determined in combination with the image data, the speech can be detected from two different dimensions, which can improve the accuracy of speech detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a network architecture diagram of a speech detection system provided in an embodiment of the present application;

[0055] Figure 2 This is a schematic diagram of an application scenario of a voice detection method provided in an embodiment of the present application;

[0056] Figure 3 It is a flow chart of a speech detection method provided in an embodiment of the present application;

[0057] Figure 4It is a flowchart of another voice detection method provided in an embodiment of the present application;

[0058] Figure 5 It is a schematic diagram of an application scenario of another voice detection method provided in an embodiment of the present application;

[0059] Figure 6 It is a schematic diagram of the structure of a speech detection device provided in an embodiment of the present application;

[0060] Figure 7 It is a schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The technical solution of the present application can be applied to the scenario of voice detection on audio data. By detecting the data frame corresponding to the starting voice in the audio data, audio detection can be performed on the data frame corresponding to the starting voice and the audio data corresponding to the subsequent data frames to determine the probability of the presence of voice in the audio data corresponding to each data frame; and the probability of the presence of voice in each data frame can be further determined in combination with the image data corresponding to each data frame, so as to determine whether there is voice in each data frame in combination with the image information and the audio information, and further determine the noise in the entire audio data, which can be subsequently applied to the scenario of noise estimation to improve the noise reduction effect. Since the voice is detected by combining image features and audio features, voice analysis can be performed from different dimensions to improve the accuracy of voice detection.

[0063] See also Figure 1 , Figure 1 is a network architecture diagram of a speech detection system provided in an embodiment of the present application, such as Figure 1 As shown, the computer device can exchange data with the terminal device, and the number of the terminal devices can be one or at least two. For example, when the number of the terminal devices is multiple, the terminal device may include Figure 1The terminal device 101a, the terminal device 101b and the terminal device 101c in the embodiment of the present invention are as follows. Taking the terminal device 101a as an example, the computer device 102 can obtain the image data corresponding to the multiple data frames, and determine the first data frame corresponding to the starting voice in the associated audio data based on the image data corresponding to the multiple data frames. The audio data associated with the image data can refer to the image data and audio data corresponding to the same data frame, such as the image data and audio data corresponding to the same data frame in the video data. Further, the computer device 102 can obtain the image features of the image data corresponding to the second data frame, and the audio features of the audio data corresponding to the second data frame; based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, determine the voice detection result corresponding to the second data frame, thereby determining the voice detection result for the entire audio data. Optionally, the computer device 102 can also send the voice detection result for the entire audio data to the terminal device 101a, so that the terminal device 101a determines which data frames in the audio data have voice, which can be used for noise estimation later to improve the noise reduction effect, and so on.

[0064] Among them, the service end mentioned in the embodiment of the present application may refer to a server or a terminal device, or it may also be a system composed of a server and a terminal device, and the embodiment of the present application does not limit this. The terminal device may be an electronic device, including but not limited to a mobile phone, a tablet computer, a laptop computer, a vehicle-mounted device, etc. The server mentioned above may be an independent physical server, or it may be a server cluster or a distributed system composed of multiple physical servers, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. When the server is an independent physical server, the server can perform voice detection independently. When the server is multiple physical servers, multiple physical servers can collaborate to perform voice detection and other processing. For example, one of the servers can determine the first data frame corresponding to the starting speech in the audio data associated with the image data set based on the image data set; another server can obtain the image features of the image data corresponding to the second data frame, and the audio features of the audio data corresponding to the second data frame; yet another server can determine the speech detection result corresponding to the second data frame based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, and so on.

[0065] For further information, see Figure 2 , Figure 2 Schematic diagram of an application scenario of a speech detection method provided in an embodiment of the present application; Figure 2As shown, the computer device can obtain the image data 21 in the video data 20, and the image data 21 can refer to any image data in the image data set. Further, the computer device can determine the first data frame corresponding to the starting voice of the audio data 22 in the video data 20 based on the image data 21. For example, if the first data frame is the 5th data frame in the video data, the computer device can obtain the image feature 23 of the image data corresponding to the 5th data frame or any data frame after the 5th data frame, and the audio feature 24 of the audio data corresponding to the data frame. Further, the image feature 23 of the image data corresponding to the data frame can be image detected based on the target image detection network 25 to obtain the image detection result 26, for example, the image detection result 26 can indicate that there is no voice in the data frame. The audio feature 24 of the audio data corresponding to the data frame is audio detected based on the target audio detection network 27 to obtain the audio detection result 28, for example, the audio detection result 28 can indicate that there is voice in the data frame. The results can be fused based on the image detection result 26 and the audio detection result 28 to determine the speech detection result 29 corresponding to the data frame. For example, if the speech detection result 29 indicates that there is no speech in the data frame, it can be determined that the speech in the audio data of the current data frame is noise, and the speech detection result can be further combined for specific applications such as noise estimation.

[0066] For further information, see Figure 3 , Figure 3 is a flow chart of a speech detection method provided in an embodiment of the present application; Figure 3 As shown, the voice detection method can be applied to a computer device, and the voice detection method includes but is not limited to the following steps:

[0067] S101, acquiring an image data set, and determining, based on the image data set, a first data frame corresponding to an initial speech in audio data associated with the image data set.

[0068] The technical solution of the present application can be applied to live broadcast scenarios, video conferencing scenarios, intelligent dialogue scenarios, online teaching scenarios, video on demand scenarios or other scenarios that require voice detection, and the embodiments of the present application do not limit this. In these scenarios, by acquiring the image data and audio data in these scenarios, the first data frame corresponding to the starting voice in the audio data can be determined based on the image data, so that the judgment can be started directly from the first data frame to determine the probability of the presence of voice in each data frame in the audio data, thereby determining which data frames have voice, or determining which data frames in the audio data have voice as noise, so as to facilitate subsequent corresponding business processing for different business scenarios.

[0069] The traditional voice detection method is to perform audio detection on the audio data corresponding to multiple continuous data frames, and the audio data of each data frame must be detected. Since the probability of the existence of voice in the audio data of the starting time period in the audio data is small, for example, the user may be adjusting the relevant equipment in the first few seconds of the audio data, so the probability of the existence of voice is small, and the amount of calculation will be increased by performing audio detection on the audio data of the starting time period. Furthermore, in actual application scenarios, such as live broadcast scenarios, when the target user is broadcasting in a noisy environment, there may be a variety of noises, such as noise in the environment, the walking sound of other users around, the voice of other users around, and so on. Traditional voice detection can only identify whether there is voice in the audio data, but cannot identify whether the voice of each data frame in the audio data is the voice of the target user (i.e., the user in the image), which will result in low accuracy of voice detection and poor effect of noise estimation. The technical solution of the present application can further determine whether the voice in the audio data corresponding to the current data frame is the voice of the target user by combining the image data for voice detection. This method can realize voice detection from different dimensions, judge whether the current voice is the voice of the target user, and improve the accuracy of voice detection and user experience.

[0070] In the embodiment of the present application, an image data set may refer to image data corresponding to a plurality of data frames, and the image data corresponding to a data frame may refer to image data included in the data frame, and one data frame may include one or more image data. The computer device may obtain video data and process the video data to obtain an image data set; or the computer device may obtain image stream data as an image data set. When obtaining video data, the computer device may obtain video data from a local database, or obtain video data from any terminal device, or obtain video data from an associated video shooting device. Further, when obtaining video data, the computer device may use image editing software to process the video data, separate the audio data and image data in the video data, and obtain an image data set and audio data. The image data and audio data in the image data set have a time correlation, that is, each data frame corresponds to image data and audio data. The audio data here may also refer to audio data corresponding to a plurality of data frames, and the audio data corresponding to a data frame may refer to audio data included in a data frame. A data frame may include audio data within a period of time, and a period of time may refer to 10 milliseconds, 30 milliseconds, 50 milliseconds, etc., which is not limited in the embodiment of the present application. Alternatively, the image portion of the video data containing the user may be intercepted to obtain an image data set. The image data may include, but is not limited to, the user's facial image data, mouth image data, and body language image data, etc. When acquiring image stream data, the computer device may acquire the image stream data from a local database, or acquire the image stream data from any terminal device, or acquire the image stream data from an associated image capture device. When the computer device acquires the image stream data from an associated image capture device, the image capture device may be used to collect the image stream data in real time, which is not limited in the embodiments of the present application.

[0071] Optionally, when acquiring audio data, the computer device may use an audio recording device to record the audio data, or acquire the audio data from the video data. It is understandable that the image data and the audio data have a time correlation, that is, each data frame includes image data and audio data, then the image data in a data frame and the audio data in the data frame have a time correlation. In one implementation, at least one data frame may also be directly acquired, and the data frame corresponding to the starting voice in the audio data corresponding to the at least one data frame may be determined based on the image data corresponding to the at least one data frame. Since the data frame includes image data and audio data, image detection may be performed on the image data in each data frame to determine the data frame where the starting voice exists. The data frame may be obtained by performing frame processing on the video data or by other means, and the embodiments of the present application are not limited to this.

[0072] In one embodiment, a method for determining a first data frame corresponding to an initial speech in audio data associated with an image data set based on an image data set may include: extracting image features of image data corresponding to each data frame in a plurality of data frames respectively; performing image detection on the image features of image data corresponding to each data frame respectively to obtain an image detection result corresponding to each data frame, wherein the image detection result corresponding to each data frame is used to indicate a probability of the presence of speech in each data frame; obtaining at least one candidate data frame in which the probability of the presence of speech is greater than a speech threshold, and determining a data frame including the earliest time in the at least one candidate data frame as the first data frame corresponding to the initial speech in the audio data.

[0073] Among them, the image features can reflect the image information in the image, for example, the image features can include the facial expression features, mouth movement features, body movement features, etc. of the user in the image, and it can be determined whether the user in the image is speaking based on the image features. The image data may include one or more users. When the image data includes one user, the user can be determined as the target user. When the image data includes multiple users, the multiple users in the image can be determined as target users. By performing image detection on the image data, image detection can be performed on each user included in the image data to determine whether the user in the image data is speaking. If each user in the image data does not speak, but the audio detection result corresponding to the image data indicates that there is voice, it can be indicated that it is the voice of the user outside the image, or the noise in the environment. In this way, the sound other than the user in the image can be excluded.

[0074] Optionally, the data frame may further include time information, and the time information in each data frame is different. When it is determined that speech exists in multiple data frames, the data frame including the earliest time may be determined as the first data frame corresponding to the starting speech.

[0075] In one implementation, image features of image data corresponding to each data frame in a plurality of data frames can be extracted based on a target image detection network; and image detection is performed on the image features of image data corresponding to each data frame to obtain the probability of speech in each data frame. The image detection result may refer to a voice activity detection (VAD) result, and the VAD detection result may reflect the probability of speech in each data frame. The target image detection network may include but is not limited to a convolutional neural network (CNN), a deep neural network (DNN), a super-resolution test sequence (Visual Geometry Group, VGG), a recurrent neural network (RNN), a residual network (ResNet), and the like.

[0076] Specifically, the image data corresponding to each data frame can be input into the target image detection network respectively, and the convolution layer in the target image detection network can perform feature extraction on the image data corresponding to each data frame, and extract the image features of the image data corresponding to each data frame. Further, image detection can be performed on the image features of the image data corresponding to each data frame based on the target image detection network to obtain an image detection result indicating the probability of the existence of speech in each data frame, thereby determining whether there is speech in each data frame. If the probability of the existence of speech in any data frame is greater than the speech threshold, it can be indicated that there is speech in the data frame. Since the image detection is performed on the image data corresponding to multiple data frames to obtain the speech detection probabilities corresponding to multiple data frames, there may be multiple data frames whose speech detection probabilities are all greater than the speech threshold, then all data frames whose probabilities of the existence of speech are greater than the speech threshold are obtained as the data frames to be selected, and the earliest data frame among the multiple data frames to be selected is determined as the first data frame corresponding to the starting speech in the audio data.

[0077] Optionally, the image data set may include image data corresponding to the first T data frames, where T is a positive integer. Since generally speaking, there may be speech in the first T data frames in a section of audio data, the starting speech can be determined by obtaining the image data corresponding to the first T data frames for judgment, avoiding directly judging the image data in all data frames, and reducing the number of image processing and the amount of calculation. If the image detection result of the image data corresponding to the first T data frames indicates that the first T data frames are not data frames corresponding to the starting speech, then the image data corresponding to T+1 data frames are obtained for image detection to determine the data frame corresponding to the starting speech. If T+1 data frames are still not data frames corresponding to the starting speech, then the image data corresponding to T+2 data frames are obtained for image detection until the data frame corresponding to the starting speech is determined. By detecting the image data corresponding to each data frame in turn, it is possible to avoid directly performing image detection on a large amount of image data, reduce the amount of image detection, and then reduce the amount of calculation, thereby improving the efficiency of speech detection.

[0078] For example, if the video data includes 100 data frames, the image detection result corresponding to the third data frame indicates that the probability of the presence of speech is 1, and the probability of the presence of speech in the second data frame is 0, and the probability of the presence of speech in the fourth data frame and the fifth data frame is 1, then the third data frame can be determined as the first data frame corresponding to the initial speech, and the third data frame and the image data and audio data after the third data frame can be further judged to determine the probability of the presence of speech in each data frame, thereby determining which data frames have speech and which data frames do not have speech, without directly performing image detection on the 100 data frames, thereby reducing the amount of calculation. Optionally, when determining the data frame corresponding to the initial speech, the audio data before the data frame does not need to be detected, which can reduce the amount of calculation.

[0079] In one embodiment, the image data and audio data in each data frame can also be detected, so as to determine the voice detection result of each data frame based on the image detection result of the image data and the audio detection result of the audio data in each data frame, that is, the probability of the existence of voice in each data frame. In this embodiment, the computer device can obtain at least one data frame, each data frame corresponding to image data and audio data; obtain the image features of the image data corresponding to each data frame, and the audio features of the audio data corresponding to each data frame; based on the image features corresponding to each data frame and the audio features corresponding to each data frame, determine the voice detection result corresponding to each data frame, and the voice detection result is used to indicate the probability of the existence of voice in each data frame. Since each data frame is detected in combination with the image data and the audio data, the accuracy of voice detection can be improved.

[0080] S102, acquiring image features of image data corresponding to the second data frame, and audio features of audio data corresponding to the second data frame.

[0081] In an embodiment of the present application, by obtaining image features of image data corresponding to the second data frame, and audio features of audio data corresponding to the second data frame, it is possible to combine the image features and audio features corresponding to the second data frame to determine whether there is speech in the second data frame. Since this method not only judges the audio features but also combines the image features for judgment, the accuracy of speech detection can be improved.

[0082] Optionally, before feature extraction is performed on the image data corresponding to the second data frame, the image data corresponding to the second data frame can also be downsampled to convert the image data into image data of a target format, so that the image data input to the target image detection network is more in line with the network input requirements. Wherein, the target format can refer to the input format of the target image detection network. For example, the input image of the image detection network is required to be 30*30, and because the resolution of different devices is different, the size of the acquired image data is different, then the acquired image data can be converted into the input format of the target image detection network, such as 30*30, so that the target image detection network is used to process the image data in the target format, such as detecting the probability of the presence of speech in the second data frame corresponding to the image data, and so on. By downsampling the image, the image information can be compressed, the important information in the image can be retained, and the network learning efficiency can be improved when the network learning is performed later. Optionally, the target image detection network can be used to extract features of the image data corresponding to the second data frame to obtain the image features corresponding to the second data frame.

[0083] In one possible scenario, if it is detected that the image data corresponding to the second data frame does not include the target user, for example, there is no person in the image data, then the image features of the image data corresponding to the second data frame cannot be extracted, and when image detection is subsequently performed based on the image features corresponding to the second data frame, the image detection result may be empty or no face is detected, and the audio detection result may be determined as a speech detection result. When obtaining the image features of the image data corresponding to the second data frame, feature extraction may be performed on the image data corresponding to the second data frame based on the target image detection network to obtain the image features of the image data corresponding to the second data frame.

[0084] Furthermore, when obtaining the audio features of the audio data corresponding to the second data frame, the audio data can be converted into the frequency domain or the Mel spectrum domain, and the converted frequency domain or Mel spectrum domain is determined as the audio feature. Among them, the audio feature can reflect the user's voice information, for example, it can reflect whether the user is speaking, and the audio feature may include but is not limited to the frequency domain amplitude and phase. Optionally, when the audio data is obtained, for example, the audio data is an audio stream, the audio stream can be framed to obtain an audio frame, thereby performing a time domain conversion on each audio frame, converting the time domain audio frame signal into a frequency domain audio frame signal, and then extracting the frequency domain features of the frequency domain audio frame signal as the audio feature. Since audio data is generally time domain information, by converting the time domain information into the frequency domain or the Mel spectrum domain, it can facilitate the feature learning of the subsequent target audio detection network.

[0085] S103: Determine a speech detection result corresponding to the second data frame based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame.

[0086] In an embodiment of the present application, since the image features corresponding to the second data frame and the audio features corresponding to the second data frame are obtained, the speech detection results corresponding to the second data frame can be determined in combination with the image features and the audio features of the second data frame. For example, the image detection results can be determined based on the image features, the audio detection results can be determined based on the audio features, and the speech detection results can be determined based on one or both of the image detection results and the audio detection results. Alternatively, the image features and the audio features are fused, and the speech detection results are determined based on the fused features. Alternatively, in some scenarios, the audio detection results are determined as the speech detection results, and in other scenarios, the image detection results are determined as the speech detection results, and so on. The audio detection result may refer to the VAD result, and the speech detection result may also refer to the VAD result of the voice endpoint detection.

[0087] It can be understood that the embodiment of the present application processes the image data and audio data of the first data frame and any data frame after the first data frame. For the image data and audio data of the first data frame and other data frames after the first data frame, the processing method of the image data and audio data of the data frame can be referred to to obtain the speech detection results corresponding to other data frames, and then determine the audio detection results of the entire audio data. The embodiment of the present application does not describe this in detail.

[0088] In one embodiment, a method for determining a speech detection result corresponding to a second data frame based on image features corresponding to the second data frame and audio features corresponding to the second data frame may include: performing image detection on the image features corresponding to the second data frame to obtain an image detection result; performing audio detection on the audio features corresponding to the second data frame to obtain an audio detection result; and determining a speech detection result corresponding to the second data frame based on the image detection result and the audio detection result.

[0089] Among them, the image detection result is used to indicate the probability that the second data frame contains the target user's voice, the audio detection result is used to indicate the probability that the second data frame contains voice, and the voice detection result is used to indicate the probability that the second data frame contains the target user's voice. It can be seen that the audio detection result indicates whether a data frame contains voice, and the image detection result indicates whether the data frame contains the voice of the target user (such as any one or more users in the image). Therefore, the combination of the two can determine whether the voice in the data frame is the voice of the user in the image, and the probability of the presence of voice.

[0090] In one implementation, the target image detection network can be used to perform image detection on the image features corresponding to the second data frame to obtain image detection results. In an optional implementation, before using the target image detection network for image detection, the initial image detection network can be pre-trained to obtain the target image detection network, so that the trained target image detection network has the ability to detect image data and determine image detection results. Specifically, the computer device can obtain a sample image set and a sample image label; use the initial image detection network to perform feature extraction on each sample image in the sample image set to obtain sample image features of each sample image; perform image detection on each sample image feature to obtain a sample image detection result of each sample image; train the initial image detection network based on the sample image detection result and sample image label of each sample image to obtain a target image detection network.

[0091] Among them, the sample image set includes a large number of sample image data, and the sample image label can refer to the sample true value. For example, the sample image label can be obtained by manually annotating the sample image data, and the sample image label can be used to indicate whether the sample image data contains speech. One sample image data corresponds to one sample image label. The sample image detection result obtained by the target image detection network detection can refer to the network output value, and the target image detection network can be trained based on the sample true value and the network output value so that the network output value is as close to the sample true value as possible. The purpose of training the initial image detection network is to make the sample true value and the network output value as consistent as possible. When the first loss function of the initial image detection network is greater than the first loss threshold, it can be indicated that the sample true value and the network output value are inconsistent, that is, the accuracy of the initial image detection network is low, and the network parameters in the initial image detection network can be further adjusted to reduce the first loss function in the initial image detection network. When the first loss function of the initial image detection network is less than or equal to the first loss threshold, it can be indicated that the sample true value and the network output value are consistent, and it can be considered that the initial image detection network training meets the requirements, and the initial image detection network at this time can be saved as the target image detection network. Therefore, the target image detection network can be used to perform image detection on the image features corresponding to the second data frame to obtain the image detection result.

[0092] Since a large amount of sample image data is used to train the initial image detection network, the initial image detection network can learn more detailed information in the sample image data, such as mouth movement change information, facial expression change information, body language change information, etc., so that the image detection results corresponding to the sample image data predicted based on these detailed information can more accurately reflect the speech situation of the sample image data. Since a large amount of sample image data is used to train the initial image detection network, the accuracy of the network can be improved. When the target image detection network is used to perform image detection on the image features of the image data corresponding to each data frame, the accuracy of image detection can be improved.

[0093] Further, the target audio detection network can be used to perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result. In an optional implementation, before using the target audio detection network for audio detection, the initial audio detection network can be pre-trained to obtain the target audio detection network, so that the trained target audio detection network has the ability to detect audio data and determine the audio detection result. Specifically, the computer device can obtain a sample audio set and a sample audio label; use the initial audio detection network to extract features of each sample audio in the sample audio set respectively to obtain a sample audio feature of each sample audio; perform audio detection on each sample audio feature to obtain a sample audio detection result of each sample audio; train the initial audio detection network based on the sample audio detection result and sample audio label of each sample audio to obtain a target audio detection network. Optionally, the target audio detection network can include but is not limited to a convolutional neural network (CNN), a deep neural network (DNN), a super-resolution test sequence (Visual Geometry Group, VGG), a recurrent neural network (RNN), a residual network (ResNet), and the like.

[0094] Among them, the sample audio set includes a large number of sample audio data, and the sample audio label can refer to the sample true value. For example, the sample audio label can be obtained by manually annotating the sample audio data, and the sample audio label can be used to indicate whether the sample audio data has speech. One sample audio data corresponds to a sample audio label. The sample audio detection result obtained by the target audio detection network detection can refer to the network output value, and the target audio detection network can be trained based on the sample true value and the network output value so that the network output value is as close to the sample true value as possible. The purpose of training the initial audio detection network is to make the sample true value and the network output value as consistent as possible. When the second loss function of the initial audio detection network is greater than the second loss threshold, it can be indicated that the sample true value and the network output value are inconsistent, that is, the accuracy of the initial audio detection network is low, and the network parameters in the initial audio detection network can be further adjusted to reduce the second loss function in the initial audio detection network. When the second loss function of the initial audio detection network is less than or equal to the second loss threshold, it can be indicated that the sample true value and the network output value are consistent, and it can be considered that the initial audio detection network training meets the requirements, and the initial audio detection network at this time can be saved as the target audio detection network. Therefore, the target audio detection network can be used to perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result.

[0095] Since a large amount of sample audio data is used to train the initial audio detection network, the initial audio detection network can learn more detailed information in the sample audio data, such as mouth movement change information, facial expression change information, body language change information, etc., so that the audio detection results corresponding to the sample audio data predicted based on these detailed information can more accurately reflect the speech situation of the sample audio data. Since a large amount of sample audio data is used to train the initial audio detection network, the accuracy of the network can be improved. When the target audio detection network is used to perform audio detection on the audio features of the audio data corresponding to each data frame, the accuracy of the audio detection can be improved.

[0096] In one implementation, if the image detection result corresponding to the second data frame indicates that the probability of the presence of speech in the second data frame is less than or equal to the speech threshold, since noise data may exist in the image data, the image detection result can be smoothed to reduce the noise in the image. Specifically, the image detection results of the first N data frames of the second data frame and the image detection results of the last M data frames of the second data frame can be obtained respectively, where M and N are both natural numbers; if the image detection results of the first N data frames indicate that the probability of the presence of speech is greater than the speech threshold, and the image detection results of the last M data frames indicate that the probability of the presence of speech is greater than the speech threshold, then the first speech probability mean value among the image detection results corresponding to the second data frame, the image detection results of the first N data frames, and the image detection results of the last M data frames is obtained, and the first speech probability mean value is determined as the image detection result corresponding to the second data frame.

[0097] Among them, the first N data frames of the second data frame, the second data frame, and the last M data frames of the second data frame may refer to several adjacent data frames. Generally speaking, a person's expression will not change too much in a short period of time, that is, the image detection results included in multiple data frames in a short period of time are similar. If the image detection result corresponding to a data frame is inconsistent with the image detection results corresponding to other data frames in a short period of time, it is likely that the image detection result of the image in the data frame is abnormal, such as noise. Then, the image detection result of the image data in the data frame can be adjusted based on the image detection results of the multiple continuous data, thereby reducing the noise in the image.

[0098] Since in some scenarios, such as video-on-demand scenarios, multiple data frames corresponding to a video data can be obtained, when performing image detection on any data frame in the video data in the scenario, the image detection can be combined with the adjacent data frames before and after the data frame to improve the accuracy of image detection. At this time, M can be a positive integer greater than or equal to 1, and the data frames before and after any data frame can be smoothed during smoothing. In other scenarios, such as live broadcast scenarios or other scenarios with strong real-time data, when performing image detection on the current data frame, since the data frame after the current data frame has not yet been obtained (there is a delay in obtaining the data frame after the current data frame), when performing image detection on the current data frame, only the historical data frame can be combined, that is, one or more data frames before the current data frame can be combined for image detection to determine the voice detection result for the current frame. At this time, M can be 0, and one or more data frames before the current data frame can be smoothed during smoothing.

[0099] For example, in a video-on-demand scenario, since multiple continuous data frames corresponding to the video data can be obtained, for example, M is equal to 2, N is equal to 2, the image detection results of the first N data frames of the second data frame are all 1 for the probability of the presence of speech, the image detection result of the second data frame is 0 for the probability of the presence of speech, and the image detection results of the last M data frames of the second data frame are 1 for the probability of the presence of speech, then the first speech probability mean can be calculated to be (1+1+0+1+1) / 5=0.8, then 0.8 can be used as the image detection result of the second data frame, and the image detection results of the first N data frames of the second data frame, the second data frame, and the last M data frames of the second data frame are 1, 1, 0.8, 1, 1, respectively. For example, the speech threshold is 0.6, because the probability of the second data frame indicating the presence of speech before adjustment is 0, which is less than the speech threshold, indicating that the second data frame does not have speech, and the probability of the second data frame indicating the presence of speech after adjustment is 0.8, which is greater than the speech threshold, indicating that the second data frame has speech. By combining the image detection results of the previous and next data frames to smooth the image detection result of the second data frame and adjusting the image detection result of the second data frame, noise in the image can be reduced and the accuracy of speech detection can be improved.

[0100] In one implementation, if the audio detection result corresponding to the second data frame indicates that the probability of the presence of speech in the second data frame is less than or equal to the speech threshold; since noise data may exist in the audio data, the audio detection result can be smoothed to reduce noise. Specifically, the audio detection results of the first N data frames of the second data frame and the audio detection results of the last M data frames of the second data frame can be obtained respectively; if the audio detection results of the first N data frames indicate that the probability of the presence of speech is greater than the speech threshold, and the audio detection results of the last M data frames indicate that the probability of the presence of speech is greater than the speech threshold, then the second speech probability mean between the audio detection result corresponding to the second data frame, the audio detection results of the first N data frames, and the audio detection results of the last M data frames is obtained, and the second speech probability mean is determined as the audio detection result corresponding to the second data frame.

[0101] Generally speaking, human speech will not stop suddenly within a short period of time, that is, the audio detection results corresponding to multiple data frames within a short period of time are similar. If the audio detection result of a data frame is inconsistent with the audio detection results of other data frames within a short period of time, it is likely that the audio detection result of the data frame is abnormal, such as noise. In this case, the audio detection result of the audio data in the data frame can be adjusted based on the audio detection results of the multiple consecutive data frames, thereby reducing the noise in the audio.

[0102] Correspondingly, in some scenarios, such as video on demand scenarios, multiple data frames corresponding to a video data can be obtained. Therefore, when performing audio detection on any data frame in the video data in the scenario, the audio detection can be combined with the adjacent data frames before and after the data frame to improve the accuracy of audio detection. At this time, M can be a positive integer greater than or equal to 1, and the data frames before and after any data frame can be smoothed during smoothing. In other scenarios, such as live broadcast scenarios or other scenarios where data has strong real-time characteristics, when performing audio detection on the current data frame, since the data frame after the current data frame has not yet been obtained (there is a delay in obtaining the data frame after the current data frame), when performing audio detection on the current data frame, only historical data frames can be combined, that is, one or more data frames before the current data frame can be combined for audio detection to determine the voice detection result for the current frame. At this time, M can be 0, and one or more data frames before the current data frame can be smoothed during smoothing.

[0103] For example, in a video-on-demand scenario, since multiple continuous data frames corresponding to the video data can be obtained, for example, M is equal to 2, N is equal to 2, for example, the audio detection results of the first N data frames of the second data frame are all 1 with a probability of audio, the audio detection result of the second data frame is 0 with a probability of audio, and the audio detection results of the last M data frames of the second data frame are 1 with a probability of speech, then the second speech probability mean can be calculated to be (1+1+0+1+1) / 5=0.8, then 0.8 can be used as the audio detection result of the second data frame, and the audio detection results of the first N data frames of the second data frame, the second data frame, and the last M data frames of the second data frame are 1, 1, 0.8, 1, 1, respectively. For example, the speech threshold is 0.6, because the probability of the second data frame indicating the presence of speech before adjustment is 0, which is less than the speech threshold, indicating that the second data frame does not have speech, and the probability of the second data frame indicating the presence of speech after adjustment is 0.8, which is greater than the speech threshold, indicating that the second data frame has speech. By combining the audio detection results of the previous and next data frames to smooth the audio detection result of the second data frame and adjusting the audio detection result of the second data frame, noise in the audio can be reduced and the accuracy of speech detection can be improved.

[0104] In a possible implementation, the weight corresponding to the image and the weight corresponding to the audio may be combined to determine the speech detection result. Specifically, the target scene associated with the image data set may be identified, and the first weight corresponding to the image and the second weight corresponding to the audio in the target scene may be obtained respectively; based on the first weight, the image detection result, the second weight and the audio detection result, the speech detection result corresponding to the second data frame may be determined.

[0105] In the embodiment of the present application, since different aspects are concerned in different scenes, for example, in the first type of scene, that is, a live broadcast scene with relatively quiet surroundings such as an indoor live broadcast scene, due to the presence of less noise, the attention to audio is higher than that to the image, and the first weight corresponding to the image is less than the second weight corresponding to the audio. Or in the second type of scene, that is, a live broadcast scene with more noise around, such as an outdoor live broadcast scene, the first weight corresponding to the image is greater than the second weight corresponding to the audio. Due to the presence of more noise, the attention to the image is higher than the audio, that is, more judgment is needed to determine whether the target user is currently speaking, so as to determine the voice detection result for the target user. Alternatively, in the third type of scene, if the pixel of the image data is lower than the pixel threshold, it may be difficult to distinguish whether the target user is currently speaking, then the first weight corresponding to the image is less than the second weight corresponding to the audio. Alternatively, if in the fourth type of scene, if there are multiple users or multiple users' voices in the scene, the first weight corresponding to the image is greater than the second weight corresponding to the audio. Alternatively, in the above-mentioned scene, the first weight corresponding to the image can also be set to be equal to the second weight corresponding to the audio. In specific applications, it can be set according to specific needs, and the embodiment of the present application does not limit this.

[0106] Among them, when identifying the target scene associated with the image data set, the target scene can be determined based on the type of the image data set. For example, when the image data is image data in video data, the image content in the video data can be judged, and the image content is matched with the image content in multiple preset scenes to determine which scene type the image in the video data belongs to. Alternatively, the video type tag corresponding to the video data can be obtained, and the target scene corresponding to the video type tag can be found from the correspondence table based on the video type tag. The correspondence table can include the correspondence between multiple video type tags and scenes. Alternatively, the scene specified by the terminal device can be obtained as the target scene. The embodiment of the present application does not limit the method for identifying the target scene.

[0107] For example, if the first weight corresponding to the image in the target scene is a1, the second weight corresponding to the audio is a2, the image detection result corresponding to the second data frame is b1, and the audio detection result corresponding to the second data frame is b2, then the speech detection result corresponding to the second data frame (that is, the probability that there is speech in the second data frame) is a1*b1+a2*b2. Then, it can be determined whether there is speech in the second data frame based on the relationship between a1*b1+a2*b2 and the speech threshold.

[0108] It can be understood that, since for any data frame, the image detection result of the image data corresponding to the data frame, the audio detection result of the audio data and the final speech detection result can be determined, if the image detection result of the image data corresponding to the data frame indicates that there is no speech in the data frame, and the audio detection result of the audio data corresponding to the data frame indicates that there is speech in the data frame, it can be said that there is noise in the data frame, and the final speech detection result of the data frame may be that there is no speech and there is noise.

[0109] In an embodiment of the present application, by acquiring image data corresponding to a plurality of data frames, the first data frame corresponding to the starting speech in the audio data associated with the image data can be determined based on the image data, so that the features of the image number and the features of the audio data of the first data frame corresponding to the starting speech and any subsequent data frame can be obtained, and the speech detection result corresponding to any data frame is determined in combination with the features of the image data and the features of the audio data, and the probability of the presence of speech in any data frame is determined, and then which data frames in the entire audio data have speech, which data frames do not have speech, and which data frames have speech that is noise. Since the image data is used to determine the data frame corresponding to the starting speech, there is no need to perform audio detection on the audio data before the data frame corresponding to the starting speech, which can reduce the amount of data processing. Furthermore, since the speech in the audio data is further determined in combination with the image data, the speech can be detected from two different dimensions, which can improve the accuracy of speech detection.

[0110] For further information, see Figure 4 , Figure 4 is a flow chart of another speech detection method provided in an embodiment of the present application; Figure 4 As shown, the voice detection method can be applied to a computer device, and the voice detection method includes but is not limited to the following steps:

[0111] S201, acquiring an image data set, and determining, based on the image data set, a first data frame corresponding to an initial speech in audio data associated with the image data set.

[0112] S202, obtaining image features of image data corresponding to the second data frame, and audio features of audio data corresponding to the second data frame.

[0113] In the embodiment of the present application, the specific implementation method of each step in step S201 to step S202 can refer to the implementation method in the above-mentioned step S101 to step S102, which will not be repeated here.

[0114] S203, performing feature fusion on the image features corresponding to the second data frame and the audio features corresponding to the second data frame to obtain fused features corresponding to the second data frame.

[0115] In an embodiment of the present application, the method of feature fusion may include but is not limited to feature splicing. Performing feature fusion on the image features corresponding to the second data frame and the audio features corresponding to the second data frame to obtain the fused features corresponding to the second data frame may include performing feature splicing on the image features corresponding to the second data frame and the audio features corresponding to the second data frame to obtain the fused features corresponding to the second data frame; it may also include performing feature fusion on the image features corresponding to the second data frame, the image features corresponding to the first P data frames of the second data frame, the audio features corresponding to the second data frame, and the audio features corresponding to the first P data frames of the second data frame to obtain the fused features corresponding to the second data frame; it may also include performing feature fusion on the image features corresponding to the second data frame, the image features corresponding to the first P data frames of the second data frame, the image features corresponding to the last Q data frames of the second data frame, the audio features corresponding to the second data frame, the audio features corresponding to the first P data frames of the second data frame, and the audio features corresponding to the last Q data frames of the second data frame to obtain the fused features corresponding to the second data frame.

[0116] Wherein, P and Q are both natural numbers, and the first P data frames of the second data frame and the last Q data frames of the second data frame may refer to data frames adjacent to the second data frame. Since a person's expression will not change much in a short period of time, and the sound will not stop suddenly, the image features and audio features of the plurality of adjacent data frames before and after the second data frame may be combined to perform feature fusion to obtain a fused feature, which may reflect the correlation between the image features of adjacent images and the correlation between the audio features of adjacent audios, so that the fused feature may more finely reflect the speech situation of the second data frame.

[0117] It is understandable that, in scenarios such as video on demand or other scenarios where the entire video data can be obtained, when the image features and audio features corresponding to any data frame in the scenario are fused, the image features and audio features corresponding to the previous and next data frames of the data frame can be fused, that is, the obtained fusion features include the image features and audio features corresponding to any data frame, and the image features and audio features corresponding to the previous and next data frames of the data frame. At this time, Q can be a positive integer greater than or equal to 1. In scenarios such as live broadcasts or other scenarios where data has a strong real-time nature, since the data frames after the current data frame have not yet been obtained, when the image features and audio features corresponding to the current data frame are fused, only the historical data frames can be combined, that is, the image data and audio data in one or more data frames before the current data frame can be combined to perform feature fusion to determine the speech detection result for the current frame, and then Q can be 0 at this time.

[0118] S204, performing speech detection on the fused features to obtain a speech detection result corresponding to the second data frame.

[0119] In an embodiment of the present application, a target speech detection network can be used to perform speech detection on the fusion feature to obtain a speech detection result corresponding to the second data frame. The target speech detection network can include but is not limited to a convolutional neural network (CNN), a deep neural network (DNN), a super-resolution test sequence (Visual Geometry Group, VGG), a recurrent neural network (RNN), a residual network (ResNet), and the like.

[0120] Before using the target speech detection network for speech detection, the initial speech detection network can be pre-trained to obtain the target speech detection network, so that the trained target speech detection network has the ability to detect the fused features and determine the speech detection results. Specifically, the computer device can obtain a sample feature set and a sample speech label set, the sample feature set includes sample audio features and sample image features corresponding to multiple data frames, and the sample speech label set includes sample speech labels corresponding to multiple data frames; use the initial speech detection network to perform feature fusion on the sample image features and sample audio features corresponding to each data frame to obtain the sample fusion features corresponding to each data frame; perform speech detection on the sample fusion features corresponding to each data frame to obtain the sample speech detection results corresponding to each data frame; based on the sample speech detection results corresponding to each data frame and the sample speech label set, train the initial speech detection network to obtain the target speech detection network; use the target speech detection network to perform speech detection on the fusion features to obtain the speech detection results corresponding to the second data frame.

[0121] Among them, the sample feature set includes a large number of sample audio features and sample image features, each sample audio feature and each sample image feature has a time correlation, and the computer device can perform feature fusion on each sample audio feature and the related sample image feature to obtain a sample fusion feature. The sample voice label set can include a sample voice label corresponding to each data frame, and the sample voice label can be used to indicate whether there is voice in each data frame. The sample voice label can refer to the sample true value, for example, the sample voice label can be obtained by manually annotating the sample fusion feature. One data frame corresponds to a sample voice label. The sample voice detection result obtained by the target voice detection network detection can refer to the network output value, and the target voice detection network can be trained based on the sample true value and the network output value so that the network output value is as close to the sample true value as possible. The purpose of training the initial voice detection network is to make the sample true value and the network output value as consistent as possible. When the third loss function of the initial voice detection network is greater than the third loss threshold, it can be indicated that the sample true value and the network output value are inconsistent, that is, the accuracy of the initial voice detection network is low, and the network parameters in the initial voice detection network can be further adjusted to reduce the third loss function in the initial voice detection network. When the third loss function of the initial speech detection network is less than or equal to the third loss threshold, it can be said that the true value of the sample is consistent with the network output value, and it can be considered that the initial speech detection network training meets the requirements, and the initial speech detection network at this time can be saved as the target speech detection network. Therefore, the target speech detection network can be used to perform speech detection on the fusion features corresponding to the second data frame to obtain the speech detection result.

[0122] Since a large amount of sample data (such as sample fusion features) is used to train the initial speech detection network, the initial speech detection network can learn more detailed information in the sample fusion features, such as mouth movement change information, facial expression change information, body language change information, etc., so that the speech detection results corresponding to the sample fusion features predicted based on these detailed information can more accurately reflect the speech situation corresponding to the sample fusion features. Since a large amount of sample data is used to train the initial speech detection network, the accuracy of the network can be improved. When the target speech detection network is used to perform speech detection on the fusion features corresponding to each data frame, the accuracy of speech detection can be improved.

[0123] See also Figure 5 , Figure 5 is a schematic diagram of an application scenario of another speech detection method provided in an embodiment of the present application; Figure 5As shown, the computer device can obtain image data 51 in the video data 50, and the image data 51 can refer to any image data in the image data set. Further, the computer device can determine the first data frame corresponding to the starting voice of the audio data 52 in the video data based on the image data 51. For example, if the first data frame is the 5th data frame in the video data, the computer device can obtain the image feature 53 of the image data corresponding to the 5th data frame or any data frame after the 5th data frame, and the audio feature 54 of the audio data corresponding to the data frame. Further, the computer device can perform feature fusion on the image feature 53 of the image data corresponding to the data frame and the audio feature 54 of the audio data corresponding to the data frame to obtain the fused feature 55 corresponding to the data frame. Further, the fused feature 55 corresponding to the data frame can be subjected to speech detection based on the target speech detection network 56 to obtain the speech detection result 57 corresponding to the data frame. For example, the speech detection result 57 can indicate that there is speech in the data frame, and then determine the speech detection result corresponding to each data frame. Subsequently, the speech detection result can be combined with the noise estimation and applied to specific scenarios such as speech enhancement.

[0124] In some possible scenarios, you can choose according to your needs Figure 3 The speech detection method in the corresponding embodiment or Figure 4 The voice detection method in the corresponding embodiment implements VAD detection. For example, in some live broadcast scenes, when the image is turned off and only the audio exists, or when the image is black in the live broadcast scene, you can select Figure 3 The corresponding embodiment of the voice detection method performs VAD detection. Because there is no image information in this scenario, that is, the image detection result corresponding to each data frame is empty, the audio data can be used for detection to obtain the audio detection result, and the audio detection result is determined as the voice detection result. In other live broadcast scenarios, when there are images and audio, you can choose Figure 3 The speech detection method in the corresponding embodiment or Figure 4 The voice detection method in the corresponding embodiment realizes VAD detection. In the case of screen or frame freeze in other live broadcast scenes, you can choose Figure 4 The speech detection method in the corresponding embodiment implements VAD detection. Because there is image information in this scenario, VAD detection can be performed after feature fusion of image features and audio features to improve the accuracy of VAD detection. By using visual information (such as image data) for VAD auxiliary detection and fusing it with audio features, the VAD detection probability can be improved, thereby improving the accuracy of noise estimation when performing noise estimation, and achieving better noise reduction effects for users outside of vision (such as outside of images). In addition, the noise reduction effect can be further optimized to improve the signal-to-noise ratio, thereby improving the sound quality.

[0125] In an embodiment of the present application, by acquiring image data corresponding to a plurality of data frames, the first data frame corresponding to the starting speech in the audio data associated with the image data can be determined based on the image data, so that the features of the image number and the features of the audio data of the first data frame corresponding to the starting speech and any subsequent data frame can be obtained, and the speech detection result corresponding to any data frame is determined in combination with the features of the image data and the features of the audio data, and the probability of the presence of speech in any data frame is determined, and then which data frames in the entire audio data have speech, which data frames do not have speech, and which data frames have speech that is noise. Since the image data is used to determine the data frame corresponding to the starting speech, there is no need to perform audio detection on the audio data before the data frame corresponding to the starting speech, which can reduce the amount of data processing. Furthermore, since the speech in the audio data is further determined in combination with the image data, the speech can be detected from two different dimensions, which can improve the accuracy of speech detection.

[0126] The method of the embodiment of the present application is introduced above, and the device of the embodiment of the present application is introduced below.

[0127] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of a speech detection device provided in an embodiment of the present application. The speech detection device can be deployed on a computer device; the speech detection device can be used to execute the corresponding steps in the speech detection method provided in an embodiment of the present application. The speech detection device 60 includes:

[0128] The start speech determination unit 601 is used to obtain an image data set, and determine, based on the image data set, a first data frame corresponding to the start speech in the audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame;

[0129] A feature acquisition unit 602 is used to acquire image features of the image data corresponding to the second data frame and audio features of the audio data corresponding to the second data frame;

[0130] The speech detection unit 603 is used to determine a speech detection result corresponding to the second data frame based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, wherein the speech detection result is used to indicate the probability that speech exists in the second data frame.

[0131] Optionally, the image data set includes image data corresponding to a plurality of data frames; the initial speech determination unit 601 is specifically configured to:

[0132] respectively extracting image features of image data corresponding to each data frame in the multiple data frames;

[0133] Performing image detection on the image features of the image data corresponding to each data frame to obtain an image detection result corresponding to each data frame, wherein the image detection result corresponding to each data frame is used to indicate the probability that speech exists in each data frame;

[0134] At least one data frame to be selected in which the probability of the existence of speech is greater than a speech threshold is obtained, and a data frame including the earliest time in the at least one data frame to be selected is determined as a first data frame corresponding to the initial speech in the audio data.

[0135] Optionally, the speech detection unit 603 is specifically configured to:

[0136] Performing image detection on the image features corresponding to the second data frame to obtain an image detection result;

[0137] Performing audio detection on the audio feature corresponding to the second data frame to obtain an audio detection result;

[0138] Based on the image detection result and the audio detection result, a speech detection result corresponding to the second data frame is determined.

[0139] Optionally, the speech detection unit 603 is specifically configured to:

[0140] Identify a target scene associated with the image data set, and obtain a first weight corresponding to the image and a second weight corresponding to the audio in the target scene;

[0141] Based on the first weight, the image detection result, the second weight and the audio detection result, a speech detection result corresponding to the second data frame is determined.

[0142] Optionally, the image detection result corresponding to the second data frame indicates that the probability that speech exists in the second data frame is less than or equal to a speech threshold; the speech detection unit 603 is specifically configured to:

[0143] Respectively obtain image detection results of the first N data frames of the second data frame and image detection results of the last M data frames of the second data frame, where M and N are both natural numbers;

[0144] If the image detection results of the first N data frames indicate that the probability of the presence of speech is greater than the speech threshold, and the image detection results of the last M data frames indicate that the probability of the presence of speech is greater than the speech threshold, then obtain the first speech probability mean between the image detection results corresponding to the second data frame, the image detection results of the first N data frames, and the image detection results of the last M data frames, and determine the first speech probability mean as the image detection result corresponding to the second data frame.

[0145] Optionally, the speech detection device 60 further includes a network training unit 604, which is used to:

[0146] Get sample image sets and sample image labels;

[0147] Using the initial image detection network to extract features from each sample image in the sample image set, to obtain sample image features of each sample image;

[0148] Performing image detection on each sample image feature to obtain a sample image detection result of each sample image;

[0149] The initial image detection network is trained based on the sample image detection result of each sample image and the sample image label to obtain a target image detection network;

[0150] The target image detection network is used to perform image detection on the image features corresponding to the second data frame to obtain an image detection result.

[0151] Optionally, the network training unit 604 is further used for:

[0152] Get sample audio sets and sample audio labels;

[0153] Using the initial audio detection network to extract features from each sample audio in the sample audio set, to obtain sample audio features of each sample audio;

[0154] Performing audio detection on each sample audio feature to obtain a sample audio detection result of each sample audio;

[0155] The initial audio detection network is trained based on the sample audio detection result of each sample audio and the sample audio label to obtain a target audio detection network;

[0156] The target audio detection network is used to perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result.

[0157] Optionally, the speech detection unit 603 is specifically configured to:

[0158] Performing feature fusion on the image feature corresponding to the second data frame and the audio feature corresponding to the second data frame to obtain a fusion feature corresponding to the second data frame;

[0159] Perform speech detection on the fused feature to obtain a speech detection result corresponding to the second data frame.

[0160] Optionally, the network training unit 604 is further used for:

[0161] Acquire a sample feature set and a sample voice label set, wherein the sample feature set includes sample audio features and sample image features corresponding to a plurality of data frames, and the sample voice label set includes sample voice labels corresponding to the plurality of data frames;

[0162] Using the initial speech detection network to perform feature fusion on the sample image features and sample audio features corresponding to each data frame, to obtain the sample fusion features corresponding to each data frame;

[0163] Perform speech detection on the sample fusion features corresponding to each data frame to obtain the sample speech detection result corresponding to each data frame;

[0164] Based on the sample speech detection result corresponding to each data frame and the sample speech label set, the initial speech detection network is trained to obtain a target speech detection network;

[0165] The target speech detection network is used to perform speech detection on the fused feature to obtain a speech detection result corresponding to the second data frame.

[0166] It should be noted that Figure 6 For the contents not mentioned in the corresponding embodiments, please refer to the description of the method embodiments, which will not be repeated here.

[0167] In an embodiment of the present application, by acquiring image data corresponding to a plurality of data frames, the first data frame corresponding to the starting speech in the audio data associated with the image data can be determined based on the image data, so that the features of the image number and the features of the audio data of the first data frame corresponding to the starting speech and any subsequent data frame can be obtained, and the speech detection result corresponding to any data frame is determined in combination with the features of the image data and the features of the audio data, and the probability of the presence of speech in any data frame is determined, and then which data frames in the entire audio data have speech, which data frames do not have speech, and which data frames have speech that is noise. Since the image data is used to determine the data frame corresponding to the starting speech, there is no need to perform audio detection on the audio data before the data frame corresponding to the starting speech, which can reduce the amount of data processing. Furthermore, since the speech in the audio data is further determined in combination with the image data, the speech can be detected from two different dimensions, which can improve the accuracy of speech detection.

[0168] See also Figure 7 , Figure 7 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7As shown, the above-mentioned computer device 70 may include: a processor 701, a network interface 704 and a memory 705. In addition, the above-mentioned computer device 70 may also include: a user interface 703, and at least one communication bus 702. Among them, the communication bus 702 is used to realize the connection and communication between these components. Among them, the user interface 703 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 703 may also include a standard wired interface and a wireless interface. The network interface 704 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 705 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 705 may optionally also be at least one storage device located away from the aforementioned processor 701. As Figure 7 As shown, the memory 705 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0169] exist Figure 7 In the computer device 70 shown, the network interface 704 can provide a network communication function; the user interface 703 is mainly used to provide an input interface for the user; and the processor 701 can be used to call the device control application stored in the memory 705 to perform the following operations:

[0170] Acquire an image data set, and determine, based on the image data set, a first data frame corresponding to an initial speech in audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame;

[0171] Acquire image features of the image data corresponding to the second data frame, and audio features of the audio data corresponding to the second data frame;

[0172] Based on the image features corresponding to the second data frame and the audio features corresponding to the second data frame, a speech detection result corresponding to the second data frame is determined, where the speech detection result is used to indicate the probability that speech exists in the second data frame.

[0173] It should be understood that the computer device 70 described in the embodiment of the present application can execute the above Figure 3 and Figure 4 The description of the above-mentioned speech detection method in the corresponding embodiment can also be performed as described above. Figure 6 The description of the above-mentioned speech detection device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0174] In an embodiment of the present application, by acquiring image data corresponding to a plurality of data frames, the first data frame corresponding to the starting speech in the audio data associated with the image data can be determined based on the image data, so that the features of the image number and the features of the audio data of the first data frame corresponding to the starting speech and any subsequent data frame can be obtained, and the speech detection result corresponding to any data frame is determined in combination with the features of the image data and the features of the audio data, and the probability of the presence of speech in any data frame is determined, and then which data frames in the entire audio data have speech, which data frames do not have speech, and which data frames have speech that is noise. Since the image data is used to determine the data frame corresponding to the starting speech, there is no need to perform audio detection on the audio data before the data frame corresponding to the starting speech, which can reduce the amount of data processing. Furthermore, since the speech in the audio data is further determined in combination with the image data, the speech can be detected from two different dimensions, which can improve the accuracy of speech detection.

[0175] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, which includes program instructions, which, when executed by a computer, causes the computer to execute the method of the aforementioned embodiment, and the computer may be part of the computer device mentioned above. For example, the processor 701 mentioned above. As an example, the program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or, alternatively, executed on multiple computer devices distributed at multiple locations and interconnected by a communication network, and multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0176] Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0177] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A speech detection method, characterized in that: The method comprises: Acquire an image data set, and determine, based on the image data set, a first data frame corresponding to an initial speech in audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame; Acquire image features of image data corresponding to the second data frame, and audio features of audio data corresponding to the second data frame; Performing image detection on the image features corresponding to the second data frame to obtain an image detection result, performing audio detection on the audio features corresponding to the second data frame to obtain an audio detection result, and determining a speech detection result corresponding to the second data frame based on the image detection result and the audio detection result, wherein the speech detection result is used to indicate a probability that speech exists in the second data frame; The performing image detection on the image features corresponding to the second data frame to obtain an image detection result includes: If the image detection result corresponding to the second data frame indicates that the probability that speech exists in the second data frame is less than or equal to the speech threshold, respectively obtaining image detection results of the first N data frames of the second data frame and image detection results of the last M data frames of the second data frame, where M and N are both natural numbers; If the image detection results of the first N data frames indicate that the probability of the presence of speech is greater than the speech threshold, and the image detection results of the last M data frames indicate that the probability of the presence of speech is greater than the speech threshold, then obtain the first speech probability mean between the image detection results corresponding to the second data frame, the image detection results of the first N data frames, and the image detection results of the last M data frames, and determine the first speech probability mean as the image detection result corresponding to the second data frame.

2. The method according to claim 1, characterized in that The image data set includes image data corresponding to a plurality of data frames; The determining, based on the image data set, a first data frame corresponding to the initial speech in the audio data associated with the image data set comprises: Respectively extracting image features of image data corresponding to each data frame in the plurality of data frames; Performing image detection on image features of the image data corresponding to each data frame to obtain an image detection result corresponding to each data frame, wherein the image detection result corresponding to each data frame is used to indicate the probability that speech exists in each data frame; At least one candidate data frame having a probability of speech being present that is greater than a speech threshold is obtained, and a data frame including the earliest time in the at least one candidate data frame is determined as a first data frame corresponding to the initial speech in the audio data.

3. The method according to claim 1, characterized in that The determining, based on the image detection result and the audio detection result, a speech detection result corresponding to the second data frame includes: Identify a target scene associated with the image data set, and obtain a first weight corresponding to the image and a second weight corresponding to the audio in the target scene; Based on the first weight, the image detection result, the second weight and the audio detection result, a speech detection result corresponding to the second data frame is determined.

4. The method according to claim 1, characterized in that: The method further comprises: Get sample image sets and sample image labels; Using an initial image detection network to extract features from each sample image in the sample image set, to obtain sample image features of each sample image; Performing image detection on each sample image feature to obtain a sample image detection result of each sample image; Training the initial image detection network based on the sample image detection result of each sample image and the sample image label to obtain a target image detection network; The performing image detection on the image features corresponding to the second data frame to obtain an image detection result includes: The target image detection network is used to perform image detection on image features corresponding to the second data frame to obtain an image detection result.

5. The method according to claim 1, characterized in that The method further comprises: Get sample audio sets and sample audio labels; Using an initial audio detection network to extract features from each sample audio in the sample audio set, to obtain sample audio features of each sample audio; Performing audio detection on each of the sample audio features to obtain a sample audio detection result of each sample audio; Training the initial audio detection network based on the sample audio detection result of each sample audio and the sample audio label to obtain a target audio detection network; The performing audio detection on the audio feature corresponding to the second data frame to obtain an audio detection result includes: The target audio detection network is used to perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result.

6. A speech detection device, characterized in that: The device comprises: a start speech determination unit, configured to acquire an image data set, and determine, based on the image data set, a first data frame corresponding to the start speech in the audio data associated with the image data set, wherein the image data set includes image data corresponding to a second data frame, and the second data frame is the first data frame or any data frame after the first data frame; a feature acquisition unit, configured to acquire image features of the image data corresponding to the second data frame, and audio features of the audio data corresponding to the second data frame; a speech detection unit, configured to perform image detection on the image features corresponding to the second data frame to obtain an image detection result, perform audio detection on the audio features corresponding to the second data frame to obtain an audio detection result, and determine a speech detection result corresponding to the second data frame based on the image detection result and the audio detection result, wherein the speech detection result is used to indicate a probability that speech exists in the second data frame; The speech detection unit is specifically used to obtain the image detection results of the first N data frames of the second data frame and the image detection results of the last M data frames of the second data frame respectively if the image detection result corresponding to the second data frame indicates that the probability of the existence of speech in the second data frame is less than or equal to the speech threshold, where M and N are both natural numbers; if the image detection results of the first N data frames indicate that the probability of the existence of speech is greater than the speech threshold, and the image detection results of the last M data frames indicate that the probability of the existence of speech is greater than the speech threshold, then obtain a first speech probability mean among the image detection results corresponding to the second data frame, the image detection results of the first N data frames, and the image detection results of the last M data frames, and determine the first speech probability mean as the image detection result corresponding to the second data frame.

7. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 5.

Images