Multi-speaker audio object localization method, device, equipment and medium

Abstract

The present invention discloses a multi-speaker audio object localization method, apparatus, device, and medium. The method comprises: obtaining video data to be processed and audio data to be processed corresponding to the video data to be processed, inputting the video data to be processed into a trained target detection model to obtain the target position of each target object, classifying the audio data to be processed using a voiceprint segmentation and clustering algorithm to obtain the speech segments of each speaker, inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaker, and determining the target position of each speaker based on the target position of the target object, the speech segment of the speaker, and the correspondence between the target object and the speech segment of the speaker.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video processing, and in particular to a multi-speaker audio object positioning method, device, equipment and medium. Background Art

[0002] With the advancement of audio production technology, digital audio has evolved from mono, stereo, surround, and finally 3D sound. The majority of audio content in existing film and television productions is mono, stereo, or surround sound. By extracting speaker position information from existing film and television productions to generate vocal object metadata for 3D sound, and then encoding the extracted vocal data from the corresponding existing film and television productions to generate 3D sound, we can enrich the audiovisual experience of existing film and television productions and increase their replay rate.

[0003] While existing technologies are relatively accurate in determining the speaker's position in audio and video for a single speaker, there are some issues with tracking multiple speakers. These issues primarily arise from the following: In multi-speaker scenarios, speaker tracking is inaccurate; Furthermore, it is difficult to locate the speaker when they are far from the camera, facing sideways, or facing away from the camera. When speaker tracking fails, manual playback of the audio and video is required to observe the speaker and manually annotate their position, significantly increasing the workload of 3D sound production.

[0004] In view of this, how to provide a solution that can improve the accuracy of locating and tracking speaker position information in audio and video in multi-speaker scenes is a technical problem that people in the relevant technical field need to solve. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a multi-speaker audio object localization method, device, equipment and medium, which can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes and reduce the workload of manual labeling of speaker position information.

[0006] In one aspect, the present invention provides a method for localizing multi-speaker audio objects, comprising:

[0007] Acquire the video data to be processed and the audio data to be processed corresponding to the video data to be processed;

[0008] Input the video data to be processed into the trained target detection model to obtain the target position of each target object;

[0009] The audio data to be processed is classified through the voiceprint segmentation and clustering algorithm to obtain the voice segments of each speaker;

[0010] Inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain a correspondence between the target object and the speech segments of the speaker, wherein the active speech detection model is trained using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels;

[0011] The target position of each speaking object is determined according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

[0012] Preferably, the target detection model is trained in the following way:

[0013] Input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time;

[0014] Predict the motion trajectory of each sample object in the sample video data to obtain the predicted bounding box of each sample object in time;

[0015] Use the loss function to calculate the loss value of the detection bounding box and the predicted bounding box;

[0016] Determine whether the initial detection model has converged based on the loss value;

[0017] If so, the initial detection model is determined to be the trained target detection model;

[0018] If not, the initial detection model is adjusted, and the process returns to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

[0019] Preferably, predicting the motion trajectory of each sample object in the sample video data to obtain a predicted bounding box of each sample object in time includes:

[0020] Obtaining a first bounding box of each sample object in the first video frame and a second bounding box of each sample object in the second video frame in the sample video data;

[0021] determining a movement direction and a movement speed of each sample object according to the first bounding box and the second bounding box;

[0022] The motion trajectory of each sample object is predicted based on the second bounding box, the motion direction, and the motion speed to obtain a predicted bounding box of each sample object in time.

[0023] Preferably, the loss function is expressed as:

[0024]

[0025] in, represents the loss function, and Represent the predicted bounding box and the detected bounding box respectively, Represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average of the normalized grayscale values ​​of the predicted bounding box and the average of the normalized grayscale values ​​of the detected bounding box.

[0026] Preferably, the audio data to be processed is classified by a voiceprint segmentation and clustering algorithm to obtain voice segments of each speaker, including:

[0027] Splitting the audio data to be processed into multiple audio segments;

[0028] Extract the voiceprint vectors from each audio clip and concatenate them into a voiceprint vector matrix;

[0029] Performing a transposition operation on the voiceprint vector matrix to obtain a transposed voiceprint vector matrix;

[0030] Perform an inner product operation on the voiceprint vector matrix and the transposed voiceprint vector matrix to obtain a similarity matrix;

[0031] The audio segments are clustered according to the similarity matrix to obtain the speech segments of each speaker.

[0032] Preferably, before dividing the audio data to be processed into a plurality of audio segments, the method further comprises:

[0033] The speech detection model is used to analyze the audio data to be processed to obtain the audio data to be processed containing the speech signal.

[0034] Preferably, the method further comprises:

[0035] Taking the face of each target object as the target, target detection is performed on the video data to be processed to obtain the face image of each target object;

[0036] The temporal feature description of the lip area is extracted from the face image as the lip movement feature of each target object.

[0037] Another aspect of the present invention provides a multi-speaker audio object localization device, comprising:

[0038] An acquisition module, configured to acquire the video data to be processed and the audio data to be processed corresponding to the video data to be processed;

[0039] The target position detection module is used to input the video data to be processed into the trained target detection model to obtain the target position of each target object;

[0040] The voice segment classification module is used to classify the processed audio data using the voiceprint segmentation and clustering algorithm to obtain the voice segments of each speaker;

[0041] a correspondence determination module, configured to input the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain a correspondence between the target object and the speech segments of the speaker, wherein the active speech detection model is trained using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels;

[0042] The output module is used to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the corresponding relationship between the target object and the speech segment of the speaking object.

[0043] Another aspect of the present invention provides an electronic device, comprising:

[0044] Memory for storing computer programs;

[0045] The processor is configured to implement the steps of the multi-speaker audio object localization method as described above when executing the computer program.

[0046] Yet another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-speaker audio object localization method are implemented.

[0047] The present invention has at least the following beneficial effects:

[0048] The present invention obtains video data to be processed and audio data to be processed corresponding to the video data to be processed, inputs the video data to be processed into a trained target detection model to obtain the target position of each target object, classifies the audio data to be processed using a voiceprint segmentation and clustering algorithm to obtain the speech segments of each speaker, and inputs the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segments of the speaker. The active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segments of each speaker in the training audio data as training samples, and the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels. The target position of each speaker is determined based on the target position of the target object, the speech segments of the speaker, and the correspondence between the target object and the speech segments of the speaker. The solution of the present invention can improve the accuracy of speaker location tracking in audio and video with multiple speakers, and reduce the workload of manual speaker location labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a multi-speaker audio object localization method provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the structure of an active speech detection model provided by an embodiment of the present invention;

[0052] Figure 3 A flowchart of a target detection model training method provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a target detection algorithm provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a flow chart of a method for obtaining a predicted bounding box provided by an embodiment of the present invention;

[0055] Figure 6 A flow chart of a voiceprint segmentation algorithm provided by an embodiment of the present invention;

[0056] Figure 7 A schematic structural diagram of a multi-speaker audio object localization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The core of the present invention is to provide a multi-speaker audio object localization method, device, equipment and medium, which can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes and reduce the workload of manual labeling of speaker position information.

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] One aspect of the present invention provides a method for locating multi-speaker audio objects. Figures 1 to 6 , the method comprising:

[0060] Step S110: Obtain the video data to be processed and the audio data to be processed corresponding to the video data to be processed.

[0061] In the embodiments of the present invention, audio and video data, also referred to as audio video, specifically refers to data produced using video data and corresponding audio data. During playback of the audio and video data, users can not only view each video frame in the video data but also hear the corresponding sound described by the audio data while viewing each video frame.

[0062] Step S120: input the video data to be processed into the trained target detection model to obtain the target position of each target object.

[0063] In the embodiments of the present invention, target detection refers to the process of identifying and locating a specific target in an image or video frame, typically outputting a bounding box for the target. The video data to be processed is input into a trained target detection model. A deep learning-based target detection algorithm is used to scan each video frame in the video data to identify and locate each target object. The bounding box for each target object, including its position information within the video frame, is then output. The target object in the embodiments of the present invention can be a person, vehicle, animal, or other object. It is understood that the target detection algorithm can more accurately locate the target position of each target object in audio and video of multi-speaker scenes.

[0064] Step S130: Classify the audio data to be processed using a voiceprint segmentation and clustering algorithm to obtain voice segments of each speaker.

[0065] In the embodiments of the present invention, voiceprint segmentation and clustering refers to identifying the voice and speaking time of each speaker in a speech segment during multi-speaker speech analysis. Using a voiceprint segmentation and clustering algorithm, the processed audio data can be classified into multiple speech segments, and a mapping relationship between the speech segments and the corresponding speakers can be generated. The speakers in the embodiments of the present invention primarily refer to the speakers. It will be appreciated that the voiceprint segmentation and clustering algorithm can accurately classify the speaker's speech even if the speaker is far away from the camera, facing sideways, or facing away from the camera.

[0066] Step S140: Input the lip movement features of each target object in the video data to be processed and the speech segments of each speaking object in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaking object, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels.

[0067] In an embodiment of the present invention, an active speech detection model can be pre-trained, using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels for training. By inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into the trained active speech detection model, a correspondence between the target object and the speech segments of the speaker can be obtained. It is understood that when the target object is a person speaking, and the speaker is facing the camera and relatively close to the camera, the active speech detection model extracts the target object's lip movement features and matches the lip movement features with the speech segments. When the match is successful, a correspondence between the target object and the speech segments of the speaker can be established.

[0068] Specifically, the active speaker detection model can be a model designed specifically for active speaker detection, such as TalkNet. The TalkNet detection model combines short-term and long-term audio and video features, utilizes an audio-visual cross-attention mechanism to enhance inter-modal interaction, and employs a self-attention mechanism to capture speech evidence over long periods of time. It extracts characteristic information from the audio and video spaces, then adds audio feature information to the video frame feature information for cross-attention processing to enhance the video feature information. Simultaneously, it adds video feature information to the audio frame feature information for cross-attention processing to enhance the audio feature information. The audio and video feature information within the same frame time is then combined to obtain audio and video feature information. The self-attention mechanism is then applied to the audio and video feature information of the preceding and following frames to obtain a frame-by-frame prediction sequence, thereby achieving active speaker detection.

[0069] Step S150 : determining the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

[0070] In this embodiment of the present invention, after determining the target position of the target object, obtaining the voice segments of each speaker, and establishing a correspondence between the target object and the speech segments of the speaker, the target position of each speaker can be determined. It is understood that for existing film and television works, by extracting speaker position information from the existing film and television works, generating vocal object metadata for producing 3D sound, and combining it with the encoded vocal data extracted from the corresponding existing film and television works to generate 3D sound, the audio-visual experience of the existing film and television works can be enriched and the replay rate of the existing film and television works can be increased.

[0071] As can be seen from the above, the multi-speaker audio object localization method provided by the embodiment of the present invention obtains to-be-processed video data and to-be-processed audio data corresponding to the to-be-processed video data, inputs the to-be-processed video data into a trained target detection model to obtain the target position of each target object, classifies the to-be-processed audio data using a voiceprint segmentation and clustering algorithm to obtain the speech segments of each speaking object, and inputs the lip movement features of each target object in the to-be-processed video data and the speech segments of each speaking object in the to-be-processed audio data into a trained active speech detection model to obtain the correspondence between the target object and the speech segments of the speaking object. The active speech detection model is trained using the set of lip movement features of each target object in the training video data and the set of speech segments of each speaking object in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaking object in the training audio data as sample labels. The target position of each speaking object is determined based on the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object. The solution of the present application can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes, and reduce the workload of manually annotating speaker position information.

[0072] Optionally, in the above embodiment, the target detection model is trained in the following manner:

[0073] Step S310: Input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time;

[0074] Step S320: predict the motion trajectory of each sample object in the sample video data to obtain a predicted bounding box of each sample object in time;

[0075] Step S330: Calculate the loss value of the detection bounding box and the prediction bounding box using the loss function;

[0076] Step S340: Determine whether the initial detection model has converged based on the loss value; if so, execute step S350; if not, execute step S360;

[0077] Step S350: determining the initial detection model as the trained target detection model;

[0078] Step S360: Adjust the initial detection model and return to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

[0079] In an embodiment of the present invention, the initial detection model is a network model based on a target detection algorithm, such as a YOLO (You Only Look Once) target detection model or an SSD (Single Shot MultiBox Detector) target detection model. The sample video data is a training video data set that may include multiple sample videos. The sample objects may be people, vehicles, animals, or other objects. The sample video data is input into the initial detection model, and after extracting image detection features, the detection bounding box of each sample object in each video frame can be obtained. At the same time, the motion trajectory of each sample object in the sample video data within the video frame can be predicted to obtain the predicted bounding box of each sample object in each video frame. Then, the loss function is used to calculate the loss value between the detection bounding box and the prediction bounding box. When the loss value is not greater than the preset value (configured according to experience, there is no limit on this), it is determined that the initial detection model has converged, and the initial detection model is determined to be a trained target detection model; when the loss value is greater than the preset value, it is determined that the initial detection model has not converged, and the initial detection model is adjusted, and the adjusted initial detection model is used to output a new detection bounding box. Based on the new detection bounding box, it is re-determined whether the adjusted initial detection model has converged. Similarly, by continuously adjusting the initial detection model, the loss value between the detection bounding box and the prediction bounding box becomes smaller and smaller, that is, the performance of the initial detection model becomes better and better, until the initial detection model converges, and the performance of the initial detection model meets the requirements, the initial detection model that meets the performance requirements can be determined as the trained target detection model, and the model training process is completed.

[0080] Furthermore, in the above embodiment, step S320 may include:

[0081] Step S3201: Obtain a first bounding box of each sample object in the first video frame and a second bounding box of each sample object in the second video frame in the sample video data;

[0082] Step S3202: Determine the movement direction and movement speed of each sample object based on the first bounding box and the second bounding box;

[0083] Step S3203: predict the motion trajectory of each sample object according to the second bounding box, motion direction, and motion speed to obtain a predicted bounding box of each sample object in time.

[0084] In an embodiment of the present invention, a first bounding box of each sample object in the first video frame and a second bounding box in the second video frame are obtained. In the second video frame, with the center point of the first bounding box as the origin, a vector consisting of a line connecting the center point of the second bounding box to the center point of the first bounding box is obtained, wherein the direction of the vector represents the direction of movement of each sample object, and the modulus of the vector represents the movement speed of each sample object. In the third video frame, the vector is translated to a position with the center point of the second bounding box as the origin to obtain a new vector, and the vertex of the new vector is used as the center point of the third bounding box, and the size of the third bounding box is the same as the size of the second bounding box, thereby obtaining a third bounding box. In the fourth video frame and subsequent video frames, the position and size of the bounding box are copied based on the above steps to obtain the temporally predicted bounding box of each sample object.

[0085] Furthermore, in the above embodiment, the loss function is expressed as:

[0086]

[0087] in, represents the loss function, and Represent the predicted bounding box and the detected bounding box respectively, Represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average of the normalized grayscale values ​​of the predicted bounding box and the average of the normalized grayscale values ​​of the detected bounding box.

[0088] In an embodiment of the present invention, a loss function may be designed based on the overlapping area, the distance between the center points, and the similarity between the predicted bounding box and the detected bounding box to determine whether the initial detection model has converged.

[0089] Specifically, It can be expressed as:

[0090]

[0091] in, represents the square of the Euclidean distance between the center point of the predicted bounding box and the center point of the detected bounding box, Represents the square of the maximum diagonal distance of the smallest enclosing box covering the predicted bounding box and the detected bounding box.

[0092] It can be expressed as:

[0093]

[0094] in, Represents the average of the normalized grayscale values ​​of the predicted bounding box, Represents the average of the normalized grayscale values ​​of the detected bounding box. It should be noted that normalizing the grayscale values ​​of the predicted bounding box and the detected bounding box first can make subsequent data processing more convenient and faster.

[0095] Optionally, in the above embodiment, step S130 may include:

[0096] Step S1301: dividing the audio data to be processed into multiple audio segments;

[0097] Step S1302: extracting voiceprint vectors from each audio clip and concatenating the voiceprint vectors into a voiceprint vector matrix;

[0098] Step S1303: performing a transposition operation on the voiceprint vector matrix to obtain a transposed voiceprint vector matrix;

[0099] Step S1304: performing an inner product operation on the voiceprint vector matrix and the transposed voiceprint vector matrix to obtain a similarity matrix;

[0100] Step S1305: cluster the audio segments according to the similarity matrix to obtain the speech segments of each speaker.

[0101] In an embodiment of the present invention, the audio data to be processed can be segmented so that each segment has only one speaker, and each segment contains a start and end time. In one embodiment, the audio data to be processed can be segmented into multiple audio segments according to a preset duration (configured based on experience, such as 1.0s or 1.5s, with no limitation). In another embodiment, the audio data to be processed can be segmented into multiple audio segments using an SCD (Speaker Change Detection) model. A voiceprint recognition model is then used to extract voiceprint features from all audio segments, obtaining voiceprint vector information (such as i-vectors and x-vectors) for each audio segment. These information is then concatenated into a voiceprint vector matrix, where the abscissa of the voiceprint vector matrix represents time length (unit: window size), and the ordinate represents the dimension of the voiceprint vector. The voiceprint vector matrix is ​​then transposed to obtain a transposed voiceprint vector matrix. The inner product operation is then performed on the voiceprint vector matrix and the transposed voiceprint vector matrix to obtain a similarity matrix, which describes the similarity between the audio segments. In this similarity matrix, each element represents the similarity between the voiceprint information of the audio clip corresponding to the row index and the voiceprint information of the audio clip corresponding to the column index. Finally, the audio clips are clustered according to the similarity matrix. Audio clips whose similarity meets a set threshold (configured based on experience and not limited) are grouped into a category. Each category corresponds to a speaker, and the speech clip corresponding to each speaker is obtained.

[0102] Furthermore, in the above embodiment, before dividing the audio data to be processed into a plurality of audio segments, the method further includes:

[0103] The speech detection model is used to analyze the audio data to be processed to obtain the audio data to be processed containing the speech signal.

[0104] In embodiments of the present invention, a voice detection model, such as a VAD (Voice Activity Detection) model, can be used to classify the audio data to be processed into two categories: data containing speech signals and data not containing speech signals. Only the audio data containing speech signals is retained. The audio data not containing speech signals may include silence, noise, and music.

[0105] Optionally, in the above embodiment, the method further includes:

[0106] Taking the face of each target object as the target, target detection is performed on the video data to be processed to obtain the face image of each target object;

[0107] The temporal feature description of the lip area is extracted from the face image as the lip movement feature of each target object.

[0108] In an embodiment of the present invention, when the target object is a person who is speaking and the speaker is facing the camera and relatively close to the camera, a video frame containing the speaker can be input, and the face can be located by using a face detection algorithm (such as Haar feature + AdaBoost, CNN, etc.). Then, the lip area is extracted from the face area, and an image of the speaker's lip area is output. Temporal feature descriptors, such as optical flow features and dynamic texture features, are extracted from the lip area image to capture the dynamic changes of the lip shape. Finally, the speaker's lip movement features are output for subsequent matching of the lip movement features with the speech segment.

[0109] Another aspect of the present invention provides a multi-speaker audio object localization device. The device described below may correspond to the method described above.

[0110] See also Figure 7 , the device comprises:

[0111] An acquisition module 710 is configured to acquire video data to be processed and audio data to be processed corresponding to the video data to be processed;

[0112] The target position detection module 720 is used to input the video data to be processed into the trained target detection model to obtain the target position of each target object;

[0113] The speech segment classification module 730 is used to classify the audio data to be processed using a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker;

[0114] Correspondence determination module 740 is configured to input the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain a correspondence between the target object and the speech segments of the speaker. The active speech detection model is trained using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels.

[0115] The output module 750 is configured to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

[0116] As can be seen from the above, the multi-speaker audio object localization device provided by the embodiment of the present invention obtains to-be-processed video data and to-be-processed audio data corresponding to the to-be-processed video data, inputs the to-be-processed video data into a trained target detection model to obtain the target position of each target object, classifies the to-be-processed audio data using a voiceprint segmentation and clustering algorithm to obtain the speech segments of each speaking object, and inputs the lip movement features of each target object in the to-be-processed video data and the speech segments of each speaking object in the to-be-processed audio data into a trained active speech detection model to obtain the correspondence between the target object and the speech segments of the speaking object. The active speech detection model is trained using the set of lip movement features of each target object in the training video data and the set of speech segments of each speaking object in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaking object in the training audio data as sample labels. The target position of each speaking object is determined based on the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object. The solution of the present application can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes, and reduce the workload of manually annotating speaker position information.

[0117] Another aspect of an embodiment of the present invention provides an electronic device, including:

[0118] Memory for storing computer programs;

[0119] The processor is configured to implement the steps of the multi-speaker audio object localization method as described above when executing the computer program.

[0120] Another aspect of the embodiments of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-speaker audio object localization method are implemented.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0122] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-speaker audio object localization method, characterized in that: include: Acquire video data to be processed and audio data to be processed corresponding to the video data to be processed; Inputting the video data to be processed into a trained target detection model to obtain the target position of each target object; Classifying the audio data to be processed by a voiceprint segmentation and clustering algorithm to obtain voice segments of each speaker; Inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain a correspondence between the target object and the speech segments of the speaker, wherein the active speech detection model is trained using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels; The target position of each speaking object is determined according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

2. The multi-speaker audio object localization method according to claim 1, characterized in that: The target detection model is trained in the following way: Input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time; Predicting a motion trajectory of each of the sample objects in the sample video data to obtain a predicted bounding box of each of the sample objects in time; Calculating loss values ​​of the detection bounding box and the prediction bounding box using a loss function; Determining whether the initial detection model has converged according to the loss value; If yes, the initial detection model is determined as the trained target detection model; If not, the initial detection model is adjusted, and the process returns to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

3. The multi-speaker audio object localization method according to claim 2, characterized in that: The predicting the motion trajectory of each of the sample objects in the sample video data to obtain a predicted bounding box of each of the sample objects in time includes: Obtaining a first bounding box of each of the sample objects in the sample video data in a first video frame and a second bounding box of each of the sample objects in a second video frame; determining a movement direction and a movement speed of each of the sample objects according to the first bounding box and the second bounding box; The motion trajectory of each of the sample objects is predicted according to the second bounding box, the motion direction, and the motion speed to obtain a predicted bounding box of each of the sample objects in time.

4. The multi-speaker audio object localization method according to claim 2, characterized in that: The expression of the loss function is: ； in, represents the loss function, and represent the predicted bounding box and the detected bounding box respectively, represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average value of the normalized grayscale values ​​of the predicted bounding box and the average value of the normalized grayscale values ​​of the detected bounding box.

5. The multi-speaker audio object localization method according to claim 1, characterized in that: The method of classifying the audio data to be processed by the voiceprint segmentation and clustering algorithm to obtain the voice segments of each speaker includes: Dividing the audio data to be processed into multiple audio segments; Extracting voiceprint vectors from each of the audio clips, and concatenating the voiceprint vectors into a voiceprint vector matrix; performing a transposition operation on the voiceprint vector matrix to obtain a transposed voiceprint vector matrix; performing an inner product operation on the voiceprint vector matrix and the transposed voiceprint vector matrix to obtain a similarity matrix; The audio segments are clustered according to the similarity matrix to obtain speech segments of each speaker.

6. The multi-speaker audio object localization method according to claim 5, characterized in that: Before dividing the audio data to be processed into a plurality of audio segments, the method further includes: The audio data to be processed is analyzed using a speech detection model to obtain the audio data to be processed containing a speech signal.

7. The multi-speaker audio object localization method according to claim 1, characterized in that: The method further comprises: Taking the face of each target object as a target, performing target detection on the video data to be processed to obtain a face image of each target object; A temporal feature description of the lip region is extracted from the face image as a lip movement feature of each target object.

8. A multi-speaker audio object localization device, characterized in that: include: An acquisition module, configured to acquire video data to be processed and audio data to be processed corresponding to the video data to be processed; A target position detection module is used to input the video data to be processed into a trained target detection model to obtain the target position of each target object; A speech segment classification module is used to classify the audio data to be processed using a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker; a correspondence determination module, configured to input the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain a correspondence between the target object and the speech segments of the speaker, wherein the active speech detection model is trained using a set of lip movement features of each target object in the training video data and a set of speech segments of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segments of each speaker in the training audio data as sample labels; The output module is configured to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the multi-speaker audio object localization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multi-speaker audio object localization method according to any one of claims 1 to 7 are implemented.

Images