Multi-speaker audio object positioning method, device, equipment and medium

Abstract

The invention discloses a multi-speaker audio object positioning method and device, equipment and a medium. The method comprises the steps of obtaining to-be-processed video data and to-be-processed audio data corresponding to the to-be-processed video data, inputting the to-be-processed video data into a trained target detection model to obtain a target position of each target object, and classifying the to-be-processed audio data through a voiceprint segmentation clustering algorithm to obtain a target position of each target object; the lip movement features of the target objects in the video data to be processed and the voice segments of the speaking objects in the audio data to be processed are input into a trained active speaking detection model, and the corresponding relation between the voice segments of the target objects and the voice segments of the speaking objects is obtained; and determining the target position of each speaking object according to the target position of the target object, the voice segments of the speaking objects and the corresponding relationship between the target object and the voice segments of the speaking objects.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video processing, and in particular to a method, device, equipment and medium for locating multi-speaker audio objects. Background Art

[0002] With the development of audio production technology, digital audio has gone through a process from mono, stereo, surround to 3D sound. Most of the audio content in existing film and television works is mono, stereo or surround sound. By extracting the speaker's position information from existing film and television works, generating human voice object metadata for producing 3D sound, and combining the human voice data extracted from the corresponding existing film and television works to encode and generate 3D sound, it can enrich the audio-visual experience of existing film and television works and improve the replay rate of existing film and television works.

[0003] In the existing technology, it is relatively accurate to judge the position information of the speaker in the audio and video of a single speaker scene, but there are some problems in the positioning and tracking of multiple speakers. The main manifestations are: in a multi-speaker scene, the positioning and tracking of the speaker's position is not accurate enough; in addition, it is difficult to locate the speaker when the speaker is far away from the camera, facing the side of the camera, or facing away from the camera. After the speaker positioning and tracking fails, it is necessary to manually replay the audio and video, observe the speaker, and manually mark the speaker's position information, which adds a lot of workload to the 3D sound production.

[0004] In view of this, how to provide a solution that can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes is a technical problem that people in the relevant technical field need to solve. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a multi-speaker audio object localization method, device, equipment and medium, which can improve the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes and reduce the workload of manually marking speaker position information.

[0006] The present invention provides a method for locating multi-speaker audio objects, comprising: Acquire the video data to be processed and the audio data to be processed corresponding to the video data to be processed; Input the video data to be processed into the trained target detection model to obtain the target position of each target object; The audio data to be processed is classified through the voiceprint segmentation and clustering algorithm to obtain the voice segments of each speaker; Inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaker, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segment of each speaker in the training audio data as sample labels; The target position of each speaking object is determined according to the target position of the target object, the speech segment of the speaking object, and the corresponding relationship between the target object and the speech segment of the speaking object.

[0007] Preferably, the target detection model is trained in the following way: Input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time; Predicting the motion trajectory of each sample object in the sample video data to obtain a predicted bounding box of each sample object in time; Use the loss function to calculate the loss value of the detection bounding box and the prediction bounding box; Determine whether the initial detection model has converged based on the loss value; If yes, the initial detection model is determined as the trained target detection model; If not, the initial detection model is adjusted, and the process returns to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

[0008] Preferably, predicting the motion trajectory of each sample object in the sample video data to obtain a predicted bounding box of each sample object in time includes: Obtaining a first bounding box of each sample object in the sample video data in a first video frame and a second bounding box of each sample object in a second video frame; Determine the moving direction and moving speed of each sample object according to the first bounding box and the second bounding box; The motion trajectory of each sample object is predicted according to the second bounding box, the motion direction and the motion speed, so as to obtain a predicted bounding box of each sample object in time.

[0009] Preferably, the loss function is expressed as:

[0010] in, represents the loss function, and Represent the predicted bounding box and the detected bounding box respectively, Represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average of the normalized grayscale values ​​of the predicted bounding box and the average of the normalized grayscale values ​​of the detected bounding box.

[0011] Preferably, the audio data to be processed is classified by a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker, including: Divide the audio data to be processed into multiple audio segments; Extract the voiceprint vectors from each audio clip and concatenate the voiceprint vectors into a voiceprint vector matrix; Performing a transposition operation on the voiceprint vector matrix to obtain a transposed voiceprint vector matrix; Perform an inner product operation on the voiceprint vector matrix and the voiceprint vector matrix after the transposition operation to obtain a similarity matrix; The audio segments are clustered according to the similarity matrix to obtain the speech segments of each speaker.

[0012] Preferably, before dividing the audio data to be processed into a plurality of audio segments, the method further comprises: The speech detection model is used to analyze the audio data to be processed to obtain the audio data to be processed containing the speech signal.

[0013] Preferably, the method further comprises: Taking the face of each target object as the target, target detection is performed on the video data to be processed to obtain the face image of each target object; The temporal feature description of the lip area is extracted from the face image as the lip movement feature of each target object.

[0014] Another aspect of the present invention provides a multi-speaker audio object localization device, comprising: An acquisition module, used to acquire the video data to be processed and the audio data to be processed corresponding to the video data to be processed; The target position detection module is used to input the video data to be processed into the trained target detection model to obtain the target position of each target object; A speech segment classification module is used to classify the audio data to be processed by using a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker; A corresponding relationship determination module is used to input the lip movement features of each target object in the video data to be processed and the speech segments of each speaking object in the audio data to be processed into a trained active speech detection model to obtain the corresponding relationship between the target object and the speech segment of the speaking object, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and the corresponding relationship between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels; The output module is used to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object and the corresponding relationship between the target object and the speech segment of the speaking object.

[0015] Another aspect of the present invention provides an electronic device, comprising: Memory for storing computer programs; The processor is used to implement the steps of the multi-speaker audio object localization method as described above when executing the computer program.

[0016] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-speaker audio object localization method are implemented.

[0017] The present invention has at least the following beneficial effects: The present invention obtains the video data to be processed and the audio data to be processed corresponding to the video data to be processed, inputs the video data to be processed into the trained target detection model, obtains the target position of each target object, classifies the audio data to be processed by the voiceprint segmentation and clustering algorithm, obtains the voice segments of each speaking object, inputs the lip movement features of each target object in the video data to be processed and the voice segments of each speaking object in the audio data to be processed into the trained active speech detection model, obtains the corresponding relationship between the target object and the voice segment of the speaking object, wherein the active speech detection model is trained with the lip movement feature set of each target object in the training video data and the voice segment set of each speaking object in the training audio data as training samples, and with the corresponding relationship between each target object in the training video data and the voice segment of each speaking object in the training audio data as sample labels, and determines the target position of each speaking object according to the target position of the target object, the voice segment of the speaking object, and the corresponding relationship between the target object and the voice segment of the speaking object. The scheme of the present application can improve the accuracy of positioning and tracking the speaker position information in the audio and video of multi-speaker scenes, and reduce the workload of manually marking the speaker position information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a multi-speaker audio object localization method provided by an embodiment of the present invention; Figure 2 A structural diagram of an active speech detection model provided by an embodiment of the present invention; Figure 3 A flowchart of a method for training a target detection model provided by an embodiment of the present invention; Figure 4 A schematic diagram of a target detection algorithm provided by an embodiment of the present invention; Figure 5 A schematic diagram of a flow chart of a method for obtaining a prediction bounding box provided by an embodiment of the present invention; Figure 6 A flow chart of a voiceprint segmentation algorithm provided by an embodiment of the present invention; Figure 7 A schematic diagram of the structure of a multi-speaker audio object localization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The core of the present invention is to provide a multi-speaker audio object positioning method, device, equipment and medium, which can improve the accuracy of positioning and tracking speaker position information in audio and video of multi-speaker scenes and reduce the workload of manually marking speaker position information.

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] An embodiment of the present invention provides a method for locating multi-speaker audio objects. Figures 1 to 6 , the method comprising: Step S110: Obtain the video data to be processed and the audio data to be processed corresponding to the video data to be processed.

[0023] In the embodiment of the present invention, the audio and video data may also be referred to as audio video, which specifically refers to data produced by using video data and corresponding audio data. During the playback of the audio and video data, the user can not only watch each video frame in the video data, but also listen to the corresponding sound described by the audio data while watching each video frame.

[0024] Step S120: input the video data to be processed into the trained target detection model to obtain the target position of each target object.

[0025] In the embodiment of the present invention, target detection refers to the process of identifying and locating a specific target in an image or video frame, and usually outputs the bounding box of the target. The video data to be processed is input into the trained target detection model, and each video frame in the video data to be processed is scanned by a target detection algorithm based on deep learning, so that each target object can be identified and located, and the bounding box of each target object is output, including its position information in the video frame. Among them, the target object in the embodiment of the present invention can be a person, a vehicle, an animal or other object. It can be understood that through the target detection algorithm, the target position of each target object can be more accurately located in the audio and video of the multi-speaker scene.

[0026] Step S130: classify the audio data to be processed by using a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker.

[0027] In the embodiment of the present invention, voiceprint segmentation and clustering refers to identifying the voice of each speaker in a speech segment and the time of his / her speaking in the speech analysis of multi-speaker interaction. Through the voiceprint segmentation and clustering algorithm, the audio data to be processed can be classified to obtain multiple speech segments, and a mapping relationship between the speech segments and the corresponding speaking objects can be generated. Among them, the speaking object in the embodiment of the present invention mainly refers to the speaker. It can be understood that through the voiceprint segmentation and clustering algorithm, even if the speaking object is far away from the camera, facing the side of the camera or facing away from the camera, the voice of the speaking object can still be accurately classified.

[0028] Step S140: input the lip movement features of each target object in the video data to be processed and the speech segments of each speaking object in the audio data to be processed into the trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaking object, wherein the active speech detection model is trained with a set of lip movement features of each target object in the training video data and a set of speech segments of each speaking object in the training audio data as training samples, and with the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels.

[0029] In an embodiment of the present invention, an active speech detection model can be pre-trained, using the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels for training. The lip movement features of each target object in the video data to be processed and the speech segments of each speaking object in the audio data to be processed are input into the trained active speech detection model, and the correspondence between the target object and the speech segment of the speaking object can be obtained. It can be understood that when the target object is a person who is speaking, and the speaker faces the camera and is close to the camera, the lip movement features of the target object are extracted through the active speech detection model, and the lip movement features are matched with the speech segment. When the match is successful, the correspondence between the target object and the speech segment of the speaking object can be established.

[0030] Specifically, the active speech detection model can be a model designed for active speaker detection, such as TalkNet. Among them, the TalkNet detection model combines the short-term and long-term features of audio and video, uses the audio-visual cross-attention mechanism to enhance the interaction between modalities, and adopts the self-attention mechanism to capture speech evidence over a long period of time, respectively extracts the characteristic information of each through the audio and video space, adds the audio feature information to the feature information of the video frame for cross-attention processing to enhance the video feature information, and adds the video feature information to the feature information of the audio frame for cross-attention processing to enhance the audio feature information, and then combines the audio and video feature information within the same frame time to obtain the audio and video feature information, and then performs the self-attention mechanism processing on the audio and video feature information of the previous and next frames to obtain a frame-by-frame prediction result sequence, thereby realizing active speaker detection.

[0031] Step S150: determining the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

[0032] In the embodiment of the present invention, after determining the target position of the target object, obtaining the voice segments of each speaking object, and establishing the corresponding relationship between the target object and the voice segments of the speaking object, the target position of each speaking object can be determined. It can be understood that for existing film and television works, by extracting the position information of the speaker from the existing film and television works, generating human voice object metadata for producing three-dimensional sound, and combining the human voice data extracted from the corresponding existing film and television works to encode and generate three-dimensional sound, it is possible to enrich the audio-visual experience of the existing film and television works and improve the replay rate of the existing film and television works.

[0033] As can be seen from the above, the multi-speaker audio object localization method provided by the embodiment of the present invention obtains the video data to be processed and the audio data to be processed corresponding to the video data to be processed, inputs the video data to be processed into the trained target detection model, obtains the target position of each target object, classifies the audio data to be processed by the voiceprint segmentation and clustering algorithm, obtains the speech segment of each speaking object, inputs the lip movement features of each target object in the video data to be processed and the speech segment of each speaking object in the audio data to be processed into the trained active speech detection model, obtains the correspondence between the target object and the speech segment of the speaking object, wherein the active speech detection model uses the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and uses the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels for training, and determines the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object. By adopting the solution of the present application, the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes can be improved, and the workload of manually marking speaker position information can be reduced.

[0034] Optionally, in the above embodiment, the target detection model is trained in the following manner: Step S310: input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time; Step S320: predicting the motion trajectory of each sample object in the sample video data to obtain a predicted bounding box of each sample object in time; Step S330, using the loss function to calculate the loss value of the detection bounding box and the prediction bounding box; Step S340, judging whether the initial detection model has converged according to the loss value; if so, executing step S350; if not, executing step S360; Step S350: determining the initial detection model as the trained target detection model; Step S360: adjust the initial detection model, and return to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

[0035] In an embodiment of the present invention, the initial detection model is a network model based on a target detection algorithm, such as a YOLO (You Only Look Once) target detection model or an SSD (Single Shot MultiBox Detector) target detection model. The sample video data is a set of video data for training, which may include multiple sample videos, and the sample objects may be people, vehicles, animals or other objects. After the sample video data is input into the initial detection model and the image detection features are extracted, the detection bounding box of each sample object in each video frame can be obtained. At the same time, the motion trajectory of each sample object in the sample video data in the video frame can be predicted to obtain the predicted bounding box of each sample object in each video frame. Then, the loss function is used to calculate the loss value between the detection bounding box and the prediction bounding box. When the loss value is not greater than the preset value (configured according to experience, there is no limit on this), it is determined that the initial detection model has converged, and the initial detection model is determined to be a trained target detection model; when the loss value is greater than the preset value, it is determined that the initial detection model has not converged, and the initial detection model is adjusted, and a new detection bounding box is output using the adjusted initial detection model, and based on the new detection bounding box, it is re-determined whether the adjusted initial detection model has converged, and so on. By continuously adjusting the initial detection model, the loss value between the detection bounding box and the prediction bounding box becomes smaller and smaller, that is, the performance of the initial detection model becomes better and better, until the initial detection model converges, and the performance of the initial detection model meets the requirements, the initial detection model that meets the performance requirements can be determined as a trained target detection model, and the model training process is completed.

[0036] Furthermore, in the above embodiment, step S320 may include: Step S3201, obtaining a first bounding box of each sample object in the first video frame and a second bounding box of each sample object in the second video frame in the sample video data; Step S3202: determining the moving direction and moving speed of each sample object according to the first bounding box and the second bounding box; Step S3203: predict the motion trajectory of each sample object according to the second bounding box, the motion direction and the motion speed, and obtain a predicted bounding box of each sample object in time.

[0037] In an embodiment of the present invention, a first bounding box of each sample object in a first video frame and a second bounding box in a second video frame are obtained. In the second video frame, a vector consisting of a line connecting the center point of the second bounding box to the center point of the first bounding box is obtained with the center point of the first bounding box as the origin, wherein the direction of the vector represents the direction of movement of each sample object, and the modulus of the vector represents the movement speed of each sample object. In the third video frame, the vector is translated to a position with the center point of the second bounding box as the origin to obtain a new vector, and the vertex of the new vector is the center point of the third bounding box, and the size of the third bounding box is the size of the second bounding box to obtain a third bounding box. In the fourth video frame and subsequent video frames, the position and size of the bounding box are copied based on the above steps to obtain the predicted bounding box of each sample object in time.

[0038] Furthermore, in the above embodiment, the expression of the loss function is:

[0039] in, represents the loss function, and Represent the predicted bounding box and the detected bounding box respectively, Represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average of the normalized grayscale values ​​of the predicted bounding box and the average of the normalized grayscale values ​​of the detected bounding box.

[0040] In an embodiment of the present invention, a loss function may be designed based on the overlapping area between the predicted bounding box and the detected bounding box, the distance between the center points, and the similarity to determine whether the initial detection model has converged.

[0041] Specifically, It can be expressed as:

[0042] in, represents the square of the Euclidean distance between the center point of the predicted bounding box and the center point of the detected bounding box, Represents the square of the maximum diagonal distance of the smallest enclosing box covering the predicted bounding box and the detected bounding box.

[0043] It can be expressed as:

[0044] in, Represents the average of the normalized grayscale values ​​of the predicted bounding box, Represents the average value of the normalized grayscale value of the detection bounding box. It should be noted that normalizing the grayscale value of the predicted bounding box and the grayscale value of the detection bounding box first can make subsequent data processing more convenient and faster.

[0045] Optionally, in the above embodiment, step S130 may include: Step S1301, dividing the audio data to be processed into multiple audio segments; Step S1302: extracting voiceprint vectors from each audio clip, and concatenating the voiceprint vectors into a voiceprint vector matrix; Step S1303, performing a transposition operation on the voiceprint vector matrix to obtain a voiceprint vector matrix after the transposition operation; Step S1304, performing an inner product operation on the voiceprint vector matrix and the voiceprint vector matrix after the transposition operation to obtain a similarity matrix; Step S1305: cluster the audio segments according to the similarity matrix to obtain the speech segments of each speaker.

[0046] In an embodiment of the present invention, after the audio data to be processed is segmented, each audio segment after segmentation can have only one speaker, and each audio segment contains a start and end time. In one embodiment, the audio data to be processed can be segmented into multiple audio segments according to a preset time length (configured according to experience, such as 1.0s or 1.5s, which is not limited); in another embodiment, the audio data to be processed can be segmented into multiple audio segments using an SCD (English full name: Speaker Change Detection, Chinese full name: Speaker Change Detection) model. Then, the voiceprint recognition model is used to extract voiceprint features from all audio segments to obtain voiceprint vector information (such as i-vector, x-vector, etc.) of each audio segment, and spliced ​​into a voiceprint vector matrix, wherein the horizontal axis of the voiceprint vector matrix is ​​the time length, the unit is the size of a window, and the vertical axis is the dimension of the voiceprint vector. Then, the voiceprint vector matrix is ​​transposed to obtain a voiceprint vector matrix after the transposition operation, and the inner product operation is performed on the voiceprint vector matrix and the voiceprint vector matrix after the transposition operation to obtain a similarity matrix for describing the similarity between each audio segment. Among them, in the similarity matrix, the value of each element represents the similarity between the voiceprint information of the audio clip corresponding to the index of the row and the voiceprint information of the audio clip corresponding to the index of the column. Finally, the audio clips are clustered according to the similarity matrix, and the audio clips whose similarity between the audio clips meets the set threshold (configured according to experience and not limited to this) are clustered into one category, each category corresponds to a speaker, and the voice clips corresponding to each speaker are obtained.

[0047] Furthermore, in the above embodiment, before dividing the audio data to be processed into a plurality of audio segments, the method further includes: The speech detection model is used to analyze the audio data to be processed to obtain the audio data to be processed containing the speech signal.

[0048] In the embodiment of the present invention, a voice detection model, such as a VAD (Voice activity detection) model, can be used to divide the audio data to be processed into two categories: audio data containing a voice signal and audio data not containing a voice signal, and only the audio data to be processed containing a voice signal is retained. The audio data not containing a voice signal may include silence, noise, music, etc.

[0049] Optionally, in the above embodiment, the method further includes: Taking the face of each target object as the target, target detection is performed on the video data to be processed to obtain the face image of each target object; The temporal feature description of the lip area is extracted from the face image as the lip movement feature of each target object.

[0050] In an embodiment of the present invention, when the target object is a person who is speaking, and the speaker is facing the camera and relatively close to the camera, a video frame containing the speaker can be input, and the face can be located by using a face detection algorithm (such as Haar feature + AdaBoost, CNN, etc.), and then the lip area is extracted from the face area, and the lip area image of the speaker is output, and temporal feature descriptors are extracted from the lip area image, such as optical flow features, dynamic texture features, etc., to capture the dynamic changes of the lip shape, and finally the lip movement features of the speaker are output for subsequent matching of the lip movement features with the speech segment.

[0051] Another aspect of the embodiment of the present invention provides a multi-speaker audio object localization device. The device described below may correspond to the method described above.

[0052] See also Figure 7 , the device comprises: An acquisition module 710 is used to acquire the video data to be processed and the audio data to be processed corresponding to the video data to be processed; The target position detection module 720 is used to input the video data to be processed into the trained target detection model to obtain the target position of each target object; A speech segment classification module 730 is used to classify the audio data to be processed by using a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker; A correspondence determination module 740 is used to input the lip movement features of each target object in the video data to be processed and the speech segments of each speaking object in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaking object, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels; The output module 750 is used to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object.

[0053] As can be seen from the above, the multi-speaker audio object positioning device provided by the embodiment of the present invention obtains the video data to be processed and the audio data to be processed corresponding to the video data to be processed, inputs the video data to be processed into the trained target detection model, obtains the target position of each target object, classifies the audio data to be processed by the voiceprint segmentation and clustering algorithm, obtains the speech segment of each speaking object, inputs the lip movement features of each target object in the video data to be processed and the speech segment of each speaking object in the audio data to be processed into the trained active speech detection model, obtains the correspondence between the target object and the speech segment of the speaking object, wherein the active speech detection model uses the lip movement feature set of each target object in the training video data and the speech segment set of each speaking object in the training audio data as training samples, and uses the correspondence between each target object in the training video data and the speech segment of each speaking object in the training audio data as sample labels for training, and determines the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object, and the correspondence between the target object and the speech segment of the speaking object. By adopting the solution of the present application, the accuracy of locating and tracking speaker position information in audio and video of multi-speaker scenes can be improved, and the workload of manually marking speaker position information can be reduced.

[0054] Another aspect of an embodiment of the present invention provides an electronic device, including: Memory for storing computer programs; The processor is used to implement the steps of the multi-speaker audio object localization method as described above when executing the computer program.

[0055] Another aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-speaker audio object localization method are implemented.

[0056] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. It should also be noted that in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the element.

[0057] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-speaker audio object localization method, characterized in that: include: Acquire video data to be processed and audio data to be processed corresponding to the video data to be processed; Inputting the to-be-processed video data into a trained target detection model to obtain the target position of each target object; Classifying the audio data to be processed by a voiceprint segmentation and clustering algorithm to obtain voice segments of each speaker; Inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaker, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaker in the training audio data as training samples, and using the correspondence between each target object in the training video data and the speech segment of each speaker in the training audio data as sample labels; The target position of each speaking object is determined according to the target position of the target object, the speech segment of the speaking object, and the corresponding relationship between the target object and the speech segment of the speaking object.

2. The multi-speaker audio object localization method according to claim 1, characterized in that: The target detection model is trained in the following way: Input the sample video data into the initial detection model to obtain the detection bounding box of each sample object in time; Predicting the motion trajectory of each of the sample objects in the sample video data to obtain a predicted bounding box of each of the sample objects in time; Calculating the loss value of the detection bounding box and the prediction bounding box using a loss function; Determining whether the initial detection model has converged according to the loss value; If yes, the initial detection model is determined as the trained target detection model; If not, the initial detection model is adjusted, and the process returns to the step of inputting the sample video data into the initial detection model to obtain the detection bounding box of each sample object.

3. The multi-speaker audio object localization method according to claim 2, characterized in that: The step of predicting the motion trajectory of each of the sample objects in the sample video data to obtain a predicted bounding box of each of the sample objects in time includes: Acquire a first bounding box of each of the sample objects in the sample video data in a first video frame and a second bounding box of each of the sample objects in a second video frame; Determining a moving direction and a moving speed of each of the sample objects according to the first bounding box and the second bounding box; The motion trajectory of each of the sample objects is predicted according to the second bounding box, the motion direction, and the motion speed to obtain a predicted bounding box of each of the sample objects in time.

4. The multi-speaker audio object localization method according to claim 2, characterized in that: The expression of the loss function is: ； in, represents the loss function, and represent the predicted bounding box and the detected bounding box respectively, represents the center point distance between the predicted bounding box and the detected bounding box, Represents the absolute value of the difference between the average value of the normalized grayscale value of the predicted bounding box and the average value of the normalized grayscale value of the detected bounding box.

5. The multi-speaker audio object localization method according to claim 1, characterized in that: The audio data to be processed is classified by the voiceprint segmentation and clustering algorithm to obtain the voice segments of each speaker, including: Dividing the audio data to be processed into multiple audio segments; Extracting voiceprint vectors from each of the audio clips, and concatenating the voiceprint vectors into a voiceprint vector matrix; Performing a transposition operation on the voiceprint vector matrix to obtain a voiceprint vector matrix after the transposition operation; Performing an inner product operation on the voiceprint vector matrix and the voiceprint vector matrix after the transposition operation to obtain a similarity matrix; The audio segments are clustered according to the similarity matrix to obtain speech segments of each speaker.

6. The multi-speaker audio object localization method according to claim 5, characterized in that: Before dividing the to-be-processed audio data into a plurality of audio segments, the method further includes: The audio data to be processed is analyzed using a speech detection model to obtain the audio data to be processed containing a speech signal.

7. The multi-speaker audio object localization method according to claim 1, characterized in that: The method further comprises: Taking the face of each target object as a target, performing target detection on the video data to be processed to obtain a face image of each target object; A temporal feature description of the lip region is extracted from the face image as a lip movement feature of each target object.

8. A multi-speaker audio object localization device, characterized in that: include: An acquisition module, used for acquiring video data to be processed and audio data to be processed corresponding to the video data to be processed; A target position detection module, used for inputting the to-be-processed video data into a trained target detection model to obtain the target position of each target object; A speech segment classification module is used to classify the audio data to be processed by a voiceprint segmentation and clustering algorithm to obtain speech segments of each speaker; A correspondence determination module, used for inputting the lip movement features of each target object in the video data to be processed and the speech segments of each speaker in the audio data to be processed into a trained active speech detection model to obtain the correspondence between the target object and the speech segment of the speaker, wherein the active speech detection model is trained using the lip movement feature set of each target object in the training video data and the speech segment set of each speaker in the training audio data as training samples, and the correspondence between each target object in the training video data and the speech segment of each speaker in the training audio data as sample labels; The output module is used to determine the target position of each speaking object according to the target position of the target object, the speech segment of the speaking object and the corresponding relationship between the target object and the speech segment of the speaking object.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the multi-speaker audio object localization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multi-speaker audio object localization method according to any one of claims 1 to 7 are implemented.

Images