Methods, devices, electronic equipment and storage media for detecting speech objects

Abstract

This application provides a method, apparatus, electronic device, and storage medium for detecting a speaking object, belonging to the field of artificial intelligence technology. The method includes: acquiring video and audio data of a conversation scene; obtaining a face bounding box sequence from the video data and extracting a lip bounding box sequence from the face bounding box sequence; generating a depth information sequence corresponding to the face bounding box sequence based on a depth estimation model, and obtaining an audio sequence corresponding to the face bounding box sequence based on the audio data; obtaining audio embedding features and video embedding features based on the audio sequence and lip bounding box sequence, respectively; weightedly fusing the depth information sequence with the audio embedding features and video embedding features, respectively; and determining the target speaking object in the conversation scene based on the obtained deep-fused audio features and deep-fused video features. The provided solution can automatically locate the speaking object in a conversation video, improving work efficiency and reducing workload.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting a speaking object. Background Technology

[0002] Currently, in video post-editing, it is often necessary to accurately locate the person speaking in the video in order to perform tasks such as subtitle matching, audio and video content alignment, and sound effect optimization. Especially in multi-person dialogues or interview scenarios, people frequently take turns speaking, and existing technology relies on manually marking video speakers, which is not only labor-intensive but also inefficient. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a method, apparatus, electronic device, and storage medium for detecting a speaking object.

[0004] In a first aspect, embodiments of this application provide a method for detecting a speaking object, the method comprising:

[0005] Acquire video and audio data from the conversation scene;

[0006] A face bounding box sequence is obtained from the video data, and a lip bounding box sequence is extracted from the face bounding box sequence.

[0007] A depth information sequence corresponding to the face bounding box sequence is generated based on the depth estimation model, and an audio sequence corresponding to the face bounding box sequence is obtained based on the audio data;

[0008] Audio embedding features and video embedding features are obtained based on the audio sequence and the lip outline sequence, respectively.

[0009] The deep information sequence is weighted and fused with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features;

[0010] The target speaker in the conversation scene is determined based on the deep fusion audio features and the deep fusion video features.

[0011] In one embodiment, a face bounding box sequence and facial key point coordinates in the video data are obtained according to a face detection algorithm;

[0012] The key points of the lip region are determined based on the face frame sequence and the coordinates of the facial key points;

[0013] The minimum bounding rectangle is obtained based on the key points of the lip region, and the minimum bounding rectangle is used as the lip frame sequence.

[0014] In one embodiment, the face detection algorithm sequentially acquires multiple faces from the video data frame by frame, and outputs the bounding box coordinates of each face.

[0015] The tracking algorithm obtains the coordinates of multiple face bounding boxes of the same face in each video frame, associates the coordinates of each face bounding box, and generates the face box sequence based on the associated face bounding box coordinates.

[0016] In one embodiment, the audio sequence is encoded using an audio timing encoder, and the encoded audio sequence is used as the audio embedding feature;

[0017] The lip box sequence is encoded using a visual temporal encoder, and the encoded lip box sequence is used as the video embedding feature.

[0018] In one embodiment, the deep information sequence and the audio embedding features are weighted and fused according to a first weighting formula to obtain the deep fused audio features;

[0019] The deep information sequence and the video embedding features are weighted and fused according to the second weighting formula to obtain the deep fused video features.

[0020] In one embodiment, the deep fusion video features and the depth information sequence are associated using a cross-attention module;

[0021] The target speaker in the conversation scenario is determined based on the association results.

[0022] In one embodiment, the detection score of each speaking object in the conversation scenario is determined by the association results;

[0023] Among the various detection scores, obtain the target detection score that is greater than the preset score threshold;

[0024] The speaking object corresponding to the target detection score is taken as the target speaking object.

[0025] Secondly, embodiments of this application provide a speaking object detection device, the speaking object detection device comprising:

[0026] The judgment module is used to obtain video and audio data of the conversation scenario;

[0027] A connection module is used to obtain a face bounding box sequence based on the video data, and extract a lip bounding box sequence from the face bounding box sequence;

[0028] The first transmission module is used to generate a depth information sequence corresponding to the face bounding box sequence according to the depth estimation model, and to obtain an audio sequence corresponding to the face bounding box sequence according to the audio data;

[0029] The selection module is used to obtain audio embedding features and video embedding features based on the audio sequence and the lip box sequence, respectively.

[0030] The second transmission module is used to perform weighted fusion of the depth information sequence with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features respectively.

[0031] The acquisition module is used to determine the target speaker in the conversation scene based on the deep fusion audio features and the deep fusion video features.

[0032] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the computer program executes the speaking object detection method provided in the first aspect when the processor is running.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a processor, executes the speaking object detection method provided in the first aspect.

[0034] The speaking object detection method provided in this application involves acquiring video and audio data of a conversation scene; obtaining a face bounding box sequence from the video data and extracting a lip bounding box sequence from the face bounding box sequence; generating a depth information sequence corresponding to the face bounding box sequence based on a depth estimation model, and acquiring an audio sequence corresponding to the face bounding box sequence based on the audio data; acquiring audio embedding features and video embedding features based on the audio sequence and the lip bounding box sequence, respectively; weightedly fusing the depth information sequence with the audio embedding features and the video embedding features to obtain deep fused audio features and deep fused video features, respectively; and determining the target speaking object in the conversation scene based on the deep fused audio features and deep fused video features. This application improves the accuracy of speaking object judgment and reduces false positives by acquiring the depth information sequence, audio embedding features, and video embedding features corresponding to the face bounding box sequence, and weightedly fusing the audio embedding features and video embedding features with the depth information sequence to obtain the target speaking object in the conversation scene. Furthermore, by automatically identifying the speaking object, it improves work efficiency. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation on the scope of protection of this application. In the various drawings, similar components are numbered similarly.

[0036] Figure 1 A flowchart of a speaking object detection method provided in an embodiment of this application is shown;

[0037] Figure 2 Another flowchart of the speaking object detection method provided in this application embodiment is shown;

[0038] Figure 3 Another schematic diagram of the speaking object detection device provided in an embodiment of this application is shown;

[0039] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0040] Icons: 300-Speaking Object Detection Device, 301-Judgment Module, 302-Connection Module, 303-First Transmission Module, 304-Selection Module, 305-Second Transmission Module, 306-Acquisition Module, 400-Electronic Device, 401-Transceiver, 402-Processor, 403-Memory. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0042] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0043] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0044] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0045] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0046] In recent years, most methods for identifying the speaker using AI technology utilize video lip-reading and audio features. The biggest drawback of these methods is that they do not provide distance information for the speaker in the frame, which leads to two problems: First, when there are multiple people in the frame, false detection is easy to occur, and relying solely on video lip-reading and audio features for identification can easily cause confusion and make it difficult to accurately identify the speaker; second, it cannot automatically focus on the main speaker and is easily affected by background speakers, leading to misjudgment. To solve these problems, this application provides a method for detecting the speaker.

[0047] Example 1

[0048] This application provides a method for detecting a speaking object.

[0049] See Figure 1 The methods for detecting the speaker include:

[0050] S101, acquire video and audio data of the conversation scene.

[0051] S102, obtain a face bounding box sequence based on the video data, and extract a lip bounding box sequence from the face bounding box sequence.

[0052] In one embodiment, the face detection algorithm sequentially acquires multiple faces in the video data according to video frames, and outputs the face bounding box coordinates corresponding to each face; the tracking algorithm acquires multiple face bounding box coordinates of the same face in each video frame, associates the face bounding box coordinates, and generates the face box sequence based on the associated face bounding box coordinates.

[0053] It should be noted that this application uses a face detection algorithm to detect and track faces appearing in each frame of video data, outputs the coordinates of the face bounding box and the coordinates of facial key points in each video frame, and determines the face box sequence through the face bounding box coordinates.

[0054] For example: Suppose that a face bounding box is detected in frame t with coordinates (x1, y1, w1, h1), and the face moves to (x2, y2, w2, h2) in frame t+1. Then the face bounding box sequence is [(x1, y1, w1, h1), (x2, y2, w2, h2)].

[0055] In one implementation, such as Figure 2 As shown, in step S1021, the face bounding box sequence and facial key point coordinates in the video data are obtained according to the face detection algorithm; in step S1022, the key points of the lip region are determined according to the face bounding box sequence and the facial key point coordinates; in step S1023, the minimum bounding rectangle is obtained according to the key points of the lip region, and the minimum bounding rectangle is used as the lip bounding box sequence.

[0056] It should be noted that, based on the obtained face bounding box sequence and the coordinates of facial key points, the key points of the lip region of each face are extracted. The smallest bounding rectangle can be obtained as the lip bounding box according to each key point, and then the lip bounding box sequence is formed.

[0057] For example, if the face bounding box sequence is [(x1,y1,w1,h1),(x2,y2,w2,h2)], where the coordinates of the smallest bounding rectangle obtained are (x1',y1',w1',h1') and (x2',y2',w2',h2'), then the corresponding lip bounding box sequence is [(x1',y1',w1',h1'),(x2',y2',w2',h2')].

[0058] This application uses a face detection algorithm to sequentially acquire multiple faces from video data frame by frame and outputs the coordinates of the face bounding boxes, enabling precise location of the specific position and range of faces in each frame. This ensures that data processing focuses on the truly face-related parts, improving the overall targeting and accuracy of the processing.

[0059] S103, Generate a depth information sequence corresponding to the face bounding box sequence based on the depth estimation model, and obtain an audio sequence corresponding to the face bounding box sequence based on the audio data.

[0060] In this embodiment, a depth map of each video frame is estimated by using a depth estimation model, and the distance between each pixel in the video frame and the camera is obtained.

[0061] It should be noted that, based on the depth map of each video frame and combined with the obtained face bounding box sequence, the depth value of the same face region in each video frame is extracted, the average of multiple depth values ​​is calculated, and the average value is used as the depth value of the face.

[0062] For example, assuming the face bounding box sequence is [(x1,y1,w1,h1),(x2,y2,w2,h2)], and the average depth of the face bounding box region of the same face in each video frame is d1,d2, then the depth sequence of the face is [d1,d2].

[0063] In this embodiment, the average depth of the faces in each video frame is extracted sequentially according to the time order of the video frames, ultimately generating a depth information sequence corresponding to the face bounding box sequence. This depth information sequence records the depth of the faces at different times, reflecting the changes in the distance between the faces and the camera in the conversation scene.

[0064] To further explain, by acquiring the timeline of the audio data, the audio data is segmented and matched according to the timeline information of the video frames. For example, the time range of each face bounding box in the face bounding box sequence is determined, and then the corresponding audio segments within this time period are extracted from the audio data. These audio segments arranged in chronological order are then integrated to form an audio sequence corresponding to the face bounding box sequence.

[0065] S104, obtain audio embedding features and video embedding features based on the audio sequence and the lip box sequence, respectively.

[0066] In one embodiment, the audio sequence is encoded using an audio temporal encoder, and the encoded audio sequence is used as the audio embedding feature; the lip box sequence is encoded using a visual temporal encoder, and the encoded lip box sequence is used as the video embedding feature.

[0067] Understandably, by encoding the audio sequence and the lip outline sequence, redundant information in the sequence is filtered out, and the complex raw data is encoded into embedded features, which facilitates subsequent processing and analysis.

[0068] S105, the deep information sequence is weighted and fused with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features respectively.

[0069] In one embodiment, the depth information sequence and the audio embedding features are weighted and fused according to a first weighting formula to obtain the deep fused audio features; the depth information sequence and the video embedding features are weighted and fused according to a second weighting formula to obtain the deep fused video features.

[0070] It should be noted that by adaptively weighting and fusing the acquired deep information sequence with video embedding features and audio embedding features, deep fused video features and deep fused audio features are obtained.

[0071] To further clarify, the first weighted formula is:

[0072]

[0073] in, To deeply integrate audio features, w i As a weighting factor, For audio embedding features.

[0074] The second weighting formula is:

[0075]

[0076] in, To deeply integrate video features, w i As a weighting factor, is the video embedding feature of the i-th lip box in the lip box sequence.

[0077] Understandably, the formula for calculating the weighting factor is:

[0078]

[0079] Among them, w i d is a weighting factor, ε is a constant, and d i It is the depth information sequence corresponding to the face bounding box.

[0080] This application uses a weighted fusion method to dynamically adjust the weighting factor based on the distance of the speaker, adaptively enhancing the features of nearby speakers and weakening the features of distant background speakers. In multi-person scenes or complex backgrounds, it increases the focus on the target speaker, helps reduce interference and false detections, and increases the accuracy of speaker identification.

[0081] S106, determine the target speaker in the conversation scene based on the deep fusion audio features and the deep fusion video features.

[0082] In one embodiment, the deep fusion video features and the deep information sequence are associated using a cross-attention module; the target speaker in the conversation scene is determined based on the association result.

[0083] It should be noted that the cross-attention module associates the deep fusion video features and the deep fusion audio features. By learning the potential relationship between the two, a detection score, i.e., a speaking probability score, is output for each person in the video. People with scores greater than a preset threshold are identified as the target speakers.

[0084] In one embodiment, the detection scores of each speaking object in the conversation scenario are determined through the association results; a target detection score greater than a preset score threshold is obtained from each detection score; and the speaking object corresponding to the target detection score is taken as the target speaking object.

[0085] In this embodiment, the cross-attention module is used to learn the dependencies between different modal features. Specifically, it learns deeply fused video and audio feature modalities to output a score for each person, uses the score to determine whether the person is speaking, and sets a score threshold, designating speakers with scores higher than the threshold as target speakers. The score threshold can be set according to actual conditions and is not limited in this application.

[0086] The speaking object detection method provided in this embodiment obtains the depth information sequence, audio embedding features, and video embedding features corresponding to the face bounding box sequence, and then performs weighted fusion of the audio embedding features and video embedding features with the depth information sequence to obtain the target speaking object in the conversation scene. This improves the accuracy of judging the speaking object, reduces false judgments, and further improves work efficiency by automatically identifying the speaking object.

[0087] Example 2

[0088] Furthermore, embodiments of this application provide a speaking object detection device.

[0089] like Figure 3 As shown, the speaking target detection device 300 includes:

[0090] The judgment module 301 is used to obtain video and audio data of the conversation scenario;

[0091] Connection module 302 is used to obtain a face bounding box sequence based on the video data, and extract a lip bounding box sequence from the face bounding box sequence;

[0092] The first transmission module 303 is used to generate a depth information sequence corresponding to the face bounding box sequence according to the depth estimation model, and to obtain an audio sequence corresponding to the face bounding box sequence according to the audio data;

[0093] Selection module 304 is used to obtain audio embedding features and video embedding features based on the audio sequence and the lip box sequence, respectively.

[0094] The second transmission module 305 is used to perform weighted fusion of the depth information sequence with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features respectively.

[0095] The acquisition module 306 is used to determine the target speaker in the conversation scene based on the deep fusion audio features and the deep fusion video features.

[0096] The speaking object detection device 300 provided in this embodiment can implement the speaking object detection method provided in Embodiment 1. To avoid repetition, it will not be described again here.

[0097] The speaking object detection device provided in this embodiment obtains the depth information sequence, audio embedding features, and video embedding features corresponding to the face bounding box sequence, and performs weighted fusion of the audio embedding features and video embedding features with the depth information sequence to obtain the target speaking object in the conversation scene. This improves the accuracy of judging the speaking object, reduces false judgments, and further improves work efficiency by automatically identifying the speaking object.

[0098] Example 3

[0099] Furthermore, this application provides an electronic device, including a memory and a processor. The memory stores a computer program, which executes the speaking object detection method provided in Embodiment 1 when the computer program is run on the processor.

[0100] For details, see Figure 4 The electronic device 400 includes a transceiver 401, a bus interface, and a processor 402. The processor 402 is configured to: acquire video data and audio data of a conversation scene; acquire a face bounding box sequence based on the video data, and extract a lip bounding box sequence from the face bounding box sequence; generate a depth information sequence corresponding to the face bounding box sequence based on a depth estimation model, and acquire an audio sequence corresponding to the face bounding box sequence based on the audio data; acquire audio embedding features and video embedding features based on the audio sequence and the lip bounding box sequence, respectively; perform weighted fusion of the depth information sequence with the audio embedding features and the video embedding features, respectively, to obtain deep fused audio features and deep fused video features; and determine the target speaker in the conversation scene based on the deep fused audio features and the deep fused video features.

[0101] In one embodiment, the processor 402 is further configured to: obtain a face bounding box sequence and facial key point coordinates in the video data according to a face detection algorithm; determine lip region key points according to the face bounding box sequence and the facial key point coordinates; obtain a minimum bounding rectangle according to the lip region key points, and use the minimum bounding rectangle as the lip bounding box sequence.

[0102] In one embodiment, the processor 402 is further configured to: sequentially acquire multiple faces in the video data according to the video frames using the face detection algorithm, and output the face bounding box coordinates corresponding to each face; acquire multiple face bounding box coordinates of the same face in each video frame according to the tracking algorithm, associate the face bounding box coordinates, and generate the face box sequence according to the associated face bounding box coordinates.

[0103] In one embodiment, the processor 402 is further configured to: encode the audio sequence using an audio temporal encoder, and use the encoded audio sequence as the audio embedding feature; and encode the lip box sequence using a visual temporal encoder, and use the encoded lip box sequence as the video embedding feature.

[0104] In one embodiment, the processor 402 is further configured to: perform weighted fusion of the depth information sequence and the audio embedding features according to a first weighting formula to obtain the deep fused audio features; and perform weighted fusion of the depth information sequence and the video embedding features according to a second weighting formula to obtain the deep fused video features.

[0105] In one embodiment, the processor 402 is further configured to: associate the deep fusion video features and the depth information sequence through a cross-attention module; and determine the target speaker in the conversation scene based on the association result.

[0106] In one embodiment, the processor 402 is further configured to: determine the detection score of each speaking object in the conversation scenario through the association result; obtain a target detection score greater than a preset score threshold from each of the detection scores; and use the speaking object corresponding to the target detection score as the target speaking object.

[0107] In this embodiment of the application, the electronic device 400 further includes a memory 403. Figure 4 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 402 and memory represented by memory 403. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 401 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. Processor 402 is responsible for managing the bus architecture and general processing, and memory 403 can store data used by processor 402 during operation.

[0108] The electronic device 400 provided in this application embodiment can execute the steps of the speaking object detection method provided in the above method embodiment 1. To avoid repetition, it will not be described again here.

[0109] The electronic device provided in this embodiment obtains the depth information sequence, audio embedding features, and video embedding features corresponding to the face bounding box sequence, and performs weighted fusion of the audio embedding features and video embedding features with the depth information sequence to obtain the target speaking object in the conversation scene. This improves the accuracy of judging the speaking object and reduces false judgments. By automatically recognizing the speaking object, it further improves work efficiency.

[0110] Example 4

[0111] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speaking object detection method provided in Embodiment 1.

[0112] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0113] The computer-readable storage medium provided in this embodiment can implement the speaking object detection method provided in Embodiment 1. To avoid repetition, it will not be described again here.

[0114] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0116] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for detecting a speaker, characterized in that, The method includes: Acquire video and audio data from the conversation scene; A face bounding box sequence is obtained from the video data, and a lip bounding box sequence is extracted from the face bounding box sequence. A depth information sequence corresponding to the face bounding box sequence is generated based on the depth estimation model, and an audio sequence corresponding to the face bounding box sequence is obtained based on the audio data; Audio embedding features and video embedding features are obtained based on the audio sequence and the lip outline sequence, respectively. The deep information sequence is weighted and fused with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features; The target speaker in the conversation scene is determined based on the deep fusion audio features and the deep fusion video features; By using the depth estimation model, the depth map of each video frame is estimated. Based on the depth map of each video frame and combined with the obtained face bounding box sequence, the depth value of the same face region in each video frame is extracted. The average of multiple depth values ​​is calculated and the average value is used as the depth value of the face. The depth average value is extracted for each face in each video frame in chronological order, and finally a depth information sequence corresponding to the face bounding box sequence is generated.

2. The speaking object detection method according to claim 1, characterized in that, The step of obtaining a face bounding box sequence based on the video data and extracting a lip bounding box sequence from the face bounding box sequence includes: The face bounding box sequence and facial key point coordinates in the video data are obtained based on the face detection algorithm; The key points of the lip region are determined based on the face frame sequence and the coordinates of the facial key points; The minimum bounding rectangle is obtained based on the key points of the lip region, and the minimum bounding rectangle is used as the lip frame sequence.

3. The speaking object detection method according to claim 2, characterized in that, The step of obtaining the face bounding box sequence from the video data according to the face detection algorithm includes: The face detection algorithm sequentially acquires multiple faces from the video data frame by frame and outputs the bounding box coordinates of each face. The tracking algorithm obtains the coordinates of multiple face bounding boxes of the same face in each video frame, associates the coordinates of each face bounding box, and generates the face box sequence based on the associated face bounding box coordinates.

4. The method for detecting a speaking object according to claim 1, characterized in that, The step of obtaining audio embedding features and video embedding features based on the audio sequence and the lip outline sequence respectively includes: The audio sequence is encoded using an audio temporal encoder, and the encoded audio sequence is used as the audio embedding feature. The lip box sequence is encoded using a visual temporal encoder, and the encoded lip box sequence is used as the video embedding feature.

5. The method for detecting a speaking object according to claim 1, characterized in that, The step of weightedly fusing the depth information sequence with the audio embedding feature and the video embedding feature to obtain deep fused audio features and deep fused video features respectively includes: The deep information sequence and the audio embedding features are weighted and fused according to the first weighting formula to obtain the deep fused audio features; The deep information sequence and the video embedding features are weighted and fused according to the second weighting formula to obtain the deep fused video features.

6. The method for detecting a speaking object according to claim 1, characterized in that, Determining the target speaker in the conversation scene based on the deep fusion audio features and the deep fusion video features includes: The deep fusion video features and the depth information sequence are associated through a cross-attention module; The target speaker in the conversation scenario is determined based on the association results.

7. The method for detecting a speaking object according to claim 6, characterized in that, Determining the target speaker in the conversation scenario based on the association results includes: The detection scores of each speaking object in the conversation scenario are determined based on the correlation results. Among the various detection scores, obtain the target detection score that is greater than the preset score threshold; The speaking object corresponding to the target detection score is taken as the target speaking object.

8. A device for detecting a speaking target, characterized in that, The device includes: The judgment module is used to obtain video and audio data of the conversation scenario; A connection module is used to obtain a face bounding box sequence based on the video data, and extract a lip bounding box sequence from the face bounding box sequence; The first transmission module is used to generate a depth information sequence corresponding to the face bounding box sequence according to the depth estimation model, and to obtain an audio sequence corresponding to the face bounding box sequence according to the audio data; by using the depth estimation model, the depth map of each video frame is estimated, and based on the depth map of each video frame and the obtained face bounding box sequence, the depth value of the same face region in each video frame is extracted, the average of multiple depth values ​​is calculated, and the average value is used as the depth value of the face. The depth average value is extracted for the faces in each video frame one by one according to the time order of the video frames, and finally a depth information sequence corresponding to the face bounding box sequence is generated. The selection module is used to obtain audio embedding features and video embedding features based on the audio sequence and the lip box sequence, respectively. The second transmission module is used to perform weighted fusion of the depth information sequence with the audio embedding feature and the video embedding feature respectively to obtain deep fused audio features and deep fused video features respectively. The acquisition module is used to determine the target speaker in the conversation scene based on the deep fusion audio features and the deep fusion video features.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that executes the speaking object detection method according to any one of claims 1 to 7 when the processor is running.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when run on a processor, executes the speaking object detection method according to any one of claims 1 to 7.

Images