Speaking object detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speaking object detection method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring video data and audio data of a session scene; acquiring a face frame sequence according to the video data, and extracting a lip frame sequence from the face frame sequence; generating a depth information sequence corresponding to the face frame sequence according to the depth estimation model, and obtaining an audio sequence corresponding to the face frame sequence according to the audio data; according to the audio sequence and the lip frame sequence, respectively obtaining audio embedding features and video embedding features; and respectively carrying out weighted fusion on the depth information sequence and the audio embedding feature and the video embedding feature, and determining a target speaking object in the session scene according to the obtained depth fusion audio feature and the depth fusion video feature. Through the provided scheme, the speaking object in the session video can be automatically positioned, the working efficiency is improved, and the workload is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for detecting a speaking object. Background Art

[0002] At present, in video post-editing, it is often necessary to accurately find the person who is speaking in the video in order to match subtitles, align audio and video content, optimize sound effects, etc. Especially in multi-person conversations or interviews, people will frequently speak alternately, and the existing technology is to manually mark the speakers in the video, which is not only labor-intensive but also inefficient. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application provide a speaking object detection method, device, electronic device and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting a speaking object, the method comprising:

[0005] Obtain video data and audio data of the conversation scene;

[0006] Acquire a face frame sequence according to the video data, and extract a lip frame sequence from the face frame sequence;

[0007] Generate a depth information sequence corresponding to the face frame sequence according to the depth estimation model, and acquire an audio sequence corresponding to the face frame sequence according to the audio data;

[0008] Acquire audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively;

[0009] The depth information sequence is weightedly fused with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively;

[0010] A target speaker in the conversation scenario is determined according to the deeply fused audio features and the deeply fused video features.

[0011] In one embodiment, a face frame sequence and facial key point coordinates in the video data are obtained according to a face detection algorithm;

[0012] Determine the key points of the lip region according to the face frame sequence and the coordinates of the facial key points;

[0013] A minimum bounding rectangle is obtained according to the key points of the lip region, and the minimum bounding rectangle is used as the lip frame sequence.

[0014] In one embodiment, the face detection algorithm is used to sequentially obtain multiple faces in the video data according to video frames, and the coordinates of the face bounding boxes corresponding to the faces are output;

[0015] A plurality of face bounding box coordinates of the same face in each of the video frames are obtained according to a tracking algorithm, the face bounding box coordinates are associated, and the face frame sequence is generated according to the associated face bounding box coordinates.

[0016] In one implementation, the audio sequence is encoded by an audio timing encoder, and the encoded audio sequence is used as the audio embedding feature;

[0017] The lip frame sequence is encoded by a visual temporal encoder, and the encoded lip frame sequence is used as the video embedding feature.

[0018] In one embodiment, the depth information sequence is weightedly fused with the audio embedding feature according to a first weighting formula to obtain the deep fused audio feature;

[0019] The depth information sequence is weightedly fused with the video embedding feature according to a second weighted formula to obtain the depth fusion video feature.

[0020] In one embodiment, the deep fusion video features and the depth information sequence are associated by a cross attention module;

[0021] The target speaking object in the conversation scenario is determined according to the association result.

[0022] In one embodiment, a detection score of each speaker in the conversation scenario is determined according to the association result;

[0023] Obtaining a target detection score greater than a preset score threshold from each of the detection scores;

[0024] The speaking object corresponding to the object detection score is used as the target speaking object.

[0025] In a second aspect, an embodiment of the present application provides a speaking object detection device, the speaking object detection device comprising:

[0026] A judgment module, used to obtain video data and audio data of the conversation scene;

[0027] A connection module, used for acquiring a face frame sequence according to the video data, and extracting a lip frame sequence from the face frame sequence;

[0028] A first transmission module, configured to generate a depth information sequence corresponding to the face frame sequence according to a depth estimation model, and to acquire an audio sequence corresponding to the face frame sequence according to the audio data;

[0029] A selection module, configured to obtain audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively;

[0030] A second transmission module is used to perform weighted fusion on the depth information sequence with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively;

[0031] An acquisition module is used to determine a target speaking object in the conversation scenario according to the deep fusion audio feature and the deep fusion video feature.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method for detecting a speaking object provided in the first aspect is executed.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed on a processor, the method for detecting a speaking object provided in the first aspect is executed.

[0034] The speaking object detection method provided by the above-mentioned application obtains video data and audio data of the conversation scene; obtains a face frame sequence according to the video data, and extracts a lip frame sequence from the face frame sequence; generates a depth information sequence corresponding to the face frame sequence according to the depth estimation model, and obtains an audio sequence corresponding to the face frame sequence according to the audio data; obtains audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively; weightedly fuses the depth information sequence with the audio embedding features and the video embedding features respectively to obtain deep fusion audio features and deep fusion video features respectively; determines the target speaking object in the conversation scene according to the deep fusion audio features and the deep fusion video features. The present application obtains the target speaking object in the conversation scene by obtaining the depth information sequence, audio embedding features and video embedding features corresponding to the face frame sequence, and weightedly fuses the audio embedding features and the video embedding features with the depth information sequence, thereby improving the judgment accuracy of the speaking object, reducing misjudgment, and further improving work efficiency by automatically identifying the speaking object. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope of protection of the present application. In each of the drawings, similar components are numbered similarly.

[0036] Figure 1 A schematic diagram of a flow chart of a method for detecting a speaking object provided in an embodiment of the present application is shown;

[0037] Figure 2 Another schematic diagram of the process of the speaking object detection method provided by the embodiment of the present application is shown;

[0038] Figure 3 Another structural schematic diagram of the speaking object detection device provided in an embodiment of the present application is shown;

[0039] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0040] Icons: 300 - speaking object detection device, 301 - judging module, 302 - connecting module, 303 - first transmission module, 304 - selecting module, 305 - second transmission module, 306 - acquiring module, 400 - electronic device, 401 - transceiver, 402 - processor, 403 - memory. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0042] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0043] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0044] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0045] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0046] In recent years, most methods of judging speaking objects through AI technology use video lip shape features and audio features. The biggest drawback is that it does not provide distance information of the speaking object in the picture, which causes two problems: first, it is easy to misdetect when there are multiple people in the picture, and it is easy to cause confusion and difficult to accurately identify the speaking object by relying solely on video lip shape features and audio features; second, it is impossible to automatically focus on the main speaking object, and it is easy to be misjudged due to interference from background speakers. In order to solve the above problems, this application provides a speaking object detection method.

[0047] Example 1

[0048] An embodiment of the present application provides a method for detecting a speaking object.

[0049] See also Figure 1 , speaking object detection methods include:

[0050] S101, obtaining video data and audio data of a conversation scene.

[0051] S102, acquiring a face frame sequence according to the video data, and extracting a lip frame sequence from the face frame sequence.

[0052] In one embodiment, the face detection algorithm is used to sequentially obtain multiple faces in the video data according to the video frames, and the face bounding box coordinates corresponding to each face are output; the multiple face bounding box coordinates of the same face in each video frame are obtained according to the tracking algorithm, the face bounding box coordinates are associated, and the face frame sequence is generated according to the associated face bounding box coordinates.

[0053] It should be noted that the present application performs face detection and tracking on the faces appearing in each frame of the video data through a face detection algorithm, outputs the face bounding box coordinates and facial key point coordinates in each video frame, and determines the face frame sequence through the face bounding box coordinates.

[0054] For example, suppose a face bounding box with coordinates (x1, y1, w1, h1) is detected in the tth frame, and the face moves to (x2, y2, w2, h2) in the t+1th frame, then the face box sequence is [(x1, y1, w1, h1), (x2, y2, w2, h2)].

[0055] In one embodiment, if Figure 2 As shown, S1021, a face frame sequence and facial key point coordinates in the video data are obtained according to a face detection algorithm; S1022, lip area key points are determined according to the face frame sequence and the facial key point coordinates; S1023, a minimum bounding rectangle is obtained according to the lip area key points, and the minimum bounding rectangle is used as the lip frame sequence.

[0056] It should be noted that based on the obtained face frame sequence and the coordinates of the facial key points, the key points of the lip area of ​​each face are extracted, and the minimum circumscribed rectangle can be obtained as the lip frame according to each key point, and then a lip frame sequence is formed.

[0057] For example, the face frame sequence is [(x1, y1, w1, h1), (x2, y2, w2, h2)], where the minimum bounding rectangle coordinates obtained are (x1', y1', w1', h1'), (x2', y2', w2', h2') respectively, then the lip frame sequence corresponding to the face frame sequence is [(x1', y1', w1', h1'), (x2', y2', w2', h2')].

[0058] This application uses a face detection algorithm to obtain multiple faces in the video data in sequence according to the video frame, and outputs the coordinates of the face boundary box, which can accurately locate the specific position and range of the face in each frame. This ensures that the data processing focuses on the part that is truly related to the face, and improves the pertinence and accuracy of the overall processing.

[0059] S103, generating a depth information sequence corresponding to the face frame sequence according to a depth estimation model, and acquiring an audio sequence corresponding to the face frame sequence according to the audio data.

[0060] In this embodiment, a depth estimation model is used to estimate the depth map of each video frame, and the distance between each pixel in the video frame and the camera is obtained.

[0061] It should be noted that based on the depth map of each video frame, combined with the acquired face frame sequence, the depth value of the same face area in each video frame is extracted, the multiple depth values ​​are averaged, and the average value is used as the depth value of the face.

[0062] For example, assuming that the face frame sequence is [(x1, y1, w1, h1), (x2, y2, w2, h2)], and the average depth of the face frame area of ​​the same face in each video frame is d1, d2, then the depth sequence of the face is [d1, d2].

[0063] In this embodiment, according to the time sequence of the video frames, the depth average value is extracted for the faces in each video frame one by one, and finally a depth information sequence corresponding to the face frame sequence is generated. The depth information sequence records the depth of the face at different times, reflecting the distance change information of the face relative to the camera in the conversation scene.

[0064] It is further explained that by obtaining the time axis of the audio data, the audio data is segmented and matched according to the time axis information of the video frame. For example, the time range of each face frame in the face frame sequence is determined, and then the audio clips corresponding to this time period are extracted from the audio data, and these audio clips arranged in time sequence are integrated to form an audio sequence corresponding to the face frame sequence.

[0065] S104: Acquire audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively.

[0066] In one embodiment, the audio sequence is encoded by an audio timing encoder, and the encoded audio sequence is used as the audio embedding feature; the lip frame sequence is encoded by a visual timing encoder, and the encoded lip frame sequence is used as the video embedding feature.

[0067] It can be understood that by encoding the audio sequence and the lip frame sequence, redundant information in the sequence is filtered, and the complex original data is encoded as embedded features, which is convenient for subsequent processing and analysis.

[0068] S105, performing weighted fusion on the depth information sequence with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively.

[0069] In one embodiment, the depth information sequence and the audio embedding feature are weightedly fused according to a first weighting formula to obtain the deep fusion audio feature; the depth information sequence and the video embedding feature are weightedly fused according to a second weighting formula to obtain the deep fusion video feature.

[0070] It should be noted that by adaptively performing weighted fusion on the acquired depth information sequence with the video embedding features and the audio embedding features, deep fusion video features and deep fusion audio features are obtained.

[0071] It is further explained that the first weighted formula is:

[0072]

[0073] in, To deeply fuse audio features, w i is the weighting factor, It is the audio embedding feature.

[0074] The second weighted formula is:

[0075]

[0076] in, is the deep fusion video feature, w i is the weighting factor, is the video embedding feature of the i-th lip frame in the lip frame sequence.

[0077] It can be understood that the weighting factor is calculated as:

[0078]

[0079] Among them, w i is the weighting factor, ε is a constant, d i It is the depth information sequence corresponding to the face frame.

[0080] The present application can dynamically adjust the size of the weighting factor according to the distance of the person through weighted fusion, adaptively enhance the characteristics of nearby people, and weaken the characteristics of distant background people. In the case of multi-person scenes or complex backgrounds, it improves the focus on the target speaking object, helps to reduce interference and false detection, and increases the accuracy of judging the speaking object.

[0081] S106: Determine a target speaker in the conversation scenario according to the deeply fused audio features and the deeply fused video features.

[0082] In one embodiment, the deep fusion video features and the depth information sequence are associated through a cross-attention module; and the target speaking object in the conversation scenario is determined based on the association result.

[0083] It should be noted that the deep fusion video features and the deep fusion audio features are associated through the cross-attention module. By learning the potential relationship between the two, a detection score, i.e., the speaking possibility score, is output for each person in the video, and those whose scores are greater than the preset threshold are regarded as target speakers.

[0084] In one embodiment, the detection score of each speaking object in the conversation scenario is determined by the association result; a target detection score greater than a preset score threshold is obtained from each detection score; and the speaking object corresponding to the target detection score is used as the target speaking object.

[0085] In this embodiment, the cross attention module is used to learn the dependencies between different modal features. That is, by learning the deep fusion video feature modality and the deep fusion audio feature modality, a score is output for each person, and the score is used to determine whether the person is speaking. At the same time, a score threshold is set, and the corresponding speakers greater than the score threshold are regarded as target speakers. The set score threshold can be set according to the actual situation, and this application does not limit it here.

[0086] The speaking object detection method provided in this embodiment obtains the depth information sequence, audio embedding features and video embedding features corresponding to the face frame sequence, and weightedly fuses the audio embedding features and the video embedding features with the depth information sequence to obtain the target speaking object in the conversation scenario, thereby improving the judgment accuracy of the speaking object, reducing the misjudgment rate, and further improving work efficiency by automatically identifying the speaking object.

[0087] Example 2

[0088] In addition, an embodiment of the present application provides a speaking object detection device.

[0089] like Figure 3 As shown, the speaking object detection device 300 includes:

[0090] The judgment module 301 is used to obtain video data and audio data of the conversation scene;

[0091] A connection module 302 is used to obtain a face frame sequence according to the video data, and extract a lip frame sequence from the face frame sequence;

[0092] A first transmission module 303 is used to generate a depth information sequence corresponding to the face frame sequence according to a depth estimation model, and to obtain an audio sequence corresponding to the face frame sequence according to the audio data;

[0093] A selection module 304 is used to obtain audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively;

[0094] The second transmission module 305 is used to perform weighted fusion on the depth information sequence with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively;

[0095] The acquisition module 306 is used to determine the target speaking object in the conversation scenario according to the deep fusion audio feature and the deep fusion video feature.

[0096] The speaking object detection apparatus 300 provided in this embodiment can implement the speaking object detection method provided in Embodiment 1, and will not be described again to avoid repetition.

[0097] The speaking object detection device provided in this embodiment obtains the depth information sequence, audio embedding features and video embedding features corresponding to the face frame sequence, and weightedly fuses the audio embedding features and the video embedding features with the depth information sequence to obtain the target speaking object in the conversation scenario, thereby improving the judgment accuracy of the speaking object, reducing the misjudgment rate, and further improving work efficiency by automatically identifying the speaking object.

[0098] Example 3

[0099] In addition, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed on the processor, the speaking object detection method provided in Embodiment 1 is executed.

[0100] For details, see Figure 4 The electronic device 400 includes: a transceiver 401, a bus interface and a processor 402, wherein the processor 402 is used to: obtain video data and audio data of a conversation scene; obtain a face frame sequence according to the video data, and extract a lip frame sequence from the face frame sequence; generate a depth information sequence corresponding to the face frame sequence according to a depth estimation model, and obtain an audio sequence corresponding to the face frame sequence according to the audio data; obtain an audio embedding feature and a video embedding feature according to the audio sequence and the lip frame sequence respectively; perform weighted fusion on the depth information sequence with the audio embedding feature and the video embedding feature respectively to obtain a deep fusion audio feature and a deep fusion video feature respectively; determine a target speaking object in the conversation scene according to the deep fusion audio feature and the deep fusion video feature.

[0101] In one embodiment, the processor 402 is also used to: obtain a face frame sequence and facial key point coordinates in the video data according to a face detection algorithm; determine lip area key points according to the face frame sequence and the facial key point coordinates; obtain a minimum bounding rectangle according to the lip area key points, and use the minimum bounding rectangle as the lip frame sequence.

[0102] In one embodiment, the processor 402 is also used to: obtain multiple faces in the video data in sequence according to the video frames through the face detection algorithm, and output the face bounding box coordinates corresponding to each of the faces; obtain multiple face bounding box coordinates of the same face in each of the video frames according to the tracking algorithm, associate the face bounding box coordinates, and generate the face frame sequence according to the associated face bounding box coordinates.

[0103] In one embodiment, the processor 402 is further used to: encode the audio sequence through an audio timing encoder, and use the encoded audio sequence as the audio embedding feature; encode the lip frame sequence through a visual timing encoder, and use the encoded lip frame sequence as the video embedding feature.

[0104] In one embodiment, the processor 402 is also used to: weightedly fuse the depth information sequence with the audio embedding feature according to a first weighted formula to obtain the deep fusion audio feature; and weightedly fuse the depth information sequence with the video embedding feature according to a second weighted formula to obtain the deep fusion video feature.

[0105] In one embodiment, the processor 402 is further used to: associate the deep fusion video features with the depth information sequence through a cross-attention module; and determine the target speaking object in the conversation scenario based on the association result.

[0106] In one embodiment, the processor 402 is further used to: determine the detection score of each speaking object in the conversation scenario through the association result; obtain a target detection score greater than a preset score threshold from each detection score; and use the speaking object corresponding to the target detection score as the target speaking object.

[0107] In the embodiment of the present application, the electronic device 400 further includes a memory 403. Figure 4 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 402 and memory represented by memory 403. The bus architecture may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 401 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 402 is responsible for managing the bus architecture and general processing, and the memory 403 may store data used by the processor 402 when performing operations.

[0108] The electronic device 400 provided in the embodiment of the present application can execute the steps of the speaking object detection method provided in the above method embodiment 1, which will not be described again to avoid repetition.

[0109] The electronic device provided in this embodiment obtains the target speaking object in the conversation scenario by obtaining the depth information sequence, audio embedding features and video embedding features corresponding to the face frame sequence, and weightedly fusing the audio embedding features and the video embedding features with the depth information sequence, thereby improving the judgment accuracy of the speaking object, reducing misjudgment, and further improving work efficiency by automatically identifying the speaking object.

[0110] Example 4

[0111] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting a speaking object provided in Embodiment 1 is implemented.

[0112] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] The computer-readable storage medium provided in this embodiment can implement the speaking object detection method provided in Embodiment 1, and will not be described again to avoid repetition.

[0114] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0116] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for detecting a speaking object, characterized in that: The method comprises: Obtain video data and audio data of the conversation scene; Acquire a face frame sequence according to the video data, and extract a lip frame sequence from the face frame sequence; Generate a depth information sequence corresponding to the face frame sequence according to the depth estimation model, and acquire an audio sequence corresponding to the face frame sequence according to the audio data; Acquire audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively; The depth information sequence is weightedly fused with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively; A target speaker in the conversation scenario is determined according to the deeply fused audio features and the deeply fused video features.

2. The method for detecting a speaking object according to claim 1, characterized in that: The step of acquiring a face frame sequence according to the video data and extracting a lip frame sequence from the face frame sequence includes: Acquire a face frame sequence and facial key point coordinates in the video data according to a face detection algorithm; Determine the key points of the lip region according to the face frame sequence and the coordinates of the facial key points; A minimum bounding rectangle is obtained according to the key points of the lip region, and the minimum bounding rectangle is used as the lip frame sequence.

3. The method for detecting a speaking object according to claim 2, characterized in that: The step of obtaining a face frame sequence in the video data according to a face detection algorithm includes: Acquire multiple faces in the video data in sequence according to the video frames using the face detection algorithm, and output the coordinates of the face bounding boxes corresponding to the faces; A plurality of face bounding box coordinates of the same face in each of the video frames are obtained according to a tracking algorithm, the face bounding box coordinates are associated, and the face frame sequence is generated according to the associated face bounding box coordinates.

4. The method for detecting a speaking object according to claim 1, characterized in that: The acquiring audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively includes: Encoding the audio sequence by an audio timing encoder, and using the encoded audio sequence as the audio embedding feature; The lip frame sequence is encoded by a visual temporal encoder, and the encoded lip frame sequence is used as the video embedding feature.

5. The method for detecting a speaking object according to claim 1, characterized in that: The step of weighting and fusing the depth information sequence with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature, respectively, includes: Performing weighted fusion on the depth information sequence and the audio embedding feature according to a first weighted formula to obtain the deep fusion audio feature; The depth information sequence is weightedly fused with the video embedding feature according to a second weighted formula to obtain the depth fusion video feature.

6. The method for detecting a speaking object according to claim 1, characterized in that: The determining the target speaking object in the conversation scenario according to the deep fusion audio feature and the deep fusion video feature includes: Associating the deep fusion video features with the depth information sequence through a cross attention module; The target speaking object in the conversation scenario is determined according to the association result.

7. The method for detecting a speaking object according to claim 6, characterized in that: The step of determining the target speaking object in the conversation scenario according to the association result includes: Determining the detection score of each speaker in the conversation scenario according to the association result; Obtaining a target detection score greater than a preset score threshold from each of the detection scores; The speaking object corresponding to the object detection score is used as the target speaking object.

8. A speaking object detection device, characterized in that: The device comprises: A judgment module, used to obtain video data and audio data of the conversation scene; A connection module, used for acquiring a face frame sequence according to the video data, and extracting a lip frame sequence from the face frame sequence; A first transmission module, configured to generate a depth information sequence corresponding to the face frame sequence according to a depth estimation model, and to acquire an audio sequence corresponding to the face frame sequence according to the audio data; A selection module, configured to obtain audio embedding features and video embedding features according to the audio sequence and the lip frame sequence respectively; A second transmission module is used to perform weighted fusion on the depth information sequence with the audio embedding feature and the video embedding feature to obtain a deep fusion audio feature and a deep fusion video feature respectively; An acquisition module is used to determine a target speaking object in the conversation scenario according to the deep fusion audio feature and the deep fusion video feature.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for detecting a speaking object according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: The device stores a computer program, which executes the speaking object detection method according to any one of claims 1 to 7 when running on a processor.

Images