Speaker detection method and device based on context multi-scale space-time diagram learning

Abstract

The invention discloses a speaker detection method and device based on context multi-scale space-time diagram learning, a storage medium and electronic equipment. The method comprises the following steps: acquiring video data, and extracting multiple pieces of video frame data from the video data; performing feature extraction on the multiple pieces of video frame data to obtain facial expression features, audio features and body language features; constructing a multi-modal dynamic space-time graph based on the facial expression features, the audio features and the body language features; wherein the multi-modal dynamic space-time graph comprises a plurality of graph snapshots, and the graph snapshots comprise a plurality of nodes; inputting the graph snapshot into a graph neural network to obtain a classification result of each node in the graph snapshot; and obtaining the speaking state of the candidate in the video data according to the classification result. The method shows higher accuracy and robustness in processing complex scenes, and brings important technical progress and application prospect to the field of active speaker detection.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and sound processing, and in particular to a speaker detection method, device, storage medium and electronic device based on contextual multi-scale spatiotemporal graph learning. Background Art

[0002] The popularity and development of video content has made active speaker detection technology increasingly important. Active speaker detection refers to identifying the person who is speaking in a video. This technology has wide application value in video conferencing, video surveillance and other fields.

[0003] Traditional active speaker detection methods are mainly based on analyzing facial expressions and audio signals. However, these methods may perform poorly in complex scenes, such as camera shaking, poor lighting, motion blur, etc.

[0004] Currently, research on active speaker detection mainly focuses on extracting facial expressions and audio signals using computer vision and deep learning techniques and combining them for detection. However, these methods often ignore other visual cues, such as body language and contextual information, as well as the characteristics of the candidate's speaking behavior over long and short time scales. This leads to poor performance in complex scenarios involving multiple candidates, camera shaking, image blur, etc. Summary of the invention

[0005] The embodiments of the present application provide a speaker detection method, device, storage medium and electronic device based on contextual multi-scale spatiotemporal graph learning, which can show higher accuracy and robustness when processing complex scenarios.

[0006] The embodiment of the present application provides a speaker detection method based on contextual multi-scale spatiotemporal graph learning, including: Acquire video data, and extract multiple video frame data from the video data; Extracting features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; Constructing a multimodal dynamic space-time graph based on the facial expression features, the audio features and the body language features; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; Inputting the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; According to the classification result, the speaking status of the candidate in the video data is obtained.

[0007] Furthermore, in the above-mentioned speaker detection method based on contextual multi-scale spatiotemporal graph learning, wherein the video frame data includes image data and audio data, after the step of extracting a plurality of video frame data from the video data, the method further includes: Performing face detection on the video frame data based on a facial recognition algorithm to obtain multiple face images; Separating audio signals from the video frame data; The video frame data is identified based on a motion detection algorithm to obtain multiple character posture information.

[0008] Furthermore, in the above-mentioned speaker detection method based on contextual multi-scale spatiotemporal graph learning, the feature extraction of the plurality of video frame data is performed respectively to obtain facial expression features, audio features and body language features, including: Inputting the face image into a 2D-ResNet model to extract features and obtain facial expression features; Inputting the audio signal into a LSTM-based deep model for audio recognition to obtain audio features; The character posture information is analyzed and encoded through MediaPipe to obtain body language features.

[0009] Furthermore, in the above-mentioned speaker detection method based on contextual multi-scale spatiotemporal graph learning, the step of constructing a multimodal dynamic space-time graph based on the facial expression features, the audio features and the body language features comprises: Constructing corresponding modal nodes based on the facial expression features, the audio features and the body language features respectively; Constructing candidate nodes, the candidate nodes corresponding to the candidates in the video frame data, the candidate nodes being connected to the corresponding modality nodes via edges; A graph snapshot is constructed based on the modal node and the candidate node, and a plurality of the graph snapshots is a multimodal dynamic space-time graph.

[0010] Furthermore, in the above-mentioned speaker detection method based on contextual multi-scale spatiotemporal graph learning, the step of inputting the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot includes: Inputting the graph snapshot into the graph neural network, and iteratively updating the edges and nodes; The classification result of the corresponding node is obtained based on the updated node information.

[0011] The embodiment of the present application also provides a speaker detection device based on contextual multi-scale spatiotemporal graph learning, comprising: An acquisition module, used for acquiring video data and extracting a plurality of video frame data from the video data; A feature extraction module, used to extract features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; A construction module, configured to construct a multimodal dynamic space-time graph based on the facial expression feature, the audio feature and the body language feature; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; A classification module, used to input the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; The detection module is used to obtain the speaking status of the candidate in the video data according to the classification result.

[0012] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned speaker detection methods based on contextual multi-scale spatiotemporal graph learning.

[0013] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in the speaker detection method based on contextual multi-scale spatiotemporal graph learning as described in any of the above items.

[0014] The present application provides a speaker detection method, device, storage medium and electronic device based on contextual multi-scale spatiotemporal graph learning. The present application constructs a multimodal dynamic space-time graph based on facial expression features, audio features and body language features, combines multimodal information with spatiotemporal relationships, and realizes comprehensive and effective active speaker detection. By fully utilizing the rich information in the video, especially considering the speaking characteristics of the candidates at different time scales, it shows higher accuracy and robustness in processing complex scenes, bringing important technical progress and application prospects to the field of active speaker detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0016] Figure 1 A flowchart of a speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application.

[0017] Figure 2 Another flowchart of a speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application.

[0018] Figure 3 A schematic diagram of the structure of a speaker detection device based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application.

[0019] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0020] Figure 5 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0022] The embodiments of the present application provide a speaker detection method, device, storage medium and electronic device based on contextual multi-scale spatiotemporal graph learning. The speaker detection device based on contextual multi-scale spatiotemporal graph learning provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0023] See also Figure 1 and Figure 2 , Figure 1 A flowchart of a speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application, Figure 1 Another flowchart of a speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application, which is applied to an electronic device, the speaker detection method based on contextual multi-scale spatiotemporal graph learning includes the following steps: S1, obtaining video data, and extracting a plurality of video frame data from the video data.

[0024] In one embodiment, after step S1, the following steps are further included: SA1, performs face detection on the video frame data based on the facial recognition algorithm to obtain multiple face images; SA2, separates the audio signal from the video frame data; SA3, based on the motion detection algorithm, identifies the video frame data and obtains multiple character posture information.

[0025] Specifically, video frames are extracted frame by frame from the input video data. In each frame, we use a facial recognition algorithm (such as Haar cascade classifier) ​​to detect the face and mark the facial expression.

[0026] Specifically, we read the video frames, convert them into grayscale images, perform face detection on the grayscale images using the trained Haar cascade classifier, draw a rectangular box on the detected face, mark the face position, and obtain the face image. Then, we separate the audio signal from the video frame and use Fourier transform to convert the audio signal into frequency domain information for subsequent processing. Finally, we analyze body language and track the key postures of the body using motion detection algorithms (e.g., optical flow method).

[0027] S2, extracting features from multiple video frame data respectively to obtain facial expression features, audio features and body language features.

[0028] In one embodiment, step S2 comprises the following steps: S21, input the face image into the 2D-ResNet model for feature extraction to obtain facial expression features.

[0029] Specifically, the 2D-ResNet model includes multiple convolutional layers, and the face image is input into the 2D-ResNet model for multi-scale convolution to obtain facial expression features.

[0030] S22, input the audio signal into the LSTM-based deep model for audio recognition to obtain audio features.

[0031] Specifically, the audio signal is first preprocessed, including sampling rate conversion, audio trimming, noise removal, etc. Then the preprocessed audio signal is analyzed by the Mel spectrum analyzer to obtain the feature map in the time-frequency domain. LSTM is combined with the acoustic model (such as DNN, HMM), and the feature map is input into the LSTM and acoustic model. The acoustic model is responsible for mapping the audio feature sequence to the vocabulary sequence, and the LSTM is responsible for learning the long-term dependency. Finally, the decoder converts the hidden state into a vocabulary sequence.

[0032] S23, analyzes and encodes the character posture information through MediaPipe to obtain body language features.

[0033] MediaPipe's full-body pose recognition (`mediapipe.solutions.pose`) can estimate 25 key points of the human body in real time, covering multiple parts such as the head, torso, limbs, etc. Whether it is simple sitting, standing, walking, or complex sports movements, this module can accurately capture and track the changes in human posture.

[0034] The purpose of step S2 is to capture and encode information related to speaking behavior. Different deep learning models are used to encode facial expressions, audio, background, and body language in video frames to extract features. For example, a 2D-ResNet model is used to analyze facial expressions to obtain features related to emotions and lip shape. Audio data is processed using a deep model based on LSTM or sound recognition to extract audio spectrum and rhythm information. Tools such as MediaPipe are used to analyze and encode body language, highlighting the key points and movement trajectories of actions.

[0035] S3, constructs a multimodal dynamic space-time graph based on facial expression features, audio features and body language features; wherein the multimodal dynamic space-time graph includes multiple graph snapshots, and the graph snapshots include multiple nodes.

[0036] In one embodiment, step S3 includes the following steps: S31, constructing corresponding modal nodes based on facial expression features, audio features and body language features respectively; S32, constructing candidate nodes, the candidate nodes correspond to candidates in the video frame data, and the candidate nodes are connected to the corresponding modality nodes through edges; S33, constructing a graph snapshot based on the modal nodes and the candidate nodes, and multiple graph snapshots are a multimodal dynamic space-time graph.

[0037] Specifically, the core of the steps to build a multimodal dynamic space-time graph (MMDSTG) is to organize the data so that the graph neural network can understand and analyze it. Create graph snapshots, each snapshot corresponds to a frame of video, containing all detected candidate speakers in that frame. If a candidate speaker's face is detected, a modal node associated with the facial features will be created in the graph, and it will be connected to the candidate node corresponding to the candidate speaker through an edge. Similarly, if sound activity is recognized in the audio data, all candidate nodes will be connected to a common audio modal node through an edge. The feature vector of each modal node is generated by its own encoder, and all candidate nodes are set to zero vectors at the initial moment.

[0038] S4, input the graph snapshot into the graph neural network to obtain the classification result of each node in the graph snapshot.

[0039] In one embodiment, step S4 includes the following steps: S41, input the graph snapshot into the graph neural network and iteratively update the edges and nodes; S42, obtaining a classification result of the corresponding node based on the updated node information.

[0040] Specifically, a graph neural network (GNN) is applied to analyze and synthesize the information in MMDSTG. GNN synthesizes features and reveals potential interactions between candidate nodes by iteratively updating the information on nodes and edges. In specific operations, GNN propagates feature information layer by layer, so that nodes in each graph snapshot can receive information from neighboring nodes. As the GNN layer goes deeper, the feature representation of each node becomes richer, reflecting the node's own characteristics and its connection with other nodes. After that, GNN classifies each node into a "speaking" or "not speaking" state.

[0041] S5, obtaining the speaking status of the candidate in the video data according to the classification result.

[0042] Specifically, we interpret and output the classification results output by GNN. We analyze the nodes classified by GNN, obtain the speaking status of each candidate node (speaking or not speaking), and output this final classification status to the outside of the system. These results can be used in various applications, such as automatic speaker recognition in video conferencing, behavioral analysis in video surveillance, etc.

[0043] This application combines multimodal information with spatiotemporal relationships to achieve comprehensive and effective active speaker detection. By fully utilizing the rich information in the video, especially considering the speaking characteristics of the candidate at different time scales, the present invention shows higher accuracy and robustness in processing complex scenes, bringing important technical progress and application prospects to the field of active speaker detection.

[0044] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a speaker detection device based on contextual multi-scale spatiotemporal graph learning. The speaker detection device based on contextual multi-scale spatiotemporal graph learning can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, server or other device, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0045] See also Figure 3 , Figure 3 The present invention specifically describes a speaker detection device based on contextual multi-scale spatiotemporal graph learning provided by an embodiment of the present invention, which is applied to an electronic device. The speaker detection device based on contextual multi-scale spatiotemporal graph learning may include: An acquisition module, used for acquiring video data and extracting a plurality of video frame data from the video data; A feature extraction module, used to extract features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; A construction module, configured to construct a multimodal dynamic space-time graph based on the facial expression feature, the audio feature and the body language feature; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; A classification module, used to input the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; The detection module is used to obtain the speaking status of the candidate in the video data according to the classification result.

[0046] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0047] In addition, the embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Figure 4 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0048] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0049] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions: Acquire video data, and extract multiple video frame data from the video data; Extracting features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; Constructing a multimodal dynamic space-time graph based on the facial expression features, the audio features and the body language features; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; Inputting the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; According to the classification result, the speaking status of the candidate in the video data is obtained.

[0050] The electronic device can implement the steps in any embodiment of the speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0051] Figure 5 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in the above embodiment. The electronic device 500 can be a terminal, a server, etc., wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0052] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0053] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0054] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0055] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0056] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0057] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0058] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0059] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Acquire video data, and extract multiple video frame data from the video data; Extracting features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; Constructing a multimodal dynamic space-time graph based on the facial expression features, the audio features and the body language features; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; Inputting the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; According to the classification result, the speaking status of the candidate in the video data is obtained.

[0060] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0061] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the speaker detection method based on contextual multi-scale spatiotemporal graph learning provided by the embodiment of the present invention.

[0062] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0063] Since the instructions stored in the storage medium can execute the steps in any embodiment of the speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in the embodiments of the present invention, the beneficial effects that can be achieved by any speaker detection method based on contextual multi-scale spatiotemporal graph learning provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0064] The above is a detailed introduction to a speaker detection method, device, storage medium and electronic device based on contextual multi-scale spatiotemporal graph learning provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speaker detection method based on contextual multi-scale spatio-temporal graph learning, It is characterized in that The method comprises: Acquire video data, and extract multiple video frame data from the video data; Extracting features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; Constructing a multimodal dynamic space-time graph based on the facial expression features, the audio features and the body language features; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; Inputting the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; According to the classification result, the speaking status of the candidate in the video data is obtained.

2. The speaker detection method based on contextual multi-scale spatiotemporal graph learning according to claim 1, It is characterized in that The video frame data includes image data and audio data, and after the step of extracting a plurality of video frame data from the video data, the method further includes: Performing face detection on the video frame data based on a facial recognition algorithm to obtain multiple face images; Separating audio signals from the video frame data; The video frame data is identified based on a motion detection algorithm to obtain multiple character posture information.

3. The speaker detection method based on contextual multi-scale spatiotemporal graph learning according to claim 2, It is characterized in that The extracting features of the plurality of video frame data respectively to obtain facial expression features, audio features and body language features includes: Inputting the face image into a 2D-ResNet model to extract features and obtain facial expression features; Inputting the audio signal into a LSTM-based deep model for audio recognition to obtain audio features; The character posture information is analyzed and encoded through MediaPipe to obtain body language features.

4. The speaker detection method based on contextual multi-scale spatiotemporal graph learning according to claim 3, It is characterized in that The constructing of a multimodal dynamic space-time graph based on the facial expression feature, the audio feature and the body language feature comprises: Constructing corresponding modal nodes based on the facial expression features, the audio features and the body language features respectively; Constructing candidate nodes, the candidate nodes corresponding to the candidates in the video frame data, the candidate nodes being connected to the corresponding modality nodes via edges; A graph snapshot is constructed based on the modal node and the candidate node, and a plurality of the graph snapshots is a multimodal dynamic space-time graph.

5. The speaker detection method based on contextual multi-scale spatiotemporal graph learning according to claim 4, It is characterized in that The step of inputting the graph snapshot into the graph neural network to obtain a classification result for each node in the graph snapshot includes: Inputting the graph snapshot into the graph neural network, and iteratively updating the edges and nodes; The classification result of the corresponding node is obtained based on the updated node information.

6. A speaker detection device based on contextual multi-scale spatiotemporal graph learning, It is characterized in that include: An acquisition module, used for acquiring video data and extracting a plurality of video frame data from the video data; A feature extraction module, used to extract features from the plurality of video frame data respectively to obtain facial expression features, audio features and body language features; A construction module, configured to construct a multimodal dynamic space-time graph based on the facial expression feature, the audio feature and the body language feature; wherein the multimodal dynamic space-time graph includes a plurality of graph snapshots, and the graph snapshots include a plurality of nodes; A classification module, used to input the graph snapshot into a graph neural network to obtain a classification result for each node in the graph snapshot; The detection module is used to obtain the speaking status of the candidate in the video data according to the classification result.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the speaker detection method based on contextual multi-scale spatiotemporal graph learning according to any one of claims 1 to 5.

8. An electronic device, It is characterized in that The invention comprises a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the speaker detection method based on contextual multi-scale spatiotemporal graph learning as described in any one of claims 1 to 5.

Images