Voice activity detection method and device, equipment and storage medium

Abstract

The invention relates to the technical field of voice recognition and finance and medical treatment, and discloses a voice activity detection method, device and equipment and a storage medium, and the method comprises the steps: carrying out the visual feature extraction of a video clip of the upper part of an individual through a deep learning model, and obtaining the visual feature representation of the video clip; generating text description related to the video clip by using a visual language model, and performing text feature extraction on the text description through a text encoder to obtain text feature representation of the text description; splicing the text feature representation and the visual feature representation to generate a multi-modal feature representation; and inputting the multi-modal feature representation into a trained voice activity detection model, and outputting a voice activity detection result through the voice activity detection model. The method does not need to depend on audio data, can achieve flexible, efficient and privacy-friendly voice activity detection on the premise of not invading individual privacy, and improves the generalization ability of the model in different scenes.

Description

Technical Field

[0001] The present invention relates to the fields of speech recognition, finance and medical technology, and in particular to a voice activity detection method, device, equipment and storage medium. Background Art

[0002] Nowadays, online communication systems have increasing requirements for high-quality voice transmission. Online communication systems can be insurance systems, banking systems, trading systems, and order systems. Among them, effectively identifying and processing human voice segments in audio signals has become a challenge that cannot be ignored. Voice Activity Detection (VAD) technology can detect whether there is voice activity in the audio signal, reduce background noise, optimize bandwidth utilization, and improve the accuracy of speech recognition. Current voice activity detection mainly relies on audio signal processing, supplemented by visual information enhancement. It has limitations in jointly modeling audio and visual information, often relies on separately encoded unimodal features, fails to fully utilize cross-modal synchronization information, and does not work well in scenarios where there are multiple speakers, high noise, or audio signals are disturbed. In addition, the existing voice activity detection methods for online communication systems mainly rely on audio signals. In some scenarios, audio data is difficult to obtain and there are privacy and security issues. Summary of the invention

[0003] The present invention provides a voice activity detection method, apparatus, device and storage medium to solve the technical problems of the existing voice activity detection technology, namely, failure to fully utilize cross-modal synchronization information, poor effect in processing scenarios with multiple speakers, high noise or audio signals being interfered with, and difficulty in obtaining audio data in certain scenarios and the existence of privacy security issues.

[0004] In a first aspect, a voice activity detection method is provided, comprising:

[0005] Extracting visual features from a video clip of the upper body of an individual using a deep learning model to obtain a visual feature representation of the video clip;

[0006] Generate a text description related to the video clip using a visual language model, and extract text features from the text description using a text encoder to obtain a text feature representation of the text description;

[0007] Concatenating the text feature representation with the visual feature representation to generate a multimodal feature representation;

[0008] The multimodal feature representation is input into a trained voice activity detection model, the voice activity detection model learns the association information of the visual features and the text features, and outputs the voice activity detection result.

[0009] In a second aspect, a voice activity detection device is provided, comprising:

[0010] Visual feature extraction module: used to extract visual features from a video clip of an individual's upper body through a deep learning model to obtain a visual feature representation of the video clip;

[0011] A text feature extraction module is used to generate a text description related to the video clip using a visual language model, and to extract text features from the text description using a text encoder to obtain a text feature representation of the text description;

[0012] Feature fusion module: used to combine the text feature representation with the visual feature representation to generate a multimodal feature representation;

[0013] Voice activity detection module: used for inputting the multimodal feature representation into a trained voice activity detection model, learning the association information of visual features and text features through the voice activity detection model, and outputting the voice activity detection result.

[0014] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned voice activity detection method when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned voice activity detection method are implemented.

[0016] In the scheme implemented by the above-mentioned voice activity detection method, device, computer equipment and storage medium, by introducing visual features and text features, visual features and text features are integrated to generate multimodal features, and multimodal features are used to detect voice activity. The present invention does not need to rely on audio data, and can achieve more flexible, efficient and privacy-friendly voice activity detection without infringing on individual privacy, improves the accuracy and robustness of voice activity detection, and enhances the model's adaptability to complex scenarios such as limited audio data acquisition or unreliable environments. The present invention does not require large-scale audio-visual data sets for pre-training, reduces the resource requirements for model training, and improves the generalization ability of the model in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is an architectural diagram of a voice activity detection system according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a voice activity detection method according to a first embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a flow chart of a voice activity detection method according to a second embodiment of the present invention;

[0021] Figure 4 is a structural schematic diagram of a voice activity detection device in one embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0023] Figure 6 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The voice activity detection method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can extract visual features from a video clip of an individual's upper body through a deep learning model to obtain a visual feature representation of the video clip; generate a text description related to the video clip using a visual language model, and extract text features from the text description through a text encoder to obtain a text feature representation of the text description; splice the text feature representation with the visual feature representation to generate a multimodal feature representation; input the multimodal feature representation into a trained voice activity detection model, learn the association information of visual features and text features through the voice activity detection model, output the voice activity detection result, and return the voice activity detection result to the client. The present invention is suitable for voice activity detection in financial / medical systems, which can be insurance systems, banking systems, trading systems, and order systems. In voice activity detection in medical systems, the present invention can remotely monitor the health status of users, analyze customer behaviors and reactions without audio data, and protect user privacy while improving medical service efficiency. In voice activity detection in insurance systems, the present invention can quickly extract key information from on-site videos, accelerate the claims process, and improve user satisfaction. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0026] See also Figure 2 , is a flow chart of a voice activity detection method in a first embodiment of the present invention. The voice activity detection method provided in the first embodiment of the present invention comprises the following steps:

[0027] S100: extracting visual features from the video clip of the upper body of the individual through a deep learning model to obtain visual feature representation of the video clip;

[0028] In this step, the deep learning model adopts the ResNet or Transformer architecture. The deep learning model is used to extract visual features of the video clip frame by frame. The visual features include the movement and posture features of the individual upper body. Then, the visual features of all frames are averaged and pooled to obtain the visual feature representation of the entire video clip.

[0029] S110: Generate a text description related to the video clip by using a visual language model through a prompting project, and extract text features from the text description through a text encoder to obtain a text feature representation of the text description;

[0030] In this step, the visual language model includes but is not limited to LLaVA-13B, etc., and the text description is generated in the following manner: first, the center frame of the video clip is obtained, that is, the frame located in the middle of the video clip, and then specific prompt words are designed through prompt engineering, and the visual language model automatically generates a text description related to the center frame of the video clip according to the prompt words. The generated text description is input into a pre-trained text encoder, and the text encoder is used to extract text features from the text description to capture semantic information related to the video clip, thereby more comprehensively capturing the individual voice activity features, which is conducive to improving the accuracy and robustness of voice activity detection.

[0031] S120: concatenating the text feature representation with the visual feature representation to generate a multimodal feature representation;

[0032] In this step, a multimodal feature representation of the video clip is generated by combining the text feature representation with the visual feature representation. No audio data is required, which can improve the accuracy and robustness of VAD in complex environments without infringing individual privacy.

[0033] S130: inputting the multimodal feature representation into a trained voice activity detection model, learning the association information between the visual features and the text features through the voice activity detection model, and outputting the voice activity detection result;

[0034] In this step, the voice activity detection model uses a deep learning model such as a multilayer perceptron (MLP) or a Transformer network. The voice activity detection model determines whether an individual is in voice activity by learning the correlation information between visual features and text features.

[0035] Furthermore, the training process of the voice activity detection model includes:

[0036] S131: Obtain a VAD data set for model training, and divide the VAD data set into a training set and a test set according to a set ratio;

[0037] S132: After the training set is labeled with speech activity types, data enhancement techniques such as random cropping and flipping are applied to perform data enhancement processing on the training set; wherein, the generalization ability of the model can be enhanced by performing data enhancement processing on the training set;

[0038] S133: inputting the training set after data enhancement processing into the deep learning model, and training the deep learning model using a binary cross entropy loss function;

[0039] S134: Use the test set to test the deep learning model, and use indicators such as F1 score and accuracy to evaluate the performance of the deep learning model to obtain a trained voice activity detection model.

[0040] Taking the insurance business application scenario as an example, the on-site video of the insurance case can be input into the deep learning model to extract the visual feature representation of the on-site video, and the visual language model can be used to generate a text description related to the on-site video through the prompt engineering, and the text feature representation of the text description can be obtained through the text encoder. After splicing the text feature representation with the visual feature representation, the voice activity detection model is used to detect the voice activity of the on-site video, thereby accelerating the claim process of the insurance case and improving user satisfaction. In the medical business application scenario, the present invention can remotely monitor the health status of users, analyze the behavior and reaction of customers without audio data, protect user privacy and improve the efficiency of medical services.

[0041] It can be seen that in the above scheme, the voice activity detection method provided by the embodiment of the present invention introduces visual features and text features, fuses visual features and text features to generate multimodal features, and uses multimodal features to detect voice activity. The present invention does not need to rely on audio data, and can achieve more flexible, efficient and privacy-friendly voice activity detection without infringing on individual privacy, improves the accuracy and robustness of voice activity detection, and enhances the model's adaptability to complex scenarios such as limited audio data acquisition or unreliable environments. The present invention does not require large-scale audio-visual data sets for pre-training, reduces the resource requirements for model training, and improves the generalization ability of the model in different scenarios.

[0042] See also Figure 3 , is a flow chart of a voice activity detection method in a second embodiment of the present invention. The voice activity detection method provided in the second embodiment of the present invention comprises the following steps:

[0043] S200: collecting video clips of the upper body of the individual and preprocessing the video clips;

[0044] In this step, a terminal device with a camera function such as a mobile phone or a camera can be used to collect a video clip of the upper body of the individual, and the frame rate and resolution of the video clip can be set according to the input requirements of the deep learning model. The preprocessing of the video clip is specifically: scaling and cropping the video clip according to the input requirements of the deep learning model to obtain a video clip that meets the preset size, and performing frame rate conversion on the cropped video clip to obtain a video clip that meets the preset frame rate to ensure the continuity and fluency of the video clip.

[0045] S210: inputting the preprocessed video clip into a pre-trained deep learning model, extracting visual features of the video clip through the deep learning model, and obtaining a visual feature representation of the video clip;

[0046] In this step, the visual feature extraction method is the same as S100 in the first embodiment, and will not be described again here to avoid redundancy.

[0047] S220: obtaining a central frame of the video clip, generating a text description related to the central frame of the video clip by using a visual language model through a prompting project, and performing text feature extraction on the text description through a text encoder to obtain a text feature representation of the text description;

[0048] In this step, the text feature extraction method is the same as S110 in the first embodiment, and will not be described again here to avoid redundancy.

[0049] S230: splicing the text feature representation and the visual feature representation to generate a multimodal feature representation;

[0050] In this step, the feature stitching process is the same as S120 in the first embodiment, and will not be described again here to avoid redundancy.

[0051] S240: inputting the multimodal feature representation into a trained voice activity detection model, learning the association information between the visual features and the text features through the voice activity detection model, and outputting a voice activity detection result;

[0052] In this step, the voice activity detection method and the training process of the voice activity detection model are the same as S130 in the first embodiment, and will not be described again here to avoid redundancy.

[0053] It can be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0054] In one embodiment, a voice activity detection device is provided, and the voice activity detection device corresponds to the voice activity detection method in the above embodiment. Figure 4 As shown, the voice activity detection device includes a video acquisition module 101, a visual feature extraction module 102, a text feature extraction module 103, a feature fusion module 104 and a voice activity detection module 105. The functional modules are described in detail as follows:

[0055] Video acquisition module 101: used to acquire video clips of the upper body of an individual and pre-process the video clips; wherein, the video acquisition module 101 can use a terminal device with a camera function such as a mobile phone or a camera to acquire video clips of the upper body of an individual, and the frame rate and resolution of the video clips can be set according to the input requirements of the deep learning model. The video acquisition module 101 pre-processes the video clips as follows: scaling and cropping the video clips according to the input requirements of the deep learning model to obtain video clips that meet the preset size, and performing frame rate conversion on the cropped video clips to obtain video clips that meet the preset frame rate, so as to ensure the continuity and fluency of the video clips.

[0056] Visual feature extraction module 102: used to extract visual features from a video clip of an individual's upper body through a deep learning model to obtain a visual feature representation of the video clip; wherein the deep learning model adopts a ResNet or Transformer architecture, and extracts visual features from the video clip frame by frame through the deep learning model, the visual features including the movement and posture features of the individual's upper body, and then, average pools the visual features of all frames to obtain a visual feature representation of the entire video clip.

[0057] Text feature extraction module 103: used to generate a text description related to the video clip using a visual language model, and extract text features from the text description through a text encoder to obtain a text feature representation of the text description; wherein the visual language model includes but is not limited to LLaVA-13B, etc., and the text description generation method is specifically as follows: first obtain the center frame of the video clip, that is, the frame located in the middle of the video clip, and then design specific prompt words through prompt engineering, and the visual language model automatically generates a text description related to the center frame of the video clip according to the prompt words. The generated text description is input into a pre-trained text encoder, and the text encoder is used to extract text features from the text description to capture semantic information related to the video clip, thereby more comprehensively capturing the individual voice activity features, which is conducive to improving the accuracy and robustness of voice activity detection.

[0058] Feature fusion module 104: used to splice the text feature representation with the visual feature representation to generate a multimodal feature representation; wherein, by combining the text feature representation with the visual feature representation to generate a multimodal feature representation of the video clip, no audio data is required, and the accuracy and robustness of VAD in complex environments can be improved without infringing on individual privacy.

[0059] The voice activity detection module 105 is used to input the multimodal feature representation into a trained voice activity detection model, learn the association information of visual features and text features through the voice activity detection model, and output the voice activity detection result; wherein the voice activity detection model adopts a deep learning model such as a multilayer perceptron (abbreviated as MLP) or a Transformer network, and the voice activity detection model judges whether an individual is in voice activity by learning the association information of visual features and text features.

[0060] Furthermore, the training process of the voice activity detection model includes:

[0061] Obtain a VAD data set for model training, and divide the VAD data set into a training set and a test set according to a set ratio;

[0062] After the training set is labeled with speech activity types, data enhancement techniques such as random cropping and flipping are applied to the training set for data enhancement processing; data enhancement processing of the training set can enhance the generalization ability of the model;

[0063] The training set after data augmentation is input into the deep learning model, and the deep learning model is trained using the binary cross entropy loss function;

[0064] The deep learning model is tested using the test set, and its performance is evaluated using indicators such as F1 score and accuracy to obtain a trained voice activity detection model.

[0065] It can be seen that in the above scheme, the voice activity detection device provided by the embodiment of the present invention introduces visual features and text features, fuses visual features and text features to generate multimodal features, and uses multimodal features to detect voice activity. The present invention does not need to rely on audio data, and can achieve more flexible, efficient and privacy-friendly voice activity detection without infringing on individual privacy, improves the accuracy and robustness of voice activity detection, and enhances the model's adaptability to complex scenarios such as limited audio data acquisition or unreliable environments. The present invention does not require large-scale audio-visual data sets for pre-training, reduces the resource requirements for model training, and improves the generalization ability of the model in different scenarios.

[0066] For the specific definition of the voice activity detection device, please refer to the definition of the voice activity detection method above, which will not be repeated here. Each module in the above-mentioned voice activity detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0067] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a voice activity detection method are implemented.

[0068] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a voice activity detection method are implemented.

[0069] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0070] Extracting visual features from a video clip of the upper body of an individual using a deep learning model to obtain a visual feature representation of the video clip;

[0071] Generate a text description related to the video clip using a visual language model, and extract text features from the text description using a text encoder to obtain a text feature representation of the text description;

[0072] Concatenating the text feature representation with the visual feature representation to generate a multimodal feature representation;

[0073] The multimodal feature representation is input into a trained voice activity detection model, the voice activity detection model learns the association information of the visual features and the text features, and outputs the voice activity detection result.

[0074] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0075] Extracting visual features from a video clip of the upper body of an individual using a deep learning model to obtain a visual feature representation of the video clip;

[0076] Generate a text description related to the video clip using a visual language model, and extract text features from the text description using a text encoder to obtain a text feature representation of the text description;

[0077] Concatenating the text feature representation with the visual feature representation to generate a multimodal feature representation;

[0078] The multimodal feature representation is input into a trained voice activity detection model, the voice activity detection model learns the association information of the visual features and the text features, and outputs the voice activity detection result.

[0079] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other storage medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0081] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0082] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A voice activity detection method, characterized in that: include: Extracting visual features from a video clip of the upper body of an individual using a deep learning model to obtain a visual feature representation of the video clip; Generate a text description related to the video clip using a visual language model, and extract text features from the text description using a text encoder to obtain a text feature representation of the text description; Concatenating the text feature representation with the visual feature representation to generate a multimodal feature representation; The multimodal feature representation is input into a trained voice activity detection model, the voice activity detection model learns the association information of the visual features and the text features, and outputs the voice activity detection result.

2. The voice activity detection method according to claim 1, characterized in that: The extracting visual features of the video clip by using a deep learning model to obtain a visual feature representation of the video clip is specifically as follows: The deep learning model adopts a ResNet or Transformer architecture, and the deep learning model is used to extract visual features of the video clip frame by frame, wherein the visual features include movement and posture features of the upper body of the individual; The visual features of all frames are average-pooled to obtain a visual feature representation of the video clip.

3. The voice activity detection method according to claim 2, characterized in that: The generating of a text description related to the video clip by using a visual language model is specifically: Obtaining a central frame of a video clip, wherein the central frame of the video clip is a frame located in the middle of the video clip; By prompting the engineering design prompt words, the visual language model generates a text description related to the central frame of the video clip according to the prompt words.

4. The voice activity detection method according to claim 3, characterized in that: The voice activity detection model adopts a multi-layer perceptron or a Transformer network, and the training process of the voice activity detection model includes: Obtain a VAD data set for model training, and divide the VAD data set into a training set and a test set according to a set ratio; After the training set is annotated with speech activity types, data enhancement processing is performed on the training set using data enhancement technology; The training set after the data enhancement processing is input into the deep learning model, and the deep learning model is trained using a binary cross entropy loss function.

5. The voice activity detection method according to claim 4, characterized in that: After inputting the training set after the data enhancement processing into the deep learning model and training the deep learning model using the binary cross entropy loss function, the method further includes: The deep learning model is tested using the test set, and the performance of the deep learning model is evaluated using evaluation indicators to obtain a trained voice activity detection model; wherein the evaluation indicators include F1 score and accuracy.

6. The voice activity detection method according to any one of claims 1 to 5, characterized in that: Before extracting visual features from the video clip of the upper body of the individual by using the deep learning model and obtaining the visual feature representation of the video clip, the method further includes: A video clip of the upper body of the individual is collected and preprocessed.

7. The voice activity detection method according to claim 6, characterized in that: The preprocessing of the video clips is specifically as follows: Scaling and cropping the video clip to obtain a video clip that meets a preset size; The trimmed video clips are subjected to frame rate conversion to obtain video clips meeting a preset frame rate.

8. A voice activity detection device, characterized in that: include: Visual feature extraction module: used to extract visual features from a video clip of an individual's upper body through a deep learning model to obtain a visual feature representation of the video clip; A text feature extraction module is used to generate a text description related to the video clip using a visual language model, and extract text features from the text description using a text encoder to obtain a text feature representation of the text description; Feature fusion module: used to combine the text feature representation with the visual feature representation to generate a multimodal feature representation; Voice activity detection module: used for inputting the multimodal feature representation into a trained voice activity detection model, learning the association information of visual features and text features through the voice activity detection model, and outputting the voice activity detection result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the voice activity detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the voice activity detection method according to any one of claims 1 to 7 are implemented.

Images