Method, device, equipment and medium for generating audio-based video description

Abstract

The present application relates to a method, device, equipment and medium for generating a video description based on audio, wherein the method comprises: extracting video frame data and audio data from a video file, and attempting to identify first text information of the video frame data and second text information of the audio data; judging the video scene in the video file according to whether the first text information and the second text information exist; selecting target data corresponding to the video scene from multimodal data for fusion to obtain data to be identified, wherein the multimodal data comprises a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, first text information and second text information; identifying the data to be identified by a preset recognition model to obtain a video description corresponding to the video file. The problem that the generated video description is not comprehensive enough due to ignoring auditory information is solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device, equipment and medium for generating audio-based video description. Background Art

[0002] In recent years, with the rapid development of Internet technology, video resources have become the most popular and favorite way for people to obtain information. Especially after the emergence of some video apps, watching videos every day has become an indispensable leisure and entertainment method for many people. In order to better serve users, efficient and reasonable analysis and understanding of videos is an important research direction in the field of computer vision, and video description is an important part of this direction.

[0003] Current video description generation technology mainly focuses on the analysis of visual information, such as capturing and understanding the content of the video through image recognition, object detection, and scene segmentation, and generating corresponding video descriptions based on this. However, as a form of multimedia, video not only contains rich visual elements, but is also often accompanied by important auditory information, such as dialogue, music, and environmental sound effects, which are also crucial for a comprehensive understanding and description of the video content. At present, when generating video descriptions, auditory information is ignored, resulting in incomplete video descriptions.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a method, device, equipment and medium for generating an audio-based video description to solve the above-mentioned technical problem of "the generated video description is not comprehensive enough due to ignoring auditory information".

[0006] According to one aspect of an embodiment of the present application, the present application provides a method for generating an audio-based video description, including: extracting video frame data and audio data in a video file, and attempting to identify first text information of the video frame data and second text information of the audio data; judging a video scene in the video file based on whether the first text information and the second text information exist; selecting target data corresponding to the video scene from multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, a first text information, and a second text information; identifying the data to be identified by a preset recognition model to obtain a video description corresponding to the video file.

[0007] Optionally, attempting to identify first text information of video frame data and second text information of audio data includes: attempting to detect text areas of each video frame in the video frame data, and extracting first text information of the text area if a text area exists; attempting to extract human voice data from audio data, and performing text recognition on the human voice data if the human voice data exists to obtain second text information.

[0008] Optionally, judging the video scene in the video file based on whether the first text information and the second text information exist includes: if the first text information exists and the second text information exists, judging that the video scene of the video file is a character dialogue; if the first text information does not exist and the second text information exists, judging that the video scene of the video file is background music; if the first text information does not exist and the second text information does not exist, judging that the video scene of the video file is a scene presentation.

[0009] Optionally, before selecting target data corresponding to the video scene from the multimodal data for fusion, the method also includes obtaining a visual feature representation in the following manner: extracting a visual feature vector sequence of the video frame data, wherein each visual feature vector in the visual feature vector sequence corresponds to a video frame in the video frame data; aligning each visual feature vector in the visual feature vector sequence to an input space to obtain a visual feature representation, wherein the input space is a data structure that can be recognized by a preset recognition model.

[0010] Optionally, before selecting target data corresponding to the video scene from the multimodal data for fusion, the method also includes obtaining an audio feature representation in the following manner: extracting an audio feature vector sequence of the audio data, wherein each audio feature vector in the audio feature vector sequence corresponds to an audio segment in the audio data; aligning each audio feature vector in the audio feature vector sequence to an input space to obtain an audio feature representation.

[0011] Optionally, target data corresponding to the video scene is selected from the multimodal data for fusion, including: if the video scene is a character dialogue, the visual feature representation and the first text information are determined as target data, and the visual feature representation and the first text feature corresponding to the first text information are fused; if the video scene is background music, the visual feature representation and the second text information are determined as target data, and the visual feature representation and the second text feature corresponding to the second text information are fused; if the video scene is a scene presentation, the visual feature representation and the audio feature representation are determined as target data, and the visual feature representation and the audio feature representation are fused.

[0012] Optionally, the method also includes obtaining a preset recognition model in the following manner: obtaining a training data set, wherein the training data set includes training data of different video scenes; training an initial model using the training data set to obtain training results; when the training results indicate that the recognition accuracy of the initial model for the video description of each video scene is greater than or equal to a target threshold, determining the initial model as the preset recognition model.

[0013] According to another aspect of an embodiment of the present application, the present application provides a device for generating an audio-based video description, including: an extraction module, used to extract video frame data and audio data in a video file, and attempt to identify first text information of the video frame data and second text information of the audio data; a judgment module, used to judge the video scene in the video file according to whether the first text information and the second text information exist; a fusion module, used to select target data corresponding to the video scene from multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, a first text information and a second text information; an identification module, used to identify the data to be identified through a preset recognition model to obtain a video description corresponding to the video file.

[0014] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0015] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application adopts a method for generating video description based on audio, including: extracting video frame data and audio data from a video file, and attempting to identify the first text information of the video frame data and the second text information of the audio data; judging the video scene in the video file according to whether the first text information and the second text information exist; selecting target data corresponding to the video scene from multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, a first text information and a second text information; identifying the data to be identified by a preset recognition model to obtain a video description corresponding to the video file. The video scene is identified by combining the audio in the video, and different multimodal data are fused based on different video scenes, and finally the multimodal data is identified to obtain a video description. Since the identification of the video scene and the use of audio data are both included in the multimodal data, the auditory information is fully considered when generating the video description, solving the problem that the generated video description is not comprehensive enough due to ignoring the auditory information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of an optional method for generating audio-based video description according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an optional framework for generating audio-based video description according to an embodiment of the present application;

[0022] Figure 3 A block diagram of an optional apparatus for generating audio-based video description according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0026] In recent years, with the rapid development of Internet technology, video resources have become the most popular and favorite way for people to obtain information. Especially after the emergence of some video apps, watching videos every day has become an indispensable leisure and entertainment method for many people. In order to better serve users, efficient and reasonable analysis and understanding of videos is an important research direction in the field of computer vision, and video description is an important part of this direction.

[0027] Current video description generation technology mainly focuses on the analysis of visual information, such as capturing and understanding the content of the video through image recognition, object detection, and scene segmentation, and generating corresponding video descriptions based on this. However, as a form of multimedia, video not only contains rich visual elements, but is also often accompanied by important auditory information, such as dialogue, music, and environmental sound effects, which are also crucial for a comprehensive understanding and description of the video content. At present, when generating video descriptions, auditory information is ignored, resulting in incomplete video descriptions.

[0028] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiment of the present application, a method for generating a video description based on audio is provided, such as Figure 1 As shown, including:

[0029] Step 101, extracting video frame data and audio data from a video file, and attempting to identify first text information of the video frame data and second text information of the audio data;

[0030] Step 103, judging the video scene in the video file according to whether the first text information and the second text information exist;

[0031] Step 105, selecting target data corresponding to the video scene from the multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, a first text information, and a second text information;

[0032] Step 107: The data to be identified is identified by using a preset identification model to obtain a video description corresponding to the video file.

[0033] A video file often contains multiple video frames and audio. This application aims to take audio into consideration when generating a video description, so it is necessary to extract both video frame data and audio data.

[0034] Specifically, a video processing library (eg, FFmpeg, OpenCV, etc.) is used to extract video frame data from a video file.

[0035] Specifically, an audio processing library (eg, Librosa, PyDub, etc.) may be used to extract audio data from a video file.

[0036] Since the video frame data may contain subtitle information, the first text information of the video frame data is attempted to be recognized, and since the audio data may contain sound information, the second text information of the audio data is attempted to be recognized.

[0037] The video scene in the video file is preliminarily determined based on whether the first text information and the second text information exist. The video scene provided in this application includes character dialogue, background music, and scene presentation. In actual application, other scenes may also be set.

[0038] According to the judgment result of the video scene, the target data that is most relevant to the current scene or most helpful for generating the video description is selected from the multimodal data for fusion. The fusion method may include feature concatenation and feature weighting.

[0039] The preset recognition model is an existing large text model, which receives the fused data to be recognized as input and generates a video description as output.

[0040] The audio-based video description generation method provided in this application combines multiple technologies such as video processing, audio processing, optical character recognition, speech recognition, and natural language processing, aiming to comprehensively understand the video content from multiple dimensions, especially the audio dimension, and generate accurate, vivid, and expressive video descriptions.

[0041] As an optional embodiment, attempting to identify first text information of video frame data and second text information of audio data includes: attempting to detect text areas of each video frame in the video frame data, and extracting first text information of the text area if a text area exists; attempting to extract human voice data from audio data, and performing text recognition on the human voice data to obtain second text information if the human voice data exists.

[0042] Attempt to detect text regions in each video frame in the video frame data, including: implementing through image processing techniques, such as edge detection, connected domain analysis, machine learning or deep learning algorithms (e.g., text detection models based on convolutional neural networks), etc., identifying potential text regions in the image, and outputting bounding boxes or coordinates of these text regions.

[0043] If there is a text area, the OCR (Optical Character Recognition) technology can be used to extract text information in the text area, specifically including: performing character segmentation, feature extraction and character classification, and generating readable first text information.

[0044] Audio data may contain a variety of sounds, such as human voices, background music, environmental noise, etc. In order to extract human voice data from audio, audio separation technology is usually required. This can be achieved through machine learning algorithms (such as support vector machines, neural networks, etc.) or deep learning models, which can distinguish different sound sources and separate them.

[0045] When there is human voice data, the human voice data is extracted from the audio data, and then the ASR (Automatic Speech Recognition) algorithm is used to recognize the second text information in the human voice data. The ASR algorithm generally includes front-end processing (such as noise suppression, feature extraction), acoustic modeling (converting sound features into speech units), language modeling (converting speech units into meaningful text), etc.

[0046] As an optional embodiment, judging the video scene in the video file according to whether the first text information and the second text information exist includes: if the first text information exists and the second text information exists, judging that the video scene in the video file is a character dialogue; if the first text information does not exist and the second text information exists, judging that the video scene in the video file is background music; if the first text information does not exist and the second text information does not exist, judging that the video scene in the video file is a scene presentation.

[0047] If first text information (such as subtitles) is detected in the video frame, and second text information corresponding to human voice is extracted from the audio data, this usually means that the video contains dialogue or narration, so it can be determined that the video scene of the video file is a character dialogue.

[0048] If the first text information is not detected in the video frame, but the text information corresponding to the human voice is extracted from the audio data, this may mean that the video mainly relies on the audio to convey information, and the video frame itself does not contain important text information. Therefore, it can be determined that the video scene of the video file is background music (i.e., a scene without subtitles but with human voice, such as a speech, a concert, etc.).

[0049] If the first text information is not detected in the video frame, and the second text information corresponding to the obvious human voice is not extracted from the audio data, this usually means that the video mainly relies on visual elements to convey information, or the video content itself does not contain enough text or human voice information for extraction. Therefore, it can be determined that the video scene of the video file is a scene presentation, such as landscape films, animations, advertisements, experimental videos and other visually dominated content.

[0050] It should be noted that, in an actual video file, the first text information may exist, but the second text information is not used. This usually means that the video is being described, and it can be understood that no video description is needed (eg, the ending of a movie).

[0051] Optionally, in practical applications, other features (such as color, texture, motion information, etc. of the video frame) may also be combined to determine the video scene.

[0052] As an optional embodiment, before selecting target data corresponding to the video scene from the multimodal data for fusion, the method also includes obtaining a visual feature representation in the following manner: extracting a visual feature vector sequence of the video frame data, wherein each visual feature vector in the visual feature vector sequence corresponds to a video frame in the video frame data; aligning each visual feature vector in the visual feature vector sequence to an input space to obtain a visual feature representation, wherein the input space is a data structure that can be recognized by a preset recognition model.

[0053] Extracting a visual feature vector sequence from video frame data includes: using a visual encoder to process each video frame, extracting visual information contained therein, and generating a corresponding visual feature vector sequence. Each visual feature vector corresponds to a video frame in the video frame data, and they together constitute a visual feature vector sequence.

[0054] The input space is a data structure that can be recognized by the preset recognition model, which defines the format and dimension of the model input data. In order for the visual feature vector to be recognized by the preset recognition model, the visual feature vector needs to be aligned to the input space. The specific alignment step involves scaling, normalizing, transforming and other operations on the visual feature vector to ensure that they meet the specifications of the input space.

[0055] The aligned visual feature vector sequence constitutes the final visual feature representation, which contains useful visual information in the video frame data and can be directly processed by the preset recognition model.

[0056] As an optional embodiment, before selecting target data corresponding to the video scene from the multimodal data for fusion, the method also includes obtaining an audio feature representation in the following manner: extracting an audio feature vector sequence of the audio data, wherein each audio feature vector in the audio feature vector sequence corresponds to an audio segment in the audio data; aligning each audio feature vector in the audio feature vector sequence to an input space to obtain an audio feature representation.

[0057] Extracting an audio feature vector sequence from audio data includes: using an audio encoder to process the audio data, extracting background sound and music type information contained therein, and generating a corresponding audio feature vector sequence. Each audio feature vector corresponds to an audio segment in the audio data, and they together constitute an audio feature vector sequence.

[0058] Similarly, in order for the audio feature vector to be correctly processed by the recognition model, the audio feature vector needs to be aligned to the input space. The alignment process involves scaling, normalizing, and transforming the audio feature vector to ensure that they meet the specifications of the input space.

[0059] The aligned audio feature vector sequence constitutes the final audio feature representation, which contains useful information in the audio data and can be directly processed by the preset recognition model.

[0060] By extracting the visual feature vectors of the video frames and the audio feature vectors of the audio data as described above, multiple aspects of the video content can be captured, thereby obtaining multimodal data. The fusion of such multimodal data can provide a more comprehensive and detailed description of the video scene.

[0061] As an optional embodiment, target data corresponding to the video scene is selected from the multimodal data for fusion, including: if the video scene is a character dialogue, the visual feature representation and the first text information are determined as target data, and the visual feature representation and the first text feature corresponding to the first text information are fused; if the video scene is background music, the visual feature representation and the second text information are determined as target data, and the visual feature representation and the second text feature corresponding to the second text information are fused; if the video scene is a scene presentation, the visual feature representation and the audio feature representation are determined as target data, and the visual feature representation and the audio feature representation are fused.

[0062] If the video scene is a conversation between people, the visual feature representation (visual information extracted from the video frame) and the first text feature corresponding to the first text information are selected for fusion. Specifically, this can be achieved through feature splicing, feature fusion algorithm or deep learning model. The fused data can more comprehensively reflect the visual and text information in the conversation scene.

[0063] If the video scene is background music, the visual feature representation and the second text feature corresponding to the second text information are selected for fusion, thereby helping the model to better understand the relationship between the background music and the video content, and the role of the background music in the video.

[0064] If the video scene is a situational presentation, then the visual feature representation and the audio feature representation are selected for fusion, which can make full use of the complementarity of visual and audio information and improve the model's ability to recognize and understand the situational presentation scene.

[0065] Specifically, the choice of fusion algorithm should be determined according to the specific application scenario and recognition task. Different fusion algorithms may be suitable for different types of features and fusion requirements.

[0066] By selecting and fusing multimodal data according to different video scenes, the recognition accuracy, robustness and adaptability of the model can be improved. At the same time, it can also make full use of the complementarity of multimodal data to provide a more comprehensive and detailed description of the video scene.

[0067] As an optional embodiment, the method also includes obtaining a preset recognition model in the following manner: obtaining a training data set, wherein the training data set includes training data of different video scenes; training an initial model using the training data set to obtain training results; when the training results indicate that the recognition accuracy of the initial model for the video description of each video scene is greater than or equal to a target threshold, determining the initial model as the preset recognition model.

[0068] The training data set contains training data of different video scenes. The training data of different video scenes are representative and can cover multiple video scenes that the model needs to recognize.

[0069] The initial model is trained using the training data set to allow the model to learn how to extract useful features from the input data and perform accurate classification or recognition based on these features.

[0070] During or after the training process, the performance of the model needs to be evaluated based on the training results. If the training results indicate that the recognition accuracy of the initial model for the video description of each video scene is greater than or equal to the target threshold, then it can be considered that the model has sufficient recognition ability, and the initial model is determined as the preset recognition model and used for subsequent video scene recognition tasks.

[0071] Figure 2 This is a schematic diagram of an optional audio-based video description generation framework provided according to an embodiment of the present application. As shown in the figure, a visual image sequence (corresponding to the video frame data of the present application) and audio data are extracted from the video data; OCR recognition is performed on the visual image sequence and ASR recognition is performed on the audio data to obtain text auxiliary information; video type classification (i.e., division of video scenes) is determined based on the text auxiliary information; corresponding multimodal information is selected for fusion based on the video scene, and the fused data is input into the text model to output the video content description of the video data. Among them, the multimodal information includes text auxiliary information, audio feature representation and video feature representation. The video feature representation is obtained by encoding the visual image sequence by the visual encoder and then mapping the visual features by the Video Q-former. The audio feature representation is obtained by encoding the audio data by the audio encoder and then mapping the audio features by the Audio Q-former. The audio feature representation and the video feature representation need to be aligned to the input space corresponding to the text model.

[0072] The present application proposes a method for generating video descriptions in combination with audio information, which divides the video into three main scenes: character dialogue, background music, and plot presentation. Each scene uses its own corresponding method to generate description information, thereby achieving a more comprehensive understanding of the video content.

[0073] The present application adopts a method for generating video description based on audio, including: extracting video frame data and audio data from a video file, and attempting to identify the first text information of the video frame data and the second text information of the audio data; judging the video scene in the video file according to whether the first text information and the second text information exist; selecting target data corresponding to the video scene from multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, a first text information and a second text information; identifying the data to be identified by a preset recognition model to obtain a video description corresponding to the video file. The video scene is identified by combining the audio in the video, and different multimodal data are fused based on different video scenes, and finally the multimodal data is identified to obtain a video description. Since the identification of the video scene and the use of audio data are both included in the multimodal data, the auditory information is fully considered when generating the video description, solving the problem that the generated video description is not comprehensive enough due to ignoring the auditory information.

[0074] According to another aspect of the embodiment of the present application, the present application provides a device for generating audio-based video description, such as Figure 3 As shown, including:

[0075] An extraction module 302 is used to extract video frame data and audio data from a video file, and attempt to identify first text information of the video frame data and second text information of the audio data;

[0076] A determination module 304, configured to determine a video scene in the video file according to whether the first text information and the second text information exist;

[0077] A fusion module 306 is used to select target data corresponding to the video scene from the multimodal data for fusion to obtain data to be recognized, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, the first text information, and the second text information;

[0078] The recognition module 308 is used to recognize the data to be recognized by using a preset recognition model to obtain a video description corresponding to the video file.

[0079] It should be noted that the extraction module 302 in this embodiment can be used to execute step 101 in the embodiment of the present application, the judgment module 304 in this embodiment can be used to execute step 103 in the embodiment of the present application, the fusion module 306 in this embodiment can be used to execute step 105 in the embodiment of the present application, and the recognition module 308 in this embodiment can be used to execute step 107 in the embodiment of the present application.

[0080] Optionally, the extraction module 302 is also used to attempt to detect text areas in each video frame in the video frame data, and extract first text information from the text area if a text area exists; attempt to extract human voice data from the audio data, and perform text recognition on the human voice data if human voice data exists to obtain second text information.

[0081] Optionally, the judgment module 304 is also used to determine that the video scene of the video file is a character dialogue if the first text information exists and the second text information exists; if the first text information does not exist and the second text information exists, then determine that the video scene of the video file is background music; if the first text information does not exist and the second text information does not exist, then determine that the video scene of the video file is a scene presentation.

[0082] Optionally, the device also includes a first acquisition module, which is used to obtain a visual feature representation in the following manner before selecting target data corresponding to the video scene from the multimodal data for fusion: extracting a visual feature vector sequence of the video frame data, wherein each visual feature vector in the visual feature vector sequence corresponds to a video frame in the video frame data; aligning each visual feature vector in the visual feature vector sequence to an input space to obtain a visual feature representation, wherein the input space is a data structure that can be recognized by a preset recognition model.

[0083] Optionally, the device also includes a second acquisition module, which is used to obtain an audio feature representation in the following manner before selecting target data corresponding to the video scene from the multimodal data for fusion: extracting an audio feature vector sequence of the audio data, wherein each audio feature vector in the audio feature vector sequence corresponds to an audio segment in the audio data; aligning each audio feature vector in the audio feature vector sequence to the input space to obtain an audio feature representation.

[0084] Optionally, the fusion module 306 is also used to determine the visual feature representation and the first text information as target data if the video scene is a character dialogue, and to fuse the visual feature representation and the first text feature corresponding to the first text information; if the video scene is background music, then determine the visual feature representation and the second text information as target data, and to fuse the visual feature representation and the second text feature corresponding to the second text information; if the video scene is a scene presentation, then determine the visual feature representation and the audio feature representation as target data, and to fuse the visual feature representation and the audio feature representation.

[0085] Optionally, the device also includes a determination module for obtaining a preset recognition model in the following manner: obtaining a training data set, wherein the training data set includes training data of different video scenes; training an initial model using the training data set to obtain training results; when the training results indicate that the recognition accuracy of the initial model for the video description of each video scene is greater than or equal to a target threshold, determining the initial model as the preset recognition model.

[0086] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0087] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0088] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0089] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0090] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0091] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0092] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0093] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0094] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0095] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0096] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0102] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for generating audio-based video description, characterized in that: include: Extracting video frame data and audio data from a video file, and attempting to identify first text information of the video frame data and second text information of the audio data; Determining a video scene in the video file according to whether the first text information and the second text information exist; Selecting target data corresponding to the video scene from the multimodal data for fusion to obtain data to be recognized, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, the first text information, and the second text information; The data to be identified is identified by using a preset identification model to obtain a video description corresponding to the video file.

2. The method according to claim 1, characterized in that The attempting to identify the first text information of the video frame data and the second text information of the audio data comprises: Trying to detect a text region of each video frame in the video frame data, and extracting the first text information of the text region if the text region exists; An attempt is made to extract human voice data from the audio data, and if the human voice data exists, text recognition is performed on the human voice data to obtain the second text information.

3. The method according to claim 1, characterized in that The determining of the video scene in the video file according to whether the first text information and the second text information exist includes: If the first text information exists and the second text information exists, determining that the video scene of the video file is a character dialogue; If the first text information does not exist, and the second text information exists, determining that the video scene of the video file is background music; If the first text information does not exist and the second text information does not exist, it is determined that the video scene of the video file is a scene presentation.

4. The method according to claim 1, characterized in that: Before selecting the target data corresponding to the video scene from the multimodal data for fusion, the method further includes obtaining the visual feature representation in the following manner: Extracting a visual feature vector sequence of the video frame data, wherein each visual feature vector in the visual feature vector sequence corresponds to a video frame in the video frame data; Each of the visual feature vectors in the visual feature vector sequence is aligned to an input space to obtain the visual feature representation, wherein the input space is a data structure that can be recognized by the preset recognition model.

5. The method according to claim 1, characterized in that Before selecting the target data corresponding to the video scene from the multimodal data for fusion, the method further includes obtaining the audio feature representation in the following manner: Extracting an audio feature vector sequence of the audio data, wherein each audio feature vector in the audio feature vector sequence corresponds to an audio segment in the audio data; Each of the audio feature vectors in the audio feature vector sequence is aligned to an input space to obtain the audio feature representation.

6. The method according to claim 3, characterized in that The selecting target data corresponding to the video scene from the multimodal data for fusion includes: If the video scene is a conversation between the characters, determining the visual feature representation and the first text information as the target data, and fusing the visual feature representation and the first text feature corresponding to the first text information; If the video scene is the background music, determining the visual feature representation and the second text information as the target data, and fusing the visual feature representation and the second text feature corresponding to the second text information; If the video scene is presented as the scenario, the visual feature representation and the audio feature representation are determined as the target data, and the visual feature representation and the audio feature representation are fused.

7. The method according to claim 1, characterized in that The method further comprises obtaining the preset recognition model in the following manner: Acquire a training data set, wherein the training data set includes training data of different video scenes; Using the training data set to train the initial model to obtain a training result; When the training result indicates that the recognition accuracy of the initial model for the video description of each of the video scenes is greater than or equal to a target threshold, the initial model is determined as the preset recognition model.

8. A device for generating audio-based video description, characterized in that: include: An extraction module, used for extracting video frame data and audio data from a video file, and attempting to identify first text information of the video frame data and second text information of the audio data; A judging module, configured to judge the video scene in the video file according to whether the first text information and the second text information exist; A fusion module, configured to select target data corresponding to the video scene from the multimodal data for fusion to obtain data to be identified, wherein the multimodal data includes a visual feature representation corresponding to the video frame data, an audio feature representation corresponding to the audio data, the first text information, and the second text information; The recognition module is used to recognize the data to be recognized by using a preset recognition model to obtain a video description corresponding to the video file.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 7.

Images