A video text generation method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and storage medium for generating video text, wherein the method includes: obtaining video data to be processed, and extracting key frame image data and audio data in the video data; obtaining text information of the video data through a preset optical character recognition model and a speech recognition model according to the key frame image data and audio data; obtaining character information of the video data through a preset biometric recognition model according to the key frame image data and audio data; and generating video text corresponding to the video data according to the text information and character information. The present application integrates biometric recognition, optical character recognition and speech recognition technologies to generate video text containing more comprehensive image, speech, text and other content information, which can greatly improve the efficiency of users' search and reading of video content, and solve the technical problem of low efficiency in existing video data search.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video text generation method, device, equipment and storage medium. Background Art

[0002] With the rise of the multimedia industry, video data has been integrated into our daily lives, and a large amount of video data is flooding our daily lives. In order to accurately search for the desired video data or quickly understand the main content of the video, the quality of video text generation is crucial.

[0003] Traditional video text generation focuses on optical character recognition or speech recognition, which makes it difficult to obtain more comprehensive video information for generating video text, resulting in technical problems of low efficiency in existing video data searches. Summary of the invention

[0004] The present application provides a video text generation method, device, equipment and storage medium, which are used to solve the technical problem of low efficiency in existing video data search.

[0005] The first aspect of the present application provides a video text generation method, comprising:

[0006] Acquire video data to be processed, and extract key frame image data and audio data from the video data;

[0007] According to the key frame image data and the audio data, the text information of the video data is obtained by using a preset optical character recognition model and a speech recognition model;

[0008] According to the key frame image data and the audio data, the character information of the video data is obtained by using a preset biometric feature recognition model;

[0009] A video text corresponding to the video data is generated according to the text information and the character information.

[0010] Preferably, obtaining the text information of the video data according to the key frame image data and the audio data through a preset optical character recognition model and a speech recognition model specifically includes:

[0011] According to the key frame image data, recognizing text characters in the key frame image data by using an optical character recognition model to obtain image text information of the video data;

[0012] According to the audio data, the audio data is converted into text data through a speech recognition model to obtain the speech and text information of the video data.

[0013] Preferably, the biometric recognition model includes: a face recognition model and a voiceprint recognition model.

[0014] Preferably, obtaining the character information of the video data according to the key frame image data and the audio data through a preset biometric feature recognition model specifically includes:

[0015] According to the key frame image data, identifying the facial features in the key frame image data through a face recognition model, so as to perform feature comparison according to the facial features to obtain the image character information of the video data;

[0016] According to the audio data, the voiceprint features in the audio data are identified through a voiceprint recognition model, so as to perform feature comparison according to the voiceprint features to obtain the voice character information of the video data.

[0017] Preferably, generating a video text corresponding to the video data according to the text information and the character information specifically includes:

[0018] Determine a first corresponding relationship according to the time information of the image text information and the time information of the voice text information, wherein the first corresponding relationship is a corresponding relationship between the image text information and the voice text information;

[0019] Determining a second corresponding relationship according to the time information of the image character information and the time information of the voice character information, wherein the second corresponding relationship is a corresponding relationship between the image character information and the voice character information;

[0020] Based on the first corresponding relationship and the second corresponding relationship, the image character information, the image text information and the voice character information, and the voice text information are summarized to obtain the video text.

[0021] Preferably, the process of acquiring the key frame image data specifically includes:

[0022] Performing frame extraction processing on the video data to obtain multiple video frame images;

[0023] Calculating the similarity between the video frame images according to the image similarity calculation method;

[0024] According to the similarity and a preset similarity threshold, the video frame images are compared and screened for similarity to obtain key frame images.

[0025] Preferably, the process of acquiring the audio data specifically includes:

[0026] Performing audio extraction processing on the video data to obtain embedded audio data of the video data;

[0027] Voice endpoint activity detection is performed on the embedded audio data, and audio segmentation processing is performed on the embedded audio data according to the detected voice endpoint to obtain audio data segmented into multiple segments.

[0028] The second aspect of the present application provides a video text generation device, comprising:

[0029] A data extraction unit, used to obtain the video data to be processed, and extract key frame image data and audio data from the video data;

[0030] A text information acquisition unit, used to obtain text information of the video data according to the key frame image data and the audio data through a preset optical character recognition model and a speech recognition model;

[0031] A character information acquisition unit, configured to obtain character information of the video data through a preset biometric feature recognition model according to the key frame image data and the audio data;

[0032] A video text generating unit is used to generate a video text corresponding to the video data according to the text information and the character information.

[0033] A third aspect of the present application provides a video text generation device, including: a memory and a processor;

[0034] The memory is used to store program codes corresponding to the video text generation method provided in the first aspect of the present application;

[0035] The processor is configured to execute the program code.

[0036] The fourth aspect of the present application provides a computer-readable storage medium, in which a program code corresponding to the video text generation method provided in the first aspect of the present application is stored.

[0037] It can be seen from the above technical solutions that this application has the following advantages:

[0038] The present application provides a method, device, equipment and storage medium for generating video text, wherein the method comprises: obtaining video data to be processed, and extracting key frame image data and audio data from the video data; obtaining text information of the video data through a preset optical character recognition model and a speech recognition model according to the key frame image data and audio data; obtaining character information of the video data through a preset biometric recognition model according to the key frame image data and audio data; generating video text corresponding to the video data according to the text information and the character information. The present application integrates biometric recognition, optical character recognition and speech recognition technologies to generate video text containing more comprehensive image, speech, text and other content information, which can greatly improve the efficiency of users' search and reading of video content, solve the technical problem of low efficiency in existing video data search, and can also accurately migrate and apply to search, recommendation and other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of a flow chart of a first embodiment of a video text generation method provided by the present application;

[0041] Figure 2 A schematic flow chart of a second embodiment of a video text generation method provided by the present application;

[0042] Figure 3 A schematic diagram of a process for obtaining key frame image data in a video text generation method provided in the present application;

[0043] Figure 4 A schematic diagram of the process of obtaining audio data in a video text generation method provided in this application;

[0044] Figure 5 A schematic structural diagram of a first embodiment of a video text generation device provided in the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application provide a video text generation method, apparatus, device and storage medium, which are used to solve the technical problem of low efficiency in existing video data search.

[0046] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] See also Figure 1 The first embodiment of the present application provides a video text generation method, comprising:

[0048] Step 101: Obtain video data to be processed, and extract key frame image data and audio data from the video data.

[0049] Step 102: According to the key frame image data and the audio data, the text information of the video data is obtained through a preset optical character recognition model and a speech recognition model.

[0050] Step 103: According to the key frame image data and the audio data, the character information of the video data is obtained through a preset biometric feature recognition model.

[0051] Step 104: Generate video text corresponding to the video data according to the text information and character information.

[0052] It should be noted that in implementing a video text generation method provided in this embodiment, first starting from step 101, the video data to be processed is obtained, and a number of key frame images and audio data are extracted from the video data. Then, based on the key frame image data and audio data obtained in step 101, corresponding content recognition operations are performed through preset optical character recognition models and speech recognition models respectively, so as to obtain text information of the video data. At the same time, the key frame image data and the character features in the audio data are identified through a preset biometric recognition model to perform character recognition and obtain character information of the video data. Finally, according to the text information obtained in step 102 and the character information obtained in step 103, the corresponding video text is generated, so as to facilitate summarization, keyword tagging and other processing according to the generated video text, thereby greatly improving the efficiency of users' search and reading of video content.

[0053] Among them, although the method flow provided in the embodiment of the present application is to obtain text information first and then obtain character information, in actual execution, the steps of obtaining text information and obtaining character information can be interchanged or executed simultaneously, which will not be repeated here.

[0054] The above is a detailed description of the first embodiment of a video text generation method provided by the present application. The following is a detailed description of the second embodiment of a video text generation method provided by the present application.

[0055] See also Figure 2 , Figure 3 and Figure 4 The second embodiment of the present application provides a video text generation method based on the above first embodiment, including:

[0056] Furthermore, step 102 in the first embodiment specifically includes:

[0057] Step 1021: Recognize text characters in the key frame image data by using an optical character recognition model according to the key frame image data to obtain image text information of the video data;

[0058] It should be noted that, in step 1021 of this embodiment, the video key frames are input into a pre-trained optical character recognition model to obtain the image text results corresponding to each video key frame; and then the image text results of similar video key frames are screened to obtain the image text information.

[0059] The optical character recognition model of this embodiment is specifically a recognition model obtained by pre-preparing image text data with text labels and combining it with deep neural network structure training.

[0060] Step 1022: According to the audio data, the audio data is converted into text data through a speech recognition model to obtain the speech and text information of the video data.

[0061] It should be noted that step 1022 of this embodiment obtains a speech-to-text result by inputting the audio feature vector in the audio data into a pre-trained speech recognition model; and then performs post-processing such as punctuation recovery, digital conversion, and hot word correction on the speech result to generate speech-to-text information.

[0062] Among them, the speech recognition model of this embodiment is specifically a recognition model obtained by pre-preparing a large amount of speech data with text labels combined with deep neural network structure training.

[0063] Further, it includes: the biometric recognition model includes: a face recognition model and a voiceprint recognition model.

[0064] Furthermore, step 103 in the first embodiment above specifically includes:

[0065] Step 1031: According to the key frame image data, the face features in the key frame image data are identified by a face recognition model, so as to perform feature comparison according to the face features to obtain image character information of the video data;

[0066] Step 1032: According to the audio data, the voiceprint features in the audio data are identified by a voiceprint recognition model, so as to perform feature comparison according to the voiceprint features to obtain the voice character information of the video data.

[0067] Furthermore, step 104 in the first embodiment specifically includes:

[0068] Step 1041: determining a first corresponding relationship according to the time information of the image text information and the time information of the voice text information, wherein the first corresponding relationship is a corresponding relationship between the image text information and the voice text information;

[0069] Step 1042: determining a second corresponding relationship according to the time information of the image character information and the time information of the voice character information, wherein the second corresponding relationship is a corresponding relationship between the image character information and the voice character information;

[0070] Step 1043: Based on the first corresponding relationship and the second corresponding relationship, the image character information, the image text information, the voice character information, and the voice text information are summarized to obtain the video text.

[0071] Furthermore, the key frame image data mentioned in the first embodiment above specifically includes the following steps of obtaining the key frame image data:

[0072] Step 201, performing frame extraction processing on the video data to obtain a plurality of video frame images;

[0073] Step 202: Calculate the similarity between video frame images according to the image similarity calculation method;

[0074] Step 203: perform similarity comparison and screening on the video frame images according to the similarity and a preset similarity threshold to obtain a key frame image.

[0075] It should be noted that this embodiment calculates image similarity based on video frame images extracted from video data, and then compares the video frame images based on the calculated similarity and a preset similarity threshold, and selects key frame images from multiple video frame images based on the obtained similarity comparison results.

[0076] Furthermore, the audio data mentioned in the first embodiment above, its acquisition process specifically includes:

[0077] Step 301: Perform audio extraction processing on video data to obtain embedded audio data of the video data;

[0078] Step 302: perform voice endpoint activity detection on the embedded audio data, and perform audio segmentation processing on the embedded audio data according to the detected voice endpoints to obtain audio data segmented into multiple segments.

[0079] The above is a detailed description of the second embodiment of a video text generation method provided by the present application. The following is a detailed description of an embodiment of a video text generation device provided by the present application.

[0080] See also Figure 5 The third embodiment of the present application provides a video text generation device, including:

[0081] The data extraction unit 301 is used to obtain the video data to be processed and extract the key frame image data and audio data in the video data;

[0082] The text information acquisition unit 302 is used to obtain the text information of the video data according to the key frame image data and the audio data through a preset optical character recognition model and a speech recognition model;

[0083] The character information acquisition unit 303 is used to obtain the character information of the video data through a preset biometric feature recognition model according to the key frame image data and the audio data;

[0084] The video text generating unit 304 is used to generate video text corresponding to the video data according to the text information and character information.

[0085] The above is a detailed description of an embodiment of a video text generation device provided by the present application. The following is a detailed description of an embodiment of a video text generation device and an embodiment of a computer-readable storage medium provided by the present application.

[0086] A fourth embodiment of the present application provides a video text generation device, including: a memory and a processor;

[0087] The memory is used to store program codes corresponding to the video text generation method provided in the first embodiment or the second embodiment of the present application;

[0088] The processor is used to execute program code.

[0089] The fifth embodiment of the present application provides a computer-readable storage medium, in which a program code corresponding to the video text generation method provided in the first embodiment or the second embodiment of the present application is stored.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0091] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0092] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0095] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0096] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video text generation method, characterized in that: include: Acquire video data to be processed, and extract key frame image data and audio data from the video data; According to the key frame image data and the audio data, the text information of the video data is obtained by using a preset optical character recognition model and a speech recognition model; According to the key frame image data, identifying the facial features in the key frame image data through a face recognition model, so as to perform feature comparison according to the facial features to obtain the image character information of the video data; According to the audio data, using a voiceprint recognition model, identifying voiceprint features in the audio data, so as to perform feature comparison according to the voiceprint features to obtain voice character information of the video data; Determine a first corresponding relationship according to the time information of the image text information and the time information of the voice text information, wherein the first corresponding relationship is a corresponding relationship between the image text information and the voice text information; Determining a second corresponding relationship according to the time information of the image character information and the time information of the voice character information, wherein the second corresponding relationship is a corresponding relationship between the image character information and the voice character information; Based on the first corresponding relationship and the second corresponding relationship, the image character information, the image text information and the voice character information, and the voice text information are summarized to obtain the video text.

2. A video text generation method according to claim 1, characterized in that: According to the key frame image data and the audio data, the text information of the video data is obtained by using a preset optical character recognition model and a speech recognition model, specifically including: According to the key frame image data, recognizing text characters in the key frame image data by using an optical character recognition model to obtain image text information of the video data; According to the audio data, the audio data is converted into text data through a speech recognition model to obtain the speech and text information of the video data.

3. A video text generation method according to claim 1, characterized in that: The key frame image data acquisition process specifically includes: Performing frame extraction processing on the video data to obtain multiple video frame images; Calculating the similarity between the video frame images according to the image similarity calculation method; According to the similarity and a preset similarity threshold, the video frame images are compared and screened for similarity to obtain key frame images.

4. A video text generation method according to claim 1, characterized in that: The process of obtaining the audio data specifically includes: Performing audio extraction processing on the video data to obtain embedded audio data of the video data; Voice endpoint activity detection is performed on the embedded audio data, and audio segmentation processing is performed on the embedded audio data according to the detected voice endpoint to obtain audio data segmented into multiple segments.

5. A video text generation device, characterized in that: include: A data extraction unit, used to obtain the video data to be processed, and extract key frame image data and audio data from the video data; A text information acquisition unit, used to obtain text information of the video data according to the key frame image data and the audio data through a preset optical character recognition model and a speech recognition model; A character information acquisition unit is used to identify the face features in the key frame image data through a face recognition model according to the key frame image data, so as to perform feature comparison according to the face features to obtain the image character information of the video data; and to identify the voiceprint features in the audio data through a voiceprint recognition model according to the audio data, so as to perform feature comparison according to the voiceprint features to obtain the voice character information of the video data; A video text generation unit is used to determine a first corresponding relationship based on time information of image text information and time information of voice text information, wherein the first corresponding relationship is the corresponding relationship between the image text information and the voice text information; determine a second corresponding relationship based on time information of the image character information and time information of the voice character information, wherein the second corresponding relationship is the corresponding relationship between the image character information and the voice character information; based on the first corresponding relationship and the second corresponding relationship, the image character information, image text information, voice character information, and voice text information are summarized to obtain a video text.

6. A video text generation device, characterized in that: include: Memory and processor; The memory is used to store a program code corresponding to the video text generation method according to any one of claims 1 to 4; The processor is configured to execute the program code.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code corresponding to the video text generation method according to any one of claims 1 to 4, and when the program code is executed by a processor, the video text generation method according to any one of claims 1 to 4 is implemented.

Images