Intelligent and accurate extraction method and system for video axis file based on OCR

Abstract

The application relates to the technical field of optical character recognition, and discloses a video axis file intelligent accurate extraction method and system based on OCR, which comprises the following steps: extracting OCR text features, image features and audio features of a target video frame; analyzing the target video frame to obtain a scene type, and fusing feature vectors to obtain a weighted feature vector based on a pre-trained adaptive attention model and the fused feature vectors; modeling the weighted feature vector to obtain hidden state sequence information; and generating a target video frame axis file based on the hidden state sequence information, a long short-term memory network model, extraction task information, the scene type, a pre-trained deep network method and the target video frame. Through implementation of the application, information and methods are comprehensively utilized, extraction strategies can be automatically adjusted according to different video types and contents, accurate and complete video frame axis files can be generated, and the problems of poor accuracy and completeness of video axis files extracted by traditional methods are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical character recognition, and particularly relates to an OCR-based intelligent and accurate video axis file extraction method and system. BACKGROUND

[0002] In today's era of digital information explosion, video as an important information carrier is widely used in various fields. Video axis file records the correspondence between video content and time, which is crucial for video editing, content retrieval, video analysis, etc. In video editing, accurate video axis file can help editors quickly locate and edit video clips; in video content retrieval, users can quickly find the required video content according to the text information in the video axis file.

[0003] However, for the text information in the video, the traditional method often relies on manual annotation, which is low in efficiency and prone to errors. There are also some automatic extraction methods, but due to the lack of comprehensive understanding and analysis of video content, the accuracy and completeness of the extracted video axis file are poor. Therefore, how to efficiently and accurately extract the video axis file is a problem to be solved. SUMMARY

[0004] Therefore, the present application provides an OCR-based intelligent and accurate video axis file extraction method and system computer device and storage medium to solve the problem of how to efficiently and accurately extract the video axis file.

[0005] In a first aspect, the present application provides an OCR-based intelligent and accurate video axis file extraction method, which comprises: obtaining a to-be-extracted video and extraction task information, preprocessing the to-be-extracted video to obtain a target video frame; extracting the OCR text features, image features and audio features of the target video frame; performing scene analysis on the target video frame to obtain a scene type; obtaining a fusion feature vector based on the scene type, the OCR text features, the image features and the audio features; obtaining a weighted feature vector based on a pre-trained adaptive attention model and the fusion feature vector; modeling the weighted feature vector using a long short-term memory network model to obtain hidden state sequence information; and generating a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, a pre-trained deep network method and the target video frame.

[0006] The method for intelligently and accurately extracting an OCR-based video axis file provided in this embodiment first reduces the amount of calculation, removes noise, and enhances the edges of the text through preprocessing operations such as decoding, sampling, and image enhancement on the video, thereby laying a foundation for subsequent accurate extraction of various features and improving the overall processing efficiency. Second, by comprehensively extracting the multi-modal features of the video frames, the method changes the traditional way of relying on manual annotation of text information, avoids the low efficiency and error-prone problems of manual annotation, and obtains video content information from multiple dimensions. Then, by analyzing the scene type, the method provides a basis for subsequent feature fusion and extraction strategy formulation, enabling the system to perform differential processing for different scenes and improving the comprehensiveness of video content understanding. Further, by fusing multi-modal features, the method makes up for the defects of existing automatic extraction methods that lack comprehensive understanding of video content, comprehensively considers multiple modal information, improves the expression ability of video content, and helps to improve the accuracy and completeness of extraction. Then, by using a pre-trained adaptive attention model, the method dynamically allocates attention weights according to the multi-modal fusion features, highlights key features, and further improves the ability to capture key video content, thereby improving the accuracy of extraction. Next, by modeling the weighted feature vector sequence using a long short-term memory network, the method effectively captures the temporal dependency of video content, enabling the system to better process information in the time dimension of the video and improving the accuracy and completeness of video content understanding. Finally, by comprehensively utilizing multiple information and pre-trained deep network methods, the system can automatically adjust the extraction strategy according to different video types and content to generate accurate and complete video frame axis files, solving the problem of poor accuracy and completeness of video axis files extracted by traditional methods. In summary, the method for intelligently and accurately extracting an OCR-based video axis file provided in this embodiment effectively solves the problem of how to efficiently, accurately, and completely extract a video axis file.

[0007] In an optional implementation, the method further includes obtaining a to-be-extracted video and extraction task information, preprocessing the to-be-extracted video to obtain target video frames, and the preprocessing includes: converting the to-be-extracted video into a video frame sequence by using a video decoding library; sampling the video frame sequence based on a preset sampling interval to obtain sampled video frames; and performing histogram equalization processing on the sampled video frames to obtain the target video frames.

[0008] In an optional implementation, the method further includes extracting OCR text features of the target video frames, and the extracting includes: recognizing characters in the target video frames by using an OCR engine to obtain text information; performing word segmentation processing, part-of-speech tagging processing, and keyword extraction processing on the text information to obtain word frequency data and inverse document frequency data of each keyword; and constructing the OCR text features based on the word frequency data and the inverse document frequency data of each keyword.

[0009] In an optional implementation, the image feature of the target video frame is extracted, including: inputting the target video frame into a convolution layer of a pre-trained convolutional neural network, generating a preliminary feature map based on the target video frame by using the convolution layer, the preliminary feature map containing edge features and texture features; inputting the preliminary feature map into a pooling layer, dividing the preliminary feature map into a plurality of small regions by using the pooling layer, performing maximum pooling processing on each small region, taking the maximum value in each small region as the output, and obtaining a reduced feature map; flattening the reduced feature map into a one-dimensional vector, mapping the one-dimensional vector into a fixed-length vector by using a fully connected layer, and obtaining the image feature.

[0010] In an optional implementation, the pre-trained convolutional neural network comprises at least three groups of convolution layers and pooling layers.

[0011] In an optional implementation, the fusion feature vector is obtained based on the scene type, the OCR text feature, the image feature, and the audio feature, including: determining a default weight based on a corresponding relationship between the default weight and the scene type and the scene type; and performing feature fusion on the OCR text feature, the image feature, and the audio feature based on the default weight to obtain the fusion feature vector.

[0012] In an optional implementation, the weighted feature vector is obtained based on the pre-trained adaptive attention model and the fusion feature vector, including: calculating an attention weight vector based on the pre-trained adaptive attention model and the fusion feature vector; and performing weighted processing on the fusion feature vector by using the attention weight vector to obtain the weighted feature vector.

[0013] In an optional implementation, the target video frame axis file is generated based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, the pre-trained deep network method, and the target video frame, including: defining environment state information based on the hidden state sequence information; defining an extraction strategy set based on the scene type; defining a reward strategy set based on the extraction task information; obtaining a target extraction strategy set based on the environment state information, the extraction strategy set, and the reward strategy set by using the pre-trained deep network method; and processing the target video frame based on the target extraction strategy set to generate the target video frame axis file.

[0014] In an alternative embodiment, the method further comprises: extracting target OCR text features, target image features and target audio features of the target video frame axis file; analyzing the target OCR text features, the target image features and the target audio features by using a fusion analysis technology based on a Transformer architecture, to obtain consistency data of the target OCR text features, the target image features and the target audio features on a time axis, and complementarity data of the target OCR text features, the target image features and the target audio features on the time axis; obtaining collaborative score data based on the consistency data and the complementarity data; and optimizing the target video frame axis file based on a reinforcement learning model, a meta-learning model and the collaborative score data, to obtain an optimized video frame axis file.

[0015] In a second aspect, the present application provides an OCR-based intelligent and accurate video axis file extraction system, which comprises: a preprocessing module for obtaining a video to be extracted and extraction task information, and preprocessing the video to be extracted to obtain a target video frame; an extraction module for extracting OCR text features, image features and audio features of the target video frame; an analysis module for performing scene analysis on the target video frame to obtain a scene type; a fusion module for obtaining a fusion feature vector based on the scene type, the OCR text features, the image features and the audio features; a calculation module for obtaining a weighted feature vector based on a pre-trained adaptive attention model and the fusion feature vector; a modeling module for modeling the weighted feature vector by using a long short-term memory network model to obtain hidden state sequence information; and a generation module for generating a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, a pre-trained deep network method and the target video frame. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0017] Figure 1 is a flowchart of an OCR-based intelligent and accurate video axis file extraction method according to an embodiment of the present application;

[0018] Figure 2 is a structural block diagram of an OCR-based intelligent and accurate video axis file extraction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0020] In today's era of digital information explosion, video as an important information carrier is widely used in various fields. Video axis file records the correspondence between video content and time, which is crucial for video editing, content retrieval, video analysis and other applications. In video editing, accurate video axis file can help editors quickly locate and edit video clips; in video content retrieval, users can quickly find the required video content according to the text information in the video axis file.

[0021] However, for the text information in the video, the traditional method often relies on manual annotation, which is inefficient and prone to errors. There are also some automatic extraction methods, but due to the lack of comprehensive understanding and analysis of video content, the accuracy and completeness of the extracted video axis file are poor. Therefore, how to efficiently and accurately extract the video axis file is a problem to be solved.

[0022] The OCR-based video axis file intelligent accurate extraction method provided in the embodiment first reduces the amount of calculation, removes noise and enhances the edges of the text through decoding, sampling and image enhancement preprocessing operations on the video, lays a foundation for subsequent accurate extraction of various features, and improves the overall processing efficiency. Secondly, by comprehensively extracting the multi-modal features of the video frames, the traditional method of relying on manual annotation of text information is changed, and the problems of low efficiency and error-prone of manual annotation are avoided, and the video content information is obtained from multiple dimensions. Then, by analyzing the scene type, the basis for subsequent feature fusion and extraction strategy is provided, so that the system can be differentially processed according to different scenes, and the comprehensiveness of the video content understanding is improved. Further, by fusing multi-modal features, the defects of the existing automatic extraction method that lack comprehensive understanding of video content are made up, various modal information is comprehensively considered, the expression ability of the video content is improved, and the accuracy and integrity of the extraction are improved. Then, by using the pre-trained adaptive attention model, the attention weight is dynamically allocated according to the multi-modal fusion features, the key features are highlighted, and the capture ability of the key content of the video is further improved, so that the accuracy of the extraction is improved. Next, by modeling the weighted feature vector sequence through the long short-term memory network, the time dependence of the video content is effectively captured, so that the system can better process the information of the video in the time dimension, and the accuracy and integrity of the video content understanding are improved. Finally, by comprehensively utilizing various information and pre-trained deep network methods, the system can automatically adjust the extraction strategy according to different video types and contents to generate accurate and complete video frame axis files, solving the problem of poor accuracy and integrity of the video axis files extracted by the traditional method. In summary, the OCR-based video axis file intelligent accurate extraction method provided in the embodiment effectively solves the problem of how to efficiently, accurately and completely extract the video axis file.

[0023] According to the embodiment of the present application, an OCR-based video axis file intelligent accurate extraction method is provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0024] In the embodiment, an OCR-based video axis file intelligent accurate extraction method is provided, Figure 1 The flowchart of the OCR-based video axis file intelligent accurate extraction method according to the embodiment of the present application is shown in Figure 1 As shown in the figure, the flow includes the following steps:

[0025] In step S101, the video to be extracted and the extraction task information are obtained, the video to be extracted is preprocessed, and the target video frame is obtained.

[0026] The video to be extracted covers various storage formats of video files, such as common MP4, AVI, MKV, etc. These videos come from a wide range of sources, such as surveillance videos, film materials, teaching videos, etc. The extraction task information includes extraction target, format requirement and precision requirement. The extraction target represents the specific information to be obtained from the video, such as extracting all dialogue text in the video or only extracting the speech of a specific character; locating the time node of a key event or identifying the scene where a specific object appears in the video, etc. For example, in the film material extraction task, it may be required to extract the dialogue of the main character and the corresponding timestamp. The format requirement specifies the format of the final generated video axis file, such as CSV, JSON, etc. Different formats are suitable for different application scenarios, such as CSV format for data table display and JSON format for data transmission and program parsing. The precision requirement involves time precision and content precision. In terms of time precision, it specifies whether the extracted timestamp is accurate to seconds, milliseconds or finer granularity; in terms of content precision, it specifies the accuracy of text extraction and the detail level of image feature recognition, etc. For example, in surveillance video analysis, it may be required to have a timestamp accurate to milliseconds to accurately capture the event occurrence moment. In addition, there may be special extraction conditions set for specific video content. For example, when extracting teaching video axis files, it is required to skip the beginning and end of the film and only extract the part containing the board content; or when processing multi-language videos, it is required to specify the extraction of text information in a certain language.

[0027] Specifically, the above step S101 comprises:

[0028] Step a1, using a video decoding library to convert the video to be extracted into a video frame sequence.

[0029] Further, videos are usually stored in specific encoding formats, such as MP4, AVI, etc. These encoding formats use compression techniques to reduce the size of video files, making them easier to store and transmit. However, when extracting video axis files, the video needs to be decoded into a series of image frames for subsequent processing. Video decoding libraries are tools that achieve this function. For example, FFmpeg, a widely used open source multimedia framework, can support decoding of multiple video encoding formats. The specific operation process is to input the file path of the video to be extracted into the video decoding library as a parameter. The decoding library will parse the video content frame by frame according to the video encoding format and frame rate, generating a continuous video frame sequence. Video frame rate represents the number of frames displayed in one second, and common frame rates include 24fps, 30fps, 60fps, etc. For example, a 10-second video with a frame rate of 30fps will generate 300 video frames after decoding.

[0030] Step a2, sampling the video frame sequence based on a preset sampling interval to obtain a sampled video frame.

[0031] Further, all frames in the video frame sequence are not equally important for the video axis file extraction task, and processing all frames greatly increases the amount of calculation and processing time. Therefore, the video frame sequence needs to be sampled, and only a part of key frames are selected for subsequent processing. The preset sampling interval is a parameter set before sampling, indicating how many frames are selected as sampling frames. For example, if the preset sampling interval is 5, then in the video frame sequence, the 1st frame, the 6th frame, the 11th frame, …, are selected as sampling frames. By reasonably setting the sampling interval, the number of video frames that need to be processed can be effectively reduced under the premise of ensuring the accuracy of the extraction result, and the processing efficiency is improved. The setting of the sampling interval needs to be determined according to the complexity of the video content and the accuracy requirement of the extraction task. For the video with slow content change, the sampling interval can be appropriately increased; and for the video with frequent content change, the sampling interval needs to be reduced to ensure that key information is not missed.

[0032] Step a3, histogram equalization processing is performed on the sampling video frame to obtain a target video frame.

[0033] Further, the image quality of the sampling video frame obtained after sampling is different, for example, the brightness is uneven, the contrast is low, etc. These problems will affect the accuracy of subsequent OCR text feature extraction and image feature extraction. Histogram equalization is a commonly used image enhancement technique, which is used to improve the contrast and brightness distribution of the image. In a digital image, each pixel has a corresponding gray value. For a color image, it is converted into a gray image before processing. The principle of histogram equalization is to redistribute the gray values of the image by transforming the gray histogram of the image, so that the gray distribution of the image is more uniform, thereby enhancing the contrast of the image. The specific calculation process is as follows: first, the frequency of each gray value in the image is counted to obtain the gray histogram. Then according to a certain formula, the original gray value is mapped to a new gray value range, thereby realizing the redistribution of the gray value. After histogram equalization processing of the sampling video frame, the image quality is improved, which is more conducive to the subsequent feature extraction operation. At this time, the video frame is the target video frame.

[0034] Step S102, extracting the OCR text feature, image feature and audio feature of the target video frame.

[0035] Specifically, the OCR text feature of the target video frame is extracted, including:

[0036] Step b1, using an OCR engine to recognize characters in the target video frame to obtain text information.

[0037] Furthermore, the Optical Character Recognition engine is a software tool that can convert the text in an image into computer-editable text. Its core principle is based on pattern recognition and machine learning technologies. When processing the target video frame, first, the image area in the video frame is divided into character candidate areas one by one, and then the image features of each candidate area are extracted, such as the outline and stroke structure of the character. Next, these features are matched with the pre-trained character models in the OCR engine, and these models contain the character sample features of various fonts, sizes, and styles. By calculating the similarity between the features, the character corresponding to the character candidate area is judged, and finally, all the recognized characters are combined into text information. For example, when recognizing a video frame containing Chinese and English, the OCR engine can accurately recognize and convert the "Hello, 你好!" in the image into the corresponding text string.

[0038] Step b2, perform word segmentation, part-of-speech tagging, and keyword extraction on the text information to obtain the word frequency data and inverse document frequency data of each keyword.

[0039] Furthermore, word segmentation refers to splitting continuous text information into independent words according to certain rules. In Chinese, since there is no obvious space separation between words, word segmentation is particularly important. Common word segmentation methods include dictionary-based matching methods, statistical model-based methods, etc. For example, using the dictionary-based matching method to segment the sentence "我喜欢吃苹果", the words "我", "喜欢", "吃", and "苹果" will be obtained. Part-of-speech tagging refers to tagging the part of speech of each segmented word, such as noun, verb, adjective, etc. Part-of-speech tagging helps to understand the grammatical structure and semantic information of the text. Commonly used part-of-speech tagging tools are based on methods such as hidden Markov models and conditional random fields. For example, when performing part-of-speech tagging on the sentence "苹果是一种水果", "苹果" will be tagged as a noun, "是" as a verb, "一种" as a quantifier, and "水果" as a noun. Keyword extraction refers to extracting the words that can represent the core content of the text from the text. Commonly used methods are based on the TF-IDF (Term Frequency-Inverse Document Frequency) method, the TextRank method, etc. After keyword extraction, each keyword will be obtained. At the same time, calculate the number of times each keyword appears in the current text to obtain the word frequency data; calculate the inverse document frequency, which represents the general importance of a word in the entire document collection. The higher the inverse document frequency, the rarer the word is in the document collection, and the higher its discrimination. For example, in an article about apple cultivation, words such as "苹果" and "种植" may be keywords, and their word frequency and inverse document frequency data can be obtained through calculation.

[0040] Step b3, constructing the OCR text feature based on the word frequency data and the inverse document frequency data of each keyword.

[0041] Further, the word frequency data and the inverse document frequency data of each keyword are combined to form a feature vector. For example, assuming that there are three keywords "apple", "fruit", and "plant", their word frequencies are 5, 3, and 2 respectively, and their inverse document frequencies are 0.5, 0.3, and 0.8 respectively, then a three-dimensional feature vector [5 0.5, 3 0.3, 2*0.8] = [2.5, 0.9, 1.6] can be constructed. This feature vector can quantitatively describe the text content in the target video frame from the semantic level, and can be used for subsequent multi-modal feature fusion, video content analysis, etc., providing important text information basis for accurate extraction of video axis files.

[0042] Specifically, the image features of the target video frame are extracted, including:

[0043] Step c1, inputting the target video frame into the convolution layer of the pre-trained convolutional neural network, and generating a preliminary feature map based on the target video frame using the convolution layer, the preliminary feature map containing edge features and texture features.

[0044] Further, the convolutional neural network is a deep learning model specially designed for processing grid-structured data. The pre-trained CNN is trained on a large image dataset, and it has learned many general image feature representations. The convolution layer is the core component of CNN, which is composed of multiple convolution kernels. The convolution kernel is a small matrix, and the common sizes are 3x3, 5x5, etc. When processing the target video frame, the convolution kernel slides over the video frame with a certain step size. For each sliding position, the convolution kernel performs weighted summation on the pixel values of the corresponding region of the video frame to obtain a new pixel value. For example, for a 3x3 convolution kernel and a video frame region containing RGB three channels, the convolution kernel will multiply and sum the corresponding pixel values of R, G, and B channels respectively, and finally obtain a new pixel value. This process is performed for each position of the video frame to generate a new feature map. Since different convolution kernels can capture different image features such as horizontal edges, vertical edges, and textures, the parallel operation of multiple convolution kernels can generate a preliminary feature map containing multiple edge features and texture features. For example, a convolution layer containing 16 3x3 convolution kernels will generate 16 different preliminary feature maps, each highlighting different types of local features in the video frame.

[0045] Step c2, inputting the preliminary feature map into the pooling layer, dividing the preliminary feature map into several small regions using the pooling layer, performing maximum pooling processing on each small region, taking the maximum value in each small region as the output, and obtaining a reduced feature map.

[0046] Further, the main role of the pooling layer is to reduce the dimension of the feature map, reduce the amount of calculation, while retaining important image features, and improve the robustness of the model. The common pooling method is maximum pooling and average pooling, and the maximum pooling is adopted here. When the maximum pooling operation is performed, the preliminary feature map is divided into non-overlapping small regions, and the common region size is 2x2 or 3x3. For each small region, the maximum value of the pixel values in the region is taken as the output of the region after pooling. For example, for a 2x2 small region, the four pixel values are 10, 20, 15, and 30, respectively. After maximum pooling, the output value of the region is 30. In this way, only the most significant features are retained in each small region, and the size of the feature map is reduced. For example, a preliminary feature map with a size of 100x100 is subjected to 2x2 maximum pooling, and the size is changed to 50x50. The new feature map generated is a reduced feature map. The reduced feature map retains the key features of the image while reducing the amount of data, thereby reducing the burden of subsequent calculations and processing.

[0047] In step c3, the reduced feature map is flattened into a one-dimensional vector, and a fully connected layer is used to map the one-dimensional vector to a fixed-length vector to obtain the image features.

[0048] Further, after the processing of the convolution layer and the pooling layer, the reduced feature map still maintains a certain spatial structure (such as a two-dimensional matrix form), but the subsequent neural network layers (such as classifiers, etc.) usually require one-dimensional vectors as input. Therefore, it is necessary to flatten the reduced feature map into a one-dimensional vector. For example, a reduced feature map with a size of 50x50x16 (height 50, width 50, and channel number 16) will become a one-dimensional vector with a length of 50x50x16 = 40000 after flattening. The fully connected layer is a common layer in neural networks, and each neuron of the fully connected layer is connected to all neurons of the previous layer. The one-dimensional vector after flattening is input into the fully connected layer, and the fully connected layer performs linear transformation on the input vector through a series of weight matrices and bias vectors, and then performs nonlinear transformation through an activation function (such as the ReLU function). Finally, the input vector is mapped to a fixed-length vector. This fixed-length vector is the extracted image feature, which contains the comprehensive feature information of the image in the video frame and can be used for subsequent multi-modal feature fusion, video content analysis, etc., providing important image information basis for accurate extraction of video axis files. For example, a 40000-dimensional vector after flattening is mapped to a 256-dimensional fixed-length vector through a fully connected layer, and the 256-dimensional vector represents the image features of the target video frame.

[0049] In some optional ways of some embodiments, the pre-trained convolutional neural network comprises at least three groups of convolutional layers and pooling layers.

[0050] In particular, a single convolutional layer and pooling layer can only extract simple and low-level image features such as edges and textures. With the increase of network depth, each combination of convolutional layer and pooling layer can extract higher-level and more abstract features. The first combination of convolutional layer and pooling layer extracts basic features such as edges and corners; the second combination can recognize more complex shapes or local structures based on the first combination; the third combination can capture features with semantic information such as parts of objects or complete object class features. Through multiple combinations, different levels of features can be comprehensively extracted from video frames, providing rich information for subsequent analysis. With the stacking of convolutional layers and pooling layers, the receptive field gradually increases. The receptive field refers to the mapping area of neurons in the convolutional neural network on the original image. In a shallow network, the neuron receptive field is small and can only focus on local details of the image; while in a deep network, the neuron receptive field is large and can obtain more image information in a larger range. At least three combinations can ensure that the network is sensitive to local details and can grasp the overall image structure, which is crucial for accurately understanding the content of the video frame. The pooling layer reduces the dimension of the feature map while preserving the key information, reducing the amount of calculation. If the number of convolutional neural network layers is too small, in order to achieve sufficient feature extraction capability, larger convolutional kernels and more parameters may be needed, which not only increases the computational burden, but also easily leads to overfitting. Through multiple combinations of convolutional layers and pooling layers, the size of the feature map is gradually reduced, the number of parameters is reduced, the generalization ability of the model is improved, the risk of overfitting is reduced, and the model can have stable performance on different video frames.

[0051] In particular, extracting audio features of the target video frame includes:

[0052] Step d1, using an audio processing library to separate the audio track from the target video.

[0053] Further, audio is usually stored in continuous time series data, and its sampling rate determines the number of audio samples collected per second. Common sampling rates include 44100 Hz, 48000 Hz, etc. The separated audio data is segmented into audio segments of fixed length, and each segment can be regarded as an independent analysis unit. For example, the audio is segmented into audio segments of 1 second in length for subsequent fine processing of each segment. In this step, the audio data is also normalized to adjust the amplitude of the audio signal to a fixed range, usually [-1, 1], to eliminate the differences in amplitude of different audio signals and provide a unified input format for subsequent feature extraction.

[0054] Step d2, performing short-time Fourier transform on each audio segment to convert the time-domain audio signal to frequency-domain representation.

[0055] Further, STFT obtains the energy distribution of the audio signal at different times and frequencies by performing Fourier transform on the audio signal in a short time window, and generates a spectrogram. For example, setting a window of 2048 sample points and a step of 512 sample points for STFT, the obtained spectrogram can clearly show the changes of the audio in the time and frequency dimensions. In order to more effectively extract audio features, the spectrogram is usually subjected to Mel frequency transform, converting linear frequency to Mel frequency, because Mel frequency is more in line with the perception characteristics of the human auditory system to frequency. Under the Mel frequency scale, Mel frequency cepstral coefficients (MFCCs) are calculated, which can well reflect the timbre and formant of the audio and other features, and are commonly used feature parameters in audio analysis. In addition, other time-frequency features such as spectral centroid and spectral bandwidth can also be calculated, which describe the frequency characteristics of the audio signal from different angles and further enrich the audio feature information.

[0056] Step d3, using principal component analysis to reduce the dimension of the extracted audio features.

[0057] Further, after time-frequency feature extraction, the obtained audio features have high dimensions and may contain redundant information. In order to reduce the amount of calculation and improve the efficiency of model training, principal component analysis projects high-dimensional data to low-dimensional space through linear transformation, which retains the main features of the data while removing noise and redundant information. For example, a high-dimensional audio feature vector containing 13 MFCCs, spectral centroid, spectral bandwidth and other features is projected to a 10-dimensional low-dimensional space through PCA. Finally, the reduced audio features are fused with the features of other modalities to form a comprehensive multi-modal feature vector, providing comprehensive data support for subsequent video content analysis and video axis file extraction based on multi-modal information.

[0058] Step S103, performing scene analysis on the target video frame to obtain a scene type.

[0059] Specifically, the above step S103 includes:

[0060] Step e1: Clean the extracted OCR text by removing special characters, garbled characters, and stop words, and purify the text content. Use a pre-trained language model based on the Transformer architecture, such as GPT-Neo, to convert the cleaned text into a high-dimensional semantic vector. In this way, the rich semantic information contained in the text can be mined, such as the keywords "goal", "match", "referee" in the sports event scene, which can be accurately captured and converted into semantic vectors to provide a basis for subsequent analysis. From the extracted image features, key information such as scene layout, object contour, and color distribution is selected. Use a combination of principal component analysis (PCA) and locally linear embedding (LLE) for dimensionality reduction. PCA is used to initially reduce the feature dimension and remove redundant information; LLE further mines the local geometric structure of the data, preserving the key features of the scene in the image and enhancing the discriminability of image features in different scenes. For example, in urban street scenes and natural scenery scenes, the feature differences between buildings and natural elements can be effectively highlighted. Normalize and standardize audio features such as MFCCs and zero-crossing rate to ensure that different audio features are analyzed on the same scale. Use wavelet transform to enhance the features of key sound elements in the audio, such as car horn sounds and siren sounds in traffic scenes, and instrument playing sounds in concert scenes, making the audio features more distinguishable.

[0061] Step e2: Choose a multi-layer perceptron (MLP) as the text analysis model. Train the model on a large amount of diverse text data from various scenarios, including but not limited to traffic, meetings, and family gatherings. The model learns the co-occurrence patterns and semantic associations of text words in different scenarios, such as "agenda", "report", and "discussion" in a meeting scenario, to determine the scene type of the text. Use a convolutional neural network based on the DenseNet architecture. Input the preprocessed image features into the model, and use densely connected convolutional layers and pooling layers to fully extract the scene features from the image. During training, use a large number of image datasets containing various scenarios to let the model learn the image features of different scenarios, such as roads and vehicles in traffic scenes, and human interactions and indoor arrangements in family gatherings, to accurately predict the scene type. Use a recurrent neural network combined with an attention mechanism, such as a long short-term memory network combined with an attention mechanism, to train the model on audio samples from different scenarios. The model learns the variation patterns of audio features in time series, such as the differences between natural sound feature sequences like rain and bird songs and industrial machine roar feature sequences, to determine the scene type of the audio.

[0062] Step e3: input the pre-processed OCR text features, image features and audio features into the respective trained models to obtain the prediction probability distribution of each model for different scene types. For example, the text analysis model outputs a traffic scene probability of 0.6, a conference scene probability of 0.2, and other scene probabilities of 0.2; the image analysis model outputs a traffic scene probability of 0.5, a conference scene probability of 0.3, and other scene probabilities of 0.2; the audio analysis model outputs a traffic scene probability of 0.4, a conference scene probability of 0.3, and other scene probabilities of 0.3. The dynamic weight fusion method is used to comprehensively judge the prediction results of the three models. The weights are dynamically adjusted according to the reliability and importance of the features of different modalities in different scenes. For example, in the traffic scene, the image and audio features may be more important, and the text analysis weight is set to 0.3, the image analysis weight is set to 0.4, and the audio analysis weight is set to 0.3; while in the conference scene, the text features may be more critical, and the corresponding weights are adjusted to text analysis weight 0.4, image analysis weight 0.3, and audio analysis weight 0.3. The weighted average probability is calculated, and the scene type with the highest probability is taken as the final result. Finally, the scene type result is recorded to provide a basis for subsequent video axis file extraction strategy formulation, improving the accuracy and pertinence of extraction.

[0063] Step S4, obtaining a fusion feature vector based on the scene type, OCR text features, image features and audio features.

[0064] Specifically, the above step S4 includes:

[0065] Step f1, determining a default weight based on the correspondence between the default weight and the scene type, and the scene type.

[0066] Further, in different scene types, the importance of OCR text features, image features, and audio features in describing the scene and extracting key information varies. For example, in a speech scene, OCR text features (such as the content of the speaker's speech) may be the most critical to understanding the video content; in a natural scenery video, image features (such as landscape, plant and animal morphology) may dominate; in a music performance scene, audio features (such as melody, rhythm) are more important. By establishing a correspondence between default weights and scene types, appropriate weights can be assigned to each modality feature according to the scene type of the video, highlighting key information and improving the accuracy of subsequent analysis and processing. A weight mapping table is pre-established, which records the correspondence between various common scene types and the default weights of each modality. For example, for "speech scene", set the OCR text feature weight to 0.5, the image feature weight to 0.3, and the audio feature weight to 0.2; for "sports event scene", set the OCR text feature weight to 0.3, the image feature weight to 0.5, and the audio feature weight to 0.2. When the scene type of the current target video frame is determined through step S3, the corresponding default weights can be quickly found in this weight mapping table. For example, if the current scene type is determined to be "speech scene", the OCR text feature weight 0.5, image feature weight 0.3, and audio feature weight 0.2 are obtained from the table. The default weight determination method based on scene type allows the system to perform differential processing for different scenes, effectively utilizing the advantages of each modality feature. Avoiding the use of fixed weights in all scenes, which may lead to some key information being ignored or some redundant information being overemphasized, thus improving the adaptability and accuracy of the entire video axis file extraction process for different scenes.

[0067] Step f2, based on the default weights, the OCR text features, image features, and audio features are fused to obtain a fused feature vector.

[0068] Further, feature fusion is the combination of features from different modalities according to certain rules to form a comprehensive feature vector. Here, a weighted sum method is used for fusion, which is based on the default weights of each modality feature, and each modality feature vector is weighted and then added to obtain a fused feature vector. This method can comprehensively consider the information of different modalities, fully utilize the complementarity of each modality, and make the fused feature vector more comprehensive and accurate in describing the content of the video frame. The fused feature vector integrates the information of multiple modalities, providing a more comprehensive and accurate description of the content of the video frame compared to a single modality feature vector. This is of great significance for subsequent video content analysis, target extraction, and other tasks based on the feature vector, and can significantly improve the accuracy and reliability of the tasks, laying a solid foundation for generating high-quality video axis files.

[0069] Step S5, based on the pre-trained adaptive attention model and the fusion feature vector, a weighted feature vector is obtained.

[0070] Specifically, the pre-trained adaptive attention model is trained using a backpropagation method, including but not limited to news reports, video programs, teaching videos, sports event videos, etc. The fusion feature vector is extracted from these videos as training data, and each fusion feature vector corresponds to a corresponding video frame, and the scene type, key information position, etc. Labels of the video frame are labeled. For example, in a news report video, the timestamp and text content corresponding to the host's speech are labeled; in a sports event video, the time and related description of key events such as goals and fouls are labeled. These labels will be used as supervision information for model training to help the model learn how to accurately assign attention weights based on the fusion feature vector.

[0071] The above step S5 includes:

[0072] Step g1, based on the pre-trained adaptive attention model and the fusion feature vector, the attention weight vector is calculated.

[0073] Further, the pre-trained adaptive attention model, which is based on the attention mechanism, draws on the human characteristic of automatically focusing on key content when processing information. In multi-modal information processing, such as video axis file extraction tasks, the fusion feature vector contains features from multiple aspects such as OCR text, images, and audio. These feature dimensions are diverse and vary in importance. The adaptive attention model can accurately calculate the importance of each feature dimension through a series of complex operations, and then generate an attention weight vector. Its core implementation is to use multiple learnable weight matrices to perform linear transformation on the fusion feature vector. First, the fusion feature vector is multiplied by the first weight matrix W1 to map the fusion feature vector from the original space to a new intermediate feature space. This process can be understood as a preliminary screening and reorganization of features. Then, the ReLU activation function is applied to filter out negative feature values and only retain and enhance valuable feature signals, resulting in a more discriminative intermediate feature representation M2. Then, M2 is multiplied by the second weight matrix W2 again to further explore the potential relationships between features and map the intermediate features to the attention weight space. Finally, the result is normalized to the 0-1 interval by the Softmax activation function, so that the sum of the weights of all dimensions is 1. This way, the attention weight corresponding to each feature dimension is obtained, and these weights together form the attention weight vector. The Softmax function acts like a "referee" to present the importance of different feature dimensions in the form of a probability distribution, allowing the model to clearly distinguish which feature dimensions are more critical to the analysis of the current video frame. Assuming that the fusion feature vector F is a d-dimensional vector, the weight matrix W1 in the adaptive attention model has a dimension of d x k, and the weight matrix W2 has a dimension of k x 1, where k is the intermediate dimension determined according to the model design, which serves as a bridge connecting the original feature space and the attention weight space. In specific operations, first, multiply the fusion feature vector F by W1, i.e., M1 = F x W1, to obtain a k-dimensional intermediate vector M1. Apply the ReLU function to M1, i.e., M2 = ReLU(M1), which will set negative values in M1 to 0 and keep positive values unchanged to enhance effective features. Then, multiply M2 by W2, i.e., M3 = M2 x W2, to obtain a one-dimensional result M3. Finally, apply the Softmax function to M3, i.e., α = Softmax(M3), to obtain an attention weight vector α with a dimension of d, where α iThe attention weight of the i-th dimension in the fusion feature vector is represented. For example, when d = 5, k = 3, F = [1, 2, 3, 4, 5], W1 is a 5x3 matrix, M1 is obtained through matrix multiplication, and M2 is obtained through the ReLU function. After multiplying W2, the attention weight vector a = [0.1, 0.2, 0.3, 0.2, 0.2] is obtained through the Softmax function. In the process of complex multi-modal information processing, not all feature dimensions are equally important for video axis file extraction. By calculating the attention weight vector, the adaptive attention model acts like a smart "filter" that can automatically identify the more critical feature dimensions in the fusion feature vector for the current task, highlighting key information, such as accurately identifying text feature dimensions closely related to the content of a speech video, image feature dimensions of facial expressions of the speaker, and audio feature dimensions of the speaker's voice tone, while suppressing redundant or secondary information such as irrelevant decorative elements in the video background. This not only helps improve the accuracy of subsequent models for video content analysis, but also greatly reduces the waste of computing resources, enabling the entire video axis file extraction system to efficiently focus on the most valuable parts when processing complex multi-modal information, improving system performance.

[0074] Step g2, weighting the fusion feature vector using the attention weight vector to obtain a weighted feature vector.

[0075] Further, the principle of weighting is based on the importance of each feature dimension reflected by the attention weight vector. Multiplying the attention weight vector and the fusion feature vector by corresponding elements essentially adjusts the dimensions of the fusion feature vector according to the size of the attention weight. The greater the attention weight, the more important the feature represented by that dimension is in describing the key information of the video frame, and the greater the proportion it occupies in the weighted feature vector; conversely, the smaller the weight, the less influence it has in the weighted feature vector. This way can re-encode the fusion feature vector according to the importance calculated by the adaptive attention model, so that the weighted feature vector can more accurately reflect the feature representation of the key information in the video frame. For example, in a sports event video, if the attention weight vector indicates that the image feature dimension of the athlete's action has a higher weight, then in the weighting process, this part of the feature dimension will be enhanced in the weighted feature vector, more prominently reflecting the key action information of the event.

[0076] Given the attention weight vector a = [a1, a2,..., a d ];

[0077] The fusion feature vector F = [F1, F2,..., F d ],

[0078] The calculation method of the weighted feature vector G is as follows: G i = a i × F i , where i = 1, 2,..., d.

[0079] For example, when the attention weight vector a = [0.2, 0.3, 0.5] and the fusion feature vector F = [10, 20, 30], the weighted feature vector G = [0.2x10, 0.3x20, 0.5x30] = [2, 6, 15]. By multiplying the corresponding elements, the attention weight is integrated into the fusion feature vector, achieving the weighted adjustment of the features. The weighted feature vector obtained after weighting is an optimization and refinement of the fusion feature vector. It highlights the feature dimensions that are important according to the adaptive attention model, so that the subsequent long short-term memory network can more effectively capture the key information in the video content when processing this feature vector. The long short-term memory network can use these weighted features to better understand the changes in the video over time, providing more representative feature representations for generating accurate video axis files. In practical applications, the weighted feature vector can help the system more accurately locate key time points, key events, etc. in the video, further improving the accuracy and reliability of video axis file extraction, and providing high-quality data support for downstream tasks such as video editing and content retrieval.

[0080] Step S6, modeling the weighted feature vector using a long short-term memory network model to obtain hidden state sequence information.

[0081] Specifically, the above step S6 includes:

[0082] Step h1, initializing the long short-term memory network.

[0083] Further, the long short-term memory network is a special recurrent neural network that is good at processing time series data and can effectively solve the gradient vanishing and gradient explosion problems in traditional RNN. Before modeling the weighted feature vector using LSTM, it needs to be initialized. The LSTM unit is composed of an input gate, a forget gate, an output gate, and a memory cell. The input gate controls the input of new information, the forget gate decides to retain or discard old information in the memory cell, and the output gate determines the output information. Multiple LSTM units are connected in sequence to form an LSTM network. For example, a network containing 3 layers of LSTM is constructed, and the number of LSTM units in each layer can be set according to task requirements and data characteristics, such as 64 units in the first layer, 32 units in the second layer, and 16 units in the third layer. The weight matrix in the LSTM network includes input weight, forget weight, output weight, and memory cell weight, which needs to be initialized. Common initialization methods include random initialization, Xavier initialization, etc. Taking Xavier initialization as an example, the initial value of the weight is determined according to the dimensions of the input and output, which can make the network converge better in the early stage of training. For example, for the input weight matrix, according to the dimensions of the weighted feature vector and the number of LSTM units, the initial weight value is calculated and set according to the Xavier initialization formula.

[0084] Step h2: input the weighted feature vector.

[0085] Further, the weighted feature vector processed by the adaptive attention model is sequentially input into the initialized LSTM network in time sequence. Assuming that the video frame sequence has T time steps, each time step corresponds to a weighted feature vector. For example, in a video with a duration of 10 seconds and a frame rate of 30 fps, there are 300 video frames, and after feature extraction and fusion processing, 300 weighted feature vectors are obtained, each time step is 1 / 30 second. At the first time step t = 1, the first weighted feature vector G1 is input into the first layer of the LSTM network. The LSTM network will update the memory cell and the hidden state through the collaborative operation of the input gate, the forget gate, the output gate, and the memory cell according to the input feature vector and the hidden state of the previous time step. For example, the input gate calculates the weight of the input signal according to G1 and the hidden state of the previous time step to decide which new information can enter the memory cell; the forget gate calculates the retention weight of the old information to decide which old information in the memory cell needs to be retained; the output gate calculates the output hidden state according to the updated memory cell and the current input.

[0086] Step h3: LSTM network operation.

[0087] Further, at each time step, the LSTM network performs complex operations on the input weighted feature vector to update the memory cell and hidden state. Input gate operation: the input gate calculates the weight of the input signal through a Sigmoid activation function. The Sigmoid function maps the output value to between 0 and 1, and the closer the value is to 1, the more important the new information of the input is, and the easier it is to enter the memory cell. Forget gate operation: the forget gate also calculates the retention weight of old information through a Sigmoid function. The closer the forget gate value is to 1, the more old information is retained in the memory cell. Memory cell update: based on the calculation results of the input gate and the forget gate, the memory cell is updated. In this way, the memory cell can both retain important old information and receive new information. Output gate operation: the output gate calculates the output hidden state according to the updated memory cell and the current input. The hidden state contains information from the current time step and previous time steps, and is the output result of the LSTM network.

[0088] Step h4: Obtain hidden state sequence information.

[0089] Further, as the time steps advance, the LSTM network sequentially processes the weighted feature vectors of each time step, and finally obtains the hidden state sequence information of the entire time sequence. After processing all T weighted feature vectors of the time steps, a hidden state sequence with a length of T is obtained, and each hidden state in the sequence contains video frame information from the first time step to the current time step, and effectively captures the long-term dependency relationship in the time sequence through the operation of the LSTM network. For example, when analyzing a continuous sports event video, the hidden state sequence can record the changes in the movements of athletes at different time points, the development of the game situation, etc. The hidden state sequence information is crucial for subsequent generation of video axis files. It can be used as input for a reinforcement learning-based policy generation model to help the model understand the changes in video content in the time dimension, thereby accurately determining the time points of key events in the video, the positions of key information, etc., and providing key basis for generating accurate and complete video axis files.

[0090] Step S7, generating a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extracted task information, the scene type, the pre-trained deep network method, and the target video frame.

[0091] Specifically, the above step S7 includes:

[0092] Step i1, defining environment state information based on the hidden state sequence information.

[0093] Further, the hidden state sequence information is modeled by a long short-term memory network from the weighted feature vector, which contains the key information of the video frame in the time sequence. The environment state information is a concept in reinforcement learning, which describes the environment state of the method model generating the video axis file. Defining the environment state information based on the hidden state sequence information is to convert the hidden state output by the LSTM into an environment description meaningful to the current video analysis task. For example, the hidden state may contain information such as the actions of the characters in the video, the language content, and the scene changes. These information is integrated into the environment state, such as the key scene of the current video and the activities of the main characters, so that the subsequent method can make decisions based on the environment state. First, feature extraction and screening are performed on the hidden state sequence. Dimensionality reduction techniques such as principal component analysis can be used to extract the main components from the high-dimensional hidden state sequence and remove redundant information. Then, according to the characteristics of the video analysis task, the feature vector is semantically mapped. For example, if a specific action or scene keyword appears in the video, it is mapped to the corresponding environment state label, such as "intense competition scene" and "speech scene". Finally, these processed information is combined into the environment state information, which is used for subsequent policy making. Accurate environment state information can help the method understand the key features and state of the current video, providing a basis for defining the extraction strategy and reward strategy. It is like providing a "map" for the method, so that the method can clearly understand its position and the situation it faces, and then formulate a targeted strategy for generating the video axis file.

[0094] Step i2, defining the extraction strategy set based on the scene type.

[0095] Further, different scene types have different characteristics and rules. Based on the scene type definition extraction strategy set, a series of possible extraction strategies are formulated for each scene type by associating scene types with extraction strategies. For example, in a meeting scenario, the extraction strategy may focus on extracting the speech content of the speaker and the text information on the PPT; in a sports event scenario, the extraction strategy may focus on the key actions of the athletes and the scoring situation of the game. By pre-analyzing the characteristics of different scene types, a mapping relationship between scene types and extraction strategies is established. First, a scene type library is established, including various common scene types such as meetings, sports events, movies, and teaching. Then, for each scene type, the characteristics and key information of the video content are analyzed. For example, for a teaching scenario, the key information may include the knowledge points explained by the teacher, the content of the blackboard writing, etc. According to these characteristics, the corresponding extraction strategy is formulated. The extraction strategy includes the selection of feature extraction methods, such as selecting specific OCR recognition parameters for blackboard content, determining the time stamp, such as dividing the time segment according to the pause of the teacher's explanation, etc. Finally, the extraction strategies formulated for each scene type are summarized into an extraction strategy set. The extraction strategy set provides multiple feasible operation schemes for the generation of video axis files. According to different scene types, appropriate extraction strategies are selected, which can improve the accuracy and efficiency of extraction. It is like a "strategy toolbox", and the method can select the most suitable tool from the toolbox to complete the generation task of the video axis file according to the current scene type of the video.

[0096] Step i3, defining a reward strategy set based on the extraction task information.

[0097] Further, the reward policy set is a mechanism used in reinforcement learning to motivate the agent to take optimal actions. According to the goals and requirements of the extraction task, a series of reward rules are formulated. For example, if the extraction task is to accurately extract the dialogue of key characters in the video, the reward policy can be to give a positive reward when the method accurately extracts the dialogue of key characters, and a negative reward when the extraction is wrong or missed. Through the reward policy, the method is guided towards the direction of meeting the requirements of the extraction task. First, the specific goals and requirements of the extraction task are clearly defined, such as the accuracy, completeness, and time precision of the extraction. Then, according to these goals and requirements, the corresponding reward rules are formulated. For example, for extraction accuracy, a positive reward value of 10 can be given when the similarity of the extracted text content to the actual video content reaches 90%; a negative reward value of -5 can be given when the similarity is below a certain threshold of 70%. For time precision, if the error between the extracted key event timestamp and the actual timestamp is within ±1 second, a reward is given; if it exceeds the range, a penalty is given. Finally, these reward rules are summarized into a reward policy set. The reward policy set provides a clear goal-oriented for the method, enabling the method to continuously adjust its behavior according to the reward feedback during the process of generating the video axis file, improving the quality of the extraction, and ultimately generating a video axis file that meets the requirements of the extraction task.

[0098] Step i4, obtaining the target extraction policy set based on the environment state information, the extraction policy set, and the reward policy set using the pre-trained deep network method.

[0099] Further, the pre-trained deep network method is used in reinforcement learning to learn the optimal policy. Through continuous interaction with the environment, in this task, the environment state information is equivalent to the environment, the extraction policy set is equivalent to the action space of the agent, the reward policy set is equivalent to the reward feedback, and learning what extraction policy to take under different environment states can obtain the maximum reward. The deep network method uses the powerful fitting ability of neural networks to learn and analyze the environment state information, the extraction policy set and the reward policy set, and find the optimal extraction policy combination. First, the environment state information, the extraction policy set and the reward policy set are input into the pre-trained deep network method. The neural network part in the deep network method extracts and processes the input information, extracts the environment state information through multiple convolutional layers and fully connected layers, and encodes the extraction policy set into a processable vector form. Then, according to the output of the neural network, the value of each extraction policy under the current environment state is calculated, that is, the reward that can be obtained by taking the policy. Through continuous iterative training, the deep network method gradually learns the optimal extraction policy under different environment states, forming the target extraction policy set. The target extraction policy set is the optimal policy combination learned by the deep network method according to the environment state, the extraction policy and the reward policy, which comprehensively considers various information of the video and requirements of the extraction task, and can guide the method to efficiently and accurately generate the video axis file.

[0100] Step i5, processing the target video frame based on the target extraction policy set to generate a target video frame axis file.

[0101] Further, the target extraction strategy set contains a series of optimal extraction strategies for the target video frames. Processing the target video frames based on these strategies means extracting key information from the target video frames according to the methods and steps specified in the strategies, and organizing them into the format of the video axis file. For example, according to the timestamp determination method specified in the strategy, the time point of the key event is extracted from the video frame sequence; according to the feature extraction method specified in the strategy, the text, image, audio and other key features in the video are extracted, and these information is organized and stored according to the format requirements of the video axis file. First, according to the timestamp determination strategy in the target extraction strategy set, the target video frame sequence is divided by time point, and the start and end time of each key event is determined. For example, in sports event video, the timestamps of key events such as goals and fouls are determined according to the strategy. Then, according to the feature extraction method in the strategy, the corresponding text, image and audio features are extracted. For example, the text information in the video is extracted using a specific OCR method, and the image features of the key scene are extracted according to the image feature extraction strategy. Finally, the extracted timestamps, text, image and audio information are organized and stored according to the format of the video axis file to generate the target video frame axis file. This step is to convert the analysis, strategy formulation and learning results into actual video axis files, and is the final goal and key link of the whole video axis file generation process. By processing the target video frames according to the target extraction strategy set, high-quality video axis files that meet the extraction task requirements can be generated, providing strong support for subsequent video editing, content retrieval and other applications.

[0102] In some optional ways of the embodiments, the above method further includes:

[0103] Step j1, extracting target OCR text features, target image features and target audio features of the target video frame axis file.

[0104] Further, the target video frame axis file has integrated key information in the video, and again extracts its OCR text features, image features and audio features, in order to conduct deep analysis on the file content from a multi-modal perspective. For the OCR text features, by re-identifying the text content in the axis file, the semantic information and potential logical relationships in the text are mined using techniques such as morphological and syntactic analysis. Image feature extraction is aimed at key images associated in the axis file, based on image color, texture, shape and other visual elements, computer vision methods are used to obtain feature vectors that can represent the content of the image. Audio feature extraction is from the audio part corresponding to the axis file, through time and frequency domain analysis of the audio signal, such as calculating the Mel frequency cepstral coefficient MFCCs, to obtain the key features of the audio. For the OCR text features, a professional OCR recognition tool is used to scan and recognize the text area in the axis file, the recognized text is cleaned to remove noise and invalid characters, and then natural language processing tools are used for word segmentation, part-of-speech tagging and keyword extraction to form the OCR text features. For image features, the image data in the axis file is read out and input into a pre-trained convolutional neural network model, after multiple convolution and pooling operations, the high-level features of the image are extracted, and finally the features are mapped to a fixed length vector through a fully connected layer. In terms of audio feature extraction, with the help of an audio processing library, the audio signal is separated from the axis file, the audio signal is framed and windowed, and then the audio feature parameters are calculated to form the audio feature vector. Extracting these multi-modal features is the basis for subsequent fusion analysis and optimization processing, which provides comprehensive data support for in-depth understanding of the content of the video frame axis file, and helps to find possible inconsistencies or incomplete information in the file.

[0105] Step j2, using a fusion analysis technology based on a Transformer architecture to analyze the target OCR text features, target image features and target audio features, to obtain consistency data of the target OCR text features, target image features and target audio features on the time axis, and complementary data of the target OCR text features, target image features and target audio features on the time axis.

[0106] Further, the Transformer architecture is based on the self-attention mechanism, which can effectively capture the long-distance dependency relationship between different modal features and realize deep fusion analysis of multi-modal information. In processing the target OCR text features, target image features, and target audio features, the Transformer takes the feature vectors of different modalities as input and calculates the correlation degree of different modal features on the time axis through the self-attention mechanism. For consistency analysis, it is determined whether the information expressed by different modalities is consistent at the same time point or time period; for complementarity analysis, it is determined whether different modalities can complement each other on the time axis to provide a more comprehensive video content description. The extracted target OCR text features, target image features, and target audio features are encoded and converted into a format suitable for input into the Transformer. For example, the text features are converted into vector representations through a word embedding layer, and the image features and audio features are normalized and then concatenated with the text feature vectors. The concatenated feature vectors are input into the Transformer model, and the multi-head attention mechanism in the model calculates the relationship between different modal features on the time dimension. Through multiple iterations of calculation, the correlation matrix of different modal features on the time axis is obtained. Based on the correlation matrix, consistency data is calculated, such as the similarity of different modal feature vectors at the same time point; by analyzing the distribution differences of different modal features on the time axis, complementary data is obtained to determine which time periods different modalities can provide unique information. Through this fusion analysis technique, consistency data and complementary data can be obtained, which can evaluate the fusion quality of multi-modal information in the video frame axis file and find problems of inconsistent or insufficient complementarity, providing direction for subsequent optimization.

[0107] Step j3, obtaining collaborative score data based on the consistency data and the complementary data.

[0108] Further, the synergy score data is a quantitative evaluation of the synergistic expression of the target OCR text features, target image features and target audio features in the time axis of the video content effect. Based on the consistency data and the complementarity data, a comprehensive score is calculated by a certain method to reflect the synergy degree of multi-modal information in the video frame axis file. For example, the weighted sum method can be used to combine the consistency score and the complementarity score according to a certain weight to obtain the final synergy score. The operation process is as follows: first, according to the actual demand and experience, the weights of the consistency data and the complementarity data are allocated, such as setting the consistency weight to 0.6 and the complementarity weight to 0.4. Then, the consistency data and the complementarity data are standardized to be in the same numerical range, which is convenient for calculation. For example, the consistency score and the complementarity score are both normalized to the interval [0, 1]. Finally, the weighted sum is performed according to the set weight to calculate the synergy score data, such as: synergy score = consistency score x 0.6 + complementarity score x 0.4. The synergy score data provides an intuitive quantitative index for evaluating the quality of the video frame axis file, which can quickly judge the fusion effect of multi-modal information in the file and provide a quantitative basis for subsequent optimization processing.

[0109] Step j4, based on the reinforcement learning model, the meta-learning model and the synergy score data, the target video frame axis file is optimized to obtain an optimized video frame axis file.

[0110] Further, the reinforcement learning model learns the optimal optimization strategy according to the synergy score data by interacting with the target video frame axis file and related data. The meta-learning model is "learning how to learn", which quickly adjusts the parameters and learning method of the reinforcement learning model using past learning experience, so that the reinforcement learning model can more efficiently optimize the current video frame axis file. Based on the synergy score data, the reinforcement learning model constantly tries different optimization operations, such as adjusting the keywords extracted by text extraction and correcting the parameters of image feature extraction, and judges the effect of the optimization operation according to the change of the synergy score, gradually finds the optimal optimization scheme, and thus obtains the optimized video frame axis file. Initialize the reinforcement learning model and the meta-learning model, set the action space and state space of the reinforcement learning model, the action space represents various possible optimization operations, and the state space represents the features of the target video frame axis file and the synergy score data. Input the related information of the target video frame axis file and the synergy score data into the reinforcement learning model, and the reinforcement learning model selects an optimization action according to the current state, such as adjusting the threshold of OCR text feature extraction. After executing the optimization action, the synergy score data of the target video frame axis file is recalculated, and the new state and reward, i.e. the change value of the synergy score, are fed back to the reinforcement learning model, and the reinforcement learning model updates its own strategy according to these information. The meta-learning model adjusts the learning rate, exploration rate and other parameters of the reinforcement learning model according to the learning situation of the reinforcement learning model in multiple tasks, and speeds up the convergence speed of the reinforcement learning model. After multiple iterations of optimization, when the reinforcement learning model converges to a certain extent, the optimized target video frame axis file obtained is the optimized video frame axis file. Through the synergy of the reinforcement learning model and the meta-learning model, the target video frame axis file can be optimized according to the synergy score data of the multi-modal information, the quality and accuracy of the file can be improved, and it can meet the needs of practical applications better.

[0111] The method for intelligently and accurately extracting an OCR-based video axis file provided in this embodiment first reduces the amount of calculation, removes noise, and enhances the edges of text through preprocessing operations such as decoding, sampling, and image enhancement on the video, lays a foundation for subsequent accurate extraction of various features, and improves the overall processing efficiency. Second, by comprehensively extracting the multi-modal features of the video frames, the traditional method of relying on manual annotation of text information is changed, avoiding the low efficiency and error-prone problems of manual annotation, and obtaining video content information from multiple dimensions. Then, by analyzing the scene type, a basis is provided for subsequent feature fusion and extraction strategy making, enabling the system to perform differential processing for different scenes and improving the comprehensiveness of video content understanding. Further, by fusing multi-modal features, the existing automatic extraction method is compensated for the lack of comprehensive understanding of video content, and various modal information is comprehensively considered to improve the expression ability of video content, which helps to improve the accuracy and integrity of extraction. Then, by using a pre-trained adaptive attention model, attention weights are dynamically allocated according to the multi-modal fusion features to highlight key features and further improve the ability to capture key video content, thereby improving the accuracy of extraction. Next, by modeling the weighted feature vector sequence using a long short-term memory network, the time-dependent relationship of the video content is effectively captured, enabling the system to better process information in the time dimension of the video and improving the accuracy and integrity of video content understanding. Finally, by comprehensively utilizing various information and pre-trained deep network methods, the system can automatically adjust the extraction strategy according to different video types and content to generate accurate and complete video frame axis files, solving the problem of poor accuracy and integrity of the video axis files extracted by traditional methods. In summary, the method for intelligently and accurately extracting an OCR-based video axis file provided in this embodiment effectively solves the problem of how to efficiently, accurately, and completely extract a video axis file.

[0112] The above is an embodiment of the OCR-based video axis file intelligent accurate extraction system provided in this application. The following describes other embodiments of the OCR-based video axis file intelligent accurate extraction provided in this application. See the following for details.

[0113] In this embodiment, an OCR-based video axis file intelligent accurate extraction system is also provided, which is used to implement the above embodiments and preferred embodiments, and has been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated.

[0114] This embodiment provides an OCR-based video axis file intelligent accurate extraction system, as shown in Figure 2 , comprising:

[0115] The preprocessing module 201 is configured to acquire a video to be extracted and extraction task information, and pre-process the video to be extracted to obtain a target video frame.

[0116] The extraction module 202 is configured to extract an OCR text feature, an image feature and an audio feature of the target video frame.

[0117] The analysis module 203 is configured to perform scene analysis on the target video frame to obtain a scene type.

[0118] The fusion module 204 is configured to obtain a fusion feature vector based on the scene type, the OCR text feature, the image feature and the audio feature.

[0119] The calculation module 205 is configured to obtain a weighted feature vector based on a pre-trained adaptive attention model and the fusion feature vector.

[0120] The modeling module 206 is configured to model the weighted feature vector by using a long short-term memory network model to obtain hidden state sequence information.

[0121] The generation module 207 is configured to generate a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, a pre-trained deep network method and the target video frame.

[0122] The OCR-based video axis file intelligent accurate extraction system provided in the embodiment first reduces the amount of calculation, removes noise and enhances the edges of the text through decoding, sampling and image enhancement and other preprocessing operations on the video, lays a foundation for subsequent accurate extraction of various features, and improves the overall processing efficiency. Secondly, by comprehensively extracting the multi-modal features of the video frames, the traditional method of relying on manual annotation of text information is changed, avoiding the low efficiency and error-prone problems of manual annotation, and obtaining video content information from multiple dimensions. Then, by analyzing the scene type, the system can provide a basis for subsequent feature fusion and extraction strategy making, so that the system can be differentially processed according to different scenes, and the comprehensiveness of video content understanding is improved. Furthermore, by fusing multi-modal features, the defects of the existing automatic extraction method that lack comprehensive understanding of video content are made up, various modal information is comprehensively considered, the expression ability of video content is improved, which helps to improve the accuracy and integrity of extraction. Then, by using the pre-trained adaptive attention model, the attention weight is dynamically allocated according to the multi-modal fusion features, the key features are highlighted, and the capture ability of the key content of the video is further improved, so as to improve the accuracy of extraction. Next, by modeling the weighted feature vector sequence through the long short-term memory network, the time dependence of the video content is effectively captured, so that the system can better process the information of the video in the time dimension, and the accuracy and integrity of the video content understanding are improved. Finally, by comprehensively utilizing various information and pre-trained deep network methods, the system can automatically adjust the extraction strategy according to different video types and contents to generate accurate and complete video frame axis files, solving the problem of poor accuracy and integrity of the video axis files extracted by the traditional method. In summary, the OCR-based video axis file intelligent accurate extraction system provided in the embodiment effectively solves the problem of how to efficiently, accurately and completely extract the video axis file.

[0123] Further function description of each module and unit is the same as the corresponding embodiment described above, which will not be repeated here.

Claims

1. An intelligent and accurate extraction method of OCR-based video axis files, characterized in that, The method comprises: obtaining a video to be extracted and extraction task information, preprocessing the video to be extracted to obtain a target video frame; extracting OCR text features, image features and audio features of the target video frame; performing scene analysis on the target video frame to obtain a scene type; the scene analysis comprises: based on the image features of the target video frame, outputting the scene type through a pre-trained scene classification model, the scene type at least including one of a speech scene, a sports event scene and a film and television scene; based on the scene type, the OCR text features, the image features and the audio features, obtaining a fusion feature vector; based on the pre-trained adaptive attention model and the fusion feature vector, obtaining a weighted feature vector; modeling the weighted feature vector using a long short-term memory network model to obtain hidden state sequence information; based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, the pre-trained deep network method and the target video frame, generating a target video frame axis file; the based on the scene type, the OCR text features, the image features and the audio features to obtain a fusion feature vector, comprising: determining a default weight based on the correspondence between the default weight and the scene type, the scene type; based on the default weight, performing feature fusion on the OCR text features, the image features and the audio features to obtain a fusion feature vector; the based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, the pre-trained deep network method and the target video frame to generate a target video frame axis file, comprising: defining environment state information based on the hidden state sequence information; defining an extraction strategy set based on the scene type; defining a reward strategy set based on the extraction task information; using a pre-trained deep network method to obtain a target extraction strategy set based on the environment state information, the extraction strategy set and the reward strategy set, the process being as follows: input the environment state information, the extraction strategy set and the reward strategy set into the pre-trained deep network method, the deep network method extracts features of the environment state information through multiple convolutional layers and fully connected layers to obtain the extraction strategy set, encodes the extraction strategy set into a processable vector form; calculate the strategy reward of each extraction strategy under the current environment state, and iteratively train to output the target extraction strategy set by the deep network method; based on the target extraction strategy set, processing the target video frame to generate a target video frame axis file.

2. The method of claim 1, wherein, the obtaining a video to be extracted and extraction task information, preprocessing the video to be extracted to obtain a target video frame, comprising: using a video decoding library to convert the video to be extracted into a video frame sequence; sampling the video frame sequence based on a preset sampling interval to obtain a sampled video frame; performing histogram equalization processing on the sampled video frame to obtain a target video frame.

3. The method of claim 1, wherein, the extracting OCR text features of the target video frame, comprising: using an OCR engine to recognize characters in the target video frame to obtain text information; The text information is subjected to word segmentation processing, part-of-speech tagging processing and keyword extraction processing to obtain word frequency data and inverse document frequency data of each keyword; Based on the word frequency data and inverse document frequency data of each keyword, an OCR text feature is constructed.

4. The method of claim 3, wherein, The image feature of the target video frame is extracted, including: The target video frame is input into the convolution layer of the pre-trained convolutional neural network, and the convolution layer is used to generate a preliminary feature map based on the target video frame, the preliminary feature map containing edge features and texture features; The preliminary feature map is input into the pooling layer, and the pooling layer is used to divide the preliminary feature map into a plurality of small regions, and the maximum pooling processing is performed on each small region, and the maximum value in each small region is taken as the output to obtain a reduced feature map; The reduced feature map is flattened into a one-dimensional vector, and the one-dimensional vector is mapped into a fixed-length vector by using the fully connected layer to obtain the image feature.

5. The method of claim 4, wherein, The pre-trained convolutional neural network includes at least three groups of convolutional layer and pooling layer combinations.

6. The method of claim 5, wherein, The pre-trained adaptive attention model and the fusion feature vector are used to obtain a weighted feature vector, including: Based on the pre-trained adaptive attention model and the fusion feature vector, an attention weight vector is calculated; The fusion feature vector is weighted using the attention weight vector to obtain a weighted feature vector.

7. The method of claim 6, wherein, The method further includes: Extracting target OCR text features, target image features and target audio features of the target video frame axis file; Using a fusion analysis technology based on the Transformer architecture to analyze the target OCR text features, target image features and target audio features to obtain consistency data of the target OCR text features, target image features and target audio features on the time axis, and complementary data of the target OCR text features, target image features and target audio features on the time axis; Based on the consistency data and complementary data, collaborative score data is obtained; Based on the reinforcement learning model, the meta-learning model and the collaborative score data, the target video frame axis file is optimized to obtain an optimized video frame axis file.

8. An OCR-based intelligent and accurate extraction system for video axis files, characterized in that, The system includes: A preprocessing module for obtaining a video to be extracted and extraction task information, preprocessing the video to be extracted to obtain a target video frame; An extraction module for extracting OCR text features, image features and audio features of the target video frame; An analysis module for performing scene analysis on the target video frame to obtain a scene type; The scene analysis includes: based on the image features of the target video frame, outputting the scene type through a pre-trained scene classification model, the scene type including at least one of a speech scene, a sports event scene and a video scene; A fusion module for obtaining a fusion feature vector based on the scene type, the OCR text features, image features and audio features; A calculation module for obtaining a weighted feature vector based on a pre-trained adaptive attention model and a fusion feature vector; A modeling module for modeling the weighted feature vector using a long short-term memory network model to obtain hidden state sequence information; The generating module is configured to generate a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, the pre-trained deep network method, and the target video frame. The fusion feature vector is obtained based on the scene type, the OCR text feature, the image feature, and the audio feature, and includes: The default weight is determined based on a corresponding relationship between a default weight and the scene type and the scene type; The default weight is determined based on a corresponding relationship between a default weight and the scene type and the scene type; The default weight is determined based on a corresponding relationship between a default weight and the scene type and the scene type; The fusion feature vector is obtained based on the default weight and the OCR text feature, the image feature, and the audio feature; The generating module is configured to generate a target video frame axis file based on the hidden state sequence information, the long short-term memory network model, the extraction task information, the scene type, the pre-trained deep network method, and the target video frame. The environment state information is defined based on the hidden state sequence information; The extraction strategy set is defined based on the scene type; The reward strategy set is defined based on the extraction task information; The target extraction strategy set is obtained based on the environment state information, the extraction strategy set, and the reward strategy set by using the pre-trained deep network method, and the process is as follows: The environment state information, the extraction strategy set, and the reward strategy set are input into the pre-trained deep network method, the deep network method extracts features of the environment state information through multiple convolutional layers and fully connected layers to obtain the extraction strategy set, encodes the extraction strategy set into a processable vector form, calculates the strategy reward of each extraction strategy under the current environment state, iteratively trains, and the deep network method outputs the target extraction strategy set; The target video frame is processed based on the target extraction strategy set to generate a target video frame axis file.

Images