Video text location positioning method, system, medium and device

Abstract

The present application provides a method, system, medium, and device for locating the position of text in a video, including: obtaining a reference frame containing text and a video frame to be located; extracting cross-modal features of the reference frame; extracting a frame feature matrix of the video frame to be located; constructing an input feature matrix based on the cross-modal features and the frame feature matrix, the input feature matrix including the cross-modal features, the frame feature matrix, and a position vector; inputting the input feature matrix into a large language model to obtain an output feature matrix; obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtaining a Gaussian distribution sample based on the mean and variance; obtaining a mask of the Gaussian distribution sample to determine the text position of the video frame to be located based on the mask. The present application can achieve the accuracy of text positioning in the video by performing feature extraction, fusion, and mask acquisition on all frames of the video text.

Description

Technical Field

[0001] The present application belongs to the technical field of video text recognition, and relates to a positioning method, and in particular to a video text position positioning method, system, medium and device. Background Art

[0002] As large amounts of video data are transmitted and consumed, the text within them needs to be processed to better understand and analyze the video content. In real life, the text within videos may include logos, subtitles, titles, and other information. This information can be very important and cannot be directly analyzed and processed. Therefore, accurately locating and recognizing text information in videos is of great significance.

[0003] Currently, video text recognition is a field of significant research and attention, with numerous mature methods and technologies. However, existing video text localization techniques primarily employ a process of defragging the video using a frame defragging code and then feeding the defragging results into a text localization model. This method results in low positioning accuracy. Furthermore, this method utilizes single-frame localization, lacking continuity and not taking into account the temporal information of previous and subsequent frames. Furthermore, due to the diversity and complexity of video content, the robustness of text processing methods also needs to be considered.

[0004] Therefore, combined with the above reasons, the existing technology has the problem of low accuracy in video text positioning technology.

[0005] Therefore, how to ensure the accuracy of video text positioning is an urgent problem to be solved. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a video text positioning method, system, medium and device for solving the problems in the existing video text positioning technology, such as the lack of timing of text positioning and low accuracy of text positioning, which reduces the efficiency of text positioning, due to the use of single-frame positioning and the failure to consider the timing information of the previous and next frames.

[0007] To achieve the above-mentioned purpose and other related purposes, in a first aspect, the present application provides a method for locating text in a video, comprising the following steps: obtaining a reference frame containing text and a video frame to be located; extracting cross-modal features of the reference frame; extracting a frame feature matrix of the video frame to be located; constructing an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; inputting the input feature matrix into a large language model to obtain an output feature matrix; obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtaining a Gaussian distribution sample based on the mean and variance; obtaining a mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask.

[0008] In an implementation of the first aspect, extracting the cross-modal features of the reference frame includes the following steps: performing text recognition on the reference frame to obtain text line content and position mask; extracting a text feature matrix of the text line content; extracting a mask feature matrix of the position mask; using the text feature matrix as a K vector and a V vector, and using the mask feature matrix as a Q vector, to obtain fused features based on a cross-attention mechanism; and passing the fused features through a multi-layer perceptron to obtain the cross-modal features.

[0009] In an implementation of the first aspect, a text feature matrix of the text line content is extracted based on a CLIP text encoder.

[0010] In an implementation of the first aspect, a frame feature matrix of the video frame to be located is extracted based on a VIT model.

[0011] In an implementation of the first aspect, constructing an input feature matrix based on the cross-modal features and the frame feature matrix includes the following steps: obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character, and a position character; passing the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character, and the position character through a multi-layer perceptron to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector, and a position vector; and concatenating the reference frame start vector, the cross-modal features, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector, and the position vector in sequence to obtain the input feature matrix.

[0012] In an implementation of the first aspect, obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix includes the following steps: inputting the output vector into a first multi-layer perceptron to obtain the mean; and inputting the output vector into a second multi-layer perceptron to obtain the variance.

[0013] In an implementation of the first aspect, obtaining the mask of the Gaussian distribution sample to determine the text position of the video frame to be located based on the mask includes the following steps: obtaining a feature map of the Gaussian distribution sample based on a decoder of a VAE model; inputting the feature map into a sigmoid function to obtain the mask of the Gaussian distribution sample; when the mask is greater than a preset value, setting the position corresponding to the mask in the video frame to be located as text; when the mask is less than the preset value, setting the position corresponding to the mask in the video frame to be located as background.

[0014] In a second aspect, the present application provides a video text position positioning system, comprising: an input module for obtaining a reference frame containing text and a video frame to be positioned; a feature extraction module for extracting cross-modal features of the reference frame; extracting a frame feature matrix of the video frame to be positioned; a feature matrix construction module for constructing an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; a matrix output module for inputting the input feature matrix into a large language model to obtain an output feature matrix; a text positioning module for obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtaining a Gaussian distribution sample based on the mean and variance; and obtaining a mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned according to the mask.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by an electronic device, the video text positioning method described in any one of the first aspects of the present application is implemented.

[0016] Finally, the present application provides a video text location device, comprising: a processor and a memory. The memory is configured to store a computer program; the processor is connected to the memory and configured to execute the computer program stored in the memory, thereby causing the video text location device to perform the video text location method.

[0017] As described above, the video text location method, system, medium, and device of the present invention have the following beneficial effects:

[0018] This application provides a method for locating text in a video. By extracting and fusing features from all text frames and obtaining a mask from the input frame, it is possible to traverse all frames and locate them. Timing information is also incorporated, effectively improving the timing and accuracy of video text positioning. Furthermore, the positioning method employed in this application can enhance the robustness of video text positioning technology, further improving positioning efficiency and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present application.

[0020] Figure 2 Shown is a flow chart of an embodiment of the video text location method described in this application.

[0021] Figure 3 Shown is a flow chart of S12 in the video text position locating method described in this application.

[0022] Figure 4 Shown is a flow chart of S14 in the video text position locating method described in this application.

[0023] Figure 5 Shown is a flow chart of S16 in the video text location system described in this application.

[0024] Figure 6 Shown is a flow chart of S18 in the video text location system described in this application.

[0025] Figure 7 Shown is a schematic diagram of the text position positioning process in one embodiment of the video text position positioning system described in this application.

[0026] Figure 8 Shown is a schematic diagram of the principle structure of the video text position positioning system described in this application in one embodiment.

[0027] Figure 9 Shown is a schematic diagram of the principle structure of the video text position positioning device described in this application in one embodiment.

[0028] Component number description

[0029] 11 cell phone 12 tablet 13 laptop 81 Input Module 82 Feature extraction module 83 Feature Matrix Building Blocks 84 Matrix output module 85 Text positioning module 91 processor 92 Memory S11~S18 step DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0032] The following embodiments of the present application provide a method, system, medium and device for locating the position of text in a video, which obtains a pixel-level matrix and a prediction feature matrix through a frame feature matrix and target text features, and performs target tracking based on the prediction feature matrix and target text features, thereby effectively tracking predefined targets of text prompts.

[0033] The video text location method of the present application can be applied to Figure 1 The electronic device described in this application may include a mobile phone 11 with wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0034] For example, the electronic device may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future-evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future-evolved Non-terrestrial Network (NTN).

[0035] For example, the electronic device can communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS).

[0036] The video text location method provided in the embodiment of the present application will be described in detail below in conjunction with the drawings in the embodiment of the present application.

[0037] See also Figure 2 , which is a flow chart of the video text location method described in this application in one embodiment. Figure 2 As shown, this embodiment provides a method for locating text positions in a video.

[0038] The video text position locating method specifically includes the following steps:

[0039] S11, obtaining a reference frame containing text and a video frame to be positioned.

[0040] In this embodiment, the first frame of the video frame to be positioned is preferably used for positioning. Positioning the first frame can complete positioning more quickly and accurately.

[0041] S12, extracting cross-modal features of the reference frame. Figure 3 , which is a flow chart of S12 in the video text location method described in this application. Figure 3 As shown, the S12 includes the following steps:

[0042] S121 , performing text recognition on the reference frame to obtain text line content and position mask.

[0043] In this example, several special characters are input, including characters representing the input frame to be located, characters indicating positions, and characters representing reference examples. An MLP (Multilayer Perceptron) is used to recognize and transform these special characters, generating different target text vectors and naming them accordingly. Target text includes text line content, text frames, and text position masks.

[0044] Specifically, the reference frame undergoes necessary preprocessing, including but not limited to denoising, binarization, and grayscaling, to improve the accuracy and robustness of text recognition. This helps eliminate interference from the image and enhance the prominence of text features. Then, image processing algorithms and other methods are used to identify the shape, structure, and pattern of the text to extract text features from the image. Next, a text recognition algorithm (such as Paddle OCR) is used to parse and identify the extracted text features. The text lines are then sorted and grouped according to their image location. Simultaneously, the precise position of each recognized text element in the image is recorded. This process can be achieved using a mask matrix of the same size as the image, where each element corresponds to a pixel in the image. At the location of the recognized text element, the corresponding element of the mask matrix is set to a specific value (e.g., 1), while remaining at the default value (e.g., 0). This position mask thus indicates the specific location and shape of the text line in the image.

[0045] It should be noted that in this implementation process, different algorithms or tools can be used, such as optical character recognition, deep learning algorithm, support vector machine, random forest algorithm, etc., and are not limited to the above technologies.

[0046] S122: Extracting a text feature matrix of the text line content.

[0047] In this embodiment, a text feature matrix of the text line content is extracted based on the CLIP text encoder.

[0048] Specifically, first, the extracted text line content is preprocessed, such as removing noise, punctuation marks, etc., and performing necessary text cleaning and normalization. This helps improve the text encoder's ability to understand and represent text content. Then, the preprocessed text is input into the CLIP text encoder to extract key features in Chinese and convert them into vectors. Through the CLIP text encoder, the text line content is converted into vectors and key feature information is extracted. Next, the extracted text feature vectors are combined into a text feature matrix. Among them, each row represents a text line, and the first column represents a feature dimension.

[0049] For example, the set of text line contents recognized by Paddle OCR is extracted with features by the CLIP image encoder to obtain a text feature matrix with a shape of (N, 256); where N represents the number of text lines.

[0050] It should be noted that the CLIP text encoder is trained based on a large-scale dataset. Therefore, when extracting text features, it can capture broader and deeper semantic information, which makes the text feature matrix extracted based on the CLIP text encoder have better performance in tasks such as text classification, clustering, and retrieval.

[0051] S123: Extracting a mask feature matrix of the position mask.

[0052] In this embodiment, the mask feature matrix of the position mask is extracted through VIT.

[0053] Specifically, based on the text feature matrix obtained in the above steps, a position mask is obtained; then, a VIT model is defined, such as a Transformer model, the position mask is input into the VIT model, the position mask is processed, and a feature matrix related to the position mask is extracted, that is, a mask feature matrix of the position mask is obtained.

[0054] S124, using the text feature matrix as the K vector and the V vector, and the mask feature matrix as the Q vector, to obtain fusion features based on the cross attention mechanism.

[0055] In this embodiment, the text feature matrix is used as the K vector and the V vector, and the mask feature matrix is used as the Q vector; the relevant parameters in the cross-attention mechanism are set, and then the attention score is calculated to obtain the attention score matrix; the elements corresponding to the mask positions in the attention score matrix are set to 0; then the attention weight is calculated to obtain the attention weight matrix, and the attention weight matrix is applied to the V vector, and the fusion feature is obtained by weighted summation and output.

[0056] S125: Pass the fusion features through a multi-layer perceptron to obtain the cross-modal features.

[0057] In this embodiment, a multi-layer perceptron structure is defined; the weights and bias parameters of the multi-layer perceptron are initialized; the fused features are input into the multi-layer perceptron for forward propagation, passing through the input layer, hidden layer, and output layer in sequence; and an output vector is generated at the output layer, which is the cross-modal feature. Finally, this cross-modal feature is named.

[0058] Specifically, based on the fused features, a multilayer perceptron structure is defined according to user needs and data size. This includes an input layer, hidden layers, and an output layer, and parameters such as the number of hidden layers, the number of neurons in each layer, and the activation function are determined. The bias parameters of the multilayer perceptron's weights are then initialized. The fused features are then input into the multilayer perceptron and forward propagated through it. During this forward propagation, the input features first pass through the input layer, then through each hidden layer in sequence, and finally reach the output layer. In each hidden layer, the fused features are processed using linear transformations and activation functions to capture the nonlinear relationships in the data. At the output layer, the multilayer perceptron generates an output vector, which is considered a cross-modal feature and is defined or named accordingly.

[0059] S13: extracting a frame feature matrix of the video frame to be located.

[0060] In this embodiment, the frame feature matrix of the video frame to be located is extracted based on the VIT model.

[0061] Specifically, first, the video frame to be located is input into the VIT model, and the input video frame is divided into image blocks or patches, thereby converting the video frame into a sequence; then a model such as a multi-layer perceptron (MLP) is used to convert it into an embedding vector; then a position code is added to each embedding vector; then the embedding vector and position code are input into the Transformer encoder for processing to obtain a feature vector; then the feature vector is aggregated through methods such as mean pooling to obtain a frame feature matrix for the entire video frame.

[0062] S14, constructing an input feature matrix based on the cross-modal features and the frame feature matrix. The input feature matrix includes the cross-modal features, the frame feature matrix and the position vector. Figure 4 , which is a flow chart of S14 in the video text location method described in this application. Figure 4 As shown, the S14 includes the following steps:

[0063] S141, obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character, and a position character.

[0064] In this embodiment, first, the formats of the reference frame start symbol, reference frame end symbol, positioning frame start symbol, and positioning frame end symbol are determined, and string matching, regular expressions, and other methods are used to search for the start symbol and end symbol of the reference frame and positioning frame in the data, and the starting and ending positions of the reference frame and positioning frame are determined based on the positions of these symbols; then, within the positioning frame, the position of the position character is determined, and the position character is extracted; then, the extracted reference frame start symbol, reference frame end symbol, positioning frame start symbol, positioning frame end symbol, and position characters are verified to ensure their accuracy and completeness.

[0065] S142, the reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, the positioning frame end symbol and the position symbol are respectively passed through a multi-layer perceptron to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector and a position vector.

[0066] In this embodiment, the reference frame start symbol, reference frame end symbol, positioning frame start symbol, positioning frame end symbol, and position character are converted into a numerical form suitable for MLP input; according to the data size and the requirements of the conversion task, a corresponding MLP model is defined for each symbol (such as the start symbol, end symbol, position character, etc.), and parameters such as the number of MLP layers, the number of neurons in each layer, and the activation function type are determined; then, the weight matrix and bias item of each MLP are randomly initialized, or pre-trained weights can be used for initialization; the numerical representations of the reference frame start symbol, reference frame end symbol, positioning frame start symbol, positioning frame end symbol, and position character are respectively passed as input to the corresponding MLP, and through forward propagation calculation, the final reference frame start vector, reference frame end vector, positioning frame start vector, positioning frame end vector, and position vector are extracted from the output layer of each MLP.

[0067] S143: Concatenate the reference frame start vector, the cross-modal feature, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector, and the position vector in sequence to obtain the input feature matrix.

[0068] In this embodiment, the reference frame start vector, cross-modal features, reference frame end vector, positioning frame start vector, frame feature matrix, positioning frame end vector and position vector are concatenated in sequence, and all vectors and frame feature matrix parts are fused into a final input feature matrix.

[0069] S15: Input the input feature matrix into a large language model to obtain an output feature matrix.

[0070] In this embodiment, the concatenated input feature matrix is preprocessed and then input into the large language model (LLM). After being processed by multiple network layers, such as an embedding layer, an attention mechanism, and a fully connected layer, the input feature matrix is transformed and calculated, and the output feature matrix is obtained through the output layer or the API interface. The output feature matrix has the same shape as the input feature matrix.

[0071] Specifically, the input feature matrix is encoded, and the encoded input feature matrix is input into the large language model LLM for processing, and the required information is extracted from the output of the LLM, and feature extraction and data conversion are performed on it to obtain the output feature matrix.

[0072] S16, obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix. Figure 5 , which is a flow chart of S16 in the video text location positioning system described in this application. Figure 5 As shown, the S16 includes the following steps:

[0073] S161: Input the output vector into a first multi-layer perceptron to obtain the mean.

[0074] S162: Input the output vector into a second multi-layer perceptron to obtain the variance.

[0075] In this embodiment, a first multilayer perceptron model is designed, and the number of layers, the number of neurons in each layer, the activation function, etc. of the first multilayer perceptron model are determined, and the weights and bias terms of the first multilayer perceptron model are initialized. Simultaneously, a second multilayer perceptron model is designed, and the number of layers, the number of neurons in each layer, the activation function, etc. of the second multilayer perceptron model are determined, and the weights and bias terms of the second multilayer perceptron model are initialized. The prepared output vector is passed as input to the first multilayer perceptron model; a forward propagation calculation is performed, passing through each layer of the first multilayer perceptron, and ultimately obtaining a predicted value of the mean. Similarly, the output vector is input into the second multilayer perceptron model, passing through each layer of the second multilayer perceptron, and ultimately obtaining a predicted value of the variance.

[0076] It should be noted that although a method using MLP to calculate the mean and variance is proposed here, in actual applications, the mean and variance are usually calculated directly using statistical methods rather than through machine learning models. The introduction of MLP may increase the complexity of the calculation, and in the absence of sufficient data or a suitable training strategy, accurate results may not be obtained. However, the calculation of the mean and variance in this application can use mathematical methods that can solve this problem and is not limited to the scope of MLP models, statistical methods, etc.

[0077] S17, obtaining a Gaussian distribution sample based on the mean and variance.

[0078] In this embodiment, a sample is sampled from the Gaussian distribution by calculating the mean and variance.

[0079] Specifically, a random value, a random sample, is generated according to the probability density function of the normal distribution using the mean and variance.

[0080] S18, obtaining the mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask. Figure 6 , which is a flow chart of S18 in the video text location positioning system described in this application. Figure 6 As shown, the S18 includes the following steps:

[0081] S181, a decoder based on the VAE model obtains a feature map of the Gaussian distribution sample;

[0082] S182, inputting the feature map into a sigmoid function to obtain a mask of the Gaussian distribution sample;

[0083] S183, when the mask is greater than a preset value, setting the position corresponding to the mask in the video frame to be positioned as text; when the mask is less than the preset value, setting the position corresponding to the mask in the video frame to be positioned as background.

[0084] In this embodiment, a VAE encoder / decoder is used to input Gaussian distribution samples into the VAE encoder for decoding, obtaining a low-dimensional feature vector. The sampled sample points are then transmitted to the VAE decoder to generate a feature map. A sigmoid algorithm is used to obtain a mask for the Gaussian distribution samples, and the relationship between the mask and a preset value is then determined. Specifically, when the mask is greater than the preset value, the position corresponding to the mask in the video frame to be located is set to text; when the mask is less than the preset value, the position corresponding to the mask in the video frame to be located is set to background.

[0085] Specifically, sigmoid is used to transform the feature map of the first frame to obtain a feature map probability value; a feature map probability mask is generated based on the feature map probability value; according to a preset value of the probability mask, it is determined whether the feature map probability mask of the first frame is text information; and then the first frame is replaced with the second frame for a new round of input processing.

[0086] The judgment rules are as follows: if the probability mask of the first frame feature map is greater than or equal to the mask preset value, the position mark is judged to be the text part; if the probability mask of the first frame feature map is less than the mask preset value, the position mark is the background part.

[0087] For example, when the preset value is set to 0.5, then: when it is greater than 0.5, it is set to 1 (text part), and when it is less than 0.5, it is set to 0 (background part), thus obtaining the text position of the first frame.

[0088] It should be noted that this application processes the first frame of text.

[0089] Therefore, after determining the position of the text in the first frame, replace the first frame with the second frame, and then process it step by step according to the above steps to obtain the mask of the second frame; and so on, repeat this process until all frames have completed text positioning, and then collect the masks of all input frames to obtain the text positioning result of the video.

[0090] It is important to note that the VAE training process minimizes reconstruction error and maximizes sample diversity in the latent space by optimizing the parameters of the encoder and decoder. Therefore, in practical applications, it is necessary to ensure that the VAE model has been fully trained and that its performance meets the requirements of the specific task. Moreover, the effectiveness of the above process depends on the quality of the feature map generated by the VAE model and whether the sigmoid function can accurately convert the feature values into probability values. If the quality of the feature map is not high or the conversion of the sigmoid function is inaccurate, the final text position may be biased. Therefore, in practical applications, multiple iterations and adjustments may be required to optimize the model output and subsequent processing steps. This process can effectively save a lot of time and cost and facilitate subsequent maintenance and updates.

[0091] The following is a detailed description of the video text location method of the present application through an embodiment. Figure 7 , which shows a flow chart of text location positioning in an embodiment of the video text location positioning system described in this application.

[0092] The first step is to obtain the reference frame containing text and the video frame to be positioned.

[0093] The second step is to extract cross-modal features of the reference frame.

[0094] Enter a few special characters: <example> ， <sample>and <location> ； <example>The content between represents reference examples. <sample>The content between represents the input frame to be positioned. <location>Indicates location.

[0095] The MLP model converts these special characters into vectors and names them a, b, c, d, and e. These vectors consist of the frame containing the text and its position mask. The position mask is determined by the position detected by Paddle OCR, with pixels containing text set to 1 and pixels without text set to 0.

[0096] Cross-modal features are then extracted. Specifically, the text line content set recognized by Paddle OCR is first extracted using the CLIP image encoder to obtain a text feature matrix with a shape of (N, 256), where N represents the number of text lines. Then, the position mask is extracted using a VIT to obtain a mask feature matrix with a shape of (, 256). Finally, the text feature matrix is used as KV and the mask feature matrix as Q, which are input into the cross-attention mechanism. The cross-attention mechanism outputs fused features, which are then mapped by an MLP to obtain the cross-modal features, named f.

[0097] Finally, VIT feature extraction is performed on the first frame to obtain a frame feature matrix with a shape of (784, 256), named g.

[0098] It should be noted that the first frame is the first frame of the position that needs to be detected.

[0099] The third step is to perform feature fusion and extraction based on the frame feature matrix to obtain the output feature matrix.

[0100] Specifically, the inputs are concatenated in the order of afbcgde to obtain an input feature matrix with a shape of (784 + 6, 256). The LLM then fuses and extracts features from each vector of the input feature matrix to obtain an output feature matrix with the same shape as the input.

[0101] The fourth step is to obtain the mask of the input frame based on the output feature matrix.

[0102] Select the vector at the last position of the output feature matrix, that is <location>The corresponding vector.

[0103] For the vector in the previous step, the mean and variance are obtained after two MLPs.

[0104] Draw a sample from a Gaussian distribution with the mean and variance.

[0105] The sample passes through the VAE decoder to generate the feature map of the first frame, and then the probability mask is obtained by sigmoid. The probability greater than 0.5 is set to 1 (text part) and the probability less than 0.5 is set to 0 (background part), thereby obtaining the text position of the first frame.

[0106] Finally, replace the first frame with the second frame to get a new round of input.

[0107] The fifth step is to traverse and collect the masks of all input frames to obtain the text positioning results of the video.

[0108] For the new input, follow steps 1 to 4 to obtain the mask for the second frame. Repeat this process until all frames have completed text localization. Collect the masks of all frames to obtain the text localization result for the video.

[0109] The video text location method provided by this application can traverse all frames and locate all frames by extracting and fusing the features of all text frames and obtaining the mask process of the input frame. At the same time, it incorporates timing information, effectively improving the timing and accuracy of video text location. At the same time, the location method adopted by this application can improve the robustness of video text location technology, further improving location efficiency and security.

[0110] The protection scope of the video text position positioning method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing, or replacing steps in the existing technology based on the principles of the present application are included in the protection scope of the present application.

[0111] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 2 The video text position positioning method.

[0112] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0113] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0114] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device. The computer program instructions for performing the operations of this application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as "C" language or similar programming languages. Computer readable program instructions can be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a computer, or entirely on a computer or server. In the case of a computer, the computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present application.

[0115] An embodiment of the present application also provides a video text position positioning system, which can implement the video text position positioning method described in the present application. However, the implementation device of the video text position positioning method described in the present application includes but is not limited to the structure of the video text position positioning system listed in this embodiment. All structural deformations and replacements of the existing technology made according to the principles of the present application are included in the protection scope of the present application.

[0116] The video text location system provided by this embodiment will be described in detail below with reference to the accompanying drawings.

[0117] This embodiment provides a video text location system, including:

[0118] See also Figure 8 , which is a schematic diagram showing the principle structure of the video text location system described in this application in one embodiment. Figure 8 As shown, the video text position positioning system includes: an input module 81, a feature extraction module 82, a feature matrix construction module 83, a matrix output module 84 and a text positioning module 85.

[0119] The input module 81 is used to obtain a reference frame containing text and a video frame to be positioned.

[0120] In this embodiment, the first frame is preferably used, because positioning the first frame can be completed more quickly and accurately.

[0121] The feature extraction module 82 is connected to the input module 81 and is used to extract the cross-modal features of the reference frame and extract the frame feature matrix of the video frame to be located.

[0122] Perform text recognition on the reference frame to obtain text line content and position mask; extract the text feature matrix of the text line content; extract the mask feature matrix of the position mask; use the text feature matrix as the K vector and V vector, and the mask feature matrix as the Q vector, and obtain fusion features based on the cross-attention mechanism; pass the fusion features through a multi-layer perceptron to obtain the cross-modal features.

[0123] In this embodiment, several special characters are input, including characters representing the input frame to be located, characters representing positions, and characters representing reference examples. These special characters are recognized and transformed using an MLP to obtain different target text vectors, each of which is named. The target text includes text line content, text frames, and text position masks. A text feature matrix for the text line content is extracted using the CLIP text encoder. The mask feature matrix for the position mask is extracted using the VIT, with a shape of ( , 256).

[0124] The text feature matrix is used as the K vector and V vector, and the mask feature matrix is used as the Q vector; the relevant parameters in the cross-attention mechanism are set, and then the attention score is calculated to obtain the attention score matrix; the elements corresponding to the mask position in the attention score matrix are set to 0; the attention weight is then calculated to obtain the attention weight matrix, and the attention weight matrix is applied to the V vector. The fusion feature is obtained and output through weighted summation.

[0125] In this embodiment, a multi-layer perceptron structure is defined; the weights and bias parameters of the multi-layer perceptron are initialized; the fused features are input into the multi-layer perceptron for forward propagation, passing through the input layer, hidden layer, and output layer in sequence; and an output vector is generated at the output layer, which is the cross-modal feature. Finally, this cross-modal feature is named.

[0126] Extracting a frame feature matrix of the video frame to be located.

[0127] In this embodiment, the frame feature matrix of the video frame to be located is extracted based on the VIT model.

[0128] The feature matrix construction module 83 is used to construct an input feature matrix based on the cross-modal features and the frame feature matrix, where the input feature matrix includes the cross-modal features, the frame feature matrix, and a position vector.

[0129] First, obtain the reference frame start character, reference frame end character, positioning frame start character, positioning frame end character, and position character. Next, pass the reference frame start character, reference frame end character, positioning frame start character, positioning frame end character, and position character through a multi-layer perceptron to obtain the reference frame start vector, reference frame end vector, positioning frame start vector, positioning frame end vector, and position vector. Then, concatenate the reference frame start vector, the cross-modal feature, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector, and the position vector in sequence to obtain the input feature matrix.

[0130] The matrix output module 84 is configured to input the input feature matrix into a large language model to obtain an output feature matrix.

[0131] In this embodiment, the concatenated input feature matrix is preprocessed and then input into the large language model (LLM). After being processed by multiple network layers, such as an embedding layer, an attention mechanism, and a fully connected layer, the input feature matrix is transformed and calculated, and the output feature matrix is obtained through the output layer or the API interface. The output feature matrix has the same shape as the input feature matrix.

[0132] Specifically, the input feature matrix is encoded, and the encoded input feature matrix is input into the large language model LLM for processing, and the required information is extracted from the output of the LLM, and feature extraction and data conversion are performed on it to obtain the output feature matrix.

[0133] The text positioning module 85 is used to obtain the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtain a Gaussian distribution sample based on the mean and variance; obtain the mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned according to the mask.

[0134] In this embodiment, the output vector is input into a first multilayer perceptron to obtain the mean, the output vector is input into a second multilayer perceptron to obtain the variance, and a Gaussian distribution sample is obtained based on the mean and variance.

[0135] In this embodiment, a sample is sampled from the Gaussian distribution by calculating the mean and variance.

[0136] Obtain a mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask; obtain a feature map of the Gaussian distribution sample by a decoder based on the VAE model; input the feature map into a sigmoid function to obtain the mask of the Gaussian distribution sample; when the mask is greater than a preset value, set the position corresponding to the mask in the video frame to be located as text; when the mask is less than the preset value, set the position corresponding to the mask in the video frame to be located as background.

[0137] In this embodiment, a VAE encoder / decoder is used to input Gaussian distribution samples into the VAE encoder for decoding, obtaining a low-dimensional feature vector. The sampled sample points are then transmitted to the VAE decoder to generate a feature map. A sigmoid algorithm is used to obtain a mask for the Gaussian distribution samples, and the relationship between the mask and a preset value is then determined. Specifically, when the mask is greater than the preset value, the position corresponding to the mask in the video frame to be located is set to text; when the mask is less than the preset value, the position corresponding to the mask in the video frame to be located is set to background.

[0138] Therefore, after determining the position of the text in the first frame, replace the first frame with the second frame, and then process it step by step according to the above steps to obtain the mask of the second frame; and so on, repeat this process until all frames have completed text positioning, and then collect the masks of all input frames to obtain the text positioning result of the video.

[0139] The video text location positioning model builds a video text location positioning system that can traverse all frames and locate all frames. At the same time, it adds timing information to it, effectively improving the timing and accuracy of video text positioning.

[0140] It should be understood that the division of the modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, or entirely in hardware. Alternatively, some modules may be implemented in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element, or integrated into a chip in the above system. Furthermore, it may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. Furthermore, these modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the modules above may be performed by hardware integrated logic circuits in the processor element or by software instructions.

[0141] The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0142] See also Figure 9 , which is a schematic diagram showing the principle structure of the video text location positioning device described in this application in one embodiment. Figure 9 As shown, this embodiment provides a video text position positioning device, which includes: a processor 91 and a memory 92; the memory 92 is used to store computer programs; the processor 91 is connected to the memory 92, and is used to execute the computer program stored in the memory 92, so that the video text position positioning device performs the various steps of the video text position positioning method described above.

[0143] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0144] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0145] In summary, the video text location method, system, medium, and device provided by this application have the following beneficial effects:

[0146] The video text location method provided by this application can traverse all frames and locate all frames by extracting and fusing the features of all text frames and obtaining the mask process of the input frame. At the same time, it incorporates timing information, effectively improving the timing and accuracy of video text location. At the same time, the location method adopted by this application can improve the robustness of video text location technology, further improving location efficiency and security.

[0147] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.< / location> < / location> < / sample> < / example> < / location> < / sample> < / example>

Claims

1. A method for locating text positions in a video, characterized in that: The following steps are involved: Obtain the reference frame containing text and the video frame to be positioned; Extracting cross-modal features of the reference frame; including: performing text recognition on the reference frame to obtain text line content and position mask; extracting a text feature matrix of the text line content based on a CLIP text encoder; extracting a mask feature matrix of the position mask; using the text feature matrix as a K vector and a V vector, and using the mask feature matrix as a Q vector, to obtain fused features based on a cross-attention mechanism; and passing the fused features through a multi-layer perceptron to obtain the cross-modal features; Extracting a frame feature matrix of the video frame to be located; An input feature matrix is constructed based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; the method includes: obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character and a position character; passing the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character and the position character through a multi-layer perceptron respectively to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector and a position vector; the reference frame start vector, the cross-modal features, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector and the position vector are sequentially concatenated to obtain the input feature matrix; wherein the position vector is the position information of the text in the positioning frame, including the starting position and the ending position of the text; the position character refers to data that directly describes the position in the positioning frame; Inputting the input feature matrix into a large language model to obtain an output feature matrix; Obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; Obtain a Gaussian distribution sample based on the mean and variance; A mask of the Gaussian distribution sample is obtained to determine the text position of the video frame to be located according to the mask.

2. The method for locating text positions in a video according to claim 1, wherein: A frame feature matrix of the video frame to be located is extracted based on the VIT model.

3. The method for locating text positions in a video according to claim 1, wherein: Obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix includes the following steps: Inputting the output vector into a first multilayer perceptron to obtain the mean; The output vector is input into a second multi-layer perceptron to obtain the variance.

4. The method for locating text positions in a video according to claim 1, wherein: Obtaining the mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask includes the following steps: A decoder based on the VAE model obtains a feature map of the Gaussian distribution sample; Input the feature map into a sigmoid function to obtain a mask of the Gaussian distribution sample; When the mask is larger than a preset value, the position corresponding to the mask in the video frame to be positioned is set as text; when the mask is smaller than the preset value, the position corresponding to the mask in the video frame to be positioned is set as background.

5. A video text location system, characterized in that: include: An input module, used to obtain a reference frame containing text and a video frame to be positioned; A feature extraction module is used to extract cross-modal features of the reference frame; the module includes: performing text recognition on the reference frame to obtain text line content and position mask; extracting a text feature matrix of the text line content based on a CLIP text encoder; extracting a mask feature matrix of the position mask; using the text feature matrix as a K vector and a V vector, and the mask feature matrix as a Q vector, to obtain fused features based on a cross-attention mechanism; passing the fused features through a multi-layer perceptron to obtain the cross-modal features; and extracting a frame feature matrix of the video frame to be located; A feature matrix construction module is used to construct an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; the module includes: obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character and a position character; passing the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character and the position character through a multi-layer perceptron respectively to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector and a position vector; concatenating the reference frame start vector, the cross-modal features, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector and the position vector in sequence to obtain the input feature matrix; wherein the position vector is the position information of the text in the positioning frame, including the starting position and ending position of the text; the position character refers to data that directly describes the position in the positioning frame; A matrix output module, configured to input the input feature matrix into a large language model to obtain an output feature matrix; A text positioning module is used to obtain the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtain a Gaussian distribution sample based on the mean and variance; obtain a mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned based on the mask.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the video text positioning method according to any one of claims 1 to 4 is implemented.

7. A video text location device, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the video text position locating device executes the video text position locating method according to any one of claims 1 to 4.

Images