Video character position positioning method, system, medium and device

Abstract

The invention provides a video character position positioning method, system, medium and device. The method comprises the following steps: acquiring a reference frame containing characters and a to-be-positioned video frame; extracting cross-modal features of the reference frame; extracting a frame feature matrix of the to-be-positioned video frame; constructing an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix comprises the cross-modal features, the frame feature matrix and a position vector; inputting the input feature matrix into a large language model to obtain an output feature matrix; obtaining a mean value and a variance of an output vector corresponding to the position vector in the output feature matrix; obtaining a Gaussian distribution sample based on the mean value and the variance; and obtaining a mask of the Gaussian distribution sample, and determining a character position of the to-be-positioned video frame according to the mask. According to the method and the device, the accuracy of character positioning in the video can be realized through technologies such as feature extraction, fusion and mask acquisition on all frames of the video character.

Description

Technical Field

[0001] The present application belongs to the technical field of video text recognition, and relates to a positioning method, and in particular to a video text position positioning method, system, medium and device. Background Art

[0002] As a large amount of video data is transmitted and used, the text in it needs to be processed in order to better understand and analyze the video content. In real life, the text in the video may involve information such as logos, subtitles, and titles. This information may be very important and cannot be directly analyzed and processed. Therefore, how to accurately locate and identify text information in the video is of great significance.

[0003] At present, video text recognition is a field that has received much attention and research, and there are many mature methods and technologies. However, in the existing video text positioning technology, the main method is to split the frame through the video frame splitting code, and input the result of the frame splitting into the text positioning model. The positioning accuracy of the positioning result of this method is low. Moreover, this positioning method uses single-frame positioning, which is not continuous and does not take into account the timing information of the previous and next frames. In addition, due to the diversity and complexity of video content, the robustness of the text processing method is also a factor that needs to be considered.

[0004] Therefore, combined with the above reasons, the existing technology has the problem of low accuracy in video text positioning technology.

[0005] Therefore, how to ensure the accuracy of video text positioning is a problem that needs to be solved urgently. Summary of the invention

[0006] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a video text positioning method, system, medium and device for solving the problems in the existing video text positioning technology, such as the lack of timing of text positioning and low accuracy of text positioning, thereby reducing the efficiency of text positioning, due to the use of single-frame positioning and failure to consider the timing information of the previous and next frames.

[0007] To achieve the above-mentioned purpose and other related purposes, in a first aspect, the present application provides a method for locating the position of text in a video, comprising the following steps: obtaining a reference frame containing text and a video frame to be located; extracting cross-modal features of the reference frame; extracting a frame feature matrix of the video frame to be located; constructing an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; inputting the input feature matrix into a large language model to obtain an output feature matrix; obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtaining a Gaussian distribution sample based on the mean and variance; obtaining a mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask.

[0008] In an implementation of the first aspect, extracting the cross-modal features of the reference frame includes the following steps: performing text recognition on the reference frame to obtain text line content and position mask; extracting a text feature matrix of the text line content; extracting a mask feature matrix of the position mask; using the text feature matrix as a K vector and a V vector, and using the mask feature matrix as a Q vector, to obtain fused features based on a cross-attention mechanism; and passing the fused features through a multi-layer perceptron to obtain the cross-modal features.

[0009] In an implementation manner of the first aspect, a text feature matrix of the text line content is extracted based on a CLIP text encoder.

[0010] In an implementation manner of the first aspect, a frame feature matrix of the video frame to be located is extracted based on a VIT model.

[0011] In an implementation of the first aspect, constructing an input feature matrix based on the cross-modal features and the frame feature matrix includes the following steps: obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character, and a position character; passing the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character, and the position character through a multi-layer perceptron respectively to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector, and a position vector; and concatenating the reference frame start vector, the cross-modal features, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector, and the position vector in sequence to obtain the input feature matrix.

[0012] In an implementation of the first aspect, obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix includes the following steps: inputting the output vector into a first multilayer perceptron to obtain the mean; and inputting the output vector into a second multilayer perceptron to obtain the variance.

[0013] In an implementation of the first aspect, obtaining the mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask includes the following steps: obtaining a feature map of the Gaussian distribution sample based on a decoder of a VAE model; inputting the feature map into a sigmoid function to obtain the mask of the Gaussian distribution sample; when the mask is greater than a preset value, setting the position corresponding to the mask in the video frame to be located as text; when the mask is less than the preset value, setting the position corresponding to the mask in the video frame to be located as background.

[0014] In a second aspect, the present application provides a video text position positioning system, comprising: an input module, used to obtain a reference frame containing text and a video frame to be positioned; a feature extraction module, used to extract cross-modal features of the reference frame; extract a frame feature matrix of the video frame to be positioned; a feature matrix construction module, used to construct an input feature matrix based on the cross-modal features and the frame feature matrix, the input feature matrix including the cross-modal features, the frame feature matrix and a position vector; a matrix output module, used to input the input feature matrix into a large language model to obtain an output feature matrix; a text positioning module, used to obtain the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtain a Gaussian distribution sample based on the mean and variance; obtain a mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned according to the mask.

[0015] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by an electronic device, the video text location method described in any one of the first aspects of the present application is implemented.

[0016] In the last aspect, the present application provides a video text location positioning device, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the video text location positioning device executes the video text location positioning method.

[0017] As described above, the video text location positioning method, system, medium and device of the present invention have the following beneficial effects: The present application provides a method for locating the position of text in a video. By extracting and fusing the features of all text frames and obtaining the mask process of the input frame, all frames can be traversed and all frames can be located. At the same time, timing information is added to it, which effectively improves the timing and accuracy of video text positioning. At the same time, the positioning method adopted by the present application can improve the robustness of the video text positioning technology and further improve the positioning efficiency and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present application.

[0019] Figure 2 Shown is a flow chart of a method for locating text positions in a video according to an embodiment of the present application.

[0020] Figure 3 Shown is a flow chart of S12 in the video text position locating method described in this application.

[0021] Figure 4 Shown is a flow chart of S14 in the video text position locating method described in this application.

[0022] Figure 5 Shown is a flow chart of S16 in the video text position positioning system described in this application.

[0023] Figure 6 Shown is a flow chart of S18 in the video text position positioning system described in this application.

[0024] Figure 7 Shown is a schematic diagram of the text position positioning process of the video text position positioning system described in this application in one embodiment.

[0025] Figure 8 Shown is a schematic diagram of the principle structure of a video text location positioning system described in this application in one embodiment.

[0026] Fig. 9 Shown is a schematic diagram of the principle structure of a video text position locating device described in this application in one embodiment.

[0027] Component number description 11 cell phone 12 Tablet 13 Laptop 81 Input Module 82 Feature extraction module 83 Feature Matrix Building Blocks 84 Matrix output module 85 Text positioning module 91 processor 92 Memory S11~S18 step DETAILED DESCRIPTION

[0028] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0030] The following embodiments of the present application provide a method, system, medium and device for locating the position of text in a video, which obtains a pixel-level matrix and a prediction feature matrix through a frame feature matrix and target text features, and performs target tracking based on the prediction feature matrix and target text features, thereby effectively tracking predefined targets of text prompts.

[0031] The video text location positioning method of the present application can be applied to Figure 1 The electronic device shown. The electronic device described in this application may include a mobile phone 11 with wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of this application does not impose any restrictions on the specific type of the electronic device.

[0032] For example, the electronic device may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a personal digital assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved public land mobile network (PLMN), or a mobile terminal in a future evolved non-terrestrial network (NTN).

[0033] For example, the electronic device can communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Satellite Navigation System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS).

[0034] The video text location method provided in the embodiment of the present application will be described in detail below in conjunction with the drawings in the embodiment of the present application.

[0035] See also Figure 2 , which is a flow chart of a method for locating text positions in a video according to an embodiment of the present application. Figure 2 As shown, this embodiment provides a method for locating a text position in a video.

[0036] The video text location positioning method specifically comprises the following steps: S11, obtaining a reference frame containing text and a video frame to be located.

[0037] In this embodiment, the video frame to be positioned preferably uses the first frame for positioning. Positioning the first frame can complete positioning more quickly and accurately.

[0038] S12, extracting cross-modal features of the reference frame. Figure 3, which is a flow chart of S12 in the video text location method described in this application. Figure 3 As shown, the S12 comprises the following steps: S121, performing text recognition on the reference frame to obtain text line content and position mask.

[0039] In this embodiment, several special characters are input respectively, including: characters of the input frame to be located, characters indicating positions, characters indicating reference examples, etc. MLP (Multilayer Perceptron) is used to recognize and transform these special characters to obtain different target text vectors, and name them respectively. Among them, the target text includes: text line content, text frame, and text position mask.

[0040] Specifically, first, the reference frame is preprocessed as necessary, including but not limited to denoising, binarization, grayscale and other operations to improve the accuracy and robustness of text recognition. This helps to eliminate interference factors in the image and make the text features more prominent. Then, the shape, structure and pattern of the text are identified by image processing algorithms and other methods to extract the text features in the image. Next, the extracted text features are parsed and identified by a text recognition algorithm (such as paddle ocr, etc.), the content of the text line is extracted from the identified text, and the text lines are sorted and grouped according to the position of the image where the text lines are located. At the same time, the exact position of each identified text element in the image is recorded; this process can be achieved by a mask matrix of the same size as the image, where each element corresponds to a pixel in the image. At the position of the identified text element, the corresponding element of the mask matrix is ​​set to a specific value (such as 1), and kept as the default value (such as 0) at other positions. In this way, the position mask can indicate the specific position and shape of the text line in the image.

[0041] It should be noted that in this implementation process, different algorithms or tools can be used to implement it, such as: optical character recognition, deep learning algorithm, support vector machine, random forest algorithm, etc., but it is not limited to the above technologies.

[0042] S122: extracting a text feature matrix of the text line content.

[0043] In this embodiment, a text feature matrix of the text line content is extracted based on the CLIP text encoder.

[0044] Specifically, first, the extracted text line content is preprocessed, such as removing noise, punctuation marks, etc., and necessary text cleaning and normalization are performed. This helps to improve the text encoder's ability to understand and represent text content. Then, the preprocessed text is input into the CLIP text encoder to extract key features in Chinese and convert them into vectors. Through the CLIP text encoder, the text line content is converted into vectors and key feature information is extracted. Next, the extracted text feature vectors are combined into a text feature matrix. Among them, each row represents a text line, and the first column represents a feature dimension.

[0045] For example, the set of text line contents recognized by Paddle OCR is extracted with features by the CLIP image encoder to obtain a text feature matrix with a shape of (N, 256), where N represents the number of text lines.

[0046] It should be noted that the CLIP text encoder is trained based on a large-scale dataset. Therefore, when extracting text features, it can capture broader and deeper semantic information, which makes the text feature matrix extracted based on the CLIP text encoder have better performance in text classification, clustering, retrieval and other tasks.

[0047] S123, extracting a mask feature matrix of the position mask.

[0048] In this embodiment, the mask feature matrix of the position mask is extracted through VIT.

[0049] Specifically, based on the text feature matrix obtained in the above steps, a position mask is obtained; then, a VIT model is defined, such as a Transformer model, the position mask is input into the VIT model, the position mask is processed, and a feature matrix related to the position mask is extracted, that is, a mask feature matrix of the position mask is obtained.

[0050] S124, using the text feature matrix as a K vector and a V vector, and using the mask feature matrix as a Q vector, and obtaining fusion features based on a cross-attention mechanism.

[0051] In this embodiment, the text feature matrix is ​​used as the K vector and the V vector, and the mask feature matrix is ​​used as the Q vector; the relevant parameters in the cross-attention mechanism are set, and then the attention score is calculated to obtain the attention score matrix; the elements corresponding to the mask positions in the attention score matrix are set to 0; then the attention weight is calculated to obtain the attention weight matrix, and the attention weight matrix is ​​applied to the V vector, and the fused features are obtained and output by weighted summation.

[0052] S125, passing the fused features through a multi-layer perceptron to obtain the cross-modal features.

[0053] In this embodiment, a multi-layer perceptron structure is defined; the weights and bias parameters of the multi-layer perceptron are initialized; the fused features are input into the multi-layer perceptron for forward propagation, passing through the input layer, hidden layer and output layer in sequence; and an output vector is generated in the output layer, which is a cross-modal feature, and finally the cross-modal feature is named.

[0054] Specifically, based on the fusion features, according to user needs and data scale, a multi-layer perceptron structure is defined; including: input layer, hidden layer and output layer, and the number of hidden layers, the number of neurons in each layer and the activation function and other parameters are determined. Then, the bias parameters of the weights of the multi-layer perceptron are initialized. Next, the fusion features are input to the multi-layer perceptron and forward propagated through the multi-layer perceptron; during the forward propagation process, the input features first pass through the input layer, then pass through each hidden layer in turn, and finally reach the output layer; and in each hidden layer, the fusion features will be processed by linear transformation and activation function to capture the nonlinear relationship of the data. Then, in the output layer, the multi-layer perceptron will generate an output vector, and the vector will be regarded as a cross-modal feature and defined or named.

[0055] S13, extracting a frame feature matrix of the video frame to be located.

[0056] In this embodiment, a frame feature matrix of the video frame to be located is extracted based on the VIT model.

[0057] Specifically, first, the video frame to be located is input into the VIT model, and the input video frame is divided into image blocks or patches, so as to convert the video frame into a sequence; then a model such as a multi-layer perceptron MLP is used to convert it into an embedding vector; then a position code is added to each embedding vector; then the embedding vector and the position code are input into the Transformer encoder for processing to obtain a feature vector; then the feature vector is aggregated by methods such as mean pooling to obtain a frame feature matrix of the entire video frame.

[0058] S14, constructing an input feature matrix based on the cross-modal features and the frame feature matrix. The input feature matrix includes the cross-modal features, the frame feature matrix and the position vector. Figure 4 , which is a flow chart of S14 in the video text location method described in this application. Figure 4 As shown, the S14 comprises the following steps: S141, obtaining a reference frame start character, a reference frame end character, a positioning frame start character, a positioning frame end character and a position character.

[0059] In this embodiment, first, the formats of the reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, and the positioning frame end symbol are determined, and the start symbol and the end symbol of the reference frame and the positioning frame are searched in the data using string matching, regular expressions, and the like, and the start and end positions of the reference frame and the positioning frame are determined based on the positions of these symbols; then, within the positioning frame, the position of the position character is determined, and the position character is extracted; then, the extracted reference frame start symbol, reference frame end symbol, positioning frame start symbol, positioning frame end symbol, and position characters are verified to ensure their accuracy and completeness.

[0060] S142, the reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, the positioning frame end symbol and the position symbol are respectively passed through a multi-layer perceptron to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector and a position vector.

[0061] In this embodiment, the reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, the positioning frame end symbol and the position character are converted into a numerical form suitable for MLP input; according to the data size and the requirements of the conversion task, a corresponding MLP model is defined for each symbol (such as: start symbol, end symbol, position character, etc.), and the number of layers of the MLP, the number of neurons in each layer, the activation function type and other parameters are determined; then, the weight matrix and bias item of each MLP are randomly initialized, or the pre-trained weights can be used for initialization; the numerical representations of the reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, the positioning frame end symbol and the position character are respectively used as input and passed into the corresponding MLP, and the final reference frame start vector, reference frame end vector, positioning frame start vector, positioning frame end vector and position vector are extracted from the output layer of each MLP through forward propagation calculation.

[0062] S143, sequentially concatenate the reference frame start vector, the cross-modal feature, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector and the position vector to obtain the input feature matrix.

[0063] In this embodiment, the reference frame start vector, cross-modal features, reference frame end vector, positioning frame start vector, frame feature matrix, positioning frame end vector and position vector are concatenated in sequence, and all vectors and frame feature matrix parts are fused into a final input feature matrix.

[0064] S15, inputting the input feature matrix into a large language model to obtain an output feature matrix.

[0065] In this embodiment, the concatenated input feature matrix is ​​preprocessed, and then the preprocessed input feature matrix is ​​input into the large language model LLM. After being processed by multiple network layers, such as: embedding layer, attention mechanism, fully connected layer, etc., the input feature matrix is ​​transformed and calculated, and the output feature matrix is ​​obtained through the output layer or API interface; and the output feature matrix has the same shape as the input feature matrix.

[0066] Specifically, the input feature matrix is ​​encoded, and the encoded input feature matrix is ​​input into the large language model LLM for processing, and the required information is extracted from the output of the LLM, and feature extraction and data conversion are performed on it to obtain the output feature matrix.

[0067] S16, obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix. Figure 5 , which is a flow chart of S16 in the video text location positioning system described in this application. Figure 5 As shown, the S16 comprises the following steps: S161, input the output vector into a first multilayer perceptron to obtain the mean.

[0068] S162: Input the output vector into a second multi-layer perceptron to obtain the variance.

[0069] In this embodiment, a first multilayer perceptron model is designed, and the number of layers, the number of neurons in each layer, the activation function, etc. of the first multilayer perceptron model are determined, and the weights and bias items of the first multilayer perceptron model are initialized; at the same time, a second multilayer perceptron model is designed, and the number of layers, the number of neurons in each layer, the activation function, etc. of the second multilayer perceptron model are determined, and the weights and bias items of the second multilayer perceptron model are initialized. The prepared output vector is passed as input to the first multilayer perceptron model; forward propagation calculation is performed, and the predicted value of the mean is finally obtained through each layer of the first multilayer perceptron. Similarly, the output vector is input into the second multilayer perceptron model, and the predicted value of the variance is finally obtained through each layer of the second multilayer perceptron.

[0070] It should be noted that although a method of using MLP to calculate the mean and variance is proposed here, in actual applications, the mean and variance are usually calculated directly using statistical methods rather than through machine learning models. The introduction of MLP may increase the complexity of the calculation, and accurate results may not be obtained without sufficient data or a suitable training strategy. However, the calculation of the mean and variance in this application can use mathematical methods that can solve this problem, and is not limited to the scope of MLP models, statistical methods, etc.

[0071] S17, obtaining a Gaussian distribution sample based on the mean and variance.

[0072] In this embodiment, a sample is sampled from a Gaussian distribution by calculating the mean and variance.

[0073] Specifically, a random sample of random values ​​is generated according to the probability density function of the normal distribution through the mean and variance.

[0074] S18, obtaining a mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask. Figure 6 , which is a flow chart of S18 in the video text location positioning system described in this application. Figure 6 As shown, the S18 comprises the following steps: S181, a decoder based on a VAE model obtains a feature map of the Gaussian distribution sample; S182, inputting the feature map into a sigmoid function to obtain a mask of the Gaussian distribution sample; S183, when the mask is greater than a preset value, setting the position corresponding to the mask in the video frame to be located as text; when the mask is less than the preset value, setting the position corresponding to the mask in the video frame to be located as background.

[0075] In this embodiment, the encoder decoder of VAE is used, and the Gaussian distribution samples are input into the encoder of VAE for decoding processing to obtain a low-dimensional feature vector; then the sample points obtained by sampling are transmitted to the decoder decoder of VAE to generate a feature map. The mask of the Gaussian distribution sample is obtained by sigmoid, and then the relationship between the mask and the preset value is determined, that is, when the mask is greater than the preset value, the position corresponding to the mask in the video frame to be located is set to text; when the mask is less than the preset value, the position corresponding to the mask in the video frame to be located is set to background.

[0076] Specifically, sigmoid is used to transform the feature map of the first frame to obtain a feature map probability value; a feature map probability mask is generated based on the feature map probability value; according to a preset value of the probability mask, it is determined whether the feature map probability mask of the first frame is text information; and then the first frame is replaced by the second frame for a new round of input processing.

[0077] The judgment rules are as follows: if the probability mask of the first frame feature map is greater than or equal to the preset mask value, the position is marked as a text part; if the probability mask of the first frame feature map is less than the preset mask value, the position is marked as a background part.

[0078] For example, when the preset value is set to 0.5, then: when it is greater than 0.5, it is set to 1 (text part), and when it is less than 0.5, it is set to 0 (background part), thus obtaining the text position of the first frame.

[0079] It should be noted that the present application processes the first frame of text.

[0080] Therefore, after the text position of the first frame is determined, the first frame is replaced by the second frame, and then the above steps are followed step by step to obtain the mask of the second frame; and so on, this process is repeated until all frames have completed text positioning, and the masks of all input frames are collected to obtain the text positioning result of the video.

[0081] It should be noted that the training process of VAE is to minimize the reconstruction error and maximize the sample diversity in the latent space by optimizing the parameters of the encoder and decoder. Therefore, in practical applications, it is necessary to ensure that the VAE model has been fully trained and its performance meets the specific task. Moreover, the effectiveness of the above process depends on the quality of the feature map generated by the VAE model and whether the sigmoid function can accurately convert the feature value into a probability value. If the quality of the feature map is not high or the conversion of the sigmoid function is inaccurate, the final text position may be biased. Therefore, in practical applications, multiple iterations and adjustments may be required to optimize the output of the model and subsequent processing steps. This process can effectively save a lot of time cost and facilitate later maintenance and updates.

[0082] The following is a detailed description of the video text location method of the present application through an embodiment. Figure 7 , which is a schematic diagram of the text position positioning process of the video text position positioning system described in this application in one embodiment.

[0083] The first step is to obtain the reference frame containing text and the video frame to be located.

[0084] The second step is to extract cross-modal features of the reference frame.

[0085] Enter a few special characters: <example> ， <sample>and <location> ； <example>The content between represents reference examples. <sample>The content between represents the input frame to be located. <location>Indicates location.

[0086] For these special characters, the MLP model converts them into vectors and names them as: a, b, c, d, e. These vectors are composed of the frame containing the text and its text position mask. The position mask is determined by the position detected by Paddle OCR. The pixel with text is 1, and the pixel without text is 0.

[0087] Then, cross-modal features are extracted. Specifically, the text line content set recognized by Paddle OCR is first extracted by CLIP image encoder to obtain a text feature matrix with a shape of (N, 256), where N represents the number of text lines; then, the position mask is extracted by a VIT to obtain a mask feature matrix with a shape of (, 256); finally, the text feature matrix is ​​used as KV and the mask feature matrix is ​​used as Q, which are input into the cross-attention mechanism. The cross-attention mechanism outputs the fusion feature, which is then mapped by an MLP to obtain the cross-modal feature, named f.

[0088] Finally, VIT feature extraction is performed on the first frame to obtain a frame feature matrix with a shape of (784, 256) and named g.

[0089] It should be noted that the first frame is the first frame of the position that needs to be detected.

[0090] The third step is to perform feature fusion and extraction based on the frame feature matrix to obtain the output feature matrix.

[0091] Specifically, we first concatenate the inputs in the order of afbcgde to obtain an input feature matrix with a shape of (784 + 6, 256). Then, LLM performs feature fusion and extraction on each vector of the input feature matrix to obtain an output feature matrix with the same shape as the input.

[0092] The fourth step is to obtain the mask of the input frame based on the output feature matrix.

[0093] Select the vector at the last position of the output feature matrix, that is <location>The corresponding vector.

[0094] For the vector in the previous step, the mean and variance are obtained after two MLPs.

[0095] Sample a Gaussian distribution using its mean and variance.

[0096] The sample passes through the VAE decoder to generate the feature map of the first frame, and then the probability mask is obtained by sigmoid, and the probability greater than 0.5 is set to 1 (text part) and the probability less than 0.5 is set to 0 (background part), so as to obtain the text position of the first frame.

[0097] Finally, replace the first frame with the second frame to get a new round of input.

[0098] The fifth step is to traverse and collect the masks of all input frames to obtain the text positioning results of the video.

[0099] For the new round of input, follow steps 1 to 4 to get the mask of the second frame. Repeat until all frames have completed text localization, and then collect the masks of all frames to get the text localization result of the video.

[0100] The video text location positioning method provided by the present application can traverse all frames and locate all frames by extracting and fusing the features of all frames of the text and obtaining the mask process of the input frame. At the same time, the timing information is added to it, which effectively improves the timing and accuracy of video text location. At the same time, the positioning method adopted by the present application can improve the robustness of the video text location technology and further improve the positioning efficiency and security.

[0101] The protection scope of the video text position positioning method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0102] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 2 The video text position positioning method.

[0103] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0104] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or protruding structures in grooves on which instructions are stored, and any suitable combination of the above. The computer-readable storage media used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0105] The computer-readable program described herein can be downloaded to each computing / processing device from a computer-readable storage medium, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device. The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcodes, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, and the programming language includes object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" language or similar programming languages. Computer readable program instructions can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a computer, or completely on a computer or server. In the case of a computer, the computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of computer readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0106] The embodiment of the present application also provides a video text position positioning system, which can implement the video text position positioning method described in the present application, but the implementation device of the video text position positioning method described in the present application includes but is not limited to the structure of the video text position positioning system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0107] The video text location positioning system provided by this embodiment will be described in detail below with reference to the diagrams.

[0108] This embodiment provides a video text location positioning system, including: See also Figure 8 , which is a schematic diagram showing the principle structure of a video text location positioning system described in this application in one embodiment. Figure 8 As shown, the video text position positioning system includes: an input module 81, a feature extraction module 82, a feature matrix construction module 83, a matrix output module 84 and a text positioning module 85.

[0109] The input module 81 is used to obtain a reference frame containing text and a video frame to be located.

[0110] In this embodiment, the first frame is preferably used. Positioning the first frame can complete positioning more quickly and accurately.

[0111] The feature extraction module 82 is connected to the input module 81 and is used to extract the cross-modal features of the reference frame; and extract the frame feature matrix of the video frame to be located.

[0112] Perform text recognition on the reference frame to obtain text line content and position mask; extract the text feature matrix of the text line content; extract the mask feature matrix of the position mask; use the text feature matrix as the K vector and V vector, and the mask feature matrix as the Q vector, and obtain fusion features based on the cross-attention mechanism; pass the fusion features through a multi-layer perceptron to obtain the cross-modal features.

[0113] In this embodiment, several special characters are input respectively, including: characters of the input frame to be located, characters representing positions, characters representing reference examples, etc. These special characters are recognized and transformed by MLP to obtain different target text vectors, and are named respectively. Among them, the target text includes: text line content, text frame, and text position mask. The text feature matrix of the text line content is extracted based on the CLIP text encoder. The mask feature matrix of the position mask is extracted by VIT, and the shape is (, 256).

[0114] The text feature matrix is ​​used as the K vector and the V vector, and the mask feature matrix is ​​used as the Q vector; the relevant parameters in the cross-attention mechanism are set, and then the attention score is calculated to obtain the attention score matrix; the elements corresponding to the mask position in the attention score matrix are set to 0; the attention weight is calculated to obtain the attention weight matrix, and the attention weight matrix is ​​applied to the V vector, and the fusion feature is obtained and output through weighted summation.

[0115] In this embodiment, a multi-layer perceptron structure is defined; the weights and bias parameters of the multi-layer perceptron are initialized; the fused features are input into the multi-layer perceptron for forward propagation, passing through the input layer, hidden layer and output layer in sequence; and an output vector is generated in the output layer, which is a cross-modal feature, and finally the cross-modal feature is named.

[0116] A frame feature matrix of the video frame to be located is extracted.

[0117] In this embodiment, a frame feature matrix of the video frame to be located is extracted based on the VIT model.

[0118] The feature matrix construction module 83 is used to construct an input feature matrix based on the cross-modal features and the frame feature matrix, where the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector.

[0119] First, obtain the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character and the position character. Secondly, pass the reference frame start character, the reference frame end character, the positioning frame start character, the positioning frame end character and the position character through a multi-layer perceptron respectively to obtain the reference frame start vector, the reference frame end vector, the positioning frame start vector, the positioning frame end vector and the position vector. Then, concatenate the reference frame start vector, the cross-modal feature, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector and the position vector in sequence to obtain the input feature matrix.

[0120] The matrix output module 84 is used to input the input feature matrix into the large language model to obtain an output feature matrix.

[0121] In this embodiment, the concatenated input feature matrix is ​​preprocessed, and then the preprocessed input feature matrix is ​​input into the large language model LLM. After being processed by multiple network layers, such as: embedding layer, attention mechanism, fully connected layer, etc., the input feature matrix is ​​transformed and calculated, and the output feature matrix is ​​obtained through the output layer or API interface; and the output feature matrix has the same shape as the input feature matrix.

[0122] Specifically, the input feature matrix is ​​encoded, and the encoded input feature matrix is ​​input into the large language model LLM for processing, and the required information is extracted from the output of the LLM, and feature extraction and data conversion are performed on it to obtain the output feature matrix.

[0123] The text positioning module 85 is used to obtain the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtain a Gaussian distribution sample based on the mean and variance; obtain the mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned according to the mask.

[0124] In this embodiment, the output vector is input into a first multi-layer perceptron to obtain the mean, the output vector is input into a second multi-layer perceptron to obtain the variance, and a Gaussian distribution sample is obtained based on the mean and variance.

[0125] In this embodiment, a sample is sampled from a Gaussian distribution by calculating the mean and variance.

[0126] Obtain a mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask; obtain a feature map of the Gaussian distribution sample based on a decoder of a VAE model; input the feature map into a sigmoid function to obtain the mask of the Gaussian distribution sample; when the mask is greater than a preset value, set the position corresponding to the mask in the video frame to be located as text; when the mask is less than the preset value, set the position corresponding to the mask in the video frame to be located as background.

[0127] In this embodiment, the encoder decoder of VAE is used, and the Gaussian distribution samples are input into the encoder of VAE for decoding processing to obtain a low-dimensional feature vector; then the sample points obtained by sampling are transmitted to the decoder decoder of VAE to generate a feature map. The mask of the Gaussian distribution sample is obtained by sigmoid, and then the relationship between the mask and the preset value is determined, that is, when the mask is greater than the preset value, the position corresponding to the mask in the video frame to be located is set to text; when the mask is less than the preset value, the position corresponding to the mask in the video frame to be located is set to background.

[0128] Therefore, after the text position of the first frame is determined, the first frame is replaced by the second frame, and then the above steps are followed step by step to obtain the mask of the second frame; and so on, this process is repeated until all frames have completed text positioning, and the masks of all input frames are collected to obtain the text positioning result of the video.

[0129] The video text position positioning model builds a video text position positioning system, which can traverse all frames and position all frames, and at the same time add timing information to it, effectively improving the timing and accuracy of video text positioning.

[0130] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0131] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0132] See also Fig. 9 , which is a schematic diagram showing the principle structure of the video text location positioning device described in this application in one embodiment. Fig. 9 As shown, this embodiment provides a video text position positioning device, which includes: a processor 91 and a memory 92; the memory 92 is used to store computer programs; the processor 91 is connected to the memory 92, and is used to execute the computer program stored in the memory 92, so that the video text position positioning device performs the various steps of the video text position positioning method as described above.

[0133] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] In summary, the video text location method, system, medium and device provided by the present application have the following beneficial effects: The video text location positioning method provided by the present application can traverse all frames and locate all frames by extracting and fusing the features of all frames of the text and obtaining the mask process of the input frame. At the same time, the timing information is added to it, which effectively improves the timing and accuracy of video text location. At the same time, the positioning method adopted by the present application can improve the robustness of the video text location technology and further improve the positioning efficiency and security.

[0136] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.< / location> < / location> < / sample> < / example> < / location> < / sample> < / example>

Claims

1. A method for locating text positions in a video, characterized in that: The following steps are involved: Obtain a reference frame containing text and a video frame to be positioned; extracting cross-modal features of the reference frame; Extracting a frame feature matrix of the video frame to be located; Constructing an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; Inputting the input feature matrix into a large language model to obtain an output feature matrix; Obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix; Obtain a Gaussian distribution sample based on the mean and variance; A mask of the Gaussian distribution sample is obtained to determine the text position of the video frame to be located according to the mask.

2. The method for locating text positions in a video according to claim 1, characterized in that: Extracting the cross-modal features of the reference frame comprises the following steps: Performing text recognition on the reference frame to obtain text line content and position mask; Extracting a text feature matrix of the text line content; extracting a mask feature matrix of the position mask; The text feature matrix is ​​used as a K vector and a V vector, the mask feature matrix is ​​used as a Q vector, and fusion features are obtained based on a cross attention mechanism; The fused features are passed through a multi-layer perceptron to obtain the cross-modal features.

3. The method for locating text positions in a video according to claim 2, characterized in that: A text feature matrix of the text line content is extracted based on the CLIP text encoder.

4. The method for locating text positions in a video according to claim 1, characterized in that: A frame feature matrix of the video frame to be located is extracted based on the VIT model.

5. The method for locating text positions in a video according to claim 1, characterized in that: Constructing an input feature matrix based on the cross-modal features and the frame feature matrix comprises the following steps: Get the reference frame start character, reference frame end character, positioning frame start character, positioning frame end character and position character; The reference frame start symbol, the reference frame end symbol, the positioning frame start symbol, the positioning frame end symbol and the position symbol are respectively passed through a multi-layer perceptron to obtain a reference frame start vector, a reference frame end vector, a positioning frame start vector, a positioning frame end vector and a position vector; The reference frame start vector, the cross-modal feature, the reference frame end vector, the positioning frame start vector, the frame feature matrix, the positioning frame end vector and the position vector are sequentially concatenated to obtain the input feature matrix.

6. The method for locating text positions in a video according to claim 1, characterized in that: Obtaining the mean and variance of the output vector corresponding to the position vector in the output feature matrix comprises the following steps: Inputting the output vector into a first multi-layer perceptron to obtain the mean; The output vector is input into a second multi-layer perceptron to obtain the variance.

7. The method for locating text positions in a video according to claim 1, characterized in that: Obtaining the mask of the Gaussian distribution sample to determine the text position of the video frame to be located according to the mask comprises the following steps: The decoder based on the VAE model obtains the feature map of the Gaussian distribution sample; Input the feature map into a sigmoid function to obtain a mask of the Gaussian distribution sample; When the mask is greater than a preset value, setting the position corresponding to the mask in the video frame to be located as text; When the mask is smaller than the preset value, the position corresponding to the mask in the video frame to be positioned is set as the background.

8. A video text location positioning system, characterized in that: include: An input module, used for acquiring a reference frame containing text and a video frame to be positioned; A feature extraction module, used to extract cross-modal features of the reference frame; Extracting a frame feature matrix of the video frame to be located; A feature matrix construction module, configured to construct an input feature matrix based on the cross-modal features and the frame feature matrix, wherein the input feature matrix includes the cross-modal features, the frame feature matrix and a position vector; A matrix output module, used for inputting the input feature matrix into a large language model to obtain an output feature matrix; A text positioning module is used to obtain the mean and variance of the output vector corresponding to the position vector in the output feature matrix; obtain a Gaussian distribution sample based on the mean and variance; obtain a mask of the Gaussian distribution sample to determine the text position of the video frame to be positioned according to the mask.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the video text location method described in any one of claims 1 to 7 is implemented.

10. A video text location positioning device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the video text position locating device executes the video text position locating method according to any one of claims 1 to 7.

Images