Video text detection method and device

Abstract

The present application discloses a method and device for detecting text in a video, which belongs to the field of AI. The method comprises: obtaining a video, wherein the video comprises n frames of pictures, where n is an integer greater than 1; identifying the text in each frame of the video, and obtaining a text recognition result of the video, wherein the process of identifying the text in the k-th frame of the video comprises: obtaining the text in the k-th frame of the video, where 1<k≤n; and matching the text in the k-th frame of the video with the text appearing in the picture before the k-th frame of the video based on the feature vector of the text in the k-th frame of the video and the feature vector of the text appearing in the picture before the k-th frame of the video. The present application can improve the efficiency of text recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a method and device for detecting text in a video. Background Art

[0002] Text detection technology (also known as text detection and recognition technology) usually requires locating and identifying text areas in digital images in various scenarios.

[0003] Currently, text detection technology for videos can only recognize each frame in the video separately. The obtained video text recognition result is only a summary of the text recognition results of each frame, resulting in a poor user experience. Summary of the Invention

[0004] The embodiment of the present invention provides a method and apparatus for detecting text in a video. The technical solution is as follows:

[0005] In a first aspect, a method for detecting text in a video is provided, the method comprising: obtaining a video, the video comprising n frames of pictures, where n is an integer greater than 1; identifying text in each frame of the video to obtain a text recognition result of the video, wherein the process of identifying text in the k-th frame of the video comprises: obtaining text in the k-th frame of the video, 1<k≤n; and matching the text in the k-th frame of the video with text appearing in pictures before the k-th frame of the video based on a feature vector of the text in the k-th frame of the video and a feature vector of text appearing in pictures before the k-th frame of the video.

[0006] When recognizing text in a video, the embodiment of the present application performs matching based on the feature vector of the text in the kth frame image and the feature vector of the text that appears continuously in the previous images. In this way, the recognition of the same text targets in different frames is achieved through the correlation relationship between the feature vectors between the images in the video. As a result, the text recognition result of the video is not only a summary of the text recognition results of each frame image, but also can reflect the correlation relationship between the text between images, providing users with more information about the video and improving the user experience.

[0007] Furthermore, by matching the feature vectors of the text in the kth frame and the feature vectors of the text that appears continuously in the previous pictures, it is also possible to associate the text corresponding to the same text target in the video, correct the text in the video, reduce the probability of misrecognition, and improve the recognition accuracy of the text in the video.

[0008] In some possible implementations, the method further includes: inputting the k-th frame image into a feature generation model to obtain a feature vector of the text in the k-th frame image. The feature generation model may be an AI model, such as a deep learning model. For example, the feature generation model is a CNN model or a DNN model.

[0009] By using a feature generation model to generate feature vectors of text, it is possible to quickly and accurately generate feature vectors and improve the efficiency of feature vector generation.

[0010] In an optional implementation, the text recognition model and the feature generation model are the same AI model. In this case, the process of obtaining the text in the k-th frame image includes: obtaining the text in the k-th frame image according to the feature generation model.

[0011] When the text recognition model and the feature generation model are the same AI model, one or more functional layers of the text recognition model and the feature generation model can be reused, thereby reducing the complexity of the model used by the video text detection device.

[0012] In another optional implementation, the text recognition model and the feature generation model are different AI models. Different AI models refer to different models in one or more of their structures, parameters, training methods, or loss functions. In this case, the process of obtaining the text in the k-th frame of the image includes: inputting the k-th frame of the image into the text recognition model, and obtaining the text in the k-th frame of the image according to the text recognition model, wherein the text recognition model and the feature generation model are two different artificial intelligence AI models.

[0013] When the text recognition model and the feature generation model are two different AI models, the models can be trained separately to reduce the mutual interference between the text recognition and feature generation processes in the training and inference stages.

[0014] In some possible implementations, the text recognition results of the video include one or more of the following results: the text recognition results of the video obtained after deduplication of text based on the matching results between pictures in the video; the text recognition results of the video obtained after text tracking based on the matching results between pictures in the video.

[0015] Among them, when the text recognition result of the video is obtained after deduplication of text based on the matching results between the pictures in the video, the text in each frame of the picture is matched with the text appearing in the picture before the kth frame of the picture based on the feature vector of the text in each frame of the picture and the feature vector of the text appearing before each frame of the picture in the video, and the successfully matched text is deduplicated, so as to achieve deduplication of the same text corresponding to the same text target in the output text recognition result of the video, reduce the probability of repeated appearance of the same text corresponding to the same text target in the recognition result, and improve the text recognition efficiency.

[0016] When the text recognition result of the video is obtained after text tracking based on the matching results between the pictures in the video, the track of the tracked text target in the video can be obtained through text tracking. The user can quickly determine the position of the text through the track, thereby improving the user experience.

[0017] In some possible implementations, the feature vector of a text includes a semantic feature vector and / or a visual feature vector. The semantic feature vector is used to reflect the semantic attributes (i.e., the meaning of the text) of the text in the image. When the text in the image is different, the extracted semantic feature vector is different. The visual feature vector is used to identify the visual features in the image, reflecting the visual effects seen by the human eye. For example, visual features include color, shape, position, and / or size.

[0018] Because visually identical or similar text in a video has a certain probability of being the same, when the text's feature vector includes both a semantic feature vector and a visual feature vector, matching the text in the kth frame with the text appearing in the image before the kth frame using these at least two dimensions further increases the accuracy of the match and optimizes the deduplication effect. Furthermore, because the visual feature vector reflects the low-level visual features of the text—i.e., how the human eye perceives the text in the image—while the semantic feature vector reflects the high-level semantic features of the text—i.e., the meaning of the text that humans can understand—these two-dimensional feature vectors are more robust and can effectively reflect the characteristics of the text.

[0019] In some possible implementations, the feature generation model is a trained AI model, and the loss function used to train the feature generation model is a ternary loss function.

[0020] The feature generation model trained with the ternary loss function generates more accurate feature vectors during the inference phase.

[0021] In some possible implementations, the method further includes: obtaining a training sample from a training sample set, the training sample being a text image, generating a triplet corresponding to the training sample; and training an initial feature generation model based on the triplet and the triplet loss function to obtain the feature generation model. Using text images as training samples to obtain triples, and training the initial feature generation model based on the triplet and the triplet loss function, can improve the robustness of text feature vectors generated by the trained feature generation model during the inference phase.

[0022] In some possible implementations, the triplet corresponding to the training sample includes: the training sample, a positive sample whose similarity to the training sample is greater than a first similarity threshold, and a negative sample whose similarity to the training sample is less than a second similarity threshold.

[0023] By training the initial feature generation model with a triplet including the training sample, a positive sample having a similarity with the training sample greater than a first similarity threshold, and a negative sample having a similarity with the training sample less than a second similarity threshold, the distance between the training sample and the positive sample can be reduced and the distance between the training sample and the negative sample can be increased during the training phase, thereby further improving the robustness of the feature vector of the text generated by the trained feature generation model during the inference phase. Furthermore, training the initial feature generation model based on the triplet and the ternary loss function can enable the feature vector obtained by the trained feature generation model during the inference phase to effectively reflect the characteristics of the images in the aforementioned video, accurately reflecting the similarity or difference between the feature vector of the text in the k-th frame image and the feature vector of the text in the image before the k-th frame image, thereby improving the effect of the trained feature generation model during the inference phase.

[0024] In some possible implementations, matching the text in the k-th image frame with the text appearing in an image before the k-th image frame includes:

[0025] Based on the edit distance between the text in the k-th frame and the picture before the k-th frame in the video, the text in the k-th frame is matched with the text appearing in the picture before the k-th frame; and / or, based on the position of the text in the k-th frame and the predicted position of the text appearing in the picture before the k-th frame in the k-th frame, the text in the k-th frame is matched with the text appearing in the picture before the k-th frame.

[0026] Based on the text's feature vector, by matching based on edit distance and / or the position of the text in the kth frame and the predicted position of text appearing in images before the kth frame, the probability of matching text corresponding to the same text target can be further improved, reducing the false positive rate and thus improving the accuracy of text recognition results. By assigning IDs to recognized text, it is easier to subsequently index the corresponding text, especially indexing text corresponding to the same text target, enabling rapid retrieval of text in the video.

[0027] Optionally, the process of obtaining the text in the k-th frame of the video further includes assigning an ID to each group of recognized text, wherein a rule for assigning the ID is pre-set. Optionally, the rule for assigning the ID is to ensure that the ID of the same text object is the same, and the IDs of different text objects are different.

[0028] In a second aspect, a device for detecting text in a video is provided, the device comprising: at least one module, which can be used to implement the method provided by the above-mentioned first aspect or various possible implementations of the first aspect.

[0029] In a third aspect, a computing device is provided, comprising a memory and a processor, wherein the memory is used to store a set of computer instructions; the processor executes the set of computer instructions stored in the memory to execute the method provided in the first aspect or various possible implementations of the first aspect.

[0030] In a fourth aspect, a non-volatile readable storage medium is provided, wherein the non-volatile readable storage medium stores computer program code. When the computer program code is executed by a computing device, the computing device executes the method provided by the above-mentioned first aspect or various possible implementations of the first aspect.

[0031] In a fifth aspect, a computer program product is provided, which includes a computer program code. When the computer program code is executed by a computing device, the computing device executes the method provided by the first aspect or various possible implementations of the first aspect.

[0032] In a sixth aspect, a chip is provided, which may include a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the text detection method for a video provided in the first aspect or various possible implementations of the first aspect.

[0033] When recognizing text in a video, the embodiment of the present application performs matching based on the feature vector of the text in the kth frame image and the feature vector of the text that appears continuously in the previous images. In this way, the recognition of the same text targets in different frames is achieved through the correlation relationship between the feature vectors between the images in the video. As a result, the text recognition result of the video is not only a summary of the text recognition results of each frame image, but also can reflect the correlation relationship between the text between images, providing users with more information about the video and improving the user experience.

[0034] Furthermore, by matching the feature vectors of the text in the kth frame and the feature vectors of the text that appears continuously in the previous pictures, it is also possible to associate the text corresponding to the same text target in the video, correct the text in the video, reduce the probability of misrecognition, and improve the recognition accuracy of the text in the video.

[0035] Furthermore, the video text detection method provided in the embodiment of the present application avoids the repetition of the same text corresponding to the same text target in the text recognition results by deduplicating the successfully matched text, thereby making the text recognition results more complete and accurate, and improving the user's viewing experience of the text recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of an application of a video text detection method provided in an embodiment of the present application in an autonomous driving scenario;

[0037] Figure 2 This is a flow chart of a method for detecting text in a video provided by an embodiment of the present application;

[0038] Figure 3 This is a schematic diagram of a scenario for recognizing text in the kth frame of a video provided by this application;

[0039] Figure 4 This is a schematic diagram of text image recognition provided by this application;

[0040] Figure 5 This is a schematic diagram of a process for identifying text in the kth frame of a video provided by this application;

[0041] Figure 6 is a structural diagram of a first text recognition model provided in an embodiment of the present application;

[0042] Figure 7 This is a schematic diagram of another process for identifying text in the kth frame of a video provided by this application;

[0043] Figure 8 is a structural diagram of a second text recognition model provided in an embodiment of the present application;

[0044] Figure 9 This is a schematic structural diagram of a first feature generation model provided in an embodiment of the present application;

[0045] Figure 10 This is a schematic structural diagram of another first feature generation model provided in an embodiment of the present application;

[0046] Figure 11 This is a schematic diagram of the structure of a second feature generation model provided in an embodiment of the present application;

[0047] Figure 12 This is a schematic diagram of the structure of a feature generation model provided in an embodiment of the present application;

[0048] Figure 13 This is a schematic diagram of the structure of another feature generation model provided in an embodiment of the present application;

[0049] Figure 14 This is a structural diagram of another feature generation model provided in an embodiment of the present application;

[0050] Figure 15 This is a structural diagram of another feature generation model provided in an embodiment of the present application;

[0051] Figure 16 This is a structural diagram of another feature generation model provided in an embodiment of the present application;

[0052] Figure 17 This is a schematic diagram of a text recognition effect provided by an embodiment of the present application;

[0053] Figure 18 This is a block diagram of a video text detection device provided by an embodiment of the present application;

[0054] Figure 19 This is a block diagram of another apparatus for detecting text in a video provided by an embodiment of the present application;

[0055] Figure 20 This is a block diagram of another apparatus for detecting text in a video provided by an embodiment of the present application;

[0056] Figure 21 A schematic diagram of the hardware structure of a computing device equipped with a video text detection apparatus is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the principles and technical solutions of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0058] Text detection technology can be applied to a variety of scenarios. When performing text detection technology in a video, by locating and identifying the text area in the digital image, it can assist users in tracking and analyzing text. For example, Figure 1 A schematic diagram of an application of a video text detection method in an autonomous driving scenario provided by an embodiment of the present application shows that a video text detection device can be installed on a vehicle to assist in autonomous driving by identifying text that appears along the vehicle's travel path. For another example, the video text detection method can be applied to text content review scenarios, such as news content review; or to text content tracking scenarios, such as tracking sensitive words in text.

[0059] The embodiment of the present application provides a method for detecting text in a video, which is performed by a video text detection device. Figure 2 This is a flow chart of a method for detecting text in a video provided by an embodiment of the present application. The method for detecting text in a video is applied to an optical character recognition (OCR) scenario. Figure 2 As shown, the text detection method of the video includes:

[0060] S201. Obtain a video, where the video includes n frames of pictures, where n is an integer greater than 1.

[0061] After obtaining the video to be identified including n frames of pictures, the video is processed to obtain a picture stream (or picture sequence) based on time sequence, and the picture stream includes n frames of pictures. The format of the picture stream obtained by processing is the format supported by the text detection device of the video. In a kind of optional example, the format of the video to be identified is different from the format supported by the text detection device of the video. For example, the format of the video to be identified is a format such as mov, mtv, dat or wmv, and the format supported by the text detection device of the video is jpg format, then the video is converted into a picture stream by format conversion. In another optional example, the format of the video to be identified is the same as the format supported by the text detection device of the video, and the text detection device of the video can directly obtain the pictures of multiple frames arranged based on time sequence in the video as a picture stream.

[0062] S202: Identify text in each frame of the video.

[0063] Among them, assuming that the k-th frame picture is a frame picture among n frames of pictures, that is, 1≤k≤n, the process of identifying the text of the k-th frame picture in the video specifically includes the following processes A1 to A2. It is worth noting that since the subsequent process of the embodiment of the present application needs to match the text in the k-th frame picture with the picture before the k-th frame picture. In one optional method, if there is no picture or no picture containing text before the k-th frame picture in the video (such as k=1), only the following process A1 is executed for the k-th frame picture. If there is a picture or a picture containing text before the k-th frame picture, processes A1 and A2 can be executed for the k-th frame picture. In another optional method, in S201, n frames of pictures are obtained from the pictures after the first frame picture in the video, the aforementioned process A1 is executed for the first frame picture in the video, and the aforementioned processes A1 and A2 are executed for each frame picture in the n frames of pictures.

[0064] The following is a detailed description of the process from A1 to A2:

[0065] A1. Get the text in the k-th frame of the video.

[0066] In the embodiment of this application, there are multiple ways to obtain the text in the kth frame image. To facilitate the reader's understanding, the following briefly introduces the text recognition principle involved in the embodiment of this application. The process of recognizing text in the video includes three sub-processes: text positioning, text image extraction, and text recognition, which are as follows:

[0067] First, the text positioning sub-process includes: determining whether there is text in a picture, if there is text in the picture, determining the text to be positioned, and determining the position and size (also known as size) of the positioned text. Based on the position and size of the positioned text, a text box is added outside the text. The text box can be the minimum circumscribed moment of the text, or any other specifications and shapes. It is worth noting that in the text positioning scenario, word segmentation rules can also be pre-set. The word segmentation rules are used to define the rules for adding text boxes when positioning text, that is, the text grouping rules. In the first optional grouping rule, for each recognized text, the distance between the text and other text is detected. When the distance between the text and other text is less than the specified distance threshold, the text and other text are divided into a group of text, that is, the same text box is added; in the second optional rule, a specified separator is added between the two texts, and the two texts are divided into different groups, that is, different text boxes are added. The specified separator can be a comma, period or semicolon. The above two optional rules can be used separately or in combination according to the actual application scenario. For example, according to the first optional grouping rule, the above Figure 1The text in the picture obtained by the vehicle is located as 6 groups of text, namely: "car", "120", "60", "large vehicle", "100" and "60". In the text location scenario, the word segmentation rule can also not be set, and the corresponding model is trained through the specified sample set, so that the text output by the trained model in the inference stage has the effect of word segmentation.

[0068] Second, the text image extraction sub-process, including: extracting the text box in the picture (such as cropping), obtaining the picture containing the area enclosed by the text box, and this picture is called the text image.

[0069] Third, the text recognition sub-process, including: determining the content of the text in the text image.

[0070] In the embodiment of the present application, the text recognition sub-process can be a process of converting the feature map (featuremap) corresponding to the text image into text content.

[0071] Figure 3 It is a schematic diagram of a scenario for recognizing the text of the k-th frame picture in a video provided by the present application. As Figure 3 shown, assume that the k-th frame picture to be recognized is picture p1. Whether there is text in picture p1 is detected through the text location process. If there is text in this picture p1, the position and size and other information of the area where the text is located in picture p1 are returned; as Figure 3 shown, Figure 3 It is assumed in that through the text location process, it is detected that there are 6 groups of text in picture p1, namely: "car", "120", "60", "large vehicle", "100" and "60". Taking "car" as an example, text images corresponding to these 6 groups of text are obtained through text image extraction. Figure 4 Taking the example of obtaining the text image of a group of text "car" to illustrate the subsequent recognition process of this group of text. Assume that the text image x1 of "car" is obtained by performing the text image extraction process; by extracting the feature map of this text image x1 and performing transcription processing on this feature map, the text "car" is obtained.

[0072] In the embodiment of the present application, all or part of the processes in the above-mentioned process of recognizing text in a video can be executed by a text recognition model, and this text recognition model can be an artificial intelligent (AI) model, such as a deep learning (DeepLearning) model. Deep learning is a machine learning technology based on deep neural network algorithms, and its main feature is to use multiple non-linear transformations to process and analyze data.

[0073] For example, the text recognition model is a convolutional neural network (CNN) model or a deep neural network (DNN) model. CNNs and DNNs are both feedforward neural networks that can be trained using a backpropagation algorithm. They are highly representative network architectures in deep learning technology. CNNs' artificial neurons can respond to surrounding cells within a certain coverage area and can process images based on image features. CNNs and DNNs include one or more convolutional layers, as well as pooling layers and / or fully connected layers.

[0074] The present application embodiment uses the following methods as examples to illustrate the process of recognizing text in a video involved in the present application embodiment:

[0075] In the first optional method, the text positioning sub-process, the text image extraction sub-process and the text recognition sub-process can all be implemented by the text recognition model. Assume that the text detection device of the video obtains the text in the k-th frame image through the first text recognition model, and the first text recognition model is used to generate the text in the received image, that is, the input end of the first text recognition model supports the input of the video image, and the input images are usually the same size. Figure 5 As shown, the recognition process includes: inputting the k-th frame image into a first text recognition model; obtaining text in the k-th frame image according to the text recognition model. The obtained text includes one or more groups of recognized text, each group of text including at least one character.

[0076] In the embodiment of the present application, the first text recognition model can be any artificial intelligence (AI) model that can perform text recognition based on an image. Figure 6It is a structural diagram of a first text recognition model provided in an embodiment of the present application. For example, the first text recognition model is a CNN model. The first text recognition model includes: feature pyramid networks (FPN) 301, a target detection special layer (RoiAlign) 302, a feature map extraction layer 303 and a decoder (decoder) 304. The FPN 301 is used to extract multiple first feature maps (also called proposal windows, proposals) of the input image. The FPN 301 is used to fuse the features of different levels of the input image. Each first feature map a1 is an area that may contain a candidate object. The area of ​​the candidate object is also called a region of interest (Roi) or a candidate area, that is, the area where the text box described in the aforementioned text image extraction sub-process is located. In an embodiment of the present application, the candidate object is a group of text, and the group of text includes one or more texts; the RoiAlign 302 is used to obtain a second feature map a2 based on the multiple first feature maps a1. For example, the RoiAlign302 obtains the corresponding second feature map a2 from multiple first feature maps a1 according to the position of the Roi in the input image (i.e., the kth frame image) for regression. The number of second feature maps a2 obtained by RoiAlign302 is the same as the number of Roi, that is, each second feature map a2 obtained by RoiAlign302 corresponds to one Roi. The second feature map extracted by RoiAlign302 is usually of a fixed size. For example, RoiAlign302 avoids quantizing the edges of the Roi and uses a bilinear interpolation method to obtain image values ​​at pixel points with floating-point coordinates, thereby converting the entire feature aggregation process into a continuous operation. For example, the position of the Roi in the input image can be represented by the coordinates [x, y, w, h] of the Roi, where x and y are the center coordinates of the Roi, w is the width of the Roi, and h is the height of the Roi. The feature map extraction layer 303 includes one or more convolutional layers. The feature map extraction layer 303 is used to obtain a semantic feature map based on the second feature map a2. For example, the semantic feature map is a two-dimensional feature map, such as a 7×7 feature map. The decoder 304 is used to transcribe the text, that is, output the text in the picture based on the extracted semantic feature map, that is, the text contained in each Roi. Optionally, the decoder 304 transcribes the text based on the attention mechanism. The use of the attention mechanism can enable the decoder 304 to output text recognition results that are closer to the human eye recognition results, thereby improving the user experience of users viewing the final text recognition results.

[0077] In the second optional way, the text recognition sub-process can be implemented by the text recognition model. The text positioning sub-process and the text image extraction sub-process are implemented by other models or modules. Assume that the text detection device of the video obtains the text in the k-th frame picture through the second text recognition model, and the second text recognition model is capable of text recognition. The second text recognition model is used to generate the text in the received text image (i.e., the picture of the text frame), that is, the input end of the second text recognition model supports the input of the video text image, and the size of the input text image can be different. Figure 7 As shown, the identification process includes:

[0078] A11. Obtain text images in the k-th frame image, where each text image includes an area where one or more texts in the k-th frame image are located.

[0079] A11 can refer to the aforementioned text positioning sub-process and text image extraction sub-process. For example, in the k-th frame image, information such as the position and size of the positioned text is determined, and based on the determined information, a text frame is added outside the text in the k-th frame image, and the area where the text frame is located is cropped to obtain a text image.

[0080] A12. Input the acquired text image into the second text recognition model.

[0081] A13. Obtain the recognition result of the text in the text box output by the second text recognition model.

[0082] In an embodiment of the present application, the second text recognition model can be any model in the AI ​​model that can perform text recognition based on text images. Figure 8 It is a structural diagram of a second text recognition model provided in an embodiment of the present application. For example, the second text recognition model is a DNN model. The second text recognition model includes: a backbone network 401, a maximum pooling layer (MaxPooling layer) 402, a fully connected layer 403 and a decoder 404. The backbone network 401 is used to extract the first feature map b1 of the input image (i.e., the cropped text image). The first feature map b1 is used to reflect the characteristics of the input image. For example, the backbone network includes one or more convolutional layers, and the backbone network can be ResNet; the maximum pooling layer 402 is used to perform dimensionality reduction processing on the feature map to obtain a second feature map b2; the fully connected layer 403 is used to perform dimensionality reduction processing on the second feature map b2 to obtain a semantic feature map b3 of the input image; the decoder 404 is used to obtain the text in the input image based on the semantic feature map b3. For example, the semantic feature map b3 is a two-dimensional feature map.

[0083] It is worth noting that in the embodiments of the present application, the process of recognizing text in a video can also be performed using other models or modules, as long as the text in each frame can be effectively recognized. The structure of the text recognition model can also be implemented in other ways, for example, FPN can be replaced with a Region Proposal Network (RPN), RoiAlign can be replaced with Roi Pooling (Roi Pooling layer); the feature map extraction layer can be replaced with other convolutional layers, etc. The connection relationship between the various functional layers in the text recognition model can also be adjusted, etc.

[0084] A2. Match the text in the kth frame image with the text that appears in the image before the kth frame image based on the feature vector of the text in the kth frame image and the feature vector of the text that appears in the image before the kth frame image in the video.

[0085] To facilitate the reader's understanding, the principles of the recognition method provided in the embodiments of this application are explained below. When a video is recorded or played, the text in its continuous image stream has a certain trajectory, which makes the display of the text have a certain correlation. For example, the text content displayed corresponding to the same text target in multiple consecutive frames of images is the same. The embodiments of this application utilize the correlation between the text in different consecutive frames of images to match the text.

[0086] In an embodiment of the present application, a text detection device for a video matches text by comparing the feature vector of the text in the k-th frame image with the feature vector of the text appearing in the image before the k-th frame image in the video, and determines the text belonging to the same text target. Further, text deduplication can be achieved. Optionally, the image before the k-th frame image in the aforementioned video can be a picture in the video that is adjacent to the k-th frame image and is located before the k-th frame image. For example, the picture before the k-th frame image includes m frames of pictures, where m is a positive integer. m can be a fixed value, for example, m=1, then the text matching is performed between the text in the k-1-th frame image and the text in the k-th frame image. For another example, m is a variable value. If there are multiple frames of pictures before the k-th frame image, then the multiple frames of pictures are obtained as m frames of pictures; or, if there are multiple frames of pictures with text before the k-th frame image, then the multiple frames of pictures with text are obtained as m frames of pictures. Optionally, the m frames include the first m (e.g., m=k-1) consecutive frames of the kth frame. In this case, the text in the kth frame needs to be matched with the text in the kmth to k-1th consecutive frames, respectively. In this way, text matching is performed based on the association between the feature vectors of the text in the kth frame and the text in the kmth to k-1th consecutive frames.

[0087] Optionally, the text that appears in the image before the k-th frame can be text that appears in every one of the m frames before the k-th frame, that is, text that appears continuously in the m consecutive frames. Continuously appearing text creates a continuous trajectory, which increases the probability of association with the k-th frame, the probability of the same text target appearing, and the probability of repetition. This allows for more accurate selection of text to be matched in the images before the k-th frame, and improves deduplication efficiency.

[0088] Before process A2, the text detection device of the video needs to obtain the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video. All or part of the process of obtaining the feature vector of the text in the video can be performed by the feature generation model. By generating the feature vector of the text through the feature generation model, it is possible to quickly and accurately generate the feature vector and improve the generation efficiency of the feature vector. The feature generation model can be an AI model, such as a deep learning model. For example, the feature generation model is a CNN model or a DNN model. The embodiment of the present application illustrates the process of obtaining the feature vector in the following two ways:

[0089] In a first optional approach, it is assumed that the video text detection device obtains a feature vector of text in the kth frame of the image through a first feature generation model. The first feature generation model is used to generate a feature vector of text in the received image. That is, the input end of the first feature generation model supports the input of video images, and the input images are usually of the same size. The process of obtaining the feature vector of text in the video includes:

[0090] A21. Input the k-th frame image into a first feature generation model to obtain a feature vector of the text in the k-th frame image.

[0091] After inputting the k-th frame image into the first feature generation model, the video text detection device extracts the feature vector of the text in the k-th frame image generated by the first feature generation model.

[0092] A22. Obtain the feature vector of the text that appears in the image before the k frames.

[0093] Each time a video text detection device inputs an image into the first feature generation model, it obtains a feature vector for the text in the corresponding image. In this embodiment of the present application, the obtained feature vector can be stored in a designated location. After obtaining the feature vectors for the text in the aforementioned k frames of images, the feature vectors for the text appearing in the image before the k frames of images can be extracted from the designated storage location.

[0094] Figure 9It is a structural diagram of a first feature generation model provided by an embodiment of the present application. For example, the first feature generation model is a CNN model. The feature vector of the aforementioned text includes a semantic feature vector. The first feature generation model includes: FPN501, RoiAlign502, a first feature map extraction layer 503 and a semantic feature extraction layer 504. The FPN501 is used to extract multiple first feature maps c1 of the input image. The FPN501 is used to fuse the features of different levels of the input image; the RoiAlign502 is used to obtain the second feature map c2 based on the multiple first feature maps c1. For example, the RoiAlign502 extracts the corresponding second feature map c2 from the multiple first feature maps c1 according to the position of the text box in the input image for regression. The first feature map extraction layer 503 includes one or more convolutional layers. The first feature map extraction layer 503 is used to obtain the semantic feature map c3 based on the second feature map c2. The semantic extraction layer 504 is used to obtain a semantic feature vector based on the semantic feature map c3. The semantic feature vector is used to reflect the semantic attribute characteristics of the text itself (i.e., the meaning of the text). When the text in the picture is different, the extracted semantic feature vector is different. Among them, the functions of FPN501, RoiAlign502 and the first feature map extraction layer 503 can respectively correspond to the functions of FPN301, RoiAlign302 and feature map extraction layer 303 described above. For detailed explanation, please refer to the corresponding functional layers described above.

[0095] Optionally, the aforementioned text feature vector also includes a visual feature vector, such as Figure 10 As shown, the first feature generation model also includes: a second feature map extraction layer 505 including one or more convolutional layers, and the second feature map extraction layer 505 is used to obtain a visual feature map c4 based on the second feature map c2. The visual feature extraction layer 506 is used to obtain a visual feature vector based on the visual feature map c4, and the visual feature vector is used to identify the visual features of the visual feature map c4 in the corresponding picture, reflecting the visual effect seen by the human eye. For example, visual features include color, shape, position and / or size. For example, the picture input into the first feature generation model is the k-th frame picture, and the visual feature vector is used to identify the visual features of the visual feature map c4 (i.e., corresponding Roi) in the k-th frame picture.

[0096] Because visually identical or similar text in a video has a certain probability of being the same, and in the embodiment of the present application, the feature vector of the text includes both a semantic feature vector and a visual feature vector, when matching the text in the k-th frame with the text appearing in the image before the k-th frame using these at least two dimensions, the accuracy of text matching is further increased, and the deduplication effect can also be optimized. Furthermore, because the visual feature vector reflects the low-level visual features of the text, i.e., the effect of the text on the image that the human eye can see; and the semantic feature vector reflects the high-level semantic features of the text, i.e., the meaning of the text that people can understand, the feature vectors of these two dimensions are more robust and can effectively reflect the characteristics of the text.

[0097] In a second alternative, assume that the video text detection device obtains text in the kth frame of the image using a second feature generation model. The second feature generation model is used to generate a feature vector for the text in the received text image. That is, the input end of the second feature generation model supports the input of text images, and the input text images can have different sizes. The process of obtaining the feature vector of the text in the video includes:

[0098] A23. Obtain text images in the k-th frame image, where each text image includes an area where one or more texts in the k-th frame image are located.

[0099] The process of acquiring the text image can refer to the aforementioned text positioning sub-process and text image extraction sub-process, and can also refer to process A11.

[0100] A24. Input the acquired text image into the second feature generation model to obtain a feature vector of the text in the k-th frame image.

[0101] After inputting each text image in the k-th frame picture into the second feature generation model, the text detection device of the video extracts the feature vector of the text in the text image (i.e., the image within the text box) generated by the second feature generation model, and finally uses the feature vectors corresponding to all text images in the k-th frame picture as the feature vector of the text in the k-th frame picture.

[0102] A25. Obtain the feature vector of the text in the text image corresponding to the text appearing in the picture before the k-th frame.

[0103] Each time the video text detection device inputs an image into the second feature generation model, it obtains a feature vector for the text in the corresponding image. In this embodiment of the present application, the obtained feature vector can be stored in a designated location. After obtaining the feature vectors for the text in the aforementioned k frames of images, the feature vectors for the text appearing in the image before the k frames of images can be extracted from the designated storage location.

[0104] Figure 11: It is a structural diagram of a second feature generation model provided in an embodiment of the present application. For example, the second feature generation model is a DNN model. The second feature generation model includes: a backbone network 601, a maximum pooling layer 602, a fully connected layer 603 and a feature extraction layer 604. The backbone network 601 is used to extract the first feature map d1 of the input image (i.e., text image). For example, the backbone network includes one or more convolutional layers, and the backbone network can be a ResNet network; the maximum pooling layer 602 is used to perform dimensionality reduction processing on the first feature map d1 to obtain a second feature map d2; the fully connected layer 603 is used to perform dimensionality reduction processing on the second feature map d2 to obtain a semantic feature map d3, and the feature extraction layer 604 is used to extract the features of the semantic feature map d3 to obtain the feature vector of the text in the input image. The feature extraction layer 604 may include one or more convolutional layers. The functions of the backbone network 601 , the maximum pooling layer 602 , and the fully connected layer 603 may correspond to the aforementioned backbone network 401 , the maximum pooling layer 402 , and the fully connected layer 403 , respectively.

[0105] It is worth noting that in the embodiments of the present application, the feature vector generation process can also be performed using other models or modules, as long as they can effectively generate the feature vector of the text. The structure of the feature generation model can also be implemented in other ways, for example, FPN can be replaced with RPN, RoiAlign can be replaced with Roi Pooling, and the feature map extraction layer can be replaced with other convolutional layers. The connection relationship between the various functional layers in the feature generation model can also be adjusted.

[0106] In an actual implementation of the embodiments of the present application, in one optional implementation, the text recognition model and the feature generation model are two different AI models. Different AI models refer to models that differ in one or more aspects, such as their structure, parameters, training methods, or loss functions. When the text recognition model and the feature generation model are two different AI models, they can be trained separately to reduce interference between the text recognition and feature generation processes during the training and inference phases.

[0107] In an optional implementation, the text recognition model and the feature generation model are the same AI model. Referring to the aforementioned process A1, since all or part of the process of recognizing text in a video is performed by a text recognition model, the text recognition model will generate a feature map, and in process A2, the feature vector of the text to be obtained may include a semantic feature vector, which can be obtained based on the semantic feature map. Therefore, one or more functional layers of the text recognition model and the feature generation model can be reused, thereby reducing the complexity of the model used by the text detection device of the video. For example, the text feature model is also used to perform text recognition on the input image, that is, the function of the aforementioned text recognition model can be implemented by the feature generation model. Then the process of obtaining the text in the kth frame of the picture in the aforementioned process A1 can be replaced by the two optional methods in A1: according to the feature generation model, the text in the kth frame of the picture is obtained. The embodiment of the present application uses the following two optional examples to illustrate the scenario where the text recognition model and the feature generation model are the same AI model:

[0108] In an alternative example, Figure 6 The FPN301, RoiAlign302 and feature map extraction layer 303 of the first text recognition model Figure 9 or Figure 10 The FPN501, RoiAlign502 and the first feature map extraction layer 503 in the first feature generation model can be reused respectively. Figure 6 The first character recognition model in Figure 10 As an example, the functional layer reuse of the first feature generation model shown in FIG is used. Assume that the reused model is Figure 12 The feature generation model shown in Figure 10 Based on the first feature generation model shown, it also includes: a decoder 507, which is used to output the text in the input image based on the feature map (such as the semantic feature map output by the first feature map extraction layer 503), and the feature map can be a two-dimensional feature map. Then the actions performed by the first text recognition model in the aforementioned process A1 can be performed by the feature generation model. Using the aforementioned Figure 12 The feature generation model shown in Figure 12 The feature generation model shown can complete the text location sub-process, text image extraction sub-process, and text recognition sub-process in one go. Optionally, the feature generation model is also used to remove duplicate text. Therefore, during the inference phase, n frames of images (i.e., the aforementioned image stream) are directly input into the feature generation model, and the first text recognition model can output the text recognition results for these n frames, and in subsequent processes, the video. This enables end-to-end video recognition.

[0109] In the embodiment of the present application, the feature generation model also includes other structures according to its specific application scenarios. Figure 13 As shown, the feature generation model also includes a classifier 508, a mask graphic extraction layer 509 and / or a prediction layer 510. The classifier 508 is used to determine the classification result and the position of the Roi based on the visual feature map c4. The classification result is used to indicate whether there is a text box in the input image. The position of the Roi can be represented by the coordinates [x, y, w, h] of the Roi; the mask graphic extraction layer 509 is used to obtain the mask graphic of the Roi in the input image, for example, based on the semantic feature map c3, obtain the mask graphic of the Roi in the input image; the prediction layer 510 is also used to obtain the segmentation mask of the Roi in the input image.

[0110] For each input frame, the mask pattern output by the first feature map extraction layer 503 corresponds to the Roi. Each mask pattern is a binary pattern obtained by binarizing the corresponding Roi. For example, the mask pattern is a binary pattern obtained by setting the grayscale value of the background pixel in the corresponding Roi to 0 (corresponding to Figure 13 The black area in the mask pattern of the text pixel is set to 1 (corresponding to Figure 13 Each mask graphic can intuitively represent the content of the corresponding Roi, so that users can understand the actual text content in the text box through the mask graphic. Among them, the segmentation mask is used to indicate the boundary position of the text in the mask graphic in the input image of the feature generation model. Since the shape of the boundary of the text is irregular, such as curved, the segmentation mask can better represent the shape of the text. The segmentation mask can be regarded as a high-level feature of the corresponding text. The segmentation mask is the area where the text pixels of the aforementioned mask graphic are located, for example, it is the area with a grayscale value of 1 in the aforementioned mask graphic.

[0111] It is worth noting that the structures of FPN501, RoiAlign502, first feature map extraction layer 503, second feature map extraction layer 505, classifier 508, mask graphic extraction layer 509 and / or prediction layer 510 can refer to the corresponding structure of mask region CNN (MaskRegion-CNN, Mask R-CNN).

[0112] Optionally, a long short-term memory network (LSTM) is further provided between the aforementioned first feature map extraction layer 503 and the decoder 507, and the LSTM is used to perform time series modeling on the semantic feature map.

[0113] Since the second feature map extracted by RoiAlign302 has performed a first regression of the position features, and the classifier 508 has performed a second regression of the position features, each group of text can be accurately located through two-stage regression (i.e., regressing the position of the text box at different layers), thereby achieving more accurate position feature acquisition and improving the accuracy of subsequent text matching.

[0114] Alternatively, as Figure 14 As shown, RoiAlign502 is also used to obtain the position of the Roi determined by the classifier in the inference stage, and based on the obtained position of the Roi, determine the second feature map a2 based on multiple first feature maps a1, that is, update the second feature map a2. Since the classifier 508 regresses the same position twice relative to RoiAlign502, the position of the Roi obtained by it is more accurate than the position of the Roi indicated by the second feature map obtained by RoiAlign502. Therefore, through the aforementioned feedback mechanism, the position of the Roi obtained by RoiAlign guides RoiAlign to extract the feature map again to obtain a new second feature map. The new second feature map will be more accurate, thereby ensuring the accurate acquisition of the text position.

[0115] In another alternative example, Figure 8 The backbone network 401, the maximum pooling layer 402 and the fully connected layer 403 of the second text recognition model in Figure 11 The backbone network 601, the maximum pooling layer 602, and the fully connected layer 603 in the second feature generation model can be reused respectively. Assume that the reused model is Figure 15 The feature generation model shown in Figure 11 The second feature generation model shown in FIG. 1 further includes a decoder 605 for acquiring text in an input image based on the semantic feature map d3. All actions performed by the second text recognition model in the aforementioned process A1 can be performed by the feature generation model.

[0116] It should be noted that the combination of the text recognition model and the feature generation model used in the text detection device of the aforementioned video is only a schematic illustration. When the embodiment of the present application is actually implemented, the text recognition model and the feature generation model used in the text detection device of the video can also have other combinations, as long as the feature vector of the text can be generated while the text is recognized.

[0117] After the text detection device of the video obtains the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video, a text matching process is performed based on the obtained feature vector. In an embodiment of the present application, the text matching process includes: matching the text in the k-th frame picture with the text appearing in the picture before the k-th frame picture based on the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video. As described in the above-mentioned process A1, after text positioning, if a frame picture includes text, then according to the word segmentation rules, the frame picture includes one or more groups of text, then based on the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video, the text in the k-th frame picture is matched with the text appearing in the picture before the k-th frame picture, including: for each group of text in the k-th frame picture, matching the feature vector of the group of text with the feature vector of each group of text appearing in the picture before the k-th frame picture in the video. For example, Q groups of text appear in the k-th frame image, and P groups of text appear in the image before the k-th frame image. Then, each group of text in the Q groups of text is matched with each group of text in the P groups of text.

[0118] Optionally, for each group of text in the k-th frame, a condition for a successful match between the group of text and another group of text appearing in the image before the k-th frame includes that the feature vector of the group of text and the feature vector of the other group of text satisfy a first matching condition. By way of example, the first matching condition includes at least one of: a similarity greater than a specified similarity threshold, or a distance less than a first distance threshold. The similarity of the feature vectors is negatively correlated with the distance between the feature vectors, that is, the smaller the distance, the greater the similarity. By way of example, the distance may be a Euclidean distance or a cosine distance.

[0119] It is worth noting that, when the feature vector of each group of characters includes a semantic feature vector, the condition for the successful matching of the group of characters with another group of characters includes that the semantic feature vector of the group of characters and the semantic feature vector of the other group of characters meet the first matching condition. When the feature vector of each group of characters includes a visual feature vector, the condition for the successful matching of the group of characters with another group of characters includes that the visual feature vector of the group of characters and the visual feature vector of the other group of characters meet the first matching condition. When the feature vector of each group of characters includes a semantic feature vector and a visual feature vector, in a first optional manner, the condition for the successful matching of the group of characters with another group of characters includes that the semantic feature vector of the group of characters and the semantic feature vector of the other group of characters meet the first matching condition, and the visual feature vector of the group of characters and the visual feature vector of the other group of characters meet the first matching condition; in a second optional manner, the feature vector of the characters is obtained by splicing the visual feature vector and the semantic feature vector. For example, for a group of characters, its visual feature vector is a 128-dimensional vector and its semantic feature vector is a 128-dimensional vector, then a 256-dimensional feature vector is obtained by splicing. Among them, in the spliced ​​feature vectors, the visual feature vector can be before or after the semantic feature vector, as long as the splicing rules of the feature vectors of each group of text in each frame of the same video are consistent. Accordingly, the condition for the successful matching of one group of text with another group of text includes that the feature vector of the one group of text and the feature vector of the other group of text meet the first matching condition. In this way, the feature vector of the text contains more elements and can better reflect the relationship between different texts in different frames. In addition, the matching process is simpler and has lower computational overhead than the first optional method mentioned above.

[0120] Optionally, the process of matching the text in the k-th frame image with the text appearing in the image before the k-th frame image further includes at least one of the following two optional methods:

[0121] In the first option, the text in the kth frame is matched against the text in the image before the kth frame based on the minimum edit distance (MED) between the text in the kth frame and the text in the video. The edit distance, also known as the Levenshtein distance, refers to the minimum number of edit operations required to transform two strings. Allowed edit operations include replacing one character with another, inserting a character, and deleting a character.

[0122] For example, the Levenshtein distance for converting "kitten" to "sitting" is 3, and the operations involved include:

[0123] 1. kitten→sitten(k→s); 2. sitten→sittin(e→i); 3. sittin→sitting(insert g).

[0124] For each group of text in the k-th frame, a condition for successful matching between the group of text and another group of text appearing in a picture before the k-th frame includes that an edit distance between the group of text and the other group of text is less than a second distance threshold.

[0125] In a second optional method, the text in the kth frame is matched with the text appearing in the picture before the kth frame based on the position of the text in the kth frame and the predicted position of the text appearing in the picture before the kth frame in the kth frame. This second optional method is based on the principle of Kalman filtering, and the position of the text appearing in the picture before the kth frame is predicted in the kth frame. For example, on a highway, as a vehicle travels, the video image stream acquired by the vehicle's video text detection device continuously changes. The text appearing in the k-1 frame may not appear in part in the kth frame. The Kalman filter can filter out the part of the text that no longer appears. Optionally, when text exists in both the kth frame and the k-1th frame, the text in the kth frame is matched with the text appearing in the picture before the kth frame based on the position of the text in the kth frame and the predicted position of the text appearing in the picture before the kth frame in the kth frame, thereby improving the matching accuracy.

[0126] For each group of text in the k-th frame image, a condition for successful matching of the group of text with another group of text appearing in the image before the k-th frame image includes that the predicted positions of the group of text and the other group of text in the k-th frame image meet a second matching condition, and the second matching condition includes that the similarity is greater than a similarity threshold, or the overlap degree is greater than a specified overlap degree, or the distance is less than a third distance threshold, and the distance can be an intersection over union (IOU) distance.

[0127] The process of obtaining the predicted position of the text that appeared in the picture before the k-th frame in the k-th frame includes: simply approximating the motion pattern of the text in the video to uniform linear motion, that is, the motion speed of each frame is the same, and the text frame where each group of text is located in the image coordinate system, the corresponding state variables include Where x and v are the center coordinates of the text box, h is the height of the text box, ε is the aspect ratio of the text box, and θ is the tilt angle of the text box (the tilt angle is relative to a specified coordinate axis in the image coordinate system, such as the horizontal axis). are the velocities of x, y, h, ε, and θ respectively.

[0128] Since the target text appears continuously in the first m frames, and the k-1 frame is the latest frame relative to the k-th frame, for each group of text, state estimation can be performed based only on the k-1 frame. Using the state estimation model, the estimated state variables of the text box containing the group of text in the k-th frame are determined. The state variables of the k-1 frame and the k-th frame satisfy the following relationship:

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] Among them, x k 、y k 、h k , ε k ,θ k are the x, y, h, ε, θ, and x of a text box in the k-th frame image. k-1 、y k-1 、h k-1 , ε k-1 ,θ k-1 are x, y, h, ε, and θ of a text box in the k-1th frame image respectively; are the x, y, h, ε, and θ velocities of a text box in the k-1th frame image, The speeds of the corresponding parameters of two adjacent frames are obtained through statistical principles, and Δt is the time interval between the k-1th frame and the kth frame in the video.

[0135] The state estimation model satisfies:

[0136]

[0137] Among them, X k Indicates the state variable corresponding to a group of text (i.e., a text box) in the k-th frame image; X k-1 It represents the state variable corresponding to a group of text in the k-1th frame image. The elements in the coefficient matrix correspond one-to-one to the elements in the aforementioned state variable. e5 is a 5-dimensional identity matrix. can be considered as a fixed coefficient.

[0138] Then, as mentioned above, for each group of text in the k-th frame image, the conditions for a successful match between the group of text and another group of text appearing in the image before the k-th frame image include: Condition 1, the feature vector of the group of text and the feature vector of the other group of text satisfy a first matching condition; and also include: Condition 2, the edit distance between the group of text and the other group of text is less than a second distance threshold, and / or, Condition 3, the predicted positions of the group of text and the other group of text in the k-th frame image satisfy a second matching condition. In an optional implementation, when the aforementioned conditions 1 to 3 are simultaneously met, it can be determined that the group of text successfully matches the other group of text appearing in the image before the k-th frame image; when any of the aforementioned conditions 1 to 3 are not met, it can be determined that the group of text fails to match the other group of text appearing in the image before the k-th frame image.

[0139] In another optional implementation, by comparing whether the distance cost between the group of characters and the other group of characters is greater than a fourth distance threshold, when the distance cost is less than the fourth distance threshold, it is determined that the group of characters successfully matches the other group of characters appearing in the picture before the k-th frame; when the distance cost is not less than the fourth distance threshold, it is determined that the group of characters fails to match the other group of characters appearing in the picture before the k-th frame. The distance cost is a weighted value of the first cost, the second cost, and the third cost, or the distance cost is a weighted value of the first cost and the second cost, or the distance cost is a weighted value of the first cost and the third cost. The first cost is the distance between the feature vector of the group of characters and the feature vector of the other group of characters; the second cost is the edit distance between the group of characters and the other group of characters; and the third cost is the distance between the predicted positions of the group of characters and the other group of characters in the k-th frame.

[0140] For example, assume that the distance cost is the weighted value of the first generation value and the third cost value, and the first cost value is the distance dist between the feature vector and the feature vector of the other group of characters. 向量距离 The third cost value is the IOU distance dist between the predicted position of the group of text and the other group of text in the k-th frame picture. iou , assuming that the weight corresponding to the first generation value is 0.9 and the weight corresponding to the third generation value is 0.1, then the weighted value of the first generation value and the third generation value = 0.1*dist iou +0.9*dist 向量距离 .

[0141] When the condition for the successful matching of the aforementioned set of text with another set of text appearing in the picture before the k-th frame is based on condition 1, the text is matched in combination with at least one of conditions 2 and 3, which can further improve the probability of successful matching of the text corresponding to the same text target.

[0142] After matching the text in the k-th frame image with the text appearing in the image before the k-th frame image, if the text in the k-th frame image successfully matches the text appearing in the image before the k-th frame image, it means that the successfully matched text is the text corresponding to the same text target. Subsequently, in S203, the text recognition result can be obtained based on the successfully matched text, for example, the deduplication result of the successfully matched text is used as the content of the text recognition result; if the text in the k-th frame image fails to match the text appearing in the image before the k-th frame image, it means that the unmatched text is not the text corresponding to the same text target. The aforementioned S202 process can be performed again for the next frame image of the k-th frame image, or the action can be stopped according to the specific situation.

[0143] S203: Obtain text recognition results of the video based on the text matching results.

[0144] After executing the text recognition process for n frames of images in the video, multiple text matches have been completed, ensuring that the text in the k-th frame image and the image before the k-th frame image belongs to the same text target. The video text detection method provided in the embodiment of the present application can be applied to different video text processing scenarios. Based on the text matching results, different processing processes can be performed to obtain the text recognition results of the video. The embodiment of the present application is illustrated by taking the following text processing scenarios as examples:

[0145] The first text processing scenario is the text deduplication scenario. The text detection device in the video can remove duplicate text based on the matching results between pictures in the video to obtain the text recognition result of the video.

[0146] When a group of text in the k-th frame image successfully matches another group of text that appears in the image before the k-th frame image in the video, it means that the group of text is similar or identical to the other group of text, and the two correspond to the same text target with a certain probability. The two can be deduplicated. After deduplication, only the group of text corresponding to the same text target is retained in the final video text recognition result.

[0147] For example, if the text that appears in the picture before the k-th frame is the text that appears in every frame before the k-th frame, when a group of text in the k-th frame successfully matches another group of text that appears continuously in the picture before the k-th frame in the video, it means that the group of text appears in multiple consecutive frames of the video. Through deduplication processing, the multiple groups of repeated identical texts are processed to obtain one group of identical text, which is output from the final text recognition result of the video, thereby ensuring the accuracy and consistency of the recognition of the text content in the video.

[0148] Related technologies for video text detection can only recognize each frame separately, resulting in repeated appearance of the same text in the video recognition results, affecting text recognition efficiency. For example, if text exists in 10 frames of a video, the text recognition results for the video will include the text in all 10 frames.

[0149] In the embodiment of the present application, based on the feature vector of the text in each frame and the feature vector of the text that appears before each frame in the video, the text in each frame is matched with the text that appears in the picture before the k-th frame, and the successfully matched text is deduplicated, so as to achieve deduplication of the same text corresponding to the same text target in the output video text recognition result, reduce the probability of repeated appearance of the same text corresponding to the same text target in the recognition result, and improve the recognition efficiency of the text. For example, if there are texts in 10 frames of pictures in a video, and the texts in 8 frames are the same, then the text recognition result of the video includes the texts in the 3 frames, of which the texts in 1 frame are the deduplication results of the texts in the aforementioned 8 frames, and the remaining 2 frames are 2 frames with different texts.

[0150] It should be noted that the process of obtaining the text in the k-th frame image in the video in the aforementioned process A1 also includes: assigning an identity (ID) to each group of recognized text, and the ID assignment rule is pre-set. Optionally, the assignment rule is: ensure that the ID of the same text target is the same, and the ID of different text targets is different. In an optional manner, in the aforementioned text recognition sub-process, each semantic feature vector stored in the database corresponds to an ID. For example, after the semantic feature vector of a group of text X successfully matches the semantic feature vector stored in the database, that is, after the similarity between the two is greater than the similarity threshold, the ID of the successfully matched semantic feature vector in the database is assigned to the group of text X. In another optional manner, after a group of text X is recognized in the k-th frame image, an ID is generated based on the position and / or semantics of the group of text X. For example, the position can be the coordinates of the text box of the group of text X. By assigning IDs to the recognized text, it is convenient to subsequently index the corresponding text, especially index the text corresponding to the same text target, and realize rapid retrieval of text in the video.

[0151] Correspondingly, in the text deduplication scenario, the text recognition result can also include the ID of the deduplicated text, which makes it easier to record the text using the ID.

[0152] Text deduplication can be used in a variety of practical scenarios, such as assisted autonomous driving and video content extraction. For assisted autonomous driving, after deduplication, driving suggestions can be generated and output in the text recognition results. For video content extraction, after deduplication, other video-related content can be obtained and output in the text recognition results.

[0153] The second text processing scenario is the text content tracking scenario. The text recognition results of the video are obtained by tracking the text based on the matching results between the images in the video.

[0154] When tracking text content, the images containing text corresponding to the same text target (i.e., successfully matched text) can be arranged in chronological order to obtain a picture sequence. The text recognition result includes the successfully matched text and the picture frame corresponding to the picture sequence, or the successfully matched text and the identification of the picture in the picture sequence. The identification of the picture can be represented by the position of the picture in the video, such as the ID or coordinates of the picture in the video. For example, the ID of each frame of the picture in the video can be represented by the picture sequence number. A group of identical texts may correspond to a picture sequence of [1, 2, 4, 7], indicating that the text corresponding to the same text target appears in the 1st, 2nd, 4th, and 7th frames of the video. Optionally, the text recognition result can also include the ID of the successfully matched text, which makes it easier to record the text using the ID. Optionally, the picture recognition result also includes the position of the successfully matched text in each frame of the picture in the picture sequence, such as the coordinates of the text in the picture. Optionally, when the text recognition result includes the picture frame corresponding to the picture sequence, the successfully matched text in the picture in the picture sequence can be highlighted when displayed, for example, by displaying a text box.

[0155] The image sequence corresponding to the same text (i.e., successfully matched text) can describe the trajectory of the same text target appearing in the video. Users can quickly determine the position of the text through this trajectory and achieve effective text tracking.

[0156] When the text recognition result includes picture frames corresponding to the picture sequence, the picture frames corresponding to the text sequence can be displayed in time sequence in the text recognition result, so that the user can view the trajectory of the same text target and find the appearance pattern of the text target more conveniently.

[0157] In the aforementioned text tracking process, in an optional manner, the text detection device of the video may also first conduct a text content review on the successfully matched text, and use the illegal text obtained from the review as a tracking target. For example, the successfully matched text obtained in the text recognition result is compared with the words in the sensitive word library, the text that is the same as the words in the sensitive word library is determined to be illegal text, and the picture frame of the text in the video is marked as an illegal picture frame. The illegal picture frame in the video can be deleted or otherwise processed later. Accordingly, the text recognition result includes the illegal text and the identifier of the picture sequence corresponding to the illegal text; or, the text recognition result includes the illegal text and the picture frame corresponding to the picture sequence corresponding to the illegal text. When the text recognition result includes the picture frame corresponding to the picture sequence corresponding to the illegal text, it can be convenient for users to view, thereby finding illegal pictures more conveniently.

[0158] In another optional manner, the text detection device of the video can synchronously review the text content during the tracking process. For example, the recognized text is compared with the words in the sensitive word library, the text that is the same as the words in the sensitive word library is determined to be illegal text, and the picture frame of the text in the video is marked as an illegal picture frame. The illegal picture frame in the video can be deleted or otherwise processed later. Accordingly, the text recognition result includes illegal text. Optionally, the text recognition result can also include the ID of the illegal text, which makes it easier to record the text using the ID. Optionally, the text recognition result can also include the position of the illegal picture frame in the video, such as the ID or coordinates of the illegal picture frame in the video. Optionally, the text recognition result also includes: illegal picture frame. In the text recognition result, not only the illegal text is output but also one or more frames of illegal pictures are output, which can be convenient for users to watch and thus find illegal pictures more conveniently.

[0159] It should be noted that the two methods of obtaining text recognition results corresponding to the above two text processing scenarios can be combined or executed separately according to actual conditions, and the embodiments of the present application do not limit this.

[0160] S204: Output the text recognition result.

[0161] By outputting the text recognition results, users can have an accurate understanding of the content they need to obtain in the video based on the text recognition results.

[0162] It's worth noting that after outputting the text recognition results, the video text detection device can further perform actions corresponding to the current text processing scenario. For example, in a video content extraction scenario, the text recognition results are inserted into the beginning of the video and output as a video overview. In an assisted autonomous driving scenario, the vehicle is controlled autonomously based on the text recognition results. For example, if the text recognition result reads "Speed ​​Limit 80," the vehicle's speed will be adjusted to below 80 kilometers per hour.

[0163] To sum up, when recognizing text in a video, the embodiment of the present application performs matching based on the feature vector of the text in the kth frame image and the feature vector of the text that appears continuously in the previous images. In this way, the recognition of the same text targets in different frames is achieved through the correlation between the feature vectors between the images in the video. Therefore, the text recognition result of the video is not only a summary of the text recognition results of each frame image, but can also reflect the correlation between the text between images, providing users with more information about the video and improving the user experience.

[0164] Furthermore, by matching the feature vectors of the text in the kth frame and the feature vectors of the text that appears continuously in the previous pictures, it is also possible to associate the text corresponding to the same text target in the video, correct the text in the video, reduce the probability of misrecognition, and improve the recognition accuracy of the text in the video.

[0165] Furthermore, the text detection method for videos provided in the embodiment of the present application avoids the repetition of the same text corresponding to the same text target in the text recognition results by deduplicating the successfully matched text, making the text recognition results more complete and accurate, and improving the user's viewing experience of the text recognition results.

[0166] As mentioned above, the model involved in the text detection method for videos provided in the embodiment of the present application may be an AI model, such as a deep learning model. The use of AI models includes two processes: training (also called pre-training) and inference. Among them, the training process is the process of optimizing the parameters of the AI ​​model using a training sample set; the inference process is the process of running the trained AI model, which is the actual use process of the AI ​​model. The use process of each model in the aforementioned S202 is the inference process. Before the aforementioned S202, reasonable training is still required to ensure the effectiveness of the trained model. For example, the aforementioned feature generation model (such as Figures 9 to 15 Any of the feature generation models shown is a trained AI model, and the loss function used in training this feature generation model is the triplet loss function. The feature generation model trained with the triplet loss function generates more accurate feature vectors during the inference phase.

[0167] Before the aforementioned S201, the video text detection method provided by the embodiment of the present application further includes: training the initial feature generation model to obtain the feature generation model. This process includes the following steps B1 to B2.

[0168] B1. Obtain a training sample from the training sample set, where the training sample is a text image, and generate a triplet corresponding to the training sample.

[0169] Optionally, the triplet (Triplet) corresponding to the training sample includes: the training sample, a positive sample (Positive) whose similarity to the training sample is greater than a first similarity threshold, and a negative sample (Negative) whose similarity to the training sample is less than a second similarity threshold. Among them, the positive sample and the training sample can be regarded as the same or similar samples, and the negative sample and the training sample can be regarded as different or dissimilar samples. The first similarity threshold and the second similarity threshold are usually different, and the second similarity threshold is usually much smaller than the first similarity threshold. In an embodiment of the present application, each sample has an ID, and samples with the same ID represent the same or similar samples, and samples with different IDs represent different or dissimilar samples. The identification of each sample is manually labeled or machine labeled. Optionally, when generating each triplet, based on the ID of the training sample, the sample with the same ID as the training sample is extracted as the positive sample, and the sample with a different ID from the training sample is extracted as the negative sample.

[0170] B2. Train the initial feature generation model based on the triples and the ternary loss function to obtain a feature generation model.

[0171] A video satisfies the following characteristics: the text corresponding to the same text target in different pictures has similar text appearance structure and the same semantics, and the text appearance structure corresponding to different text targets is different. The principle of the ternary loss function is: through training, the similarity between each training sample and its positive sample is improved, and the similarity between each training sample and its negative sample is reduced, that is, the distance between the training sample and the positive sample is reduced, and the distance between the training sample and the negative sample is increased. Training the initial feature generation model according to the triples and the ternary loss function can make the feature vectors obtained by the trained feature generation model in the inference stage effectively reflect the characteristics of the pictures in the aforementioned video, and accurately reflect the similarity or difference between the feature vectors of the text in the k-th frame picture and the feature vectors of the text in the picture before the k-th frame picture, thereby improving the effect of the trained feature generation model in the inference stage. The ternary loss function can be expressed as:

[0172] L=[d a，p +d a，n +a] + ;

[0173] Among them, a represents the training sample, p represents the positive sample corresponding to the training sample a, n represents the negative sample corresponding to the training sample a, d a，p Indicates the distance between the training sample a and the positive sample p, d a，n Indicates the distance between training sample a and negative sample n. a is a hyperparameter with a preset value, such as 0.2 or 0.3. + indicates that when the value in [] is greater than zero, the value is taken as the loss value. When the value in [] is less than zero, the loss value is zero.

[0174] like Figure 16 As shown, when training the initial feature generation model based on the triples and the ternary loss function, the aforementioned process B1 can be executed multiple times to obtain multiple triples, and then process B2 can be executed based on the obtained multiple triples. Process B2 can also be executed after each execution of process B1. This embodiment of the present application does not limit this.

[0175] Take the example of executing process B2 after executing process B1 each time. The training process can be executed multiple times until the loss value of the ternary loss function converges within a specified range, wherein each training process includes: positive samples whose similarity to the training samples is greater than a first similarity threshold, and negative samples whose similarity to the training samples is less than a second similarity threshold, inputting the triplet into the initial feature generation model, and performing backpropagation training based on the initial feature generation model to reduce the distance between the training samples and the positive samples and increase the distance between the training samples and the negative samples. Each training process includes: calculating the loss value based on the ternary loss function, and then updating the network parameters of the initial feature generation model after calculating the gradient based on the loss value.

[0176] by Figure 10 、 Figure 13 or Figure 14Taking the initial feature generation model shown in as an example, since the initial feature generation model generates both semantic feature vectors and visual feature vectors, when training the initial feature generation model based on the triples and the aforementioned ternary loss function, a ternary loss function L1 can be set for the semantic feature generation layer 503, and another ternary loss function L2 can be set for the position feature generation layer 504. The input triples of the two are the same, but the hyperparameters of the two are different, namely a1 and a2, and finally an overall ternary loss function L is determined, L=L1+L2. The training process is then performed multiple times until the loss value of the ternary loss function L converges within the specified range corresponding to a1+a2, that is, the training process is completed. In each training process, the distance between the visual feature vector of the training sample and the visual feature vector of the positive sample is reduced, the distance between the visual feature vector of the training sample and the visual feature vector of the negative sample is increased, and the distance between the semantic feature vector of the training sample and the semantic feature vector of the positive sample is reduced, and the distance between the semantic feature vector of the training sample and the semantic feature vector of the negative sample is increased.

[0177] Because adjacent frames in the same sample video are more closely related, optionally, the training samples in the training sample set belong to the same sample video, each training sample and its corresponding positive sample belong to adjacent frames of the sample video, and each training sample and its corresponding negative sample belong to adjacent frames of the sample video. In this way, a batch of training data in the training phase comes from two adjacent images in the same sample video, which can improve the accuracy of the correlation between the feature vectors generated by the feature generation model for adjacent frames during the inference process.

[0178] It is worth noting that the timing of executing the aforementioned process S202 may vary depending on the specific scenario. In one optional implementation, after all n frames in the video are recognized, text-based feature vector matching is performed on each of the n frames. Specifically, after executing the aforementioned process A1 with each of the n frames as the kth frame, the aforementioned process A2 is then executed with each of the n frames as the kth frame.

[0179] In another optional manner, each frame of the video is taken as the k-th frame, and the aforementioned processes A1 and A2 are executed on the k-th frame. That is, the text recognition process of the image is performed in a loop iteration manner. Optionally, during the loop iteration process, the text can also be deduplicated and / or tracked synchronously. Assuming that the image before the k-th frame is the previous frame of the k-th frame, the aforementioned first feature generation model is used for reasoning. The schematic process of S202 of a video text detection method provided in an embodiment of the present application includes:

[0180] B1. Get the kth frame image.

[0181] B2. Identify and obtain h1 ROIs in the k-th frame image, where h1 is a positive integer. Each ROI corresponds to a position where text may appear, that is, one ROI corresponds to a group of text. This process can refer to the process performed by the first text recognition model or the second text recognition model in the aforementioned process A1. For example, when recognizing the k-th frame image, when h ROIs are obtained, the confidence of each ROI is calculated. When the confidence of any ROI is greater than the confidence threshold corresponding to the ROI, the ROI is determined as the final ROI. In this way, h1 more reliable ROIs are screened out from the h ROIs.

[0182] B3. Match the text in the k-th frame with the text in the k-1-th frame, and remove duplicates from the matched text. Assuming there are h2 ROIs in the k-1-th frame, where h2 is a positive integer, process B3 includes: for each of the h1 ROIs, match the feature vector of the text in each ROI with the feature vector of the text in the h2 ROIs, obtaining h2 distances (or similarities), and ultimately determining h1×h2 first distances (or similarities) between the h1 ROI and the h2 ROI.

[0183] Optionally, process B3 further includes: for each of the h1 ROIs, obtaining the edit distance between the feature vectors of the text of each ROI and the text of h2 ROIs, obtaining h2 edit distances, and finally determining h1×h2 edit distances between the h1 ROI and the h2 ROIs.

[0184] Optionally, process B3 also includes: for each Roi in the h1 Roi, obtaining the distance (or degree of overlap or similarity) between the position of each Roi and the predicted position of the text of the h2 Roi on the k-th frame image, obtaining h2 distances, and finally determining h1×h2 second distances (or degree of overlap or similarity) between the h1 Roi and the h2 Roi.

[0185] Match h1 ROIs with h2 ROIs based on the obtained parameters, where the obtained parameters include at least h1×h2 first distances (or similarities) and, optionally, further include at least one of h1×h2 edit distances and h1×h2 second distances (or overlap degrees or similarities).

[0186] When a certain Roi in the k-th frame image is successfully matched with a certain Roi in the k-1-th frame image, the same text target corresponding to the two is determined, and the same ID is assigned to the Roi of the same text target.

[0187] Process B3 can refer to the text matching process in the aforementioned process A2. Optionally, only one group of multiple groups of text assigned the same ID can be retained to achieve text deduplication simultaneously. Optionally, based on the ID corresponding to the same text target, the image sequence corresponding to the text with the same ID is updated to achieve text tracking simultaneously. Among them, when the text target appears for the first time, the update process refers to establishing a new text sequence; when the text target does not appear for the first time, the update process refers to adding the ID or coordinates of the kth frame image to the image sequence corresponding to the text target.

[0188] B4. Update k so that the updated k = k + 1. When the updated k > n, output the text recognition result of the video. When the updated k ≤ n, repeat B1 to B4 until the updated k > n, and output the text recognition result of the video.

[0189] like Figure 17 As shown, when the n frames of a video include picture p1 and picture p2, the text recognition results obtained by using traditional technology are: "passenger car", "120", "60", "large car", "100" and "60" corresponding to picture p1; and "passenger car", "120", "6", "large car", "100" and "6" corresponding to picture p2. For example, for the same text target "passenger car", one group of text should be output, but using traditional technology will produce two groups of text outputs. However, by using the video recognition method provided by the embodiment of the present application, since the correlation relationship between the feature vectors between the pictures in the video is established, the deduplication of the text corresponding to the same text target is achieved, so the text recognition results of the video obtained are: "passenger car", "120", "60", "large car", "100" and "60", which avoids the repetition of the same text corresponding to the same text target and achieves the recognition of the complete text recognition result.

[0190] The video text detection method provided in the embodiment of the present application, when recognizing text in the video, performs matching based on the feature vector of the text in the k-th frame image and the feature vector of the text that appears continuously in the previous image, and deduplicates the successfully matched text. In this way, through the correlation between the feature vectors between the images in the video, the repeated appearance of the same text corresponding to the same text target in the text recognition results is avoided, thereby improving the text recognition efficiency.

[0191] Moreover, during the inference stage, the visual feature vector and the semantic feature vector are concatenated into a feature vector of the text, which serves as a robust feature vector that combines low-level visual features and high-level semantic features. This can effectively reflect the characteristics of the text and optimize the deduplication effect.

[0192] Training the initial feature generation model based on the triples and ternary loss function can enable the feature vectors obtained by the trained feature generation model in the inference stage to effectively reflect the characteristics of the pictures in the aforementioned video, and accurately reflect the similarity or difference between the feature vectors of the text in the k-th frame picture and the feature vectors of the text in the picture before the k-th frame picture, thereby improving the effect of the trained feature generation model in the inference stage.

[0193] The order of steps of the video text detection method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with this technical field can easily think of different methods within the technical scope disclosed in this application, and they should all be covered by the protection scope of this application, so they will not be repeated here.

[0194] The embodiment of the present application provides a video text detection device 70, such as Figure 18 The device 70 shown comprises:

[0195] A first acquisition module 701 is configured to acquire a video, where the video includes n frames of images, where n is an integer greater than 1;

[0196] The recognition module 702 is configured to recognize text in each frame of the video and obtain a text recognition result of the video. The process of the recognition module 702 recognizing text in the k-th frame of the video includes:

[0197] Get the text in the k-th frame of the picture, 1<k≤n;

[0198] Based on the feature vector of the text in the k-th frame image and the feature vector of the text appearing in the image before the k-th frame image in the video, the text in the k-th frame image is matched with the text appearing in the image before the k-th frame image.

[0199] The text detection device provided in the embodiment of the present application has a recognition module that, when recognizing text in a video, performs matching based on the feature vectors of the text in the k-th frame image and the feature vectors of the text that appears continuously in the previous images, and deduplicates the successfully matched text. In this way, the recognition of the same text targets in different frames is achieved through the correlation between the feature vectors between the images in the video, so that the text recognition result of the video is not only a summary of the text recognition results of each frame image, but can also reflect the correlation between the text between images, providing users with more information about the video and improving the user experience.

[0200] Furthermore, by matching the feature vectors of the text in the kth frame and the feature vectors of the text that appears continuously in the previous pictures, it is also possible to associate the text corresponding to the same text target in the video, correct the text in the video, reduce the probability of misrecognition, and improve the recognition accuracy of the text in the video.

[0201] Alternatively, as Figure 19 As shown, the device 70 further includes:

[0202] The second acquisition module 703 is configured to input the text image in the k-th frame into a feature generation model to obtain a feature vector of the text in the k-th frame.

[0203] Optionally, the recognition module 702 obtains the text in the k-th frame image, including: obtaining the text in the k-th frame image according to the feature generation model; or, inputting the k-th frame image into a text recognition model, and obtaining the text in the k-th frame image according to the text recognition model, wherein the text recognition model and the feature generation model are two different artificial intelligence AI models.

[0204] Optionally, the text recognition result of the video includes one or more of the following results: the text recognition result of the video obtained after deduplication of text based on the matching results between pictures in the video; the text recognition result of the video obtained after text tracking based on the matching results between pictures in the video.

[0205] Optionally, the feature vector of the text includes a semantic feature vector and a visual feature vector.

[0206] Optionally, the feature generation model is a trained artificial intelligence AI model, and the loss function used to train the feature generation model is a ternary loss function.

[0207] Alternatively, as Figure 20 As shown, the device 70 also includes: a third acquisition module 704, which is used to obtain a training sample from a training sample set, where the training sample is a text image, and generate a triplet corresponding to the training sample; a training module 705, which is used to train the initial feature generation model according to the triplet and the ternary loss function to obtain the feature generation model.

[0208] Optionally, the triplet corresponding to the training sample includes: the training sample, a positive sample whose similarity to the training sample is greater than a first similarity threshold, and a negative sample whose similarity to the training sample is less than a second similarity threshold.

[0209] Optionally, the recognition module 702 matches the text in the k-th frame image with the text appearing in the image before the k-th frame image, further comprising:

[0210] Matching the text in the k-th frame with the text that appears in the picture before the k-th frame based on the edit distance between the text in the k-th frame and the picture before the k-th frame in the video;

[0211] And / or, based on the position of the text in the k-th frame image and the predicted position of the text appearing in the image before the k-th frame image in the k-th frame image, the text in the k-th frame image is matched with the text appearing in the image before the k-th frame image.

[0212] Figure 21 A hardware structure diagram of a computing device 800 deployed with a video text detection device 70 provided in an embodiment of the present application is shown. Figure 21 The computing device 800 shown includes a memory 801 , a processor 802 , a communication interface 803 , and a bus 804 . The memory 801 , the processor 802 , and the communication interface 803 are communicatively connected to each other via the bus 804 .

[0213] The memory 801 can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 801 can store programs. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the text detection method for the video. The memory can also store data sets. For example, a portion of the storage resources in the memory 801 is divided into a data set storage module 106 for storing the data set required by the video text detection device 70, and a portion of the storage resources in the memory 801 is divided into an AI model storage module 107 for storing the AI ​​model library.

[0214] The processor 802 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0215] The processor 802 can also be an integrated circuit chip with signal processing capabilities. During implementation, the functions of the video text detection device 70 of the present application can be completed by hardware integrated logic circuits or software instructions in the processor 802. The above-mentioned processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments below of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments below of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and combines its hardware to complete the function of the video text detection device 70 of the embodiment of the present application.

[0216] The communication interface 803 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device 800 and other devices or communication networks. For example, a data set can be obtained through the communication interface 803.

[0217] The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).

[0218] It should be noted that the above embodiment provides a text detection device for a video. When performing text recognition, the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text detection device for a video provided in the above embodiment and the text detection method embodiment for a video are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0219] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance. The term "plurality" refers to two or more, unless expressly limited otherwise. A referring to B means that A is the same as B or that A is a simple variant of B.

[0220] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0221] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for detecting text in a video, characterized in that: The method comprises: Obtain a video, where the video includes n frames of pictures, where n is an integer greater than 1; Identify the text in each frame of the video to obtain a text recognition result of the video, wherein the process of identifying the text in the k-th frame of the video includes: Get the text in the k-th frame of the picture, 1<k≤n; Based on the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video, the text in the k-th frame picture and the text appearing in the picture before the k-th frame picture are matched. The feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture are both feature vectors obtained using a feature generation model. The feature vector of the text is obtained by splicing a semantic feature vector and a visual feature vector. The semantic feature vector is used to reflect the semantic attribute characteristics of the text in the picture, and the visual feature vector is used to identify the visual features of the text in the picture.

2. The method according to claim 1, characterized in that The method further comprises: The k-th frame image is input into the feature generation model to obtain a feature vector of the text in the k-th frame image.

3. The method according to claim 2, characterized in that The obtaining of the text in the k-th frame of the picture includes: Generating a model based on the features, obtaining text in the k-th frame of the image; or, The k-th frame image is input into a text recognition model, and the text in the k-th frame image is obtained according to the text recognition model, wherein the text recognition model and the feature generation model are two different artificial intelligence AI models.

4. The method according to any one of claims 1 to 3, characterized in that: The text recognition result of the video includes one or more of the following results: the text recognition result of the video obtained after deduplication of text based on the matching results between the pictures in the video; The text recognition result of the video is obtained by tracking the text based on the matching results between the pictures in the video.

5. The method according to claim 2 or 3, characterized in that The feature generation model is a trained artificial intelligence AI model, and the loss function used to train the feature generation model is a ternary loss function.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a training sample from a training sample set, where the training sample is a text image, and generate a triplet corresponding to the training sample; An initial feature generation model is trained according to the triples and the ternary loss function to obtain the feature generation model.

7. The method according to claim 6, characterized in that The triples corresponding to the training samples include: The training sample, a positive sample whose similarity to the training sample is greater than a first similarity threshold, and a negative sample whose similarity to the training sample is less than a second similarity threshold.

8. The method according to any one of claims 1-3, 6-7, characterized in that: Matching the text in the k-th frame image with the text appearing in the image before the k-th frame image includes: Matching the text in the k-th frame with the text appearing in the picture before the k-th frame based on the edit distance between the text in the k-th frame and the picture before the k-th frame in the video; And / or, based on the position of the text in the k-th frame image and the predicted position of the text appearing in the image before the k-th frame image in the k-th frame image, the text in the k-th frame image is matched with the text appearing in the image before the k-th frame image.

9. A device for detecting text in a video, characterized in that: The device comprises: A first acquisition module is used to acquire a video, where the video includes n frames of pictures, where n is an integer greater than 1; A recognition module is configured to recognize text in each frame of the video and obtain a text recognition result of the video, wherein the process of the recognition module recognizing text in the kth frame of the video includes: Get the text in the k-th frame of the picture, 1<k≤n; Based on the feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture in the video, the text in the k-th frame picture and the text appearing in the picture before the k-th frame picture are matched. The feature vector of the text in the k-th frame picture and the feature vector of the text appearing in the picture before the k-th frame picture are both feature vectors obtained using a feature generation model. The feature vector of the text is obtained by splicing a semantic feature vector and a visual feature vector. The semantic feature vector is used to reflect the semantic attribute characteristics of the text in the picture, and the visual feature vector is used to identify the visual features of the text in the picture.

10. The device according to claim 9, characterized in that The device further comprises: The second acquisition module is used to input the text image in the k-th frame image into the feature generation model to obtain the feature vector of the text in the k-th frame image.

11. The device according to claim 10, characterized in that The recognition module obtains the text in the k-th frame image, including: Generating a model based on the features, obtaining text in the k-th frame of the image; or, The k-th frame image is input into a text recognition model, and the text in the k-th frame image is obtained according to the text recognition model, wherein the text recognition model and the feature generation model are two different artificial intelligence AI models.

12. The device according to any one of claims 9 to 11, characterized in that: The text recognition result of the video includes one or more of the following results: the text recognition result of the video obtained after deduplication of text based on the matching results between the pictures in the video; The text recognition result of the video is obtained by tracking the text based on the matching results between the pictures in the video.

13. The device according to claim 10 or 11, characterized in that The feature generation model is a trained artificial intelligence AI model, and the loss function used to train the feature generation model is a ternary loss function.

14. The device according to claim 13, characterized in that The device further comprises: A third acquisition module is used to acquire a training sample from the training sample set, where the training sample is a text image, and generate a triplet corresponding to the training sample; A training module is used to train the initial feature generation model according to the triples and the ternary loss function to obtain the feature generation model.

15. The device according to claim 14, characterized in that The triples corresponding to the training samples include: The training sample, a positive sample whose similarity to the training sample is greater than a first similarity threshold, and a negative sample whose similarity to the training sample is less than a second similarity threshold.

16. The device according to any one of claims 9-11, 14-15, characterized in that: The recognition module matches the text in the k-th frame image with the text appearing in the image before the k-th frame image, including: Matching the text in the k-th frame with the text appearing in the picture before the k-th frame based on the edit distance between the text in the k-th frame and the picture before the k-th frame in the video; And / or, based on the position of the text in the k-th frame image and the predicted position of the text appearing in the image before the k-th frame image in the k-th frame image, the text in the k-th frame image is matched with the text appearing in the image before the k-th frame image.

17. A computing device, characterized in that The computing device includes a memory and a processor, the memory being configured to store a set of computer instructions; The processor executes a set of computer instructions stored in the memory to perform the method according to any one of claims 1 to 8.

18. A non-transitory readable storage medium, characterized in that The non-transitory readable storage medium stores computer program code. When the computer program code is executed by a computing device, the computing device performs the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product comprises a computer program code. When the computer program code is executed by a computing device, the computing device performs the method according to any one of claims 1 to 8.

Images