A video OSD character recognition method, apparatus and medium

Abstract

This invention provides a video OSD character recognition method, comprising the following steps: S1, acquiring an OSD video image; S2, inputting the OSD video image to a character position detection module to obtain character position information; S3, simultaneously inputting the character position information and the OSD video image to a character position cropping module to obtain a cropped image; the character position cropping module randomly offsets the center coordinates of character image blocks in the OSD video image by M character lengths; S4, inputting the cropped image to a character content recognition module to output the recognized string information. This invention utilizes the position information of OSD characters during the character recognition process, which can accelerate model convergence and improve recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of character recognition, in particular to a video OSD character recognition method and device and medium. BACKGROUND

[0002] With the wide application of intelligent security systems based on video monitoring, a large number of OSD (Onscreen display) videos are generated every day. In practice, it is necessary to identify the OSD character information in these OSD video images, which is used to determine whether the characters displayed in real time in the OSD video are consistent with the characters to be displayed actually set, or is used for video archiving, or is used for database indexing, etc.

[0003] There are currently two solutions as follows: The first one is to perform OCR (optical character recognition) character recognition based on traditional image processing methods. Gradient feature extraction, HOG feature extraction, dilation, erosion, image binarization and other computer vision technologies are used to detect and segment the OSD characters in the video into single characters, then template matching is performed for each OSD character, and finally the recognized characters are obtained. The main disadvantage of this method is that the character segmentation effect is poor due to the influence of the light changes of the video acquisition environment and the complex background in the image, incomplete character segmentation and missing segmentation and missegmentation problems are easy to occur, the recognition accuracy is low, and the online deployment effect is unstable, especially the segmentation threshold is difficult to determine due to the scene change between day and night.

[0004] The second one is based on deep learning method, which first detects the position of OSD characters through a character detection network, and then inputs the character segment image into a deep learning network for character recognition. The main disadvantage of this method is that it requires a large number of training samples and computing resources, and is affected by the image background and light.

[0005] The existing deep learning method directly applied to OSD character recognition has the following three main disadvantages.

[0006] First, the current OSD character recognition model has low accuracy.

[0007] Second, the current OCR optical character recognition method does not fully utilize the unique information of OSD character recognition, which is mainly because the data of traditional OCR character recognition comes from artificial annotation, which is time-consuming and laborious, and generally does not finely annotate the position information of each character, while the data synthesized by OSD technology has both position and content information.

[0008] Third, the deep learning model based on the OCR method is not optimized for OSD character recognition. The OSD character is composed of a continuous block of square characters. There are no character deformation or character scaling in the same picture in the video. The color of the character is mostly black or white. The OCR recognition is often in the natural world, such as outdoor billboard characters, mobile phone shooting paper characters, and characters with different colors and textures. If the method is not designed to strengthen the learning of the unique features of OSD characters and weaken the influence of the features of the background characters, the OCR character recognition method will be directly applied to OSD character recognition, which will often misidentify the background characters in the video picture. When these irrelevant background characters are sent to the OSD recognition module, it is easy to cause the performance degradation of the entire OSD character recognition system. Secondly, the pixels of OSD characters are monochrome, but the background directly comes from the video background, which makes the background of OSD characters complex and variable, increasing the difficulty of recognition. The background of the characters in the OCR research object is often single, which makes the OCR character recognition method not sensitive to OSD characters, and easy to cause missed detection.

[0009] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The material described herein in this section is not prior art to the claims of the present application and is not admitted to be prior art by inclusion in this section. SUMMARY

[0010] In view of the above technical problems in the related art, the present application provides a video OSD character recognition method, which comprises the following steps:

[0011] S1, obtaining an OSD video image;

[0012] S2, inputting the OSD video image into a character position detection module to obtain character position information;

[0013] S3, inputting the character position information and the OSD video image into a character position cropping module to obtain a cropped image; the character position cropping module randomly offsets the center coordinates of the character image block in the OSD video image by M character lengths;

[0014] S4, inputting the cropped image into a character content recognition module to output recognized string information.

[0015] Specifically, the character content recognition module has the following specific structure: a first CNN feature map (101), a down-sampling module (102), a spatial attention module (103), an up-sampling module (104), a multiplier (105), and a second CNN feature map (106), wherein the multiplier (105) performs multiplication operation on the output from the up-sampling module (104) and the CNN feature map of the first CNN feature map (101).

[0016] Specifically, the second CNN feature map (106) is input into a CRNN network for character recognition.

[0017] Specifically, the training data generation method of the character position detection module is as follows:

[0018] The background region and the character region are simultaneously divided into blocks with a height of k1 and a width of k2, denoted as block, and a single character background average method is applied to superimpose characters for each block block to obtain an OSD character image of a synthetic character.

[0019] Specifically, edge filtering is performed on the OSD character image of the synthetic character.

[0020] In a second aspect, another embodiment of the present application discloses a video OSD character recognition device, which comprises the following units:

[0021] An OSD video acquisition unit is configured to acquire an OSD video image.

[0022] A character position acquisition unit is configured to input the OSD video image into a character position detection module to obtain character position information.

[0023] A character cropping unit is configured to input the character position information and the OSD video image into a character position cropping module to obtain a cropped image, and the character position cropping module randomly offsets the center coordinates of the character image block in the OSD video image by M character lengths.

[0024] A character recognition unit is configured to input the cropped image into a character content recognition module to output recognized character string information.

[0025] Specifically, the character content recognition module has the following specific structure: a first CNN feature map (101), a down-sampling module (102), a spatial attention module (103), an up-sampling module (104), a multiplier (105), and a second CNN feature map (106), wherein the multiplier (105) performs multiplication operation on the output from the up-sampling module (104) and the CNN feature map of the first CNN feature map (101).

[0026] Specifically, the character recognition unit further comprises inputting the second CNN feature map (106) into a CRNN network for character recognition.

[0027] Specifically, the training data generation method of the character position detection module is as follows:

[0028] The background region and the character region are simultaneously divided into blocks with a height of k1 and a width of k2 size pixels, denoted as block, and a single character background average method is applied to superimpose characters for each block block to obtain an OSD character image of the synthesized characters.

[0029] In a third aspect, another embodiment of the present application discloses a non-volatile memory, which stores instructions, and the instructions are executed by a processor to implement the video OSD character recognition method.

[0030] In the process of recognizing characters, the position information of the OSD characters is used at the same time, which can accelerate the model convergence and improve the recognition accuracy. The prior art directly uses a segmentation auxiliary network to jointly position information outside the recognition network for training, removes the segmentation network after training, and then directly uses the recognition network. This method may have the problem that the character position in the OSD video image is not clear enough, and more character information may be learned and distributed in the segmentation network, and the supervision effect on the recognition network is limited. The position information is directly integrated into the recognition network through a two-level position attention mechanism in the present application, which guides the recognition network to appropriately pay attention to the position information of the characters in the recognition process. The network architecture in the training stage and the deployment stage is completely consistent, which maximizes the network to learn and remember the position information of the characters, and the obtained recognition network has better effect and higher accuracy. At the same time, compared with externally using a segmentation auxiliary network to guide the network to pay attention to the position information, the method of the present application has the advantages of smaller model size and less resource consumption in training. The method for synthesizing OSD in the present embodiment can large-scale and realistically synthesize the OSD style effect in a real scene, and can simulate the effect of the character being edge blurred in the OSD image due to the information degradation problem in the signal acquisition process to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0032] Figure 1 is a flowchart of a video OSD character recognition method provided by an embodiment of the present application;

[0033] Figure 2 is a kind of OSD character schematic diagram provided by the embodiment of the present application;

[0034] Figure 3 is a kind of OSD character synthesis schematic diagram provided by the embodiment of the present application;

[0035] Figure 4 is the schematic diagram of character recognition module provided by the embodiment of the present application;

[0036] Figure 5 is the schematic diagram of CBAM module provided by the embodiment of the present application;

[0037] Figure 6 is a kind of video OSD character recognition device schematic diagram provided by the embodiment of the present application;

[0038] Figure 7 is a kind of video OSD character recognition device schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0040] Embodiment one

[0041] Reference Figure 1 The embodiment discloses a video OSD character recognition method, which comprises the following steps:

[0042] S1, obtaining OSD video image;

[0043] Specifically, the present embodiment obtains video stream with OSD from monitoring camera or cloud server storing video stream.

[0044] The present embodiment obtains OSD video image from video stream by using video analysis method, and converts corresponding frame object in video stream into image.

[0045] S2, the OSD video image is input to the character position detection module, and the position information of character image block is obtained, which is represented by (x_center, y_center, h, w). Wherein x_center represents the horizontal coordinate of the center position of character image block, y_center represents the vertical coordinate of the center position of character image block, h represents the height of character image block, and w represents the width of character image block;

[0046] The character position detection module of the embodiment is constructed based on a convolutional neural network, which needs to be trained.

[0047] The training process of the character position modeling module of the embodiment is as follows:

[0048] In the training stage, a large-scale synthetic training data set is first generated, and the method is as follows:

[0049] In the video image, one or more character image blocks are randomly selected, and the size of each character image block is random but needs to ensure that it can completely contain multiple characters.

[0050] The character image block of the embodiment refers to an image block composed of one or more characters, wherein the image block composed of multiple characters has no obvious space between the multiple characters, i.e., the multiple characters are continuous.

[0051] The character image block is generally located at the bottom of the video image, such as the form of a commonly used subtitle. Of course, the character image block can also be located at other positions of the video image.

[0052] According to each selected character image block, multiple characters are randomly selected and filled into the character image block using the OSD technology, so that the character image block is just filled. All character image blocks are filled in this way until all character image blocks are filled, and the filled OSD video image, character code, and character image block position information are obtained, wherein the character image block position information is represented by (x_center, y_center, h, w).

[0053] The process of filling characters into the video image using the OSD technology is to fill the characters into the character image block one by one. If the selected character image block is filled with multiple characters, the position information of the character image block is called the image block position information of the multiple characters. In the character image block containing multiple characters, for each character, there is a single character image block position information corresponding to it.

[0054] In the process of filling characters into the image using the OSD technology, the color, font size, and font of the characters are randomly selected.

[0055] The data synthesis tool of the embodiment is Python, and the data synthesis is in the process of superimposing the font image onto the video image.

[0056] The commonly used superimposition method in the prior art is the single character background average method, and the specific method is as follows:

[0057] Let the height of the character be h, the width of the character be w, the pixel value of the character region before superimposition be P, and the pixel value of the character region after superimposition be X. The height of the background be H, the width of the background be W, the pixel value of the background region before superimposition be Q, and the pixel value of the background region after superimposition be Y.

[0058] Before superimposing the character onto the background, the background is a block of image region, and the average value of the pixels of the background image is calculated as:

[0059]

[0060] The formula q is the value of each pixel point, and n is the total number of pixels, wherein n = H x W.

[0061] If Q mean <T0, then X = 255. If Q mean >T1, then X = 0, and otherwise, the value of X is determined by a Bernoulli distribution with a probability of 0.5. Specifically, in one Bernoulli trial, if the number of successes is r and the number of failures is 1-r, the value of P(r) is calculated according to the following formula, wherein p = 0.5. If P(r) is greater than 0.5, then the value of X is 255, otherwise, the value of X is 0.

[0062] P(r) = p r *(1-p) (1-r)

[0063] T0 and T1 are threshold values obtained from a Gaussian distribution with a mean of 0 and a variance of 1. Specifically, the Gaussian distribution probability P(x) is calculated using the following formula. If the randomly sampled x is between [-1, 1], i.e., the confidence distribution interval of the standard normal distribution is 67.27%, then T0 = T*P(x), T1 = 255-T*P(x) = 255-T0, wherein T is a hard threshold value, and in the embodiment, T is 100.

[0064]

[0065] In some cases, the character background changes too much, and the superimposed character obtained by using the single character background average method is prone to have some problems of local pixel fusion with the background, causing the characters in the OSD video image to be unclear. In this embodiment, assuming that the width of the background region is W and the height is H, the adaptive character pixel superimposition technology is to divide the background region and the character region into blocks with a height of k1 and a width of k2, denoted as block, and apply the single character background average method to superimpose the character for each block. After calculating one block region, the next block region is calculated, and the calculation of all divided block regions is completed. The value of k1 ranges from 3 to H, and the value of k2 ranges from 3 to W. For example, in Figure 3In the middle, the image A with the size of H*W is selected as the background of the synthesized OSD video image, which is divided into 5*5 blocks, and the single character background average method is applied to each block respectively. For example, in Q11, the position corresponds to a non-character region in the corresponding character image, so the Q11 image block of the background is directly copied to the synthesized OSD character image. In the Q24 block of the background, the position corresponds to a character region in the corresponding character image, and the pixel of the Q24 block of the background is 255 (black), while the pixel of the character region is 255 (black). According to the single character background average method, the character region of the position block in the synthesized OSD character image will be changed to pixel value 0 (white) with a certain probability.

[0066] After the OSD character obtained by the adaptive character pixel superposition technology, the edge of the OSD character is filtered to make the background and the font edge fuse with each other. The specific method is to use a 3*3 size median filter kernel, move the median filter kernel along the boundary between the character region and the background region in the OSD video image, calculate the mean value of the pixels in the region where the median filter kernel is located, and fill the entire region where the median filter kernel is located with the mean value. When the calculation of a place is completed, move to the next position with a step of 1 pixel unit and repeat the above calculation until the calculation of all the boundaries between the character region and the background region is completed.

[0067] The training process of the character position detection module is to send the synthesized data after data augmentation into the character position detection module to obtain the predicted position information of the character image block, compare it with the real synthesized position information of the character image block, calculate the loss function, obtain the error gradient, and then update the parameters of the entire character position detection module by back propagation until the error between the predicted position information of the character image block by the character position detection module and the real synthesized position information of the character image block reaches a set threshold, and the training is stopped.

[0068] S3, input the character position information and the OSD video image into the character position cropping module simultaneously to obtain a cropped image; the character position cropping module randomly offsets the center coordinates of the character image block in the OSD video image by M character lengths. The specific method is to randomly select M characters for all OSD characters in the character image block, add the widths of the M characters to obtain the offset of the horizontal coordinate, and select the height of the character with the maximum height in the M characters as the offset of the vertical coordinate.

[0069] The character position cropping module is to crop the OSD video image combined with the position information of the character image block obtained in step S2, and finally output the cropped image. The cropping method is the enlargement random cropping method, and the purpose is to guide the character content recognition module to appropriately pay attention to the position change of the characters.

[0070] ReferenceFigure 2 In this embodiment, the center coordinates of the image block position information of the plurality of characters are randomly offset by M character lengths, and in the implementation case, M is 3, 5, or 8. At the same time, the width and height of the image block position information of the plurality of characters are also expanded or reduced accordingly, so as to meet the requirement that the cropped character image block contains complete character information and has a random size of redundant blank area. The specific method of expanding or reducing the width and height of the image block position information of the plurality of characters is as follows: assuming that the original center coordinates in the position information of the image block of the plurality of characters are (x_center, y_center), after being randomly offset by M character lengths, the center coordinates become (x_center±α, y_center±β), where α represents the horizontal coordinate offset, and β represents the vertical coordinate offset. Correspondingly, the width in the position information of the image block of the plurality of characters is adjusted to w+2*|α|+Δ, and the height is adjusted to h+2*|β|+Δ, where Δ is a random size of redundant amount, and the purpose is to meet the requirement that the cropped character image block contains complete character information and has a blank area. That is, the original position information of the image block of the plurality of characters is (x_center, y_center, h, w), and the position cropping model adjusts it to (x_center±α, y_center±β, h+2*|β|+Δ, w+2*|α|+Δ), and then crops.

[0071] S4, input the cropped image into a character content recognition module, and output recognized string information;

[0072] The character content recognition module of this embodiment specifically adopts a model designed by deep learning and neural network. Such a network is generally composed of a CNN (Convolutional Neural Network) layer, an RNN (Recurrent Neural Network) and its variants LSTM (Long short Term Memory), and a self-attention mechanism layer, and a training loss function often adopts a CTC Loss or a classification loss function. Such a network is represented by CRNN (Convolutional Recurrent Neural Network).

[0073] The present application optimizes the character content recognition module by using a position supervision mechanism method. Specifically, a two-level position attention module is designed as follows: Figure 4 The module can be directly embedded into a character recognition network, such as a CRNN network. The calculation process of the module is as follows:

[0074] (1) Obtain CNN features 101 from the deep neural network and send them to the downsampling module 102. In the implementation example, CNN features refer to the features from the intermediate layer of the CRNN network based on VGG13. It should be understood that the CNN features of the present invention are not limited to a specific type of network or a specific layer. Related network transformation and replacement modules with a matrix output format of C*H*W (C is the number of channels, H is the feature map height, and W is the feature map width) should all be considered as CNN features. For example, in the transformer network, the data output after the self-attention layer can be rearranged into a C*H*W matrix format, and this feature should also be considered as a CNN feature.

[0075] (2) The downsampling module 102 processes the CNN feature map output by 101 through convolution, batch regularization, activation, pooling, and downsampling to obtain a feature map with 1 channel. In this embodiment, the final downsampling ratio is 0.25, which means that the length and width of the CNN feature map output by 101 are each reduced by a factor of 4. Then, supervised learning of the image block position information of multiple characters is performed on this feature map. The supervised learning method is to calculate the corresponding two-dimensional Gaussian probability distribution heatmap of the image block position information (x_center, y_center, h, w) of multiple characters in the image input to the network, and perform supervised learning with the two-dimensional feature map output by 102. The loss function is Focal Loss. It should be understood that the downsampling of the length and width by a factor of 4 in this embodiment is not limited to 4. In this embodiment, the downsampling module refers to a module that may include convolution, batch regularization, activation, pooling, and downsampling.

[0076] (3) The result of 102 is input to the spatial attention module 103. The 103 module consists of convolutional layers, pooling layers, regularization layers, etc. Its function is to focus attention on the feature map output by the downsampling module 102 and improve the position accuracy of a single character.

[0077] In this embodiment, the spatial attention module is composed of a CBAM (Convolutional Block Attention Module) module that does not include channel attention (e.g.) Figure 5 ).

[0078] (4) The up-sampling module 104 obtains a feature map with a channel number of 1 through convolution, batch normalization, activation, pooling, up-sampling, and the like. In the embodiment, the final up-sampling ratio is 4, and the up-sampling ratio is kept consistent with the down-sampling, so as to ensure that the output size of the feature map 101 and the input size of the multiplier 105 remain consistent. Specifically, the length and width of the output CNN feature map of 103 are each enlarged by 4 times, and then the feature map is subjected to single-character image block position information supervised learning. The supervised learning method is to calculate the corresponding two-dimensional Gaussian probability distribution heat map of the single-character image block position information (x_center, y_center, h, w) in the input network image, and perform supervised learning on the 2D feature map output by 103, and the loss function is Focal Loss. It should be understood that the up-sampling in the embodiment is not limited to 4 times. In the embodiment of the present module, the up-sampling module refers to a module that may include convolution, batch normalization, activation, pooling, up-sampling, and the like.

[0079] (5) The multiplier 105 performs multiplication operation on the output from 104 and the CNN feature map 101, so that the network can focus on the position feature of the character, and finally outputs the CNN feature map 106. The CNN feature map 106 matrix and the CNN feature map 101 matrix remain consistent in the number of dimensions and shape.

[0080] The OSD recognition method of the embodiment can effectively improve the OSD recognition accuracy in different scenes, and the results are shown in Table 1.

[0081]

[0082] The present embodiment uses the position information of OSD characters in the character recognition process, which can accelerate model convergence and improve recognition accuracy. The prior art directly uses a segmentation auxiliary network to jointly train position information outside the recognition network, removes the segmentation network after training, and then directly uses the recognition network. This method may have the problem that the character position in the OSD video image is not clear enough, and more character information may be learned in the segmentation network, while the supervision of the recognition network is limited. The present embodiment directly integrates position information into the recognition network through a two-level position attention mechanism to guide the recognition network to appropriately focus on the position information of the characters during the recognition process. The network architecture in the training stage and the deployment stage is completely consistent, which maximizes the learning and memory of the position information of the characters, and the obtained recognition network has better effect and higher accuracy. At the same time, compared with externally using a segmentation auxiliary network to guide the network to focus on position information, the method of the present embodiment has the advantages of smaller model size and less resource consumption in training. The method for synthesizing OSD of the present embodiment can large-scale and realistically synthesize OSD style effects in real scenes, and can simulate the edge blur effect of characters in OSD images caused by information degradation in the signal acquisition process to a certain extent.

[0083] Embodiment two

[0084] Reference Figure 6 The present embodiment discloses a video OSD character recognition device, which comprises the following units:

[0085] An OSD video acquisition unit is configured to acquire an OSD video image.

[0086] A character position acquisition unit is configured to input the OSD video image to a character position detection module to obtain character position information.

[0087] The character position detection module of the present embodiment is constructed based on a convolutional neural network, which needs to be trained.

[0088] The training process of the character position modeling module of the present embodiment is as follows:

[0089] In the training stage, first, a large-scale training data set is synthesized, and the method is as follows:

[0090] In the video image, one or more character image blocks are randomly selected, and the size of each character image block is random but needs to be ensured to be able to completely contain multiple characters.

[0091] The character image block of the present embodiment refers to an image block composed of one or more characters, wherein the image block composed of multiple characters has no obvious space between the multiple characters, i.e., the multiple characters are continuous.

[0092] The character image blocks are generally located below the video image, for example, in the form of a common subtitle. Of course, the character image blocks can also be located in other positions of the video image.

[0093] According to each selected character image block, a plurality of characters are randomly selected and filled into the character image block by using the OSD technology, so that the character image block is just filled. All the character image blocks are filled in this way until all the character image blocks are filled, and a filled OSD video image, character encoding, and position information of the character image block are obtained, wherein the position information of the character image block uses (x_center, y_center, h, w).

[0094] The process of filling characters into the video image by using the OSD technology is that the characters are filled into the character image block one by one. If the selected character image block is filled with a plurality of characters, the position information of the character image block is referred to as the image block position information of the plurality of characters. In the character image block containing a plurality of characters, for each character, there is a single character image block position information corresponding to the character.

[0095] In the process of filling characters into the image by using the OSD technology, the color, font size, and font of the characters are randomly selected.

[0096] The data synthesis tool of the embodiment is Python, and the data synthesis is in the process of superimposing the font image onto the video image.

[0097] The commonly used superimposition method in the prior art is a single character background average method, and the specific method is as follows:

[0098] Suppose that the height of the character is h, the width of the character is w, the pixel value of the character region before superimposition is P, and the pixel value of the character region after superimposition is X. The height of the background is H, the width of the background is W, the pixel value of the background region before superimposition is Q, and the pixel value of the background region after superimposition is Y.

[0099] Before the character is superimposed on the background, the background is an image region, and the average value of the pixels of the background image is calculated as:

[0100]

[0101] The formula q is the value of each pixel point, and n is the total number of pixels, wherein n = H x W.

[0102] If Q mean <T0, then X = 255. If Q meanT1, then X=0, otherwise, X is determined by a Bernoulli distribution with probability 0.5, in particular, in one Bernoulli trial, if the success number is r, the failure number is 1-r, the P(r) value is calculated according to the following formula, where p=0.5, if P(r) is greater than 0.5, then X value is 255, otherwise, X value is 0.

[0103] P(r)=p r *(1-p) (1-r)

[0104] T0, T1 are threshold values obtained from a Gaussian distribution with mean 0 and variance 1, in particular, in calculating the Gaussian distribution probability P(x) using the following formula, if the x obtained by random sampling is between [-1, 1], that is, the confidence distribution interval of the standard normal distribution is 67.27%, then T0=T*P(x), T1=255-T*P(x)=255-T0, where T is the hard threshold value, and in the embodiment, T is 100.

[0105]

[0106] In some cases, the background of the character changes too much, and the superimposed character obtained by using the single character background average method is easy to cause some local pixels of the character to be fused with the background, resulting in unclear characters in the OSD video image. In this embodiment, assuming that the width of the background area is W and the height is H, the adaptive character pixel superposition technology is to divide the background area and the character area into blocks with a height of k1 and a width of k2, denoted as block, and apply the single character background average method to superimpose characters for each block. When a block area is calculated, the next block area is calculated, and the calculation of all divided block areas is completed. The value of k1 ranges from 3 to H, and the value of k2 ranges from 3 to W. For example, in Figure 3 , an image A with a size of H*W is selected as the background of the synthesized OSD video image, which is divided into 5*5 blocks, and the single character background average method is applied to each block, for example, in Q11, the position corresponds to a non-character area in the character image, so the Q11 image block of the background is directly copied to the synthesized OSD character image. In the Q24 block of the background, the position corresponds to a character area in the character image, and the pixels of the Q24 block of the background are 255 (black), while the pixels of the character area are 255 (black). According to the single character background average method, the character area of the position block in the synthesized OSD character image will be changed to pixel value 0 (white) with a certain probability.

[0107] The OSD character obtained after the adaptive character pixel superposition technology is filtered to make the background and the font edge fuse with each other. Specifically, a 3*3 median filter core is used, the median filter core is moved along the boundary between the character region and the background region in the OSD video image, the mean value of the pixels in the region where the median filter core is located is calculated, and the mean value is used to fill the region where the median filter core is located, when the calculation of one place is completed, the next position is moved according to a step of 1 pixel unit, and the above calculation is repeated until the boundary between the entire character region and the background region is calculated.

[0108] a character cropping unit configured to input the character position information and the OSD video image into a character position cropping module to obtain a cropped image, and the character position cropping module is configured to randomly offset the center coordinates of the character image block in the OSD video image by M character lengths;

[0109] The character position cropping module is configured to crop the OSD video image combined with the position information of the character image block obtained in step S2, and finally output the cropped image. The cropping method is a random cropping method, and the purpose is to guide the character content recognition module to appropriately pay attention to the position change of the character.

[0110] Reference Figure 2 In the embodiment, the center coordinates of the image block position information of the plurality of characters are randomly offset by M character lengths, and in the implementation case, M is 3, 5 or 8. At the same time, the width and height of the image block position information of the plurality of characters are also correspondingly expanded or reduced, wherein the width adjustment is w+2*|a|+D, and the height adjustment is h+2*|b|+D, so as to satisfy that the cropped character image block contains complete character information and has a random size redundant blank area.

[0111] a character recognition unit configured to input the cropped image into a character content recognition module to output recognized string information.

[0112] The character content recognition module specifically includes a first CNN feature map (101), a down-sampling module (102), a spatial attention module (103), an up-sampling module (104), a multiplier (105), and a second CNN feature map (106), wherein the multiplier (105) performs multiplication operation on the output from the up-sampling module (104) and the CNN feature map of the first CNN feature map (101).

[0113] The character recognition unit further includes inputting the second CNN feature map (106) into a CRNN network for character recognition.

[0114] The embodiment uses the position information of OSD characters in the character recognition process, which can accelerate model convergence and improve recognition accuracy. The prior art directly uses a segmentation auxiliary network to jointly train position information outside the recognition network, removes the segmentation network after training, and then directly uses the recognition network. This method may have the problem that the character position in the OSD video image is not clear enough, and more character information may be learned in the segmentation network, while the supervision of the recognition network is limited. The embodiment directly integrates the position information into the recognition network through a two-level position attention mechanism, guides the recognition network to appropriately pay attention to the position information of the characters during the recognition process, and the network architecture in the training stage and the deployment stage is completely consistent, which maximizes the learning and memory of the position information of the characters by the network, and the obtained recognition network has better effect and higher accuracy. At the same time, compared with externally using a segmentation auxiliary network to guide the network to pay attention to the position information, the method of the embodiment has the advantages of smaller model size and less resource consumption in training. The method for synthesizing OSD of the embodiment can large-scale and realistically synthesize OSD style effects in real scenes, and can simulate the edge blur effect of characters in OSD images caused by information degradation in the signal acquisition process to a certain extent.

[0115] Embodiment three

[0116] Reference Figure 7 , Figure 7 is a structural schematic diagram of a video OSD character recognition device of the embodiment. The video OSD character recognition device 20 of the embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. The processor 21 implements the steps in the above method embodiments when executing the computer program. Alternatively, the processor 21 implements the functions of each module / unit in the above device embodiments when executing the computer program.

[0117] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can complete a specific function, which are used to describe the execution process of the computer program in the video OSD character recognition device 20. For example, the computer program can be divided into the modules in embodiment two, and the specific functions of each module are described in the working process of the device in the above embodiments, which will not be repeated here.

[0118] The video OSD character recognition device 20 can include, but is not limited to, a processor 21, a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the video OSD character recognition device 20, and does not constitute a limitation on the video OSD character recognition device 20, and can include more or less components than the diagram, or combine certain components, or different components, for example, the video OSD character recognition device 20 can also include an input and output device, a network access device, a bus, etc.

[0119] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 21 is the control center of the video OSD character recognition device 20, and connects various parts of the entire video OSD character recognition device 20 through various interfaces and lines.

[0120] The memory 22 can be used to store computer programs and / or modules, and the processor 21 realizes various functions of the video OSD character recognition device 20 by running or executing computer programs and / or modules stored in the memory 22, and calling data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0121] The modules / units of the video OSD character recognition device 20 are stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. The computer program is executed by the processor 21, and the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. that can carry the computer program code. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0122] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0123] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for video OSD character recognition, comprising the following steps: S1, obtaining an OSD video image; S2, inputting the OSD video image into a character position detection module to obtain character position information; S3, inputting the character position information and the OSD video image into a character position cropping module to obtain a cropped image; the character position cropping module randomly offsets the center coordinates of a character image block in the OSD video image by M character lengths; S4, inputting the cropped image into a character content recognition module to output recognized character string information; the character content recognition module comprises a first CNN feature map, a down-sampling module, a spatial attention module, an up-sampling module, a multiplier, and a second CNN feature map, wherein the multiplier performs multiplication operation on the output from the up-sampling module and the CNN feature map of the first CNN feature map; wherein the image block position information of a plurality of characters in an input network image is calculated to obtain a corresponding two-dimensional Gaussian probability distribution heat map, and supervised learning is performed on the 2D feature map output by the down-sampling module and the up-sampling module, and the loss function is FocalLoss. 2.The method of claim 1, wherein the second CNN feature map (106) is input into a CRNN network for character recognition. 3.The method of claim 1, wherein a training data generation method of the character position detection module is as follows: Divide a background region and a character region into blocks with a height of k1 and a width of k2, denoted as block, and apply a single character background average method to superimpose characters for each block to obtain an OSD character image of a synthetic character. 4.The method of claim 3, wherein edge filtering is performed on the OSD character image of the synthetic character. 5.A device for video OSD character recognition, comprising the following units: an OSD video obtaining unit configured to obtain an OSD video image; a character position obtaining unit configured to input the OSD video image into a character position detection module to obtain character position information; a character cropping unit configured to input the character position information and the OSD video image into a character position cropping module to obtain a cropped image; the character position cropping module randomly offsets the center coordinates of a character image block in the OSD video image by M character lengths; The character recognition unit is configured to input the cropped image into a character content recognition module, and output recognized character string information; the character content recognition module comprises a first CNN feature map (101), a down-sampling module (102), a spatial attention module (103), an up-sampling module (104), a multiplier (105), and a second CNN feature map (106), wherein the multiplier (105) performs multiplication operation on the output from the up-sampling module (104) and the CNN feature map of the first CNN feature map (101); wherein the image block position information of the multi-character in the input network image is calculated to obtain a corresponding two-dimensional Gaussian probability distribution heat map, and the two-dimensional Gaussian probability distribution heat map is subjected to supervised learning with the 2-dimensional feature map output by the down-sampling module and the up-sampling module, and the loss function is FocalLoss.

6. The apparatus of claim 5, wherein the character recognition unit further comprises inputting the second CNN feature map (106) into a CRNN network for character recognition.

7. The apparatus of claim 5, wherein a training data generation method of the character position detection module is as follows: dividing a background region and a character region into blocks according to a height of k1 and a width of k2, denoted as block, and applying a single character background average method to superimpose characters to obtain an OSD character image of a synthesized character for each block.

8. A non-volatile memory, wherein the memory stores instructions, and the instructions are executed by a processor to implement the video OSD character recognition method of any one of claims 1-4. ​

Images