A method and system for OCR recognition of audio and video

Abstract

The present application relates to the field of image processing and recognition technology, and specifically to an OCR recognition method and system for audio and video, the method comprising: extracting potential Chinese character areas in the grayscale image of each current frame; determining the Chinese character certainty of each potential Chinese character area in each current frame; based on the difference between the average distribution of the grayscale values ​​of all pixels in the two target areas of each potential Chinese character area in each current frame and its comparison area, combined with the Chinese character certainty, constructing a Chinese character evaluation index for each potential Chinese character area in each current frame to achieve text recognition of audio and video. The present application is based on the morphological texture features of the Chinese character area and the spatiotemporal context features of the video data to adjust the training parameters of the neural network, improve the accuracy of character recognition in audio and video and the training accuracy of the neural network model.

Description

Technical Field

[0001] The present application relates to the field of image processing and recognition technology, and in particular to an OCR recognition method and system for audio and video. Background Art

[0002] The OCR recognition method for audio and video is to use audio processing technology and video processing technology to extract text information contained in audio and video files, which is helpful for the retrieval, classification and supervision of audio and video content. For the voice signal in the audio data, the voice tool can be used to directly translate it into text data, but due to the complex characteristics of video data, such as complex background, diverse fonts and blurred video, the accuracy and efficiency of OCR recognition of video data cannot achieve the ideal effect.

[0003] In traditional OCR text recognition for video data, the video data is decomposed into frame images, and the text information of the entire video data is obtained by extracting text from each frame image. However, in actual video data, continuous multi-needle images have certain similarities and associations, and therefore have a spatiotemporal contextual association feature. In addition, the arrangement of text blocks in the frame image has a certain typesetting style, so the line segmentation method in traditional text detection cannot accurately locate the features of Chinese characters, resulting in low Chinese character recognition accuracy and efficiency in the traditional OCR recognition process. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide an OCR recognition method and system for audio and video. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides an OCR recognition method for audio and video, the method comprising the following steps:

[0006] Get the grayscale image of each current frame in the audio and video;

[0007] Extract potential Chinese character areas and stroke lines in each current frame grayscale image through edge detection and connected domain analysis;

[0008] Divide the local range by taking the intersection of the diagonals of the minimum circumscribed rectangle of each potential Chinese character area in each current frame as the center;

[0009] Determine the Chinese character certainty of each potential Chinese character region in each current frame according to the size of the minimum bounding rectangle of each potential Chinese character region in each current frame, the number of potential Chinese character regions in the local range, and the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the edge of the nearest potential Chinese character region;

[0010] The potential Chinese character region closest to the central pixel point of the local range of each potential Chinese character region in each current frame is recorded as the target region of each potential Chinese character region in each current frame;

[0011] The potential Chinese character region in the next frame that is closest to the central pixel point of each potential Chinese character region in the current frame is recorded as the comparison region of each potential Chinese character region in each current frame;

[0012] According to the difference between the average distribution of the grayscale values ​​of all pixels in the two target areas of each potential Chinese character area in each current frame and its comparison area, combined with the Chinese character confidence, a Chinese character evaluation index is constructed for each potential Chinese character area in each current frame to achieve text recognition in audio and video.

[0013] Preferably, the method for determining the Chinese character certainty of each potential Chinese character region in each current frame is:

[0014] Determine the Chinese character morphology weight of each potential Chinese character region in each current frame according to the size of the minimum bounding rectangle of each potential Chinese character region in each current frame and the number of potential Chinese character regions in the local range;

[0015] Determine the Chinese character texture coefficient of each potential Chinese character region in each current frame according to the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the edge of the nearest potential Chinese character region;

[0016] The expression of the Chinese character confidence of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character confidence of the ith potential Chinese character region in the jth frame image; Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the Chinese character texture coefficient of the i-th potential Chinese character area in the j-th frame image.

[0017] Preferably, the expression of the Chinese character morphology weight of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the length of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the width of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the number of potential Chinese character regions within the local range of the i-th potential Chinese character region in the j-th frame image.

[0018] Preferably, the Chinese character texture coefficient of each potential Chinese character region in each current frame is the reciprocal of the average of the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the nearest edge of the potential Chinese character region.

[0019] Preferably, the process of constructing the Chinese character evaluation index of each potential Chinese character area in each current frame is as follows:

[0020] Constructing the correlation deviation of each potential Chinese character region in each current frame according to the difference between the average distribution of the grayscale values ​​of all pixels in the two target regions of each potential Chinese character region in each current frame and the comparison region;

[0021] Determine a confidence evaluation index for each potential Chinese character region in each current frame according to the Chinese character confidence of each potential Chinese character region in each current frame;

[0022] A Chinese character evaluation index for each potential Chinese character region in each current frame is determined based on the association deviation and the confidence evaluation index.

[0023] Preferably, the construction process of the association deviation of each potential Chinese character region in each current frame is:

[0024] Analyze the mean of the grayscale values ​​of all pixels in the target area of ​​each potential Chinese character area in each current frame, and record it as the first mean of the target area of ​​each potential Chinese character area in each current frame;

[0025] The associated deviation of each potential Chinese character region in each current frame is the difference between the first mean values ​​of the two target regions of each potential Chinese character region and its comparison region in each current frame.

[0026] Preferably, the confidence evaluation index of each potential Chinese character region in each current frame is the maximum value between the confidence evaluation index of each potential Chinese character region in each current frame and the confidence evaluation index of its comparison region.

[0027] Preferably, the Chinese character evaluation index of each potential Chinese character region in each current frame is the ratio of the confidence evaluation index of each potential Chinese character region in each current frame to the associated deviation.

[0028] Preferably, the method of realizing text recognition of audio and video includes:

[0029] Taking the Chinese character evaluation indexes of all potential Chinese character regions in each current frame as the input of the threshold segmentation algorithm, obtaining the segmentation threshold of each current frame;

[0030] If the Chinese character evaluation index of the potential Chinese character region in each current frame is greater than the segmentation threshold, then the potential Chinese character region is a real Chinese character region, otherwise, it is a background interference region.

[0031] In a second aspect, an embodiment of the present application further provides an OCR recognition system for audio and video, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0032] This application has at least the following beneficial effects:

[0033] The present application performs morphological analysis and texture analysis on potential Chinese character regions, and determines the Chinese character certainty of each potential Chinese character region in each current frame according to the size of the minimum bounding rectangle of each potential Chinese character region in each current frame, the number of potential Chinese character regions in the local range, and the distance from all pixel points on the stroke line of each potential Chinese character region in each current frame to the edge of the nearest potential Chinese character region, so as to reflect whether the potential Chinese character region contains text information;

[0034] This application utilizes the temporal and spatial contextual association features between adjacent frames, and obtains the Chinese character evaluation index of each potential Chinese character area in each current frame based on the grayscale value of all pixels in the target area of ​​each potential Chinese character area in each current frame and the difference in the grayscale value of all pixels in the target area of ​​the mirror-symmetrical potential Chinese character area in the next frame, combined with the Chinese character confidence, to reflect the degree of distinction between the Chinese characters and the background in the potential Chinese character area and adjust the inclination in the Sigmoid function in the BP neural network. This application combines the morphological texture features of the Chinese character area and the temporal and spatial contextual features of the video data to adjust the training parameters of the neural network, improve the training accuracy of the model, and facilitate accurate recognition of audio and video content OCR. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A flowchart of a method for OCR recognition of audio and video;

[0037] Figure 2 Extraction diagram for Chinese character evaluation index;

[0038] Figure 3 Schematic diagram of the audio and video character recognition process. DETAILED DESCRIPTION

[0039] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of the OCR recognition method and system for audio and video proposed in accordance with the present application, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0041] The following is a detailed description of a specific solution of an OCR recognition method and system for audio and video provided by the present application in conjunction with the accompanying drawings.

[0042] See also Figure 1 , which shows a flowchart of a method for OCR recognition of audio and video provided by an embodiment of the present application, the method comprising the following steps:

[0043] Step S1: Obtain video data that needs to be OCR recognized through the audio and video management platform.

[0044] This application obtains audio and video data that needs to be identified and detected through an audio and video management platform, wherein the audio and video data is a complete audio and video content. By decoding a single audio and video data, audio data and video data are obtained respectively, and corresponding processing is performed to extract text.

[0045] For the audio data in a single audio or video data, this embodiment uses STT (Speech-to-Text) in NLP to convert speech to text. The optional processing tools include iFlytek, Otter.ai, HappyScribe and Speechnotes, thereby obtaining the speech text of the audio or video data.

[0046] For the video data in a single audio and video data, since the frame images in the video data have certain temporal and spatial correlation characteristics, especially the frame images in the same segment reflect the same content information, different segments are often distinguished by transition frames in the video data, where the transition frames are characterized by being completely black. Therefore, all the transition frames in the entire video data are obtained through pixel comparison, and all video frames of adjacent transition frames are called a video segment. Based on the further analysis of the image frames contained in the video segment, the text information in the frame images is extracted.

[0047] Step S2: Determine the Chinese character certainty of each potential Chinese character area in each current frame based on the size of the minimum circumscribed rectangle of each potential Chinese character area in each current frame, the number of potential Chinese character areas in the local range, and the distance from all pixel points on the stroke line of each potential Chinese character area in each current frame to the edge of the nearest potential Chinese character area.

[0048] For each current frame in the same video clip in the video data, its content may contain multiple text information, which is generally called scene text, that is, the text information contained in the natural scene captured by the camera during the shooting process, showing a specific arrangement and embedded in the image background. In addition, scene text often has a specific layout distribution for a certain artistic style or scene highlighting, and does not only adopt row and column distribution.

[0049] This embodiment mainly aims at extracting Chinese character information, and Chinese characters basically present the morphological structure of square characters, and Chinese characters have certain stroke texture characteristics. Therefore, it is necessary to analyze the content of each current frame and extract the potential Chinese character area. The specific processing is as follows:

[0050] Each current frame is converted into a grayscale image, and the closed contours obtained through edge detection algorithm and connected domain analysis are several potential Chinese character areas in each current frame; the several edge lines of each potential Chinese character area in each current frame obtained by using skeleton extraction algorithm are stroke lines.

[0051] It should be noted that there are many commonly used edge detection algorithms and connected domain analysis methods. In this embodiment, the Canny edge detection algorithm and morphological operations are used to extract the potential Chinese character areas and their edges and stroke lines in each current frame. Implementers can also use other edge detection algorithms and connected domain analysis algorithms in the prior art, and this embodiment does not impose any special restrictions.

[0052] Among them, the Canny edge detection algorithm and morphological operations are well-known technologies in image processing technology, and the specific process of extracting edges will not be repeated; the skeleton extraction algorithm is a well-known technology, and the specific process of extracting the skeleton structure of an object will not be repeated.

[0053] Since the potential Chinese character area contains real Chinese character font blocks and interference factors in some background information, the potential Chinese character area needs to be further analyzed.

[0054] Chinese characters have the characteristics of square characters, and the overall size of Chinese characters is basically the same within a similar range. Therefore, in the real Chinese character area, there is a Chinese character area whose minimum circumscribed rectangle is close to a square. At the same time, Chinese characters have certain stroke texture characteristics. A single stroke has a certain width, and the distance between the pixel point at the edge of the stroke and the pixel point at the center of the stroke is often close.

[0055] Therefore, according to the size of the minimum circumscribed rectangle of each potential Chinese character area in each current frame and the number of potential Chinese character areas in the local range, as well as the distances from all pixel points on the stroke line of each potential Chinese character area in each current frame to the edge of the nearest potential Chinese character area, the Chinese character certainty of each potential Chinese character area in each current frame is determined, and the Chinese character certainty of each potential Chinese character area in each current frame is obtained to determine the possibility that the potential Chinese character area is a real Chinese character area, and exclude the influence of the real Chinese character font block and background information on the potential Chinese character area, specifically:

[0056] (1) The Chinese character morphological weight of each potential Chinese character region in each current frame.

[0057] In actual audio and video, Chinese characters are usually square in shape and have a certain degree of symmetry; and Chinese characters do not appear in isolation, but are arranged according to certain typesetting rules. Therefore, according to the size of the minimum bounding rectangle of each potential Chinese character area in each current frame and the number of potential Chinese character areas in the local range, the Chinese character morphological weight of each potential Chinese character area in each current frame is constructed to determine whether the shape of the potential Chinese character area meets the characteristics of Chinese characters and characterize the importance of each potential Chinese character area in the overall recognition task, specifically:

[0058] A local range is divided with each potential Chinese character area in each current frame as the center, wherein the local range is a circular range with the intersection of the diagonals of the minimum circumscribed rectangle of each potential Chinese character area in each current frame as the center and the maximum value of the length and width of the minimum circumscribed rectangle of each potential Chinese character area in each current frame as the radius.

[0059] It should be noted that the value of the multiple k is artificially set. In this embodiment, the value of k is 2. The implementer can set it according to the specific situation. This embodiment does not impose any special restrictions.

[0060] The expression of the Chinese character morphological weight of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the length of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the width of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the number of potential Chinese character regions within the local range of the i-th potential Chinese character region in the j-th frame image.

[0061] If the potential Chinese character region is a real Chinese character region, due to the characteristics of Chinese characters as square characters, the minimum circumscribed rectangle of the potential Chinese character region is close to a square, so the aspect ratio is close to 1. At the same time, because Chinese characters have a certain typesetting style, that is, a single Chinese character does not appear alone, there are often other potential Chinese character regions within the local range of the potential Chinese character region, so the greater the Chinese character morphological weight of the potential Chinese character region.

[0062] (2) Chinese character texture coefficient of each potential Chinese character area in each current frame.

[0063] The basic components of Chinese characters are strokes. The stroke line can accurately locate the stroke structure of Chinese characters. The distance from the pixel point on the stroke line to the edge of the Chinese character reflects the detail of the Chinese character texture. According to the distance from all the pixel points on the stroke line of each potential Chinese character area in each current frame to the edge of the nearest potential Chinese character area, the Chinese character texture coefficient of each potential Chinese character area in each current frame is determined to better understand the internal structure of Chinese characters and reduce the possibility of misrecognition. Specifically:

[0064] The Chinese character texture coefficient of each potential Chinese character region in each current frame is the reciprocal of the average of the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the nearest edge of the potential Chinese character region.

[0065] If the potential Chinese character area is a real Chinese character area, since Chinese characters are composed of basic strokes, the strokes occupy a certain pixel width in the frame image, and the stroke lines are obtained by the skeleton extraction algorithm. The distance from the points on the stroke lines to the edge contour of the Chinese character strokes is usually small, so the value of the Chinese character texture coefficient in this area is larger.

[0066] (3) The Chinese character certainty of each potential Chinese character area in each current frame.

[0067] In each frame of the audio and video, the real Chinese character font block and the background area may be similar to the potential Chinese character area. In order to improve the recognition accuracy, the Chinese character certainty of each potential Chinese character area in each current frame is determined based on the Chinese character morphological weight and the Chinese character texture coefficient, specifically:

[0068] The expression of the Chinese character confidence of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character confidence of the ith potential Chinese character region in the jth frame image; Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the Chinese character texture coefficient of the i-th potential Chinese character area in the j-th frame image.

[0069] If the current area is an interference area formed in the frame image, since the background information is often complex, the height-to-length ratio of the minimum circumscribed rectangle of the current area is relatively small, and the number of potential defect areas in the local range may be small, resulting in a smaller Chinese character morphological weight value in the area. In addition, the interference area of ​​the background information is not necessarily as fine as the Chinese character strokes, so the pixel points on the stroke line are far away from the edge contour, the value of the Chinese character texture coefficient is small, and finally the value of the Chinese character confidence in the current area is small.

[0070] Step S3: constructing a Chinese character evaluation index for each potential Chinese character area in each current frame based on the difference between the average distribution of the grayscale values ​​of all pixels in the two target areas of each potential Chinese character area in each current frame and its comparison area, combined with the Chinese character confidence.

[0071] In step S2, by analyzing the pixel-level and region-level features of the potential Chinese character regions in each current frame, an evaluation is made as to whether the potential Chinese character regions contain Chinese characters. However, some real Chinese characters may be too close to the background color of each frame in the image, so the obtained Chinese character confidence value is small, which affects the identification of the real Chinese character blocks.

[0072] The same scene is often presented in the same segment of video data, so the same information may be repeated in adjacent frame images, but the degree of distinction between real Chinese characters and background information in different frame images is different. Based on this, the Chinese characters are further confirmed according to the different performances of the same Chinese character area in adjacent frames.

[0073] Audio and video data often have a high frequency, and the video rate can often reach 30FPS, 60FPS or even higher. Therefore, the displacement of Chinese characters between adjacent frames is not large, but the background in the image may change greatly due to factors such as background light changes and perspective changes. Therefore, a comprehensive analysis is performed on the frame after each current frame.

[0074] It should be noted that if the current frame image is the end frame, it often has certain transition information, and the end frame is not analyzed and processed.

[0075] If there is no other potential Chinese character area in the local range of the potential Chinese character area of ​​the next frame of the current frame, then the area is recorded as the background area;

[0076] If there are multiple potential Chinese character areas in the local range of the potential Chinese character area of ​​the frame after each current frame, only the potential Chinese character area closest to the center pixel point in the local range of each potential Chinese character area in each current frame is analyzed and recorded as the target area of ​​each potential Chinese character area in each current frame.

[0077] The potential Chinese character region in the subsequent frame that is closest to the central pixel point of each potential Chinese character region in the current frame is recorded as the comparison region of each potential Chinese character region in the current frame.

[0078] According to the difference between the average distribution of the grayscale values ​​of all pixels in the two target areas of each potential Chinese character area in each current frame and the comparison area, combined with the Chinese character confidence, a Chinese character evaluation index for each potential Chinese character area in each current frame is constructed, specifically:

[0079] (1) The correlation deviation of each potential Chinese character region in the comparison set of each current frame.

[0080] In audio and video data, the appearance of the Chinese character area may change due to lighting or camera shaking. Therefore, the associated deviation of each potential Chinese character area in each current frame is constructed based on the difference between the grayscale values ​​of all pixels in the target area of ​​each potential Chinese character area in the current frame and the grayscale values ​​of all pixels in the target area of ​​the mirror-symmetrical potential Chinese character area in the next frame to determine whether these changes are beyond the normal range, so as to evaluate the stability of the potential Chinese character area, specifically:

[0081] Analyze the mean of the grayscale values ​​of all pixels in the target area of ​​each potential Chinese character area in each current frame, and record it as the first mean of the target area of ​​each potential Chinese character area in each current frame;

[0082] The associated deviation of each potential Chinese character region in each current frame is the difference between the first mean values ​​of the two target regions of each potential Chinese character region and its comparison region in each current frame.

[0083] It should be noted that there are many ways to measure the difference. In this embodiment, the difference between the current frame and the next frame is measured by taking the absolute value of the difference between the first mean of the target area of ​​each potential Chinese character area in the current frame and the first mean of the target area of ​​the mirror-symmetric potential Chinese character area in the next frame. The implementer may also adopt other methods of measuring the difference such as ratio according to the specific situation, and this embodiment does not impose any special restrictions.

[0084] (2) Chinese character evaluation index of each potential Chinese character region in each current frame.

[0085] The confidence evaluation index of each potential Chinese character area in each current frame is determined according to the Chinese character confidence of each potential Chinese character area in each current frame. The confidence evaluation index of each potential Chinese character area in each current frame is the maximum value between the confidence evaluation index of each potential Chinese character area in each current frame and the confidence evaluation index of the comparison area.

[0086] The Chinese character evaluation index of each potential Chinese character region in each current frame is the ratio of the confidence evaluation index of each potential Chinese character region in each current frame to the associated deviation.

[0087] If the potential Chinese character area is a real Chinese character area, the displacement of Chinese characters in adjacent frame images is relatively close, and the strokes and colors of Chinese characters are relatively fixed, so the correlation deviation of the potential Chinese character area in the comparison set is relatively small. At the same time, due to background factors, the value of the Chinese character certainty of each potential Chinese character area in each current frame is small, but with the change of viewing angle and light, the distinction between Chinese characters and background in adjacent frames is large, and finally the value of the Chinese character evaluation index of each potential Chinese character area in each current frame is large.

[0088] Preferably, as an embodiment of the present application, the Chinese character evaluation index extraction schematic diagram is as follows Figure 2 shown.

[0089] Step S4: Based on the Chinese character evaluation index of each potential Chinese character area, a BP training model is obtained.

[0090] The spatiotemporal correlation of all potential Chinese character regions in each current frame is used as the input of the threshold segmentation algorithm to obtain the segmentation threshold.

[0091] If the Chinese character evaluation index of the potential Chinese character region in each current frame is greater than the segmentation threshold, then the potential Chinese character region is a real Chinese character region, otherwise, it is a background interference region.

[0092] It should be noted that there are many commonly used threshold algorithms. In this embodiment, the Otsu threshold segmentation algorithm is used to obtain the segmentation threshold to achieve the segmentation purpose. The implementer may also use other threshold segmentation algorithms such as the maximum inter-class variance in the prior art according to the specific situation. This embodiment does not impose any special restrictions.

[0093] Among them, the Otsu threshold segmentation algorithm is a well-known technology, and the specific implementation process of its segmentation is not repeated here.

[0094] The Chinese character evaluation index can be used to determine whether the current area is a real Chinese character and how it can be distinguished from the background.

[0095] Furthermore, a preset number of frames of images with Chinese characters are obtained, and the text blocks in the image (i.e., the minimum bounding rectangle of the potential Chinese character area) are obtained by projection method. The BP neural network is used for training, and the optimizer is set to Adam, and the cross entropy loss function is used. The Sigmoid function is used as the activation function, and the training epoch is a preset value.

[0096] It should be understood that the values ​​of the preset quantity and the preset value are both manually set. In this embodiment, the value of the preset quantity is 10,000, and the value of the preset value is 20. The implementer can set them according to the specific situation, and this embodiment does not impose any special restrictions.

[0097] It should be noted that the normalized result of the Chinese character evaluation index of each text block (potential Chinese character area) is used as the value of the inclination α of the Sigmoid function to achieve training of the video data.

[0098] If the value of the Chinese character evaluation index corresponding to the potential Chinese character area is large, it indicates that the current area is more distinguishable from the background. Therefore, the slope α of the Sigmoid function is increased, that is, the linearity of the model is improved, and the training speed of the model is improved; if the value of the Chinese character evaluation index corresponding to the potential Chinese character area is small, it indicates that the current area is less distinguishable from the background. Therefore, the slope α is reduced, the nonlinear ability of the model is improved, which is conducive to the model learning the detailed features of the text blocks.

[0099] Among them, the projection method, BP neural network and cross entropy loss function are well-known technologies, and the specific process will not be repeated here.

[0100] Step S5: Using the BP training model for OCR recognition of the video data to be detected.

[0101] The BP neural network model with the highest test accuracy in step S4 is used for deployment, and the video data that needs to be OCR recognized is obtained. The key frames in the video data are extracted through the DBNet target detection deep learning model, and the frame images are obtained using the projection method to obtain text blocks. All text blocks in all key frames are used as inputs of the trained BP neural network. The model outputs all text information in the current video data. Finally, the order of the key frames and the spatial position of the text blocks in the key frames are used to typeset the text information to obtain the text information of the video data, and combined with the translated text of the voice data, the text data after OCR recognition of the current audio and video data is obtained.

[0102] Among them, the DBNet target detection deep learning model is a well-known technology, and its specific process of predicting the location and category of the target will not be repeated here.

[0103] At this point, OCR recognition of audio and video is achieved based on the Chinese character evaluation index and BP neural network model.

[0104] Preferably, as an embodiment of the present application, the audio and video character recognition process diagram is as follows: Figure 3 shown.

[0105] Based on the same inventive concept as the above method, an embodiment of the present application also provides an OCR recognition system for audio and video, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above-mentioned OCR recognition methods for audio and video are implemented.

[0106] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0108] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. An OCR recognition method for audio and video, characterized in that: The method comprises the following steps: Get the grayscale image of each current frame in the audio and video; Extract potential Chinese character areas and stroke lines in each current frame grayscale image through edge detection and connected domain analysis; Divide the local range by taking the intersection of the diagonals of the minimum circumscribed rectangle of each potential Chinese character area in each current frame as the center; Determine the Chinese character certainty of each potential Chinese character region in each current frame according to the size of the minimum bounding rectangle of each potential Chinese character region in each current frame, the number of potential Chinese character regions in the local range, and the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the edge of the nearest potential Chinese character region; The potential Chinese character region closest to the central pixel point of the local range of each potential Chinese character region in each current frame is recorded as the target region of each potential Chinese character region in each current frame; The potential Chinese character region in the next frame that is closest to the central pixel point of each potential Chinese character region in the current frame is recorded as the comparison region of each potential Chinese character region in each current frame; The construction process of the Chinese character evaluation index of each potential Chinese character area in each current frame is as follows: Analyze the mean of the grayscale values ​​of all pixels in the target area of ​​each potential Chinese character area in each current frame, and record it as the first mean of the target area of ​​each potential Chinese character area in each current frame; The associated deviation of each potential Chinese character region in each current frame is the difference between the first mean values ​​of the two target regions of each potential Chinese character region and its comparison region in each current frame; The confidence evaluation index of each potential Chinese character region in each current frame is determined according to the Chinese character confidence of each potential Chinese character region in each current frame, and the Chinese character evaluation index of each potential Chinese character region in each current frame is the ratio of the confidence evaluation index of each potential Chinese character region in each current frame to the associated deviation.

2. The OCR recognition method for audio and video according to claim 1, characterized in that: The method for determining the Chinese character confidence of each potential Chinese character area in each current frame is: Determine the Chinese character morphology weight of each potential Chinese character region in each current frame according to the size of the minimum bounding rectangle of each potential Chinese character region in each current frame and the number of potential Chinese character regions in the local range; Determine the Chinese character texture coefficient of each potential Chinese character region in each current frame according to the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the edge of the nearest potential Chinese character region; The expression of the Chinese character confidence of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character confidence of the ith potential Chinese character region in the jth frame image; Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the Chinese character texture coefficient of the i-th potential Chinese character area in the j-th frame image.

3. The OCR recognition method for audio and video according to claim 2, characterized in that: The expression of the Chinese character morphological weight of each potential Chinese character area in each current frame is: In the formula, Represents the Chinese character morphological weight of the i-th potential Chinese character region in the j-th frame image; Represents the length of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the width of the minimum bounding rectangle of the ith potential Chinese character area in the jth frame image; Represents the number of potential Chinese character regions within the local range of the i-th potential Chinese character region in the j-th frame image.

4. The OCR recognition method for audio and video according to claim 2, characterized in that: The Chinese character texture coefficient of each potential Chinese character region in each current frame is the reciprocal of the average of the distances from all pixel points on the stroke line of each potential Chinese character region in each current frame to the nearest edge of the potential Chinese character region.

5. The OCR recognition method for audio and video according to claim 1, characterized in that: The confidence evaluation index of each potential Chinese character region in each current frame is the maximum value between the confidence evaluation index of each potential Chinese character region in each current frame and the confidence evaluation index of its comparison region.

6. The OCR recognition method for audio and video according to claim 1, characterized in that: Realize text recognition of audio and video, including: Taking the Chinese character evaluation indexes of all potential Chinese character regions in each current frame as the input of the threshold segmentation algorithm, obtaining the segmentation threshold of each current frame; If the Chinese character evaluation index of the potential Chinese character region in each current frame is greater than the segmentation threshold, then the potential Chinese character region is a real Chinese character region, otherwise, it is a background interference region.

7. An OCR recognition system for audio and video, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images