Text recognition method, device, electronic device and storage medium

Abstract

The present invention provides a text recognition method, apparatus, electronic device, and storage medium. The method includes: obtaining a video to be recognized and determining key frames in the video to be recognized; classifying the shooting states of the key frames to obtain the shooting states of the key frames; and integrating candidate recognition texts in overlapping regions of the key frames based on the shooting states of the key frames to obtain the recognition text of the video to be recognized. The method, apparatus, electronic device, and storage medium provided by the present invention collect the entire picture of a multi-page scenario in the form of a recorded video, thereby avoiding the problem of missed collection in the multi-page scenario. Moreover, the video to be recognized itself contains the collection order, eliminating the need for manual sorting and reducing the manual requirements for text recognition. In addition, integrating the candidate recognition texts in the overlapping regions based on the shooting states of the key frames realizes automatic integration of texts in the overlapping regions, ensuring the reliability of the recognition text.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a text recognition method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of technology, OCR (Optical Character Recognition) technology has been widely used in people's life and work.

[0003] However, in some multi-page text recognition scenarios, such as text recognition scenarios of long pictures, books, etc., a single photo cannot capture the entire scene to be recognized. It may be necessary to take multiple photos and perform text recognition on each photo separately. After that, the recognized texts of multiple photos need to be sorted manually to obtain the text recognition result of the entire scene.

[0004] Considering that the content in multiple photos taken for a single scene may overlap or be missing, manual processing is required after text recognition, resulting in an increase in labor costs; for scenes with a large number of pages, a large number of photos are needed to cover the scene content, and correspondingly, a large number of photos need to be sorted and recognized sequentially during the text recognition process, resulting in low recognition efficiency. Summary of the Invention

[0005] The present invention provides a text recognition method, apparatus, electronic device, and storage medium to solve the problem that manual assistance is required for text recognition of multiple images in a multi-page scene in the prior art, which is time-consuming and laborious.

[0006] The present invention provides a text recognition method, including:

[0007] Obtain a video to be recognized, and determine each key frame in the video to be recognized;

[0008] Classify the shooting states of each key frame to obtain the shooting states of each key frame;

[0009] Based on the shooting states of each key frame, integrate the candidate recognition texts in the overlapping regions of each key frame to obtain the recognition text of the video to be recognized.

[0010] According to the text recognition method provided by the present invention, the step of integrating the candidate recognition texts in the overlapping regions of each key frame based on the shooting states of each key frame to obtain the recognition text of the video to be recognized includes: [[ID=)36]]

[0011] Based on the shooting states of each key frame and the confidence levels of the candidate recognition texts in the overlapping regions of each key frame, integrate the candidate recognition texts in the overlapping regions of each key frame to obtain the recognition text of the video to be recognized.

[0012] According to a text recognition method provided by the present invention, integrating the candidate recognition texts of the overlapping regions in each key frame based on the shooting states of each key frame and the confidence levels of the candidate recognition texts of the overlapping regions in each key frame to obtain the recognition text of the video to be recognized includes:

[0013] If there is a stable key frame among the key frames including the overlapping region, determining the candidate recognition text of the overlapping region in the stable key frame as the region recognition text of the overlapping region, where the stable key frame is a key frame with a stable shooting state;

[0014] Otherwise, based on the confidence levels of the candidate recognition texts of the key frames including the overlapping region, selecting the region recognition text of the overlapping region from the candidate recognition texts of the key frames including the overlapping region;

[0015] Based on the region recognition texts of each overlapping region and the region recognition texts of the non-overlapping regions in each key frame, determining the recognition text of the video to be recognized.

[0016] According to a text recognition method provided by the present invention, classifying the shooting states of each key frame to obtain the shooting states of each key frame includes:

[0017] Classifying the shooting lengths of each key frame to obtain the shooting length categories of each key frame, where the shooting length categories are the beginning of the length, the middle of the length, the end of the length, or others;

[0018] Based on the shooting length categories of each key frame, determining the shooting states of each key frame.

[0019] According to a text recognition method provided by the present invention, determining the shooting states of each key frame based on the shooting length categories of each key frame includes:

[0020] Deleting the key frames with the shooting length category of others, and determining the shooting states of the key frames with the shooting length categories of the beginning of the length or the middle of the length as unstable states, and determining the shooting states of the key frames with the shooting length category of the end of the length as stable states.

[0021] According to a text recognition method provided by the present invention, before integrating the candidate recognition texts of the overlapping regions in each key frame based on the shooting states of each key frame, it further includes:

[0022] Based on the shooting direction of the video to be recognized, locating the text regions of each key frame, where the text regions include overlapping regions and non-overlapping regions;

[0023] Perform text recognition on the text regions of each key frame to obtain candidate recognition texts for the overlapping regions in each key frame and region recognition texts for the non-overlapping regions in each key frame.

[0024] According to a text recognition method provided by the present invention, the determination of each key frame in the video to be recognized includes:

[0025] Obtain the image features of each image frame in the video to be recognized;

[0026] Based on the similarity between the image features of each image frame, select key frames from each image frame.

[0027] The present invention also provides a text recognition device, including:

[0028] A key frame determination unit, configured to obtain a video to be recognized and determine each key frame in the video to be recognized;

[0029] A state determination unit, configured to classify the shooting states of each key frame to obtain the shooting states of each key frame;

[0030] An overlapping integration unit, configured to integrate the candidate recognition texts of the overlapping regions in each key frame based on the shooting states of each key frame to obtain the recognition text of the video to be recognized.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the text recognition method as described in any one of the above are implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the text recognition method as described in any one of the above are implemented.

[0033] The text recognition method, device, electronic device, and storage medium provided by the present invention collect the entire picture of a multi-page scenario in the form of a recorded video, and thus apply the video to be recognized to text recognition, avoiding the problem of missed collection in a multi-page scenario. Moreover, since the video to be recognized itself contains the collection order, the obtained key frames do not need to be manually sorted, reducing the manual requirements for text recognition. In addition, integrating the candidate recognition texts of the overlapping regions based on the shooting states of each key frame realizes the automatic integration of the text for the overlapping regions, ensuring the reliability of the recognition text. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly describe the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0035] Figure 1 is a schematic flowchart of the text recognition method provided by the present invention;

[0036] Figure 2 is a schematic flowchart of step 130 in the text recognition method provided by the present invention;

[0037] Figure 3 is a schematic flowchart of step 120 in the text recognition method provided by the present invention;

[0038] Figure 4 is a schematic flowchart of the text recognition method based on text regions provided by the present invention;

[0039] Figure 5 is a schematic structural diagram of the shooting length classification model provided by the present invention;

[0040] Figure 6 is a schematic structural diagram of the text recognition device provided by the present invention;

[0041] Figure 7 is a schematic structural diagram of the electronic device provided by the present invention. Detailed Embodiments

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0043] In view of some multi-length text recognition scenarios, such as text recognition scenarios for long pictures, books, etc., a single photo cannot capture the entire scene to be recognized. Related technologies usually need to take pictures in blocks to obtain a set of images of the entire scene, and then perform OCR recognition on them separately. Finally, manual sorting and processing of the recognition results are required. However, the image content obtained by the above operations may have overlapping or missing situations, which need to be manually processed after text recognition, resulting in an increase in labor costs; and for scenarios with a large number of lengths, a large number of photos are required to cover the scene content, corresponding to a large number of photos that need to be sorted and recognized in sequence during the text recognition process, resulting in low recognition efficiency.

[0044] In view of the above problems, an embodiment of the present invention provides a text recognition method, which can be used in text recognition scenarios of multiple pages. Figure 1 It is a schematic flowchart of the text recognition method provided by the present invention, as Figure 1 shown, the method includes:

[0045] Step 110, obtain a video to be recognized and determine each key frame in the video to be recognized.

[0046] Specifically, considering that it is very likely to miss the acquisition when collecting the whole picture of a multi-page scenario in the form of taking pictures in blocks, in the embodiment of the present invention, the form of recording a video is used to collect the whole picture of the multi-page scenario. Compared with the form of taking pictures in blocks that needs to be executed intermittently, the form of recording a video can achieve uninterrupted collection throughout the process, so as to ensure that there is no problem of missing collection for the multi-page scenario.

[0047] By applying the form of recording a video to collect the whole picture of the multi-page scenario, the video to be recognized can be obtained. The video to be recognized here covers all the information that needs to be recognized in the multi-page scenario. Considering that the number of image frames contained in the video to be recognized is huge, directly applying all the image frames in the video to be recognized for text recognition requires a large amount of computing resources, and the information reflected by adjacent multiple frames of images in the video to be recognized tends to be the same. Therefore, before text recognition, a key frame extraction operation can be performed on all the image frames in the video to be recognized, so as to reduce the number of image frames that need to be recognized subsequently and improve the text recognition efficiency on the premise of ensuring that the text information to be recognized is complete and without omission.

[0048] Here, determining each key frame in the video to be recognized can be achieved by comparing the similarity between each image frame in the video to be recognized. Specifically, one frame can be selected from multiple image frames with a similarity greater than or equal to a preset threshold as the key frame, and an image frame with a similarity less than the preset threshold to other image frames can be used as the key frame. Here, the calculation of the similarity between image frames can be to first extract the image features of each image frame, and then calculate the similarity between the image features. The extraction of image features can first perform grayscale processing on the image frame, and then perform discrete cosine transform (DCT) on the grayscale processed image frame to obtain the image features of the image frame. It can also be to input the image frame into a pre-trained image feature extraction model to obtain the image features of the image frame. For example, the image feature extraction model can be constructed based on convolutional neural networks (CNN). The similarity between each image feature can be achieved through similarity algorithms such as Euclidean distance, Pearson correlation coefficient, and cosine similarity. The embodiments of the present invention do not make specific limitations on this.

[0049] Step 120: Classify the shooting states of each key frame to obtain the shooting states of each key frame.

[0050] Step 130: Based on the shooting states of each key frame, integrate the candidate recognition texts in the overlapping regions of each key frame to obtain the recognition text of the video to be recognized.

[0051] Specifically, during the shooting of the video to be recognized, the state of the shooting device or the shooting object may change. For example, when shooting a long picture, the shooting device may need to move for shooting, or the long picture needs to be scrolled for display. Another example is that there is a page-turning process when shooting the content of a book. As a result, in the video to be recognized obtained by shooting, some image frames may be shot in an unstable state, and the text information displayed at this time may be ambiguous and unclear. Correspondingly, there will also be differences in the reliability of the results obtained by text recognition for key frames obtained under different shooting states. For example, key frames obtained by shooting in a stable state can usually clearly display the collected text information, and the recognition text obtained by text recognition based on this also has relatively high reliability. However, the text information displayed in key frames obtained by shooting in an unstable state may be ambiguous, and the reliability of the recognition text obtained by text recognition based on this will also be relatively low.

[0052] Therefore, after obtaining each key frame in the video to be recognized, it is necessary to classify the shooting state of each key frame. The shooting state referred to here is used to characterize whether the corresponding key frame is shot in a stable state. The shooting state classification can be achieved through a pre-trained shooting state classification model, or it can first determine whether the key frame belongs to the start, middle, end, or the process of switching between any of the multiple sections of the video, and then determine the shooting state of the key frame.

[0053] Overlapping text regions, that is, overlapping areas, may be captured between key frames. In the embodiments of the present invention, the recognized text obtained by performing text recognition on the overlapping area for different key frames is denoted as candidate recognized text. Since the shooting states of different key frames may be different, whether the display of the overlapping area is clear will directly affect the effect of text recognition. Therefore, the candidate recognized texts for the same overlapping area by different key frames may be different. In view of this situation, during the process of integrating the recognized texts of each key frame for each area, especially during the integration process for the overlapping area, since the overlapping area may correspond to multiple different candidate recognized texts, the shooting states of the key frames that make up the overlapping area can be applied to determine the recognized text of the overlapping area, thereby realizing the integration of the recognized texts of the video to be recognized.

[0054] For example, for any overlapping area, among the key frames that make up the overlapping area, if there is a key frame with a shooting state of stable state, the candidate recognized text of this key frame for this overlapping area can be used as the final recognized text of this overlapping area. If there is no key frame with a shooting state of stable state, among the candidate recognized texts of the key frames shot in non-stable states for this overlapping area, the candidate recognized text with the highest confidence in text recognition can be selected as the final recognized text, or the candidate recognized text with a confidence in text recognition higher than a preset threshold can be selected as the final recognized text.

[0055] The method provided by the embodiments of the present invention uses the form of recording a video to collect the whole picture of a multi-section scene, and thus applies the video to be recognized to text recognition, avoiding the problem of missed collection in the multi-section scene. Moreover, the video to be recognized itself contains the collection order, so the obtained key frames do not need to be sorted manually, reducing the manual requirements for text recognition. In addition, the candidate recognized texts of the overlapping area are integrated based on the shooting states of each key frame, realizing the automatic integration of the text for the overlapping area and ensuring the reliability of the recognized text.

[0056] Based on the above embodiments, step 130 includes:

[0057] Integrate the candidate recognition texts in the overlapping regions of each key frame based on the shooting state of each key frame and the confidence of the candidate recognition texts in the overlapping regions of each key frame, to obtain the recognition text of the video to be recognized.

[0058] Specifically, for the regions that overlap among the key frames, when integrating the candidate recognition texts in such overlapping regions, not only the shooting states of each key frame need to be relied on, but also the confidence of the candidate recognition texts obtained by text recognition for the overlapping regions in each key frame can be considered. The confidence here is used to reflect the reliability of text recognition, that is, the credibility of the candidate recognition texts obtained thereby.

[0059] For any key frame, the shooting state of this key frame can reflect whether the text information shown in this key frame is clear, and thus reflect whether the candidate recognition text for the overlapping region obtained by text recognition based on this is reliable, that is, analyze whether the candidate recognition text is reliable from the display characteristics of the key frame itself. And the confidence of the candidate recognition text obtained by text recognition of this key frame for the overlapping region reflects whether the candidate recognition text obtained by text recognition is reliable, that is, analyze whether the candidate recognition text is reliable from the characteristics of text recognition itself. By combining these two to analyze the reliability of the candidate recognition texts for the overlapping regions in each key frame, the relatively most reliable candidate recognition text can be selected therefrom, so as to obtain the final recognition text for this overlapping region.

[0060] For example, based on the shooting states of each key frame and the confidence of the candidate recognition texts, the candidate recognition texts of each key frame can be scored, and the final recognition text can be selected based on the scores; or the shooting state can be taken as the highest priority, and the candidate recognition text of the key frame in the stable state can be preferentially selected. If there is no key frame in the stable state, then the text with the highest confidence is selected from the candidate recognition texts of the key frames in each non - stable state as the final recognition text.

[0061] Based on any of the above embodiments, Figure 2 is a schematic flowchart of step 130 in the text recognition method provided by the present invention, as Figure 2 shown, step 130 includes:

[0062] Step 131, if there is a stable key frame among the key frames containing the overlapping region, determine the candidate recognition text of the overlapping region in the stable key frame as the regional recognition text of the overlapping region, where the stable key frame is a key frame with a shooting state of a stable state;

[0063] Step 132, otherwise, based on the confidence of the candidate recognition texts of the key frames containing the overlapping region, select the regional recognition text of the overlapping region from the candidate recognition texts of the key frames containing the overlapping region;

[0064] Step 133: Determine the recognition text of the video to be recognized based on the region recognition text of each overlapping region and the region recognition text of non-overlapping regions in each key frame.

[0065] Specifically, for the case where there are overlapping regions in multiple key frames, it can first be determined whether there is a key frame with a shooting state of a stable state, that is, whether there is a stable key frame, among the key frames containing overlapping regions:

[0066] If there is a stable key frame, the candidate recognition text for the overlapping region in the stable key frame can be directly determined as the final recognition text for the overlapping region, denoted here as the region recognition text;

[0067] If there is no stable key frame, that is, the shooting states of all key frames containing overlapping regions are non-stable states. At this time, referring to the confidence levels of the candidate recognition texts obtained by performing text recognition on the overlapping regions in each key frame, the region recognition text for the overlapping region is selected from the candidate recognition texts of each key frame. For example, the candidate recognition text with the highest confidence level can be selected, or the candidate recognition text with a confidence level greater than a preset threshold can be selected as the region recognition text.

[0068] After obtaining the region recognition text for each overlapping region respectively, the region recognition text for each overlapping region and the region recognition text obtained by performing text recognition on non-overlapping regions in each key frame can be spliced to obtain the recognition text that can reflect all text information in the multi-page scenario.

[0069] Based on any of the above embodiments, Figure 3 is a schematic flowchart of step 120 in the text recognition method provided by the present invention, as Figure 3 shown, step 120 includes:

[0070] Step 121: Classify the shooting pages of each key frame to obtain the shooting page category of each key frame, where the shooting page category is page start, page middle, page end, or others.

[0071] Specifically, by classifying the shooting length of the key frames, it can be determined at which stage of any shooting process in a multi-length scene the shooting of the key frames specifically occurs. The obtained shooting length category can be one of the beginning of the length, the middle of the length, the end of the length, and others. Here, the beginning of the length means that the shooting of the key frame occurs at the beginning stage of a length shooting, the middle of the length means that the shooting of the key frame occurs at the middle stage of a length shooting, the end of the length means that the shooting of the key frame occurs at the ending stage of a length shooting, and others mean that the shooting of the key frame occurs outside the process of a length shooting. For example, when the key frame is shot, the length shooting has not started yet or has already ended. At this time, the content displayed in the key frame is the content outside the multi-length scene, that is, noise content. Another example is that the key frame shooting occurs during the switching process between two length shootings.

[0072] The above-mentioned classification of the shooting length of the key frames can be implemented through a pre-trained classification model, or it can be to first extract the image features of the key frames and then perform the shooting length classification based on the extracted image features. The embodiments of the present invention do not make specific limitations on this. For example, the classification model can be based on a four-class CNN network architecture and trained based on the sample images with pre-marked shooting length classifications.

[0073] Step 122: Determine the shooting states of the key frames based on the shooting length categories of the key frames.

[0074] Specifically, after obtaining the shooting length categories of the key frames, the shooting states can be determined based on the shooting length categories. For example, considering that during the shooting process of a single length, the stability in the initial stage is usually poor, and as the shooting progresses, the stability tends to be better as the progress goes further. Therefore, it can be considered that the shooting state of the key frame at the beginning of the length is an unstable state, and the shooting state of the key frame at the end of the length is a stable state. Another example is that considering other states, the content included in the key frame may not be the content in the multi-length. The key frames in other states can be directly deleted, and only the key frames of the three shooting length types of the beginning of the length, the middle of the length, and the end of the length are analyzed for the shooting state.

[0075] The method provided by the embodiments of the present invention performs shooting state analysis on the basis of obtaining the shooting length types of the key frames, which helps to improve the reliability and accuracy of the shooting state analysis.

[0076] Based on any of the above embodiments, step 122 includes:

[0077] Delete the key frames with the shooting length category of others, and determine the shooting state of the key frames with the shooting length category of the beginning of the length or the middle of the length as the unstable state, and determine the shooting state of the key frames with the shooting length category of the end of the length as the stable state.

[0078] Specifically, on the premise that the shooting length types of each key frame are known, each key frame can be screened and the shooting state can be analyzed based on the shooting length types of each key frame.

[0079] Among them, for the key frames with the shooting length type of "other", considering that such key frames are mostly captured outside the process of shooting a single length, the content of multi-length scenarios may not be included in such key frames. Even if such key frames contain the content of multi-length scenarios, due to factors such as length switching, the captured information may be blurred and difficult to identify. Since the key frames with the shooting length type of "other" are useless for obtaining the text information in the video to be recognized and will instead cause additional consumption of computing resources, such key frames can be directly screened out, thereby further reducing the computing amount required for text recognition.

[0080] For the key frames with the shooting length type of "length start" or "length middle", most of such key frames are captured at the start or in the middle of shooting a single length. The start and middle stages are relatively less stable. Therefore, the shooting state of the key frames with the shooting length category of "length start" or "length middle" is determined as an unstable state.

[0081] For the key frames with the shooting length category of "length end", most of such key frames are captured at the end of shooting a single length. As the shooting process progresses, the end stage becomes more stable compared to the start and middle stages. Therefore, the shooting state of the key frames with the shooting length category of "length end" is determined as a stable state.

[0082] Based on any of the above embodiments, Figure 4 is a schematic flowchart of the text recognition method based on text regions provided by the present invention. As Figure 4 shown, before step 130 is executed, it further includes:

[0083] Step 410, based on the shooting direction of the video to be recognized, locate the text regions of each key frame, where the text regions include overlapping regions and non-overlapping regions.

[0084] Step 420, perform text recognition on the text regions of each key frame to obtain the candidate recognition text of the overlapping regions and the region recognition text of the non-overlapping regions in each key frame.

[0085] Specifically, before integrating the recognized text in each key frame, text recognition needs to be performed on each key frame, and the process of text recognition needs to be further divided into two stages: text region location and text recognition.

[0086] Considering that during the shooting process of multi-page scenarios, the distribution method of text regions in multi-page scenarios will directly affect the shooting direction of the video. For example, for a vertically long picture, the shooting direction is usually from top to bottom or from bottom to top. When a book is unfolded, two adjacent pages are unfolded side by side, and the shooting direction is usually from left to right or from right to left. Correspondingly, when positioning the text region in the key frame, the shooting direction of the video to be recognized can also be referred to.

[0087] Specifically, when positioning the text region in each key frame, the shooting direction of the video to be recognized can be referred to to determine the pixel point scanning direction when scanning and positioning the text region. For example, for a shooting direction from top to bottom or from bottom to top, the corresponding pixel point scanning direction can also be vertical scanning. For a shooting direction from left to right or from right to left, the corresponding pixel point scanning direction can also be horizontal scanning.

[0088] After determining the pixel point scanning direction, the pixel point scanning direction can be followed to determine the positions of the black pixel points in each row or each column in the pixel point scanning direction, so as to achieve the positioning of the text region. For example, the size of the key frame is x*y, and the pixel point scanning direction is horizontal scanning. By performing horizontal scanning on each row in the key frame, the number of black pixel points in each row can be obtained. Among them, the rows with the number of black pixel points significantly lower than that of most rows can form the intervals between each line of text. Specifically, when positioning, the following rules can be executed: If the number of black pixel points in the previous row is 0, then query downward to record the y coordinate of the row where the number of the first black pixel point is not 0 as the starting point of a line of text region; if the number of black pixel points in the previous row is not 0, then query downward to record the y coordinate of the row where the number of the first black pixel point is 0 as the ending point of a line of text region. Based on the adjacent starting point and ending point, the positioning of the text region can be achieved. When the pixel point scanning direction is vertical scanning, the rule for positioning the text region is similar to the rule when the pixel point scanning direction is horizontal scanning, which will not be elaborated here.

[0089] After locating the text regions of the key frames, text recognition can be performed on each text region. Specifically, each text region in the key frame can be cut, and the image of the cut text region can be input into a pre-trained recognition model for multi-classification. For example, the text region can be subdivided into character regions, and the recognition model can perform multi-classification on the image of each character region to obtain the probability that each character region belongs to each character, and then determine the character corresponding to each character region. Alternatively, the image of the text region can be directly input into an OCR model for recognition to obtain the text corresponding to the text region. In addition, after obtaining the text corresponding to the text region, for some text with a prediction probability lower than the threshold, such as several words or phrases, multiple candidate texts can be screened out and confirmed through a semantic model to obtain the final region recognition text.

[0090] It should be noted that for the text regions in a single key frame, the text region can be divided into an overlapping region or a non-overlapping region according to whether the text region exists in other key frames. Whether it is an overlapping region or a non-overlapping region, text recognition can be performed based on the above examples to obtain the recognition text of the overlapping region, that is, the candidate recognition text, or the recognition text of the non-overlapping region, that is, the region recognition text.

[0091] Based on any of the above embodiments, in step 110, determining the key frames in the video to be recognized includes:

[0092] Obtain the image features of each image frame in the video to be recognized;

[0093] Based on the similarity between the image features of the image frames, select key frames from the image frames.

[0094] Specifically, for each image frame in the video to be recognized, the extraction of its image features can first perform gray-scale processing on the image frame, and then perform discrete cosine transform DCT on the gray-scale processed image frame to obtain it. Alternatively, the image frame can be input into a pre-trained image feature extraction model to obtain the image features of the image frame. For example, the image feature extraction model can be constructed based on a convolutional neural network, and the embodiments of the present invention do not make specific limitations on this.

[0095] On this basis, the similarity between the image features of each image frame can be calculated, for example, through similarity algorithms such as Euclidean distance, Pearson correlation coefficient, and cosine similarity. After obtaining the similarity between the image features of each image frame, key frames can be selected based on this. For example, one frame can be selected as a key frame from multiple image frames with a similarity greater than or equal to a preset threshold, and an image frame with a similarity less than the preset threshold to other image frames can be used as a key frame.

[0096] The method provided by the embodiment of the present invention selects key frames based on the similarity between image features, reduces the number of image frames that need to be text-recognized subsequently, and improves the text recognition efficiency on the premise of ensuring the integrity and non-omission of the text information to be recognized.

[0097] Based on any of the above embodiments, in step 110, obtaining the image features of each image frame in the video to be recognized includes:

[0098] Based on the following formula, grayscale each image frame:

[0099] Gray = (Red + Green + Blue) / 3

[0100] In the formula, Gray is the grayscale value obtained after grayscaling, and Red, Green, and Blue are the channel values of the image frame in the three channels of the red channel, green channel, and blue channel, respectively.

[0101] After completing grayscaling, perform DCT conversion on each image frame based on the following formula, and use the obtained result as the image feature:

[0102]

[0103]

[0104] In the formula, F(u, v) is the result after conversion, N is the conversion parameter representing the block size, f(x, y) represents the grayscaled image frame, x and y are the horizontal and vertical coordinates of the pixel points in the image frame, and u and v are the feature dimensions after conversion.

[0105] On this basis, the similarity between the image features of two image frames can be determined by the following formula:

[0106]

[0107] In the formula, sim is the similarity, A i and B i respectively represent the image features of two image frames.

[0108] Based on any of the above embodiments, the text recognition method can be implemented through the following steps:

[0109] For multi-page scenarios that require text recognition, the entire picture can be collected in the form of recording a video, so as to obtain a video to be recognized that contains all the information that needs to be text-recognized in the multi-page scenario.

[0110] For the video to be recognized, each image frame in the video to be recognized can be grayscale-converted respectively, and DCT conversion is performed on each image frame after grayscale conversion, so as to obtain the image features of each image frame. On this basis, the similarity of the image features is calculated between every two image frames, and key frames are extracted from each image frame based on the similarity between the image features of every two image frames.

[0111] For each key frame, the shooting segment category of each key frame can be obtained through the shooting segment classification model. Here, the shooting segment category can be the beginning of the segment, the middle of the segment, the end of the segment, or others. For the key frames with the shooting segment type being others, they are directly screened out; for the key frames with the shooting segment type being the beginning or the middle of the segment, the shooting status of such key frames can be determined as an unstable state; for the key frames with the shooting segment category being the end of the segment, the shooting status of such key frames can be determined as a stable state. For example, Figure 5 is a schematic structural diagram of the shooting segment classification model provided by the present invention. As Figure 5 shown, the shooting segment classification model can be constructed based on the CNN model, and the convolutional pooling layers in each layer are pooled after CNN convolution, Figure 5 where the input is the key frame, and the output is the probability that the key frame belongs to each shooting segment category, Figure 5 The model shown is a four-class CNN model. For the 4 probabilities output by it, the shooting segment category with the highest probability can be selected as the shooting segment category of the key frame.

[0112] In addition, for each key frame, it is also necessary to locate the text area in the key frame and perform text recognition, so as to obtain the candidate recognition text of the overlapping area and the area recognition text of the non-overlapping area in each key frame.

[0113] In the subsequent integration process of the candidate recognition text of the overlapping area, the recognition text of the overlapping area can be corrected by the shooting status of the key frame with a later timestamp and the confidence of the candidate recognition text of the overlapping area.

[0114] Specifically, for the overlapping area that needs to be integrated currently, it can be judged whether the key frame containing the overlapping area before is in a stable state. If the key frame containing the overlapping area before is in an unstable state, it is necessary to compare the confidence of the overlapping area in the previous key frame and the current key frame. If the confidence of the overlapping area in the current key frame is greater than the confidence of the overlapping area in the previous key frame, then the candidate recognition text and confidence are replaced, otherwise the candidate recognition text and confidence of the overlapping area in the previous key frame are retained. For example, the adjustment strategies of serial numbers 2, 3, and 4 in the table.

[0115] If the previous key frame containing the overlapping area is in a stable state, the recognition result of the overlapping area will not be appended, such as the adjustment strategies with serial numbers 1 and 5 in the table.

[0116] It should be noted that the recognition results in the table are the candidate recognition texts corresponding to the key frames.

[0117]

[0118]

[0119] Based on the above table, the text recognition results in a multi-page scenario can be obtained, that is, [Region 1: Recognition result T4-1; Region 2: Recognition result T4-2; Region 3: Recognition result T3-3; Region 4: Recognition result T4-4; Region 5: Recognition result T5-5].

[0120] Based on any of the above embodiments, Figure 6 is a schematic structural diagram of the text recognition device provided by the present invention, as Figure 6 shown, the device includes:

[0121] A key frame determination unit 610, configured to obtain a video to be recognized and determine each key frame in the video to be recognized;

[0122] A state determination unit 620, configured to classify the shooting states of each key frame to obtain the shooting states of each key frame;

[0123] An overlapping integration unit 630, configured to integrate the candidate recognition texts of the overlapping areas in each key frame based on the shooting states of each key frame to obtain the recognition text of the video to be recognized.

[0124] The device provided by the embodiments of the present invention collects the overall view of a multi-page scenario in the form of a recorded video, and thus applies the video to be recognized to text recognition, avoiding the problem of missed collection in a multi-page scenario. Moreover, the video to be recognized itself contains the collection order, so there is no need to manually sort the obtained key frames, reducing the manual requirements for text recognition. In addition, integrating the candidate recognition texts of the overlapping areas based on the shooting states of each key frame realizes the automatic integration of texts for the overlapping areas, ensuring the reliability of the recognition text.

[0125] Based on any of the above embodiments, the overlapping integration unit 630 is configured to:

[0126] Integrate the candidate recognition texts of the overlapping areas in each key frame based on the shooting states of each key frame and the confidence levels of the candidate recognition texts of the overlapping areas in each key frame to obtain the recognition text of the video to be recognized.

[0127] Based on any of the above embodiments, the overlapping integration unit 630 is configured to:

[0128] If there is a stable key frame among the key frames including the overlapping region, determine the candidate recognition text of the overlapping region in the stable key frame as the region recognition text of the overlapping region, where the stable key frame is a key frame with a stable shooting state;

[0129] Otherwise, based on the confidence levels of the candidate recognition texts of the key frames including the overlapping region, select the region recognition text of the overlapping region from the candidate recognition texts of the key frames including the overlapping region;

[0130] Based on the region recognition texts of the overlapping regions and the region recognition texts of the non-overlapping regions in the key frames, determine the recognition text of the video to be recognized.

[0131] Based on any of the above embodiments, the state determination unit 620 is configured to:

[0132] Classify the shooting lengths of the key frames to obtain the shooting length categories of the key frames, where the shooting length categories are the start of the length, the middle of the length, the end of the length, or others;

[0133] Based on the shooting length categories of the key frames, determine the shooting states of the key frames.

[0134] Based on any of the above embodiments, the state determination unit 620 is configured to:

[0135] Delete the key frames with the shooting length category of others, and determine the shooting states of the key frames with the shooting length category of the start of the length or the middle of the length as the unstable state, and determine the shooting state of the key frame with the shooting length category of the end of the length as the stable state.

[0136] Based on any of the above embodiments, the apparatus further includes an identification unit, configured to:

[0137] Based on the shooting direction of the video to be recognized, locate the text regions of the key frames, where the text regions include overlapping regions and non-overlapping regions;

[0138] Perform text recognition on the text regions of the key frames to obtain the candidate recognition texts of the overlapping regions in the key frames and the region recognition texts of the non-overlapping regions in the key frames.

[0139] Based on any of the above embodiments, the key frame determination unit 610 is configured to:

[0140] Obtain the image features of the image frames in the video to be recognized;

[0141] Select key frames from the respective image frames based on the similarity between the image features of the respective image frames.

[0142] Figure 7 An exemplary schematic physical structure diagram of an electronic device is shown as Figure 7 shown. The electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute a text recognition method, which includes: obtaining a video to be recognized and determining key frames in the video to be recognized; classifying the shooting states of the key frames to obtain the shooting states of the key frames; based on the shooting states of the key frames, integrating candidate recognition texts in overlapping regions of the key frames to obtain the recognition text of the video to be recognized.

[0143] In addition, when the logic instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0144] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the text recognition method provided by the above-mentioned various methods. The method includes: obtaining a video to be recognized and determining key frames in the video to be recognized; classifying the shooting states of the key frames to obtain the shooting states of the key frames; based on the shooting states of the key frames, integrating candidate recognition texts in overlapping regions of the key frames to obtain the recognition text of the video to be recognized.

[0145] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the text recognition method provided above. The method includes: obtaining a video to be recognized and determining key frames in the video to be recognized; classifying the shooting states of each key frame to obtain the shooting state of each key frame; and integrating candidate recognition texts in overlapping regions of each key frame based on the shooting states of each key frame to obtain the recognition text of the video to be recognized.

[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0147] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text recognition method, characterized in that, Including: Obtain the video to be recognized, and determine each key frame in the video to be recognized, where the video to be recognized is obtained by comprehensively collecting multiple scene sections in the form of an application recording video; Classify the shooting states of each key frame to obtain the shooting states of each key frame, where the shooting state is used to represent whether the key frame is obtained under a stable state; Based on the shooting states of each key frame and the confidence levels of the candidate recognition texts in the overlapping regions of each key frame, integrate the candidate recognition texts in the overlapping regions of each key frame to obtain the recognition text of the video to be recognized, including: If there is a stable key frame among the key frames including the overlapping region, determine the candidate recognition text of the overlapping region in the stable key frame as the regional recognition text of the overlapping region, where the stable key frame is a key frame with a shooting state of a stable state; Otherwise, based on the confidence levels of the candidate recognition texts of the key frames including the overlapping region, select the regional recognition text of the overlapping region from the candidate recognition texts of the key frames including the overlapping region; Based on the regional recognition texts of each overlapping region and the regional recognition texts of the non-overlapping regions of each key frame, determine the recognition text of the video to be recognized.

2. The text recognition method according to claim 1, characterized in that, The classifying the shooting states of each key frame to obtain the shooting states of each key frame includes: Classify the shooting sections of each key frame to obtain the shooting section categories of each key frame, where the shooting section category is section start, section middle, section end, or other; Based on the shooting section categories of each key frame, determine the shooting states of each key frame.

3. The text recognition method according to claim 2, characterized in that The determining the shooting states of each key frame based on the shooting section categories of each key frame includes: Delete the key frames with the shooting section category of other, and determine the shooting states of the key frames with the shooting section category of section start or section middle as non-stable states, and determine the shooting state of the key frame with the shooting section category of section end as a stable state.

4. The text recognition method according to any one of claims 1 to 3, characterized in that, Before the integrating the candidate recognition texts in the overlapping regions of each key frame based on the shooting states of each key frame, it further includes: Based on the shooting direction of the video to be recognized, locate the text regions of each key frame, where the text regions include overlapping regions and non-overlapping regions; Perform text recognition on the text regions of each key frame to obtain the candidate recognition texts of the overlapping regions in each key frame and the regional recognition texts of the non-overlapping regions of each key frame.

5. The text recognition method according to any one of claims 1 to 3, characterized in that, The determining each key frame in the video to be recognized includes: Obtain the image features of each image frame in the video to be recognized; Based on the similarity between the image features of each image frame, select key frames from each image frame.

6. A text recognition device, characterized in that, Including: A key frame determination unit, configured to obtain the video to be recognized and determine each key frame in the video to be recognized, where the video to be recognized is obtained by comprehensively collecting multiple scene sections in the form of an application recording video; A state determination unit, configured to classify the shooting states of each key frame to obtain the shooting states of each key frame, where the shooting state is used to represent whether the key frame is obtained under a stable state; An overlapping integration unit, configured to integrate the candidate recognition texts of the overlapping regions in each key frame based on the shooting states of the key frames and the confidence levels of the candidate recognition texts of the overlapping regions in each key frame, so as to obtain the recognition text of the video to be recognized; if there are stable key frames among the key frames including the overlapping regions, determining the candidate recognition text of the overlapping region in the stable key frame as the regional recognition text of the overlapping region, where the stable key frame is a key frame with a stable shooting state; otherwise, based on the confidence levels of the candidate recognition texts of the key frames including the overlapping regions, selecting the regional recognition text of the overlapping region from the candidate recognition texts of the key frames including the overlapping regions; determining the recognition text of the video to be recognized based on the regional recognition texts of the overlapping regions and the regional recognition texts of the non-overlapping regions in each key frame.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the text recognition method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the text recognition method according to any one of claims 1 to 5 are implemented.

Images