OCR system performance evaluation method and apparatus, device, and readable storage medium

Abstract

The present application relates to an OCR system performance evaluation method and apparatus, a device, and a readable storage medium. The method comprises: inputting a plurality of video frames into an OCR system to obtain ND character recognition targets; on the basis of a character labeling result for each video frame, obtaining NG character labeling targets; matching the ND character recognition targets with the NG character labeling targets to obtain Nmapped matching pairs; calculating a first recognition precision parameter between the character recognition target and the character labeling target in each matching pair; and, according to each first recognition precision parameter, evaluating the performance of the OCR system.

Description

Performance evaluation method, device, equipment and readable storage medium of OCR system

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 202310943429.3, filed on July 28, 2023, entitled “Performance evaluation method, device, apparatus and readable storage medium for OCR system,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of optical character recognition technology, and in particular to a performance evaluation method, apparatus, device, and readable storage medium for an OCR system. Background Art

[0004] Optical Character Recognition (OCR) is a key research topic in the field of computer vision and holds immense value in real-world production environments. With the advancement and widespread adoption of artificial intelligence (AI), OCR technology is being applied in an increasing number of scenarios. The use of OCR for text recognition is no longer limited to high-definition images; many scenarios that record video data are also beginning to incorporate OCR technology for intelligent transformation, such as video text retrieval and video text information verification. However, the development of a technology cannot be separated from a comprehensive evaluation system. It is the tireless pursuit of OCR evaluation metrics by experts that has enabled the continuous iteration and development of OCR technology.

[0005] Currently, OCR technology evaluation metrics are limited to images and target specific, single subtasks such as OCR detection or recognition. However, in real-world applications, systematic, business-focused OCR systems are often used, resulting in a lack of evaluation methods for OCR systems in video scenarios.

[0006] Summary of the Invention

[0007] According to various embodiments disclosed in the present application, a method, apparatus, device, and readable storage medium for evaluating the performance of an OCR system are provided.

[0008] A performance evaluation method for an OCR system includes:

[0009] Input multiple video frames into the OCR system for text recognition and obtain N Dcharacter recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer;

[0010] Based on the text annotation results for each video frame, we get N G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N G is a positive integer;

[0011] N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G );

[0012] Calculating a first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair; and

[0013] The performance of the OCR system is evaluated according to each of the first recognition accuracy parameters.

[0014] A performance evaluation device for an OCR system includes:

[0015] Recognition module, used to input multiple video frames into the OCR system for text recognition, and obtain N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer;

[0016] The annotation module is used to obtain N based on the text annotation results for each video frame. G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, NG is a positive integer;

[0017] Matching module, used to N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G );

[0018] a calculation module, configured to calculate a first recognition accuracy parameter between a text recognition target and a text annotation target in each matching pair; and

[0019] An evaluation module is configured to evaluate the performance of the OCR system according to each of the first recognition accuracy parameters.

[0020] A computer device includes a memory and one or more processors, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processors, the one or more processors perform the following steps:

[0021] Input multiple video frames into the OCR system for text recognition and obtain N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer;

[0022] Based on the text annotation results for each video frame, we get N G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N G is a positive integer;

[0023] N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G );

[0024] Calculating a first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair; and

[0025] The performance of the OCR system is evaluated according to each of the first recognition accuracy parameters.

[0026] One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps:

[0027] Input multiple video frames into the OCR system for text recognition and obtain N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer;

[0028] Based on the text annotation results for each video frame, we get N G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N G is a positive integer;

[0029] N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G );

[0030] Calculating a first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair; and

[0031] The performance of the OCR system is evaluated according to each of the first recognition accuracy parameters.

[0032] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0034] FIG1 is a schematic flow chart of a method for evaluating the performance of an OCR system according to one or more embodiments;

[0035] FIG2 is a flow chart of a method for matching a text recognition target with a text annotation target according to one or more embodiments;

[0036] FIG3 is a flow chart of a method for calculating text content matching according to one or more embodiments;

[0037] FIG4 is a schematic flow chart of a method for evaluating the performance of an OCR system according to one or more embodiments;

[0038] FIG5 is a structural block diagram of a performance evaluation device for an OCR system according to one or more embodiments;

[0039] FIG6 is a structural block diagram of an electronic device according to one or more embodiments. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] In this application, reference is made to FIG1 , which is a flow chart of a first embodiment of a method for evaluating the performance of an OCR system provided by an embodiment of the present application. The execution subject of the embodiment of the present application can be any electronic device with processing capabilities. The performance evaluation method of the OCR system provided by the embodiment may include the following steps:

[0042] S101, multiple video frames are input into the OCR system for text recognition to obtain N D A text recognition target.

[0043] Wherein, each text recognition target corresponds to at least one first text area and the first text content in the first text area, the first text contents corresponding to each text recognition target are the same, and the first text areas corresponding to each text recognition target are within the preset area error range, N D Is a positive integer.

[0044] Specifically, a video can be captured by an image capture device, and multiple video frames can be determined from the video; the multiple video frames obtained are input into an OCR system for text recognition to obtain a text recognition result corresponding to each video frame, wherein each text recognition result may include a first text area and the first text content within the first text area; and from these text recognition results, texts with the same first text content and whose first text area is within a preset area error range are determined as the same text recognition target.

[0045] Specifically, since there may be many identical shooting scenes between the video frames in the same video, and if the text recognition results of different video frames corresponding to the same shooting scene are different, it can reflect the performance of the OCR system. Therefore, by determining the text with the same first text content in the text recognition results of different video frames and the first text area within the preset area error range as the same text recognition target, the text recognition results of the same shooting scene can be grouped together, so that the grouped text recognition results can be compared with the text annotation results later.

[0046] For example, suppose that after the OCR system performs text recognition on video frame 5, it is determined that the text recognition result corresponding to video frame 5 includes "beautiful flowers" (first text content) located at position 1 (first text area) and "blue sky and white clouds" located at position 2; after the OCR system performs text recognition on video frame 7, it is determined that the text recognition result corresponding to video frame 7 includes "beautiful flowers" located at a position close to position 1 (i.e., within the preset regional error range of position 1); after the OCR system performs text recognition on video frame 9, it is determined that the text recognition result corresponding to video frame 9 includes "beautiful flowers" located at a position close to position 1 (i.e., within the preset regional error range of position 1) and "blue sky and white clouds" located at a position close to position 2 (i.e., within the preset regional error range of position 2); after the OCR system performs text recognition on video frame 10, it is determined that the text recognition result corresponding to video frame 10 includes "blue sky and white clouds" located at a position close to position 2 (i.e., within the preset regional error range of position 2).

[0047] Then, the "beautiful flowers" in the text recognition results corresponding to video frame 5, the "beautiful flowers" in the text recognition results corresponding to video frame 7, and the "beautiful flowers" in the text recognition results corresponding to video frame 9 can be determined as the same text recognition target "beautiful flowers" (text recognition target 1); at the same time, the "blue sky and white clouds" in the text recognition results corresponding to video frame 5, the "blue sky and white clouds" in the text recognition results corresponding to video frame 9, and the "blue sky and white clouds" in the text recognition results corresponding to video frame 10 can also be determined as the same text recognition target "blue sky and white clouds" (text recognition target 2).

[0048] Therefore, text recognition target 1 corresponds to: "beautiful flowers" at position 1 in video frame 5, "beautiful flowers" at a position close to position 1 in video frame 7, and "beautiful flowers" at a position close to position 1 in video frame 9; text recognition target 2 corresponds to: "blue sky and white clouds" at position 2 in video frame 5, "blue sky and white clouds" at a position close to position 2 in video frame 9, and "blue sky and white clouds" at a position close to position 2 in video frame 10.

[0049] S102, based on the text annotation results for each video frame, obtain N G A text-labeled target.

[0050] Among them, each text annotation target corresponds to at least one second text area and the second text content in the second text area, the second text content corresponding to each text annotation target is the same, and the second text area corresponding to each text annotation target is within the preset area error range, N G Is a positive integer.

[0051] Specifically, after determining multiple video frames, each video frame can be annotated using an annotation tool or manual annotation to obtain a text annotation result corresponding to each video frame, wherein each text annotation result may include a second text area and a second text content within the second text area; and from these text annotation results, the text with the same second text content and the second text area within the preset area error range is determined as the same text annotation target.

[0052] For example, assuming that after text annotation is performed on video frame 5, it is determined that the text annotation results corresponding to video frame 5 include "beautiful flowers" (second text content) located at position 1 (second text area), "blue sky and white clouds" located at position 2, and "distant house" located at position 3; after text annotation is performed on video frame 7, it is determined that the text annotation results corresponding to video frame 7 include "beautiful flowers" located at a position close to position 1 (i.e., located within the preset regional error range of position 1) and "distant house" located at a position close to position 3 (i.e., located within the preset regional error range of position 3); after text annotation is performed on video frame 9, it is determined that the text annotation results corresponding to video frame 9 include "beautiful flowers" located at a position close to position 1 (i.e., located within the preset regional error range of position 1) and "blue sky and white clouds" located at a position close to position 2 (i.e., located within the preset regional error range of position 2); after text annotation is performed on video frame 10, it is determined that the text annotation results corresponding to video frame 10 include "blue sky and white clouds" located at a position close to position 2 (i.e., located within the preset regional error range of position 2).

[0053] Then, the "beautiful flowers" in the text annotation results corresponding to video frame 5, the "beautiful flowers" in the text annotation results corresponding to video frame 7, and the "beautiful flowers" in the text annotation results corresponding to video frame 9 can be determined as the same text annotation target "beautiful flowers" (text annotation target 1); the "blue sky and white clouds" in the text annotation results corresponding to video frame 5, the "blue sky and white clouds" in the text annotation results corresponding to video frame 9, and the "blue sky and white clouds" in the text annotation results corresponding to video frame 10 can also be determined as the same text annotation target "blue sky and white clouds" (text annotation target 2); the "house in the distance" in the text annotation results corresponding to video frame 5 and the "house in the distance" in the text annotation results corresponding to video frame 7 can also be determined as the same text annotation target "house in the distance" (text annotation target 3).

[0054] Therefore, text annotation target 1 corresponds to: "beautiful flowers" at position 1 in video frame 5, "beautiful flowers" at a position close to position 1 in video frame 7, and "beautiful flowers" at a position close to position 1 in video frame 9; text annotation target 2 corresponds to: "blue sky and white clouds" at position 2 in video frame 5, "blue sky and white clouds" at a position close to position 2 in video frame 9, and "blue sky and white clouds" at a position close to position 2 in video frame 10; text annotation target 3 corresponds to: "distant house" at position 3 in video frame 5 and "distant house" at a position close to position 3 in video frame 7.

[0055] S103, N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs.

[0056] Among them, each matching pair includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G ).

[0057] Specifically, the N obtained in S101 can be D character recognition targets and N obtained by S102 G The text annotation targets are matched so that the text recognition target and the text annotation target can achieve double matching of text content and text position. Among them, since there may be text recognition targets that cannot match the text annotation targets, N mapped Less than or equal to min(N D , N G ).

[0058] S104: Calculate a first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair.

[0059] Specifically, since each text recognition target corresponds to the text recognition result of at least one video frame, and each text annotation target corresponds to the text annotation result of at least one video frame, by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, the recognition accuracy of each text recognition target in the corresponding different video frames can be calculated, that is, the cross-frame detection and recognition accuracy of each text recognition target can be calculated.

[0060] S105 : Evaluate the performance of the OCR system according to each first recognition accuracy parameter.

[0061] Specifically, the text recognition performance of the OCR system can be evaluated by the cross-frame detection and recognition accuracy of each text recognition target obtained by performing text recognition on the OCR system.

[0062] The performance evaluation method of the OCR system provided in the embodiment of the present application is to input multiple video frames into the OCR system for text recognition to obtain N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer; based on the text annotation results for each video frame, we get N G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N G is a positive integer; N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, where each matching pair includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G); calculate the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair; and evaluate the performance of the OCR system based on each first recognition accuracy parameter. Thus, the text recognition results and the text annotation results of the same shooting scene in the video can be respectively collected, and then the collected text recognition results and the text annotation results can be matched to achieve text recognition and text tracking; by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, and evaluating the performance of the OCR system based on each first recognition accuracy parameter, text detection for missed detection, false detection, and multiple detection is achieved, thereby achieving high-precision evaluation of the OCR system in video scenarios by disassembling the text detection, text tracking, and text recognition processes.

[0063] In some embodiments, the above N D character recognition targets and N G Match the text annotation targets and get N mapped Matching pairs can include: calculating each character recognition target and N G The distance between the text-labeled targets is matched according to the distance to obtain N mapped matching pairs.

[0064] Specifically, the distance between the text recognition target and the text annotation target can represent the correlation between the text recognition target and the text annotation target. If the correlation is large, the text recognition target and the text annotation target can be matched to obtain N mapped matching pairs.

[0065] Specifically, assuming that 20 text recognition targets (text recognition target 1, text recognition target 2...text recognition target 20) and 25 text annotation targets (text annotation target 1, text annotation target 2...text annotation target 25) are obtained, the distance between each text recognition target and the 25 text annotation targets is calculated respectively, that is: the distance between text recognition target 1 and text annotation target 1, the distance between text recognition target 1 and text annotation target 2...the distance between text recognition target 1 and text annotation target 25 is calculated respectively, the distance between text recognition target 2 and text annotation target 1, the distance between text recognition target 2 and text annotation target 2...the distance between text recognition target 2 and text annotation target 25 is calculated respectively...the distance between text recognition target 20 and text annotation target 1, the distance between text recognition target 20 and text annotation target 2...the distance between text recognition target 20 and text annotation target 25 is calculated respectively.

[0066] In the embodiment of the present application, each character recognition target and N are calculated separately. GThe distance between the text-labeled targets is matched according to the distance to obtain N mapped Thus, the correlation between the text recognition target and the text annotation target can be determined based on the distance between them, and the text recognition target and the text annotation target corresponding to the greater correlation can be matched to obtain a matching pair, thereby improving the accuracy of text tracking.

[0067] In some embodiments, the above calculations are performed for each character recognition target and N G The distance between the text annotation targets may include: for each video frame, calculating the overlap between the first text area corresponding to the text recognition target j and the second text area corresponding to the text annotation target i, where j = 1, 2...N D , i=1,2…N G The accumulated overlap between the first text area corresponding to the text recognition target j and the second text area corresponding to the text annotation target i in each video frame is used as the distance between the text recognition target j and the text annotation target i.

[0068] Specifically, the text area overlap between the text recognition target and the text annotation target can represent the distance between the text recognition target and the text annotation target. Therefore, the text area overlap between the text recognition target and the text annotation target can also represent the correlation between the text recognition target and the text annotation target. If the correlation is large, the text recognition target and the text annotation target can be matched to obtain a matching pair.

[0069] Specifically, the degree of text region overlap between the text recognition target and the text annotation target in each video frame can be determined by calculating the degree of overlap between a first text region corresponding to the text recognition target in each video frame and a second text region corresponding to the text annotation target in the same video frame. The calculated text region overlaps between the same text recognition target and the text annotation target in all video frames are then accumulated to obtain the degree of overlap between each text recognition target and all text annotation targets.

[0070] Specifically, in order to reduce the amount of calculation, it can also be determined in advance whether the first text area corresponding to the text recognition target in each video frame overlaps with the second text area corresponding to the text annotation target in the video frame. If so, the intersection-and-union ratio (degree of overlap) between the overlapping first text area and the second text area is calculated, and the intersection-and-union ratios between the same text recognition targets and text annotation targets in all video frames are accumulated to obtain the degree of overlap between each text recognition target and all text annotation targets.

[0071] In the embodiment of the present application, the overlap between the first text area corresponding to the text recognition target j and the second text area corresponding to the text annotation target i is calculated for each video frame, where j = 1, 2...N D , i=1,2…N G The accumulated overlap between the first text region corresponding to the text recognition target j and the second text region corresponding to the text annotation target i in each video frame is used as the distance between the text recognition target j and the text annotation target i. Thus, the correlation between the text recognition target and the text annotation target can be determined based on the overlap between the text recognition target and the text annotation target. The text recognition target and the text annotation target corresponding to the greater correlation are then matched to obtain a matching pair, thereby improving the accuracy of text tracking.

[0072] Specifically, the Hungarian algorithm can be used to calculate the distance to obtain a matching pair with a high matching degree. For example, the size can be defined as (N G , N D ) of a matrix M, any element M ij Represents the distance between the i-th text annotation target and the j-th text recognition target. Through the matching method shown in Figure 2, the values ​​of each element in the matrix M are calculated and used as the basis for matching the text recognition target with the text annotation target. As shown in Figure 2, Figure 2 is a matching method for text recognition targets and text annotation targets provided in an embodiment of the present application. The matching method for text recognition targets and text annotation targets provided in this embodiment can include the following steps:

[0073] S201, initializing the value of each element in the matrix M to 0.

[0074] S202, traversing the video starting from the first video frame.

[0075] S203: Determine whether there is an overlapping first text area and second text area.

[0076] If yes, execute S204; if no, execute S206.

[0077] S204: Calculate the intersection-over-union ratio between the first and second overlapping text regions.

[0078] S205, corresponding M ij The intersection-and-union ratios are accumulated.

[0079] S206: Determine whether it is the last video frame.

[0080] If yes, execute S207; if no, execute S208.

[0081] S207: Using the Hungarian algorithm to perform matching calculation on the distance, thereby obtaining a matching pair.

[0082] S208, take the next video frame.

[0083] Specifically, after executing S208 , the process returns to executing S203 .

[0084] In the embodiment of the present application, through the above-mentioned matching method of the text recognition target and the text annotation target, the Hungarian algorithm can be used to perform matching calculation on the distance, thereby obtaining a matching pair with a higher matching comparison.

[0085] In some embodiments, the calculation of the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair may include: determining at least one target video frame in which the text recognition target in matching pair k matches the text annotation target in matching pair k, where k = 1, 2...N mapped ; For each target video frame, calculate the product of the degree of overlap between the first text area corresponding to the text recognition target in matching pair k and the second text area corresponding to the text annotation target in matching pair k, and the degree of match between the first text content corresponding to the text recognition target in matching pair k and the second text content corresponding to the text annotation target in matching pair k; based on each product and the number of target video frames, calculate the first recognition accuracy parameter between the text recognition target and the text annotation target in matching pair k.

[0086] Specifically, based on the above text recognition and text annotation results, it can be determined that the text recognition target in each matching pair matches at least one target video frame of the text annotated target in the matching pair. It can also be determined based on the above matching process that the text recognition target in each matching pair matches at least one target video frame of the text annotated target in the matching pair. For example, assuming that matching pair 2 includes text recognition target 2 and text annotated target 4, it can be determined that text recognition target 2 matches text annotated target 4 in video frames 1, 10, and 20.

[0087] Specifically, for each target video frame, the degree of overlap between the first text area corresponding to the text recognition target in matching pair k and the second text area corresponding to the text annotation target in matching pair k can characterize the positional correlation between the text recognition target and the text annotation target in matching pair k in each target video frame.

[0088] Moreover, for each target video frame, the matching degree between the first text content corresponding to the text recognition target in matching pair k and the second text content corresponding to the text annotation target in matching pair k can represent the recognition accuracy of the text content of the text recognition target and the text annotation target in matching pair k in each target video frame.

[0089] Therefore, based on the product of the degree of overlap between the first text area corresponding to the text recognition target in the matching pair k in each target video frame and the second text area corresponding to the text annotation target in the matching pair k, and the degree of match between the first text content corresponding to the text recognition target in the matching pair k in each target video frame and the second text content corresponding to the text annotation target in the matching pair k, the position correlation between the text recognition target and the text annotation target in the matching pair k in each target video frame and the recognition accuracy of the text content can be obtained simultaneously.

[0090] Specifically, based on the above product corresponding to each target video frame and the number of target video frames, the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k can be calculated. The first recognition accuracy parameter can characterize the ability of the OCR system to recognize each text recognition target.

[0091] Specifically, for each target video frame, the second text content corresponding to the text annotation target in matching pair k can be used as a reference, and the first text content corresponding to the text recognition target in matching pair k can be compared with the second text content, and the degree of match between the text contents can be determined based on the comparison result. In a specific implementation, the comparison can determine whether there are missing characters, over-recognized characters, or misrecognized characters in the first text content. The degree of match between the text contents can be determined based on at least one of the missing characters, over-recognized characters, and misrecognized characters.

[0092] For example, the text content matching degree calculation method shown in FIG3 can be used to obtain the matching degree between the first text content corresponding to the text recognition target in the matching pair k and the second text content corresponding to the text annotation target in the matching pair k in each target video frame. The text content matching degree calculation method provided in this embodiment may include the following steps:

[0093] S301 , for each target video frame, taking the second text content corresponding to the text annotation target in matching pair k as a benchmark, and comparing it with the first text content corresponding to the text recognition target in matching pair k.

[0094] S302: Determine whether there are any missing characters in the first text content.

[0095] If yes, execute S303; if no, execute S304.

[0096] S303: Determine that the number of missing characters is A.

[0097] Wherein, A is an integer greater than 0.

[0098] S304: Determine that the number of missing characters is 0.

[0099] Specifically, after executing S303 and S304, S305 is executed.

[0100] S305: Determine whether there are multiple recognized characters in the first text content.

[0101] If yes, execute S306; if no, execute S307.

[0102] S306: Determine the number of over-recognized characters as D.

[0103] Wherein, D is an integer greater than 0.

[0104] S307: Determine that the number of over-recognized characters is 0.

[0105] Specifically, after executing S306 and S307, S308 is executed.

[0106] S308: Determine whether there are any misrecognized characters in the first text content.

[0107] If yes, execute S309; ​​if no, execute S310.

[0108] S309: Determine the number of misrecognized characters as F.

[0109] Wherein, F is an integer greater than 0.

[0110] S310: Determine that the number of misrecognized characters is 0.

[0111] Specifically, after executing S309 and S310, S311 is executed.

[0112] S311, calculating the number of characters of the second text content as S.

[0113] Wherein, S is an integer greater than 0.

[0114] S312: Determine whether the sum of the number of missing characters, over-recognized characters, and incorrectly recognized characters is less than the number of characters in the second text content.

[0115] If yes, execute S313; if no, execute S314.

[0116] S313, calculate the matching degree

[0117] S314: Calculate the matching degree to be 0.

[0118] In the embodiment of the present application, it is determined that the text recognition target in the matching pair k matches at least one target video frame of the text annotation target in the matching pair k, where k=1, 2...N mappedFor each target video frame, calculate the product of the degree of overlap between the first text region corresponding to the text recognition target in matching pair k and the second text region corresponding to the text annotation target in matching pair k, and the degree of match between the first text content corresponding to the text recognition target in matching pair k and the second text content corresponding to the text annotation target in matching pair k; calculate the first recognition accuracy parameter between the text recognition target and the text annotation target in matching pair k based on each product and the number of target video frames. In this way, the recognition accuracy of each text recognition target in the corresponding target video frames can be calculated, that is, the cross-frame detection and recognition accuracy of each text recognition target can be calculated.

[0119] In some embodiments, calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k based on each product and the number of target video frames may include: calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k based on a first formula, wherein the first formula is:

[0120] Among them, SingleAcc k is the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k, The number of target video frames that match the text recognition target in matching pair k to the text annotation target in matching pair k, To match the first text region corresponding to the text recognition target in k in the target video frame t, To match the second text area corresponding to the text annotation target in k in the target video frame t, is the matching degree between the first text content corresponding to the text recognition target and the second text content corresponding to the text annotation target in the matching pair k in the target video frame t.

[0121] In an embodiment of the present application, the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k is calculated by the above-mentioned first formula. Thus, the recognition accuracy of each text recognition target in the corresponding target video frames can be calculated relatively accurately by the above-mentioned first formula, that is, the accuracy of cross-frame detection and recognition of each text recognition target can be calculated.

[0122] In some embodiments, the above-mentioned evaluation of the performance of the OCR system based on each first recognition accuracy parameter may include: calculating a second recognition accuracy parameter of the OCR system based on each first recognition accuracy parameter; and evaluating the performance of the OCR system based on the second recognition accuracy parameter.

[0123] Specifically, the second recognition accuracy parameter representing the text recognition performance of the OCR system can be calculated by counting the cross-frame recognition accuracy parameters of each text recognition target obtained through the OCR system.

[0124] In the embodiment of the present application, a second recognition accuracy parameter of the OCR system is calculated using each first recognition accuracy parameter; the performance of the OCR system is evaluated based on the second recognition accuracy parameter. Thus, by statistically analyzing the cross-frame recognition accuracy parameters of each text recognition target obtained by the OCR system, a second recognition accuracy parameter representing the text recognition performance of the OCR system can be calculated, thereby achieving high-precision evaluation of the OCR system in video scenarios.

[0125] In some embodiments, calculating the second recognition accuracy parameter of the OCR system according to each first recognition accuracy parameter may include: calculating the second recognition accuracy parameter of the OCR system according to a second formula, wherein the second formula is:

[0126] Among them, TotalAcc is the second recognition accuracy parameter.

[0127] Specifically, the second recognition accuracy parameter of the OCR system is obtained by accumulating the first recognition accuracy parameter between the text recognition targets and the text annotation targets in each matching pair, and then dividing it by the average of the number of text recognition targets and the number of text annotation targets. In the above second formula, the numerator can reflect the detection and recognition accuracy of the matching targets. The higher the detection and recognition accuracy value, the higher the overall score of the OCR system, indicating better recognition performance of the OCR system. However, when multiple detections and missed detections occur, the numerator does not change, but the denominator increases significantly, which will result in a decrease in the overall score of the OCR system.

[0128] Specifically, the performance level of the OCR system and the corresponding relationship between the performance level of the OCR system and the second recognition accuracy parameter of the OCR system can be preset. Then, after obtaining the second recognition accuracy parameter of the OCR system, the current performance level of the OCR system can be determined based on the above corresponding relationship.

[0129] Specifically, the second recognition accuracy parameter can also be compared with a preset threshold. If the second recognition accuracy parameter is greater than or equal to the preset threshold, the performance of the OCR system is determined to be "passing"; if the second recognition accuracy parameter is less than the preset threshold, the performance of the OCR system is determined to be "failed".

[0130] Specifically, the second recognition accuracy parameter of the current OCR system and the second recognition accuracy parameter of the improved OCR system may be calculated and compared to determine whether the text recognition performance of the improved OCR system is improved.

[0131] Specifically, the second recognition accuracy parameters of different OCR systems may be calculated and compared to determine which OCR system has better text recognition performance.

[0132] In the embodiment of the present application, the second recognition accuracy parameter of the OCR system is calculated using the second formula. Thus, by decomposing the text detection, text tracking, and text recognition processes, a high-precision evaluation based on the OCR system in a video scenario can be achieved.

[0133] As a specific example, FIG4 is a flow chart of a second embodiment of a performance evaluation method for an OCR system provided in an embodiment of the present application. As shown in FIG4 , the performance evaluation method for an OCR system provided in this embodiment may include the following steps:

[0134] S401, multiple video frames are input into the OCR system for text recognition to obtain N D text recognition targets and a text detection frame in at least one video frame corresponding to each text recognition target and first text content in the text detection frame.

[0135] The above-mentioned text detection frame is the above-mentioned first text area.

[0136] S402: Based on the text annotation results for each video frame, obtain N G text annotation targets and a text annotation frame in at least one video frame corresponding to each text annotation target and second text content in the text annotation frame.

[0137] Among them, the above-mentioned text annotation box is the aforementioned second text area.

[0138] S403, calculate each character recognition target and N G The distance between the text-annotated targets is calculated by using the Hungarian algorithm to match the distance, and the text recognition target and the text-annotated target are matched to obtain N mapped matching pairs.

[0139] Specifically, the matching method shown in FIG2 can be used to calculate the value of each element in the matrix M, and use it as a basis for matching the text recognition target with the text annotation target.

[0140] S404: Determine whether the text recognition target in each matching pair matches at least one target video frame of the text annotation target in the matching pair.

[0141] S405 , calculating a first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair using a first formula.

[0142] Among them, through the first formula, the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair is calculated based on the degree of overlap between the text detection box corresponding to the text recognition target in each matching pair and the text annotation box corresponding to the text annotation target in the matching pair, and the degree of matching between the first text content corresponding to the text recognition target in each matching pair and the second text content corresponding to the text annotation target in each matching pair.

[0143] S406 , calculating a second recognition accuracy parameter of the OCR system according to each first recognition accuracy parameter using a second formula.

[0144] S407: Evaluate the performance of the OCR system according to the second recognition accuracy parameter.

[0145] In an embodiment of the present application, the text recognition results of the same shooting picture in the video and the text annotation results of the same shooting picture in the video can be respectively collected, and then the collected text recognition results and the text annotation results are matched to achieve text recognition and text tracking; by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, and evaluating the performance of the OCR system according to each first recognition accuracy parameter, text detection for missed detection, false detection, and multiple detection is achieved, and thus by disassembling the text detection, text tracking and text recognition processes, high-precision evaluation based on the OCR system in the video scene is achieved.

[0146] FIG5 is a schematic diagram of the structure of a performance evaluation device for an OCR system provided in an embodiment of the present application. As shown in FIG5 , the performance evaluation device 50 for an OCR system may include: a recognition module 510 , a labeling module 520 , a matching module 530 , a calculation module 540 , and an evaluation module 550 .

[0147] The recognition module 510 can be used to input multiple video frames into the OCR system for text recognition to obtain N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer; the annotation module 520 can be used to obtain N based on the text annotation results for each video frame G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N Gis a positive integer; the matching module 530 can be used to D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, where each matching pair includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G ); a calculation module 540, which can be used to calculate a first recognition accuracy parameter between a text recognition target and a text annotation target in each matching pair; an evaluation module 550, which can be used to evaluate the performance of the OCR system based on each first recognition accuracy parameter.

[0148] The performance evaluation device of the OCR system provided in the embodiment of the present application inputs multiple video frames into the OCR system for text recognition through the recognition module, and obtains N D character recognition targets, wherein each character recognition target corresponds to at least one first character region and a first character content within the first character region, the first character contents corresponding to each character recognition target are the same, and the first character regions corresponding to each character recognition target are within a preset regional error range, N D is a positive integer; through the annotation module, based on the text annotation results for each video frame, N is obtained G text annotation targets, wherein each text annotation target corresponds to at least one second text area and second text content within the second text area, the second text contents corresponding to each text annotation target are the same, and the second text areas corresponding to each text annotation target are within the preset area error range, N G is a positive integer; through the matching module, N D character recognition targets and N G Match the text annotation targets and get N mapped matching pairs, where each matching pair includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G); by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair; by using the evaluation module, the performance of the OCR system is evaluated based on each first recognition accuracy parameter. In this way, the text recognition results of the same shot in the video and the text annotation results of the same shot in the video can be respectively collected, and then the collected text recognition results and the text annotation results are matched to achieve text recognition and text tracking; by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, and evaluating the performance of the OCR system based on each first recognition accuracy parameter, text detection for missed detection, false detection, and multiple detection is achieved, thereby achieving high-precision evaluation of the OCR system in the video scene by disassembling the text detection, text tracking, and text recognition processes.

[0149] In addition, corresponding to the performance evaluation method of the OCR system provided in the above embodiment, the embodiment of the present application also provides an electronic device, as shown in Figure 6, the electronic device 60 may include: a memory 610, a processor 620, and computer-readable instructions stored on the memory 610 and executable on the processor 620. When the processor 620 executes the computer-readable instructions, all steps of the performance evaluation method of the OCR system provided in the embodiment of the present application are implemented.

[0150] In the above-mentioned electronic device, the text recognition results of the same shooting picture in the video and the text annotation results of the same shooting picture in the video can be respectively collected together, and then the collected text recognition results and the text annotation results are matched to realize text recognition and text tracking; by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, and evaluating the performance of the OCR system according to each first recognition accuracy parameter, the text detection of missed detection, false detection and multiple detection is realized, and thus by disassembling the text detection, text tracking and text recognition processes, high-precision evaluation based on the OCR system in the video scene is realized.

[0151] In addition, corresponding to the performance evaluation method of the OCR system provided in the above embodiment, the embodiment of the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor, all steps of the performance evaluation method of the OCR system in the embodiment of the present application are implemented.

[0152] In the above-mentioned computer-readable storage medium, the text recognition results of the same shooting picture in the video and the text annotation results of the same shooting picture in the video can be respectively collected, and then the collected text recognition results and the text annotation results are matched to realize text recognition and text tracking; by calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in each matching pair, and evaluating the performance of the OCR system according to each first recognition accuracy parameter, the text detection of missed detection, false detection and multiple detection is realized, and thus by disassembling the text detection, text tracking and text recognition processes, high-precision evaluation based on the OCR system in the video scene is realized.

[0153] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit instructions for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the computer-readable instructions are printed, since the computer-readable instructions may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or otherwise processing in a suitable manner as necessary, and then stored in a computer memory.

[0154] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A performance evaluation method for an OCR system, comprising: Input multiple video frames into the OCR system for text recognition and obtain N D text recognition targets, wherein each text recognition target corresponds to at least one first text area and a first text content within the first text area, each first text content corresponding to each text recognition target is the same, and each first text area corresponding to each text recognition target is within a preset area error range, N D is a positive integer; Based on the text annotation results for each video frame, we get N G text annotation targets, wherein each text annotation target corresponds to at least one second text area and a second text content in the second text area, each second text content corresponding to each text annotation target is the same, and each second text area corresponding to each text annotation target is within a preset area error range, N G is a positive integer; N D text recognition targets and N G Match the text annotation targets to get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G );and Calculating a first recognition accuracy parameter between a text recognition target and a text annotation target in each of the matching pairs; The performance of the OCR system is evaluated according to each of the first recognition accuracy parameters.

2. The method according to claim 1, wherein: The N D text recognition targets and N G Match the text annotation targets to get N mapped Matching pairs, including: Calculate each text recognition target and N separately G The distance between the text-annotated targets is calculated, and matching is performed based on the distance to obtain N mapped Matching pairs.

3. The method according to claim 2, wherein: The calculation of each character recognition target and N G The distance between text-annotated targets, including: For each video frame, calculate the overlap between the first text area corresponding to the text recognition target j and the second text area corresponding to the text annotation target i, where j = 1, 2...N D , i=1,2…N G ;and The accumulated overlap between the first text area corresponding to the text recognition target j and the second text area corresponding to the text annotation target i in each video frame is used as the distance between the text recognition target j and the text annotation target i.

4. The method according to claim 3, wherein: The Hungarian algorithm is used to perform matching calculation on the distance.

5. The method according to any one of claims 1 to 4, wherein: The calculating of a first recognition accuracy parameter between a text recognition target and a text annotation target in each matching pair includes: Determine that the text recognition target in the matching pair k matches at least one target video frame of the text annotation target in the matching pair k, where k=1, 2...N mapped ; For each target video frame, calculating the product of the overlap between the first text region corresponding to the text recognition target in the matching pair k and the second text region corresponding to the text annotation target in the matching pair k and the matching between the first text content corresponding to the text recognition target in the matching pair k and the second text content corresponding to the text annotation target in the matching pair k; and A first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k is calculated based on each product and the number of target video frames.

6. The method according to claim 5, wherein: The step of calculating the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k according to the products and the number of target video frames includes: According to the first formula, the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k is calculated, where: The first formula is: Among them, SingleAcc k is the first recognition accuracy parameter between the text recognition target and the text annotation target in the matching pair k, The number of target video frames that match the text recognition target in matching pair k to the text annotation target in matching pair k, To match the first text region corresponding to the text recognition target in k in the target video frame t, To match the second text area corresponding to the text annotation target in k in the target video frame t, is the degree of matching between the first text content corresponding to the text recognition target in the match pair k and the second text content corresponding to the text annotation target in the target video frame t.

7. The method according to claim 6, wherein: The evaluating the performance of the OCR system according to each of the first recognition accuracy parameters includes: Calculating a second recognition accuracy parameter of the OCR system according to each of the first recognition accuracy parameters; and The performance of the OCR system is evaluated according to the second recognition accuracy parameter.

8. The method according to claim 7, wherein: The calculating, according to each of the first recognition accuracy parameters, a second recognition accuracy parameter of the OCR system comprises: According to the second formula, a second recognition accuracy parameter of the OCR system is calculated, wherein the second formula is: Among them, TotalAcc is the second recognition accuracy parameter.

9. A performance evaluation device for an OCR system, comprising: The recognition module is used to input multiple video frames into the OCR system for text recognition and obtain N D text recognition targets, wherein each text recognition target corresponds to at least one first text area and a first text content within the first text area, each first text content corresponding to each text recognition target is the same, and each first text area corresponding to each text recognition target is within a preset area error range, N D is a positive integer; The annotation module is used to obtain N based on the text annotation results for each video frame. G text annotation targets, wherein each text annotation target corresponds to at least one second text area and a second text content in the second text area, each second text content corresponding to each text annotation target is the same, and each second text area corresponding to each text annotation target is within a preset area error range, N G is a positive integer; Matching module, used to N D text recognition targets and N G Match the text annotation targets to get N mapped matching pairs, each of which includes a text recognition target and a text annotation target, N mapped Less than or equal to min(N D , N G ); A calculation module, used to calculate a first recognition accuracy parameter between a text recognition target and a text annotation target in each matching pair; and An evaluation module is used to evaluate the performance of the OCR system according to each of the first recognition accuracy parameters.

10. An electronic device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images