Video-based document multi-row header character recognition and matching method and system

Abstract

The invention provides a video-based document multi-row header character recognition and matching method and system, and the method comprises the steps: carrying out the target recognition of a document title text in a video, and obtaining a title textbox; a cross-frame tracking mechanism is adopted, target tracking is conducted on the title textbox, in each frame, the newly detected title textbox is matched with an existing tracking box, position information, state information and tracking duration of the successfully matched tracking box are updated, and the unsuccessfully matched title textbox serves as a new tracking target. If the tracking duration exceeds a preset threshold value, the tracking target is taken as an effective target; performing text recognition on the effective target to obtain a single-line text; and performing multi-line text matching on the single-line text according to the textbox detection information to obtain a multi-line title text. According to the invention, efficient and automatic detection and management of the compliance of the document in the business process video are realized, and the problems of low efficiency and error proneness of traditional manual examination are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a video-based document multi-line header text recognition and matching method and system, as well as a corresponding computer terminal and computer-readable storage medium. Background Art

[0002] In the field of administrative business, document compliance detection is crucial. Video cloud is a system solution for a network application system based on cloud computing. The solution uses audio and video signals as information carriers. End users obtain the required services from the cloud through the network in an on-demand and easily scalable manner. It has now become an important service means in the administrative approval process. Business process videos need to contain specific document materials and stay in the picture for a certain period of time to ensure compliance. However, in the existing technology, compliance detection mainly relies on manual review of document materials in the video, which is not only inefficient but also prone to human errors. In addition, traditional methods lack effective automation means when processing video content, and it is difficult to ensure that the time that document materials stay in the video meets the requirements, thus affecting the compliance and effectiveness of business processes.

[0003] After searching, we found:

[0004] The Chinese invention patent application with publication number CN112766246A, "Document title recognition method, system, terminal and medium based on deep learning", inputs the image document into a pre-trained deep neural network model to obtain the text line position information and confidence information in the image document; constructs the structural features of the image document based on the text line position information; adds the image document name and whether the text line is a title to the structural features to form a training sample to train the machine learning model and obtain a machine learning classification model; obtains the structural features of the image document to be recognized, and inputs the features into the classification model to output the text line position information where the image document title is located and the confidence information that the line is a title. The document title recognition technology still has the following technical problems:

[0005] This document title recognition technology requires text detection on the entire text of the material, not just the title. It needs to be combined with the determination of the text type, that is, the title text is determined by whether each text line is a title line. This method results in significant deficiencies in recognition accuracy and timeliness.

[0006] This document title recognition technology does not take into account the continuity of document materials and cannot meet the requirements for document compliance testing.

[0007] The video may contain multiple documents at the same time. The documents may appear in the picture at different angles or with partial occlusion. It is necessary to identify multiple document materials in the picture at the same time. However, the document title recognition technology does not have the ability to handle complex situations and cannot truly meet the recognition of specific document materials contained in the business process video. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for recognizing and matching multi-line header characters in a document based on video, and also provides a corresponding computer terminal and computer-readable storage medium.

[0009] According to one aspect of the present invention, a method for recognizing and matching multi-line header characters of a document based on video is provided, comprising:

[0010] Performing target recognition on the document title text in the video to obtain a title text box; assigning a unique ID to the title text box, and recording the initial frame ID and position information of the title text box;

[0011] Using a cross-frame tracking mechanism, the title text box is tracked, and the successfully tracked title text box is used as a tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information;

[0012] In each frame, the newly detected title text box is matched with the existing tracking target, and the position information, state information and tracking duration of the successfully matched tracking target are updated, and the unsuccessfully matched title text box is used as the new tracking target;

[0013] Determine the tracking time of the tracking target, and if the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target;

[0014] Performing text recognition on the valid target to obtain a single line of text;

[0015] Multi-line text matching is performed on the single-line text according to the text box position information of the single-line text to obtain multi-line title text.

[0016] Preferably, the target recognition of the document title text in the video to obtain the title text box includes:

[0017] Decoding the video to obtain image frames, and preprocessing the image frames;

[0018] A pre-trained YOLO_OBB model is provided, and the pre-processed image frame is used as input. The pre-trained YOLO_OBB model is used for forward propagation to identify the document title text in the image frame, obtain the position coordinates and confidence of the rotated text detection box, and obtain the title text box.

[0019] Preferably, the cross-frame tracking mechanism is used to track the title text box, and the successfully tracked title text box is used as the tracking target, wherein each tracking target is configured with a unique ID and initial location information and status information, including:

[0020] A tracker is initialized, and the title text box is tracked by the tracker through a cross-frame tracking mechanism to obtain a successfully tracked title text box in each frame, obtain a tracking target, and generate a tracking frame for each tracking target.

[0021] Preferably, in each frame, the newly detected title text box is matched with the existing tracking target, and the position information, state information and tracking duration of the successfully matched tracking target are updated, and the unsuccessfully matched title text box is used as a new tracking target, including:

[0022] The intersection-and-union ratio matching algorithm is used to obtain the regional overlap between the current frame title text box and the existing tracking target; when the regional overlap is greater than or equal to the set threshold, it indicates a successful match, and the position information, status information and tracking duration of the successfully matched tracking target are updated; when the regional overlap is less than the set threshold, it indicates an unsuccessful match, and the unsuccessfully matched title text box is used as a new tracking target.

[0023] Preferably, performing text recognition on the valid target to obtain a single line of text includes:

[0024] Preprocessing the effective target to obtain a target image;

[0025] A pre-trained OCR text recognition model is provided, the target image is used as the input of the pre-trained OCR text recognition model, and forward propagation is performed through the pre-trained OCR text recognition model to obtain a text recognition result, which is a single line of text.

[0026] Preferably, performing multi-line text matching on the single-line text according to the text box position information of the single-line text to obtain multi-line title text includes:

[0027] According to the text box position information of the single-line text, the text box angle difference and the center point distance of two adjacent single-line texts are obtained, the text box angle difference and the center point distance are judged, and the text contents of the two adjacent single-line texts that meet the preset splicing conditions are spliced ​​forward and backward to form a candidate title text;

[0028] Using an edit distance algorithm, similarity calculation is performed between the candidate title text and the predefined template title;

[0029] When the similarity between the candidate title text and the predefined template title exceeds a set threshold, the match is determined to be successful, and the matching multiple lines of title text information are recorded.

[0030] According to another aspect of the present invention, a video-based document multi-line header text recognition and matching system is provided, comprising:

[0031] A target recognition module, which is used to perform target recognition on the document title text in the video to obtain a title text box; assign a unique ID to the title text box, and record the initial frame ID and position information of the title text box;

[0032] A target tracking module, which uses a cross-frame tracking mechanism to track the title text box, and uses the successfully tracked title text box as a tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information;

[0033] Tracking and matching module, which is used to match the newly detected title text box with the existing tracking target in each frame, and update the position information, state information and tracking duration of the successfully matched tracking target, and use the unsuccessfully matched title text box as the new tracking target;

[0034] An effective target determination module, which is used to determine the tracking time of the tracking target. If the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target;

[0035] The title text recognition module is used to perform text recognition on the valid target to obtain a single line of text; perform multi-line text matching on the single line of text according to the text box position information of the single line of text to obtain multi-line title text.

[0036] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0037] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein a processor is used to execute the method described in any one of the above-mentioned methods of the present invention, or to run the system described in the above-mentioned system of the present invention.

[0038] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0039] The video-based document multi-line header text recognition and matching method and system provided by the present invention can automatically extract the document title text appearing in the video, accurately identify and match it with a predefined template, ensure the compliance display time of the document in the video, and realize efficient and automated detection and management of document compliance in business process videos, successfully solving the problems of low efficiency and error-proneness of traditional manual review.

[0040] The video-based document multi-line header text recognition and matching method and system provided by the present invention not only significantly improves the efficiency and accuracy of compliance detection through automated processing, reduces the workload of manual processing, but also enhances the reliability of audit archiving by automatically generating evidence images.

[0041] The video-based document multi-line header text recognition and matching method and system provided by the present invention automatically extracts the title text content appearing in the video and matches it with a predefined template, ensuring that the display time of the document material in the video is compliant, and automatically generates evidence images for review and archiving, which significantly improves the efficiency and accuracy of compliance detection, reduces manual workload, and enhances the reliability and security of business processes.

[0042] The video-based document multi-line header text recognition and matching method and system provided by the present invention not only optimizes the security and stability of business processes, but also provides a reliable solution for document compliance checking in administrative business.

[0043] The video-based document multi-line header text recognition and matching method and system provided by the present invention directly treat the single-line title as a conventional image recognition target for target detection, annotate massive title texts, use the YOLO_OBB model to achieve rotation target detection, perform potential layout analysis, distinguish between titles and texts, and have higher work efficiency.

[0044] The video-based document multi-line header text recognition and matching method and system provided by the present invention fully consider the continuity of document materials, that is, the document materials need to appear in the video and appear continuously for a specified time. Through title text tracking and matching and tracking duration judgment, an effective tracking target is obtained to meet the document compliance detection requirements.

[0045] The video-based document multi-line header text recognition and matching method and system provided by the present invention can effectively handle the complex situation that multiple documents may be simultaneously included in the video screen and the documents may appear in the screen at different angles, partially blocked, etc., to meet the needs of simultaneously identifying multiple document materials in the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 The present invention is a flowchart of a video-based document multi-line header text recognition and matching method in a preferred embodiment of the present invention.

[0048] Figure 2 A schematic diagram of the work of obtaining a title text box in a preferred embodiment of the present invention.

[0049] Figure 3 This is a workflow diagram for target tracking in a preferred embodiment of the present invention.

[0050] Figure 4 This is a workflow diagram for matching multiple lines of text using a single line of text in a preferred embodiment of the present invention.

[0051] Figure 5 The figure is a schematic diagram of the component modules of a video-based document multi-line header text recognition and matching system in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0053] Video cloud has become an important service in the administrative approval process. However, in existing technologies, compliance detection mainly relies on manual review of document materials in videos, which is not only inefficient but also prone to human errors. In addition, traditional methods lack effective automation when processing video content, making it difficult to ensure that the time that document materials stay in the video meets the requirements, thus affecting the compliance and effectiveness of business processes.

[0054] In response to the above problems, an embodiment of the present invention provides a video-based document multi-line header text recognition and matching method. The method automatically extracts and recognizes title text content from the video through automated processing technologies such as title text detection, text box tracking, text recognition and matching, and matches it with a predefined template to ensure that the display time of the document material in the video meets compliance requirements. Compared with traditional methods, the method significantly improves the efficiency and accuracy of compliance detection, reduces manual workload, and enhances the reliability and security of business processes.

[0055] Specifically, the video-based document multi-line header text recognition and matching method provided in this embodiment may include the following operations:

[0056] S1, target recognition is performed on the document title text in the video to obtain the title text box; a unique ID is assigned to the title text box, and its initial frame ID and position information (i.e., position coordinates) are recorded;

[0057] S2, using the cross-frame tracking mechanism to track the title text box, and taking the successfully tracked title text box as the tracking target, where each tracking target is configured with a unique ID and initial location information and status information;

[0058] S3, in each frame, the newly detected title text box is matched with the existing tracking target, and the position information, state information and tracking duration of the successfully matched tracking target are updated. The unsuccessfully matched title text box will become a new tracking target;

[0059] S4, judging according to the tracking time of the tracking target; if the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target;

[0060] S5, perform text recognition on the valid target to obtain a single line of text;

[0061] S6, performing multi-line text matching on the single-line text according to the text box position information of the single-line text to obtain multi-line title text.

[0062] The technical solutions provided by the above embodiments of the present invention are further described in detail below in conjunction with preferred embodiments.

[0063] The above embodiment of the present invention provides a video-based document multi-line header text recognition and matching method, such as Figure 1 As shown, the following steps are included:

[0064] S1, target recognition is performed on the document title text in the video to obtain the title text box; a unique ID is assigned to the title text box, and its initial frame ID and position information (i.e., position coordinates) are recorded. Specifically:

[0065] After the system is started, it is initialized, including loading the detection model and setting the interactive interface. After the user uploads the video file that needs to be checked for compliance, the compliance check starts and the video frame is decoded. If the video frame is decoded successfully, the document title text in each frame image is recognized through the target detection model to obtain the title text box. Each detected title text box is assigned a unique ID, and its initial frame ID and location information are recorded.

[0066] S2, using the cross-frame tracking mechanism, tracks the title text box, and uses the successfully tracked title text box as the tracking target, where each tracking target is configured with a unique ID and initial location information and status information. Specifically:

[0067] The cross-frame tracking mechanism is used to track the title text box. All successfully tracked title text boxes are used as tracking targets (Track), and each Track is configured with a unique ID, initial position information and status information.

[0068] S3, in each frame, matches the newly detected title text box with the existing tracking target, and updates the position information, state information and tracking duration of the successfully matched tracking target, and the unsuccessfully matched title text box will become the new tracking target. Specifically:

[0069] In each frame, the newly detected title text box is matched with the existing tracking target. When the match is successful, the location information, state information and tracking duration of the tracking target are updated; if the match is unsuccessful or there is no existing tracking target, a new tracking target is initialized and used as the new tracking object.

[0070] S4, judging according to the tracking time of the tracking target; if the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target. Specifically:

[0071] The judgment is made based on the tracking duration of each tracking target. If the tracking duration of a tracking target exceeds the preset threshold, it is considered a valid target. For all tracking targets currently in the MISSING state, the system adds 1 to their missing count; if the missing count exceeds the preset maximum missing count, the tracking target is deleted to avoid continuing to process invalid targets.

[0072] S5, perform text recognition on the valid target to obtain a single line of text. Specifically:

[0073] For the tracking target that is determined to be a valid target, the region (ROI) containing the title text is extracted from the original video frame using its stored text box position information, and text recognition is performed. After the text is successfully recognized, the state of the tracking target is updated to the LABELED state, indicating that the text recognition is completed.

[0074] S6, performing multi-line text matching on the single-line text according to the text box position information of the single-line text to obtain multi-line title text. Specifically:

[0075] Utilize all tracking targets in the LABELED state to match multi-line texts based on their text box position information. By identifying text boxes that are approximately parallel and adjacent to each other, determine whether they constitute potential multi-line title texts. All single-line recognized texts and potential multi-line title texts are organized into a set of texts to be matched. Each element in the set of texts to be matched is matched with the material name in the predefined template for similarity, and the edit distance is calculated. If the edit distance is less than the preset threshold, the match is considered successful, and the successfully matched tracking target is updated to the MATCHED state, indicating that the material meets the compliance requirements.

[0076] Finally, check whether the user chooses to exit. If the user chooses to exit, stop the video analysis; if not, show the user all the matching information and give the compliance check conclusion.

[0077] In some preferred implementations, the above S1, performing target recognition on the document title text in the video to obtain the title text box, may further include the following operations:

[0078] S11, decoding the video, obtaining image frames, and preprocessing the image frames;

[0079] S12, providing a pre-trained YOLO_OBB model, taking the pre-processed image frame as input, performing forward propagation through the pre-trained YOLO_OBB model, identifying the document title text in the image frame, obtaining the position coordinates and confidence of the rotated text detection box, and obtaining the title text box.

[0080] S1 is mainly used to detect single-line title text. Commonly used text detectors usually perform text detection on the entire text of the material, not just the title. It is also necessary to combine the judgment of the text type (that is, whether each text line is a title line) to determine the title text. S1 directly treats the single-line title as a regular image recognition target for target detection (considering multi-line titles, the entire title can also be used as a detection target, but the subsequent text recognition steps will not be able to proceed normally because the text recognition model usually recognizes single-line text). The data set preparation process before training collected a variety of pictures containing document materials to ensure that the scenes of the data set are as rich as possible and improve the generalization performance of the model. In order to facilitate subsequent text recognition (the corrected text recognition accuracy is higher), a rotated target detection model (YOLO_OBB) is adopted. The rotating target box can provide angle information, so that text correction can be performed. Figure 2 shown.

[0081] In some preferred implementations, the above S11, preprocessing the image frame, may further include: resizing and format conversion;

[0082] In some preferred implementations, the above S12, providing a pre-trained YOLO_OBB model, may further include the following operations:

[0083] S121, obtaining a diverse document dataset including documents with and without headers, and annotating them to obtain a sample set;

[0084] S122, performing data enhancement processing on the sample set to obtain training samples;

[0085] S123, training a YOLO_OBB model using the training samples to obtain a pre-trained YOLO_OBB model.

[0086] The above S1 is the step of detecting the title text. This step uses the YOLO_OBB model to achieve the goal of accurately identifying the document title text in the video. Traditional text detection methods usually recognize the entire text and cannot effectively distinguish and extract the title part of the document. Accurately detecting the document title is one of the key steps in multi-line header text recognition and matching technology.

[0087] To address the above issues, during the model training phase, a variety of document datasets were used for annotation, including document samples with and without headers. Special attention was paid to the precise annotation of the rotated target box to ensure that the model can accurately recognize the title text at various document rotation angles. Data augmentation techniques include image rotation, scaling, and cropping, aiming to improve the robustness and generalization ability of the model.

[0088] In the actual processing flow, the trained YOLO_OBB model goes through the following core steps: first, the decoded image frame is preprocessed, including size adjustment and format conversion, to meet the input requirements of the YOLO_OBB model; then the model is inferred, that is, the preprocessed image is input into the YOLO_OBB model for forward propagation to obtain the position coordinates and confidence of the rotated text detection box.

[0089] In some preferred implementations, the above S2 uses a cross-frame tracking mechanism to track the title text box, and uses the successfully tracked title text box as the tracking target, wherein each tracking target is configured with a unique ID and initial location information and status information, and may further include the following operations:

[0090] Initialize a tracker, use the tracker, and track the title text box through a cross-frame tracking mechanism to obtain the successfully tracked title text box in each frame, obtain the tracking target, and generate a tracking box for each tracking target.

[0091] The status information includes the target's starting detection frame, tracking length, and current tracking status (tracking, tracking lost, and recognized, i.e., the current target, i.e., the tracking frame, has undergone text recognition). The starting detection frame and tracking length can be used to locate the time in the video when the material appears, and to determine the moment it appears and disappears in the video. The tracking status indicates that the target is being continuously tracked; the tracking lost indicates that the target is temporarily lost (possibly due to missed detection or occlusion, etc., and if the tracking is lost for a long time, the tracking is exited), i.e., no title text box that can match the original tracked target appears in the newly appeared frame; the recognized indicates that the tracking duration of the tracked target has reached the shortest continuous duration and the title text has been recognized.

[0092] The compliance check requirements include: (1) the specified material must appear in the video, and the duration of continuous appearance must meet the requirements; (2) if the duration meets the requirements, the start and end time of the material appearing in the video can be obtained. Therefore, the continuous appearance time of the material in the video can be determined by tracking the title text box. Each title detection box that appears in each frame of the video must be tracked. If the title detection box can match the existing tracked title box, the tracked title box is updated, otherwise a new tracked title box is initialized. The location information is recorded to track the title box; the initial frame ID is recorded to obtain the time when the material starts to appear in the video; the tracking length is recorded to obtain the time when the material last appears in the video; the status information is recorded to determine the current processing progress. The status includes (TRACKING, MISSING, LABELED, MATCHED). TRACKING means that the current tracker is in normal tracking state, MISSING means that the target is lost, LABELED indicates that the text box has been recognized, and MATCHED indicates that the title text in the text box matches the material name in the predefined template and no further processing is required. Existing target tracking methods only focus on the spatial position and motion trajectory of the target, but do not include state information management. The method provided by the embodiment of the present invention not only tracks the target, but also performs text content recognition and state management specifically for the title box. It provides more detailed state information and more accurate tracking by recording the state of each title box (such as TRACKING, MISSING, LABELED, MATCHED). By recording the initial frame ID and tracking length of the title box, the continuous appearance duration of the material in the video can be accurately calculated.

[0093] The number of title text boxes can be 0 or more. Each frame in the video is tested to detect the title text boxes in it. Since the business requires the specified material to appear, it must appear continuously for a certain period of time to be compliant. The title text box is used as the basis for target tracking. By tracking and detecting the title text box, the tracking time of the tracked text box, that is, the duration, is recorded. Only when the duration reaches the required time (that is, the shortest continuous time that the material appears in the video), the title text is recognized.

[0094] In some preferred implementations, the above S3, in each frame, matches the newly detected title text box with the existing tracking target, and updates the position information, state information and tracking duration of the successfully matched tracking target, and the unsuccessfully matched title text box becomes a new tracking target, and can further include the following operations:

[0095] The intersection-and-union ratio matching algorithm is adopted to obtain the regional overlap between the title text box of the current frame and the tracking box of the existing tracking target; when the regional overlap is greater than or equal to the set threshold, it indicates a successful match, and the tracking box of the successfully matched tracking target updates its position information, state information and tracking duration; when the regional overlap is less than the set threshold, it indicates an unsuccessful match, and the unsuccessfully matched title text box is used as a new tracking target.

[0096] For the title text box that appears in each frame, if no matching tracking target is found, it will automatically become a new tracking target. For example, for the first frame detection, all the title text boxes that appear are new tracking targets. For subsequent frame detections, if the detected title text box matches the tracking target, the tracking box corresponding to the tracking target is updated, otherwise the detected title text box becomes the new tracking target.

[0097] In some preferred implementations, the above S4, in the process of acquiring the valid target, may further include the following operations:

[0098] Check all tracking targets and remove those that do not meet the tracking time or are invalid.

[0099] S3 is mainly designed to achieve stable and fast text box tracking, ensuring that every text box target that appears in the video frame can be reliably tracked. General target tracking usually uses Kalman filtering, which assumes that the system dynamics are linear and may not perform well when dealing with complex or nonlinear motion. For rotating text boxes or targets with significant angle changes, this linear assumption may lead to inaccurate tracking.

[0100] In the video document recognition process, the main difficulties are: (1) judging the validity of the text box and avoiding unnecessary processing during long video processing; (2) in real scenes, the text box may not be detected temporarily due to occlusion, fast movement or other interference factors.

[0101] In order to solve the above difficulties, the method provided by the above embodiment of the present invention adopts the following technical means: (1) Setting a duration threshold to ensure that only text boxes that appear stably enter the subsequent processing stage, thereby improving the efficiency and processing accuracy of the system. Regularly check the status of the tracking frame and remove the invalid tracking frame to ensure the efficient use of system resources and the validity of the tracking frame list. (2) Set max_miss_count in the MISSING state, which is used to define the maximum number of frames that a tracking frame can lose in the MISSING state. If the tracking frame fails to match the corresponding text frame in consecutive max_miss_count frames, the system will decide to terminate further tracking and processing of the tracking frame. By setting max_miss_count, the system can tolerate a certain number of lost frames, thereby improving the robustness and stability of tracking. By limiting the number of consecutive frames lost, text box tracking failures caused by short-term losses or detection errors can be reduced.

[0102] The above steps S2 to S4 are steps of target tracking of the title text box. In a preferred embodiment, the specific process of target tracking is as follows: Figure 3 As shown, the following steps are included:

[0103] First, based on S1, the current frame is decoded to obtain an image frame and preprocessed. Then, the preprocessed image frame is input into the pre-trained YOLO_OBB model to identify the document title in the image frame, obtain the position coordinates and confidence of the rotated text detection box, and thus obtain the title text box.

[0104] After obtaining the title text box, create or update the tracking object, namely "Track". Each Track represents the tracking track of a title text box, including its unique identifier, current position information, status information (such as TRACKING, MISSING, etc.), tracking length and other properties. First check whether there is an unprocessed Track. If so, process each unprocessed Track in turn.

[0105] When processing each Track, initialize the current maximum intersection-over-union (IOU) to 0, and initialize the title detection box ID corresponding to the maximum IOU to -1. Then, traverse all unmatched title detection boxes in the current frame, and calculate the IOU (intersection-over-union) between each title detection box and the tracking box represented by the Track. If the IOU of a title detection box is greater than the preset minimum matching threshold and greater than the current maximum IOU value, update the current maximum IOU value and record the corresponding title detection box ID.

[0106] After traversing all unmatched title detection boxes, check the current maximum IOU value. If the maximum IOU is still 0, it means that no title detection box matching the Track is found. The state of the Track is marked as MISSING, and its tracking loss count is increased by 1. If the maximum IOU is not 0, the title detection box corresponding to the maximum IOU is marked as matched to prevent other Tracks from matching it again. Then, the information of the detection box (such as position, tracking length, etc.) is used to update the relevant properties of the current Track. If the Track was previously in the MISSING state, its loss count is reset to 0. Finally, the state of the current Track is updated to TRACKING, indicating normal tracking; if the state of the Track is other (such as LABELED or MATCHED), the state is no longer updated.

[0107] When all existing Tracks are processed, all unmatched title detection frames are traversed, and a new Track is created for each unmatched title detection frame in turn, and its position information and status information are initialized.

[0108] Traverse all Tracks in the MISSING state. If the tracking loss count of a Track exceeds the preset maximum allowed number of losses, the target of the title text box is considered lost and the Track is deleted. At this point, the target tracking step of the title text box is completed.

[0109] In this step, based on the fact that S1 has already used the YOLO_OBB model to detect rotated text boxes, each detected text box is assigned a unique identifier in the initial state, and its initial frame ID and position coordinates are recorded.

[0110] Further, based on the text boxes detected by S1, the system initializes a tracker. This means that each detected text box will be tracked, and each tracking target has its own unique ID, as well as its initial position and state information. Subsequently, the continuity and stability of the text box are maintained through a cross-frame tracking mechanism. In each frame, the system attempts to match the newly detected text box with the existing tracking box. This matching relies on the intersection over union (IoU), which is the area overlap measure between the current frame detection box and the existing tracking box. The successfully matched tracking box will have its position, state, and tracking duration updated to maintain its continuity in the video.

[0111] In addition, the duration of the document title in the video is determined based on the trajectory information of the text box. If a text box continues to appear in the video for more than a preset threshold (for example, 10 seconds), the text box is considered a valid target. This duration judgment ensures that only text boxes that appear stably will enter the subsequent text recognition or other processing stages, thereby effectively improving the operating efficiency. Finally, the algorithm regularly checks all tracking boxes and removes those that do not meet the tracking time or are invalid. This step ensures the validity of the tracking list and effectively manages system resources to keep the system running efficiently.

[0112] In some preferred implementations, the above S5, performing text recognition on the valid target to obtain a single line of text, may further include the following operations:

[0113] S51, preprocessing the effective target to obtain a target image;

[0114] S52, providing a pre-trained OCR text recognition model, taking the target image as the input of the pre-trained OCR text recognition model, performing forward propagation through the pre-trained OCR text recognition model, and obtaining a text recognition result, which is a single line of text.

[0115] In some preferred implementations, the above S51, pre-processing the valid targets, may further include the following operations:

[0116] S511, calculating the width and height of the text box according to the text box position coordinates of the valid target, determining affine transformation parameters, and acquiring a corrected title text image from the original video frame based on the affine transformation parameters;

[0117] S512, processing the corrected title text image, the processing including: scaling and normalization.

[0118] The corrected title text image can be obtained by performing an affine transformation on the original video frame using the affine transformation matrix. The affine transformation matrix is ​​obtained by using the getAffineTransform function provided by opencv given the original point set and the target point set. The original point set is the coordinates of the four corner points of the text detection box, and the target point set is a rectangle with (0,0) as the starting point and the length and width of the minimum circumscribed rectangle of the text detection box as the length and width.

[0119] The above S5 is the step of title text recognition, which recognizes stably appearing text boxes based on OCR technology, including but not limited to PaddleOCR. Specifically. In the model training stage, we first use the tools provided by PaddleOCR to synthesize a large number of title text datasets, covering text images of various fonts and rotation angles. Through data enhancement techniques such as rotation, scaling, and noise addition, the training dataset is further enriched to improve the generalization ability and recognition accuracy of the model in various scenarios. Then use the dataset to fine-tune the PaddleOCR v3 Chinese pre-trained model to ensure high-precision recognition of document title content.

[0120] In the actual processing flow, the trained OCR model goes through the following core steps: First, perform necessary text correction based on the position of the stable text box. Specifically, first calculate the width and height of the text box based on the text box coordinates, determine the affine transformation parameters, and apply the affine transformation to obtain the corrected title text image from the original image; then perform necessary preprocessing such as scaling and normalization on the corrected image to meet the model input requirements; then perform model inference, that is, input the preprocessed image into the OCR model for forward propagation to obtain the recognition result.

[0121] In some preferred implementations, the above S6, performing multi-line text matching on a single-line text according to the text box detection information to obtain multi-line title text, may further include the following operations:

[0122] S61, according to the text box position information of the single-line text, obtain the text box angle difference and center point distance of two adjacent single-line texts, judge the text box angle difference and center point distance, and splice the text contents of two adjacent single-line texts that meet the preset splicing conditions forward and backward to form a candidate title text; for example, 2024 Financial Report and Company: XYZ Co., Ltd., the candidate title text for forward splicing is: "2024 Financial Report Company: XYZ Co., Ltd.", and the candidate title text for reverse splicing is: "Company: XYZ Co., Ltd. 2024 Financial Report", then no matter which order is used when entering the full name of the material in the predefined template, it can be correctly identified (for example, enter according to "XYZ Co., Ltd. 2024 Financial Report"). If the full name of the material is entered strictly in the title order, it is unnecessary to identify them separately after forward and reverse splicing;

[0123] S62, using an edit distance algorithm to calculate the similarity between the candidate title text and the predefined template title;

[0124] S63, when the similarity between the candidate title text and the predefined template title exceeds the set threshold, the match is determined to be successful, and the matching multiple lines of title text information are recorded. Further, the multiple lines of title text information include: text content, similarity score and video frame number when matching.

[0125] In some preferred embodiments, the preset splicing conditions in S61 include:

[0126] The distance between the center points of the text boxes of two adjacent single-line texts is less than the set multiple of the text height;

[0127] The text box angle difference between two adjacent single-line texts is less than the set angle.

[0128] Further preferably, the multiple is set to 5 times, which takes into account the maximum line (space) distance between multiple lines of title text.

[0129] Further preferably, the angle is set to 5 degrees. This is because the actual title text may have a certain angle deviation from the text detection frame, and therefore the multi-line title text detection frames may not be completely parallel, that is, the angle difference is 0 degrees.

[0130] The predefined template title is the full name of the material that should appear in the video. Since the material name may be very long and may be multi-line in the material, the detected single-line text is first combined based on the position information (text boxes are adjacent and have the same angle, which may constitute a combination), and then the combined text is used to match the material name. Compliance requirements mean that the video must contain certain materials, and the time the materials appear in the video must reach the shortest continuous time. In addition, evidence is required, that is, the frames where the materials first appear and last appear, or a video clip of the part where the materials appear is captured. Through: (1) Target tracking: Ensure that the material appears in the video for the shortest continuous time. Each tracking target has a "tracking length" field to identify the tracking time, and a "start tracking frame" field to identify the frame where the material first appears. When the target tracking is completed, the end tracking frame, that is, the frame where the material last appears, can be determined by the start tracking frame and the tracking length. (2) Text recognition and matching: The text box with the shortest continuous time will be recognized. The recognized single-line text will be combined as much as possible according to the position relationship, and then matched with the predefined template. If the match is successful, it means that the material meets the compliance requirements.

[0131] The above S6 is a step of matching multi-line title text. The above-mentioned steps of title text detection and OCR text recognition are only for single-line text, while the document title is likely to be multi-line text. In order to realize the recognition of multi-line text, this step matches the recognized single-line text according to the text box detection information to find the possible multi-line text. Specifically, this step selects candidate multi-line title texts according to the text box angle difference and the center point distance. When the conditions are met, the text content is spliced ​​forward and backward to form a candidate title text so as to match it with the predefined template. The matching process calculates the similarity between the spliced ​​text and the predefined template title through the edit distance algorithm. This method can effectively handle spelling errors, omissions and insertions, thereby improving the accuracy and robustness of the match. When the similarity between the text and the template exceeds the set threshold, the algorithm will determine that the match is successful and record the detailed information of the match, including the text content, similarity score, and the video frame number at the time of matching.

[0132] Since the long title in the document material may be presented in the form of multiple lines, and the rotating object detection model used can only detect single-line title text, and considering the possible false detection of the detection model (i.e., non-title text lines are identified as title text lines), it is necessary to match the single-line title text to obtain possible multi-line title text, and then compare it with the document material name appearing in the predefined template to determine whether the specified document material appears in the video.

[0133] In a preferred embodiment, a specific process is matched, such as Figure 4 As shown, the following steps are included:

[0134] First, check whether there is a qualified track, that is, a track in the LABELED state or the TRACKING state and whose tracking duration is greater than the preset minimum continuous appearance time. If there is no such track, there is no need to perform the matching process and it ends directly.

[0135] If there is a Track that meets the conditions, first create an empty text set to be matched. For all Tracks that meet the conditions, if their status is TRACKING, perform text recognition on them. After successful recognition, update the Track status to LABELED and add the recognized text to the text set to be matched; if the Track is already in the LABELED state, directly add its recognized text to the set.

[0136] Next, create an empty candidate text pair set S to store text pairs that may have matching relationships (i.e., text pairs that may form multi-line titles). This set actually stores the Track ID corresponding to the text, because the corresponding text box information and recognized text can be obtained through the Track ID.

[0137] Then, the Track ID combinations corresponding to all elements in the text set to be matched are obtained to form text pair combinations. For each text pair combination, the system sequentially retrieves the corresponding text box position information and determines whether the following two conditions are met at the same time: (1) the angle difference between the text boxes is less than the preset minimum angle threshold (indicating that the text boxes are approximately parallel); (2) the vertical distance between the center points of the text boxes is less than the preset maximum line spacing (indicating that the text boxes are adjacent). If both of the above conditions are met, the Track ID combination corresponding to the text pair is saved in the candidate text pair set S, and the forward and reverse spliced ​​text of the text pair is saved in the text set to be matched.

[0138] Taking into account the situation that a multi-line title may contain more than two lines, and the above steps can only find potential two-line titles, the candidate text pair set S is then judged for fusion. If the set S is empty, it means that no potential matching text pairs are found, and the process ends. If the set S is not empty, it will be judged for fusion until further fusion is impossible. Specifically, if two text pairs in the set S share a common Track ID, the two text pairs can be fused. The fused text is concatenated forward and backward and saved in the text set to be matched, and the fused Track ID combination is saved in the set S, and the two elements before the fusion are deleted.

[0139] After the fusion judgment is completed, each element in the text set to be matched will be used to perform similarity matching with the material names in the predefined template. For each text, the edit distance between it and the template name is calculated. If the edit distance is less than or equal to the preset threshold, the match is considered successful, indicating that the material meets the compliance requirements. The Track corresponding to the successfully matched text will be marked as MATCHED.

[0140] The main difficulties in the matching process are: (1) there may be more than one text material appearing in a single frame of the video, and there may be matching keys between which single-line texts; and (2) the processing of multi-line titles of more than two lines.

[0141] In order to solve the above difficulties, the method provided by the above embodiment of the present invention adopts the following technical means: (1) Considering that the lines of multiple title texts are parallel and the line spacing between title texts is small, while there is usually a large line spacing between the main text, the matching is performed by checking the angle difference between the text boxes (to ensure parallelism, because there may be deviations in text box detection, they may not be completely parallel, so a minimum angle difference is defined, which is set to 5 degrees in the system) and the vertical distance between the center points of the detected text boxes (the line height is approximately determined according to the height of the detected text boxes, so as to define a maximum line spacing, which is set to 5 times the line height in the system). (2) According to the angle difference between the text boxes and the distance difference between the center points of the text boxes, only the case where the title is exactly two lines can be matched, while the title in the actual document material may exceed two lines. Therefore, after the two-line title is matched, a fusion step is added, that is, to determine whether there is a possibility of fusion between the two matched lines of title (see the example of Figure 6 for details). If there is a common title text line between the two two-line titles, they can be fused into a three-line title, and so on, until the fusion cannot be continued.

[0142] An embodiment of the present invention provides a video-based document multi-line header text recognition and matching system.

[0143] Specifically, Figure 5 As shown, the video-based document multi-line header text recognition and matching system provided in this embodiment may include the following modules:

[0144] A target recognition module, which is used to perform target recognition on the document title text in the video and obtain a title text box; assign a unique ID to the title text box and record its initial frame ID and position information;

[0145] A target tracking module, which uses a cross-frame tracking mechanism to track the title text box, and uses the successfully tracked title text box as a tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information;

[0146] Tracking and matching module, which is used to match the newly detected title text box with the existing tracking target in each frame, and update the position information, state information and tracking duration of the successfully matched tracking target, and the unsuccessfully matched title text box is used as the new tracking target;

[0147] An effective target determination module, which is used to determine the tracking time of the tracking target. If the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target;

[0148] The title text recognition module is used to perform text recognition on the valid target to obtain a single line of text; perform multi-line text matching on the single line of text according to the text box position information of the single line of text to obtain multi-line title text.

[0149] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0150] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0151] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0152] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0153] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0154] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0155] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0156] The video-based document multi-line header text recognition and matching method and system provided in the above-mentioned embodiments of the present invention can accurately extract, recognize and match multi-line header text from videos through the integration and optimization of title text detection, text box tracking, text recognition and matching algorithms, especially in terms of accuracy and robustness when processing multi-line text in videos and documents with different rotation angles.

[0157] The video-based document multi-line header text recognition and matching method and system provided in the above-mentioned embodiments of the present invention automatically extracts, identifies and matches document title text from the video, ensures the compliance display time of the document in the video, realizes an automated compliance detection process, ensures the compliance display of document materials in the business process video, and thus improves the efficiency and reliability of compliance detection.

[0158] The video-based document multi-line header text recognition and matching method and system provided by the above-mentioned embodiments of the present invention realizes efficient and automated detection and management of document compliance in business process videos, and successfully solves the problems of low efficiency and error-proneness of traditional manual review. Specifically, the method and system provided by the above-mentioned embodiments of the present invention can automatically extract the document title text appearing in the video, accurately identify and match it with the predefined template, and ensure the compliance display time of the document in the video. This automated processing not only significantly improves the efficiency and accuracy of compliance detection and reduces the workload of manual processing, but also enhances the reliability of audit archives by automatically generating evidence images. Therefore, the method and system provided by the above-mentioned embodiments of the present invention not only optimizes the security and stability of business processes, but also provides a reliable compliance solution for administrative business.

[0159] All matters not covered in the above embodiments of the present invention are well known in the art.

[0160] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A video-based document multi-line header text recognition and matching method, characterized in that: include: Perform target recognition on the document title text in the video to obtain the title text box; Assign a unique ID to the title text box, and record the initial frame ID and position information of the title text box; Using a cross-frame tracking mechanism, the title text box is tracked, and the successfully tracked title text box is used as a tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information; In each frame, the newly detected title text box is matched with the existing tracking target, and the position information, state information and tracking duration of the successfully matched tracking target are updated, and the unsuccessfully matched title text box is used as the new tracking target; Determine the tracking time of the tracking target, if the tracking time of the tracking target exceeds a preset threshold, the tracking target is regarded as a valid target; Performing text recognition on the valid target to obtain a single line of text; Multi-line text matching is performed on the single-line text according to the text box position information of the single-line text to obtain multi-line title text.

2. The video-based document multi-line header text recognition and matching method according to claim 1 is characterized in that: The target recognition of the document title text in the video to obtain the title text box includes: Decoding the video to obtain image frames, and preprocessing the image frames; A pre-trained YOLO_OBB model is provided, and the pre-processed image frame is used as input. The pre-trained YOLO_OBB model is used for forward propagation to identify the document title text in the image frame, obtain the position coordinates and confidence of the rotated text detection box, and obtain the title text box.

3. The video-based document multi-line header text recognition and matching method according to claim 2 is characterized in that: Also includes any one or more of the following: - the preprocessing of the image frame includes: resizing and format conversion; -Providing a pre-trained YOLO_OBB model, including: Obtain a diverse set of document data including those with and without headers, and annotate them to obtain a sample set; Performing data enhancement processing on the sample set to obtain training samples; A YOLO_OBB model is trained using the training samples to obtain a pre-trained YOLO_OBB model.

4. The video-based document multi-line header text recognition and matching method according to claim 1 is characterized in that: The cross-frame tracking mechanism is used to track the title text box, and the successfully tracked title text box is used as the tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information, including: Initialize a tracker, use the tracker to track the title text box through a cross-frame tracking mechanism, obtain the successfully tracked title text box in each frame, and obtain the tracking target; In each frame, the newly detected title text box is matched with the existing tracking target, and the position information, state information and tracking duration of the successfully matched tracking target are updated, and the unsuccessfully matched title text box is used as a new tracking target, including: The intersection-and-union ratio matching algorithm is used to obtain the regional overlap between the current frame title text box and the existing tracking target; when the regional overlap is greater than or equal to the set threshold, it indicates a successful match, and the position information, status information and tracking duration of the successfully matched tracking target are updated; when the regional overlap is less than the set threshold, it indicates an unsuccessful match, and the unsuccessfully matched title text box is used as a new tracking target.

5. The video-based document multi-line header text recognition and matching method according to claim 1 is characterized in that: The process of obtaining effective targets also includes: Check all tracking targets and remove those that do not meet the tracking time or are invalid.

6. The video-based document multi-line header text recognition and matching method according to claim 1, characterized in that: The performing text recognition on the valid target to obtain a single line of text includes: Preprocessing the effective target to obtain a target image; Providing a pre-trained OCR text recognition model, taking the target image as input of the pre-trained OCR text recognition model, performing forward propagation through the pre-trained OCR text recognition model, and obtaining a text recognition result, which is a single line of text; The step of performing multi-line text matching on the single-line text according to the text box position information of the single-line text to obtain multi-line title text includes: According to the text box position information of the single-line text, the text box angle difference and the center point distance of two adjacent single-line texts are obtained, the text box angle difference and the center point distance are judged, and the text contents of the two adjacent single-line texts that meet the preset splicing conditions are spliced ​​forward and backward to form a candidate title text; Using an edit distance algorithm, similarity calculation is performed between the candidate title text and the predefined template title; When the similarity between the candidate title text and the predefined template title exceeds a set threshold, the match is determined to be successful, and the matching multiple lines of title text information are recorded.

7. The video-based document multi-line header text recognition and matching method according to claim 6 is characterized in that: Also includes any one or more of the following: -Preprocessing the effective target includes: Calculating the width and height of the text box according to the text box position coordinates of the valid target, determining affine transformation parameters, and acquiring a corrected title text image from the original video frame based on the affine transformation parameters; Processing the corrected title text image, the processing comprising: scaling and normalizing; -The preset splicing conditions include: The distance between the center points of the text boxes of two adjacent single-line texts is less than the set multiple of the text height; The text box angle difference between two adjacent single-line texts is less than the set angle; -The multiple lines of title text information include: text content, similarity score and video frame number when matching.

8. A video-based document multi-line header text recognition and matching system, characterized in that: include: A target recognition module, which is used to perform target recognition on the document title text in the video and obtain the title text box; Assign a unique ID to the title text box, and record the initial frame ID and position information of the title text box; A target tracking module, which uses a cross-frame tracking mechanism to track the title text box, and uses the successfully tracked title text box as a tracking target, wherein each tracking target is configured with a unique ID and initial position information and status information; Tracking and matching module, which is used to match the newly detected title text box with the existing tracking target in each frame, and update the position information, state information and tracking duration of the successfully matched tracking target, and use the unsuccessfully matched title text box as the new tracking target; An effective target determination module, which is used to determine the tracking time of the tracking target. If the tracking time exceeds a preset threshold, the tracking target is regarded as a valid target; The title text recognition module is used to perform text recognition on the valid target to obtain a single line of text; perform multi-line text matching on the single line of text according to the text box position information of the single line of text to obtain multi-line title text.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images