Method and apparatus for comparing and extracting image frames in a video sequence of document review

Abstract

This application provides a method for comparing and extracting image frames from a document flipping video sequence. A trained convolutional neural network is used to identify the state type of each frame in the video sequence to be identified, and based on this, the video sequence is divided into multiple image sequences that repeatedly alternate between two state types: document flattened and document flipping. A recurrent neural network is used to identify the action type of the image sequence with the state type of document flipping. Based on the identification results, image sequences with incomplete page flipping action type are deleted, and the two preceding and following image sequences are merged to obtain a processed video sequence. Page segmentation is performed based on the processed video sequence. This scheme proposes a method for detecting the integrity of page flipping actions, which can not only effectively segment video sequences with ordinary page numbers, but also handle the segmentation of consecutive blank page numbers, and can effectively select the highest quality image from the image sequence on the same page.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a method and apparatus for comparing and extracting image frames in a video sequence of document review. Background Technology

[0002] With the rapid development of information technology, electronic certificates have gradually become an indispensable part of people's lives due to their portability, ease of access and storage. Naturally, how to digitize physical certificates has become a research hotspot in recent years.

[0003] In existing technologies, a video stream is typically created by flipping through a physical document and recording the entire process. Then, image processing and recognition technologies are used to extract images, recognize text, and extract key information from the video stream to convert the physical document into electronic document data for storage, thus achieving the digitization of the physical document. In the process of digitizing physical documents, the detection of page-turning states is particularly important. The most common method is the pixel average difference method, which uses the difference in the average pixel values ​​of consecutive frames as the basis for determining whether a page-turning state is present or not. Specifically, if the pixel difference is large, it is determined to be in a page-turning state; if the pixel difference is small, it is determined to be in a stable, non-page-turning state.

[0004] In the process of researching and practicing existing technologies, the inventors of this application discovered that existing page-turning detection methods are easily affected by noise and have poor robustness. Moreover, since blank pages and the page-turning images between blank pages are very similar, they cannot handle the division of page numbers for consecutive blank pages, often resulting in misjudgment or omission, leading to poor page recognition performance and greatly affecting the accuracy of electronic document data. Summary of the Invention

[0005] This invention provides a method and apparatus for comparing and extracting image frames in a video sequence of document flipping. It can not only effectively divide video sequences with ordinary page numbers, but also handle the division of video sequences with consecutive blank page numbers, greatly improving the pagination recognition effect, enhancing the accuracy of electronic document data, and improving its robustness.

[0006] This invention provides a method for comparing and extracting image frames from a video sequence of document review, including:

[0007] Obtain a video sequence to be identified, which records the process of flipping through physical documents;

[0008] A pre-trained convolutional neural network (CNN) is used to identify the state type of each frame in the video sequence to be identified. The state types include document flattened and document being flipped.

[0009] Based on the position and state type of each frame in the video sequence to be identified, the video sequence to be identified is divided into multiple image sequences that alternate between two state types: document flattened and document flipped open. The image sequence with the state type of document flattened is the first type of image sequence, and the image sequence with the state type of document flipped open is the second type of image sequence.

[0010] A pre-trained recurrent neural network (RNN) is used to identify the action types of the second type of image sequence, including complete page turning and incomplete page turning;

[0011] Delete the second type of image sequence with incomplete page turning action type, and merge the two first type of image sequences before and after the second type of image sequence with incomplete page turning action type to obtain the processed video sequence;

[0012] Page segmentation recognition is performed based on the processed video sequence.

[0013] Optionally, in this embodiment, dividing the video sequence to be identified into multiple image sequences that alternate between two state types—"document flattened" and "document flipped open"—based on the position and state type of each frame image in the video sequence to be identified, wherein the image sequence with the state type "document flattened" is a first type of image sequence and the image sequence with the state type "document flipped open" is a second type of image sequence includes:

[0014] Based on the position and state type of each frame in the video sequence to be identified, frames with the same state type, adjacent and consecutive positions are divided into the same image sequence.

[0015] Image sequences with the status type "document flattened" are set as the first type of image sequences, and image sequences with the status type "document flipping" are set as the second type of image sequences, resulting in multiple image sequences that alternate between the two status types of "document flattened" and "document flipping".

[0016] Optionally, in this embodiment of the application, the step of using a preset trained recurrent neural network to identify the action type of the second type of image sequence includes:

[0017] The hidden layer coding features of the second type of image sequence are extracted using a pre-trained recurrent neural network.

[0018] The action type of the second type of image sequence is identified based on the hidden layer coding features.

[0019] Optionally, in this embodiment, the hidden layer coding feature reflects whether the page-turning action enables the document to display the next page of unfolded content. The step of identifying the action type of the second type of image sequence based on the hidden layer coding feature includes:

[0020] When the page-turning action is determined to enable the document to display the next page of flattened content based on the hidden layer coding features, the action type of the second type of image sequence is determined to be a complete page turn.

[0021] When it is determined from the hidden layer coding features that the page-turning action cannot display the next page of the document, the action type of the second type of image sequence is determined to be incomplete page turning.

[0022] Optionally, in this embodiment of the application, before performing pagination recognition based on the processed video sequence, the method further includes:

[0023] Obtain the number of image frames for each second-type image sequence in the processed video sequence;

[0024] If the number of image frames is greater than a preset threshold, it is determined that the second type of image sequence has page-turning lag, and an early warning is issued for the page-turning lag.

[0025] Optionally, in this embodiment of the application, the pagination recognition based on the processed video sequence includes:

[0026] The first type of image sequence to be identified is determined from the processed video sequence to obtain the current sequence;

[0027] The image that meets the preset conditions is selected from the current sequence as the target image corresponding to the current sequence, and the process returns to the step of determining the first type of image sequence to be identified from the processed video sequence, until all first type of image sequences have been identified.

[0028] The target images are merged according to their order in the processed video sequence, and page content recognition is performed on the merged target images to obtain the electronic certificate data corresponding to the video sequence to be recognized.

[0029] Optionally, in this embodiment of the application, the step of selecting images from the current sequence that meet preset conditions as target images corresponding to the current sequence includes:

[0030] An image quality assessment model is used to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence.

[0031] Calculate the overall quality score of each frame in the current sequence based on the detection results;

[0032] The image with the highest overall quality score is selected as the target image for the current sequence.

[0033] Optionally, in this embodiment of the application, the step of using an image quality assessment model to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence includes:

[0034] The current frame is determined from the current sequence to obtain the current frame;

[0035] Divide the current frame's image into multiple squares;

[0036] An image quality assessment model is used to identify the sharpness type of each block in the current frame. The sharpness type includes sharp and blurry. The number of blocks with the sharpness type is counted, and the ratio of the number of blocks with the sharpness type to the total number of blocks in the current frame is calculated to obtain the image sharpness of the current frame.

[0037] An image quality assessment model is used to identify the flatness type of each block in the current frame. The flatness type includes flat and uneven. The number of blocks with the flatness type of flat is counted, and the ratio of the number of blocks with the flatness type of flat to the total number of blocks in the current frame is calculated to obtain the flatness of the image in the current frame.

[0038] An image quality assessment model is used to identify the reflectivity type of each block in the current frame. The reflectivity type includes reflective and non-reflective. The number of blocks with the reflectivity type of non-reflective is counted, and the ratio of the number of blocks with the reflectivity type of non-reflective to the total number of blocks in the current frame is calculated to obtain the reflectivity of the image in the current frame.

[0039] The image quality assessment model is used to detect the coordinates of the four vertices of the document in the current frame. Based on the coordinates of the four vertices, it is determined whether the four vertices are in the current frame. The number of vertices in the current frame is counted, and the ratio of the number of vertices in the current frame to 4 is calculated to obtain the boundary detection result of the image in the current frame.

[0040] Return to the step of determining the frame that needs to be detected in the current sequence, until all frames in the current sequence have been detected.

[0041] Optionally, in this embodiment of the application, before merging the target images according to their order in the processed video sequence, the method further includes:

[0042] Determine whether the sharpness, smoothness, reflectivity, and boundary detection results of the target image all meet the preset quality requirements;

[0043] If so, then proceed with the step of merging the target images according to their order in the processed video sequence;

[0044] If not, a message indicating that the image does not meet quality requirements will be generated.

[0045] Accordingly, embodiments of this application also provide an image frame comparison and extraction device in a document review video sequence, comprising:

[0046] An acquisition unit is used to acquire a video sequence to be identified, wherein the video sequence to be identified records the process of flipping through physical documents;

[0047] The state recognition unit is used to identify the state type of each frame image in the video sequence to be recognized by using a preset trained convolutional neural network. The state type includes "document flattened" and "document being flipped".

[0048] The segmentation unit is used to divide the video sequence to be identified into multiple image sequences that alternate between two state types: "document flattened" and "document flipped open," based on the position and state type of each frame image in the video sequence to be identified. The image sequence with the state type "document flattened" is the first type of image sequence, and the image sequence with the state type "document flipped open" is the second type of image sequence.

[0049] An action recognition unit is used to identify the action type of the second type of image sequence using a preset trained recurrent neural network, wherein the action type includes complete page turning and incomplete page turning;

[0050] The processing unit is used to delete the second type of image sequence with the action type of incomplete page turning, and merge the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain the processed video sequence;

[0051] The pagination recognition unit is used to perform pagination recognition based on the processed video sequence.

[0052] Furthermore, embodiments of this application also provide a storage medium storing multiple instructions, which are adapted for loading by a processor to execute operations in any of the image frame comparison and extraction methods provided in the document review video sequence of this application embodiments.

[0053] In this embodiment, after obtaining the video sequence to be identified, which records the process of flipping through a physical document, a trained convolutional neural network can be used to identify the state type of each frame in the video sequence. Based on the position and state type of each frame in the video sequence, the video sequence is divided into multiple image sequences that alternate between two state types: document flattened and document flipping. Then, a recurrent neural network is used to identify the action type of the second type of image sequence (i.e., the image sequence with the state type of document flipping). Based on the identification result, the second type of image sequence with the action type of incomplete page flipping is deleted, and the two preceding and following first type of image sequences (i.e., the image sequence with the state type of document flattened) are merged to obtain the processed video sequence. Then, based on this processed video sequence... This solution performs page-turning recognition. Because it considers multiple factors, such as the document's flipping status and the completeness of the page-turning action, rather than solely relying on the average pixel difference between consecutive frames, it can detect page turns even in similar images, such as consecutive blank pages. This means the solution can effectively segment video sequences with ordinary page numbers, as well as those with consecutive blank pages, significantly improving page-turning recognition and enhancing the accuracy of electronic document data. Furthermore, since it considers the completeness of the page-turning action as a criterion, it can accurately identify page turns even in the event of external noise interference, such as incomplete page turns due to paper quality issues, demonstrating good robustness. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the image frame comparison and extraction method in the document review video sequence provided in the embodiments of this application;

[0056] Figure 2 This is an example diagram of image sequence segmentation in an embodiment of this application;

[0057] Figure 3 This is an example diagram of the processed video sequence in an embodiment of this application;

[0058] Figure 4 This is another flowchart of the method for comparing and extracting image frames in a video sequence for reviewing documents provided in this application embodiment;

[0059] Figure 5 This is a framework diagram of the image frame comparison and extraction method in the document review video sequence in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of the structure of the image frame comparison and extraction device in the video sequence of document browsing provided in the embodiments of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] This application provides a method and apparatus for comparing and extracting image frames from a video sequence of document review. These will be described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0063] This embodiment will describe the image frame comparison and extraction device in the document review video sequence from the perspective of the device. In a specific implementation, the image frame comparison and extraction device in the document review video sequence can be integrated into an electronic device, such as a server or terminal. The terminal may include a mobile phone, tablet computer, laptop computer and / or personal computer (PC).

[0064] A method for comparing and extracting image frames from a video sequence of document flipping includes: acquiring a video sequence to be identified that records the process of flipping through a physical document; then, using a pre-trained post-convolutional neural network, identifying the state type (including document flattened and document flipping forward) of each frame in the video sequence to be identified; and dividing the video sequence to be identified into multiple image sequences that repeatedly alternate between the two state types of document flattened and document flipping forward, based on the position and state type of each frame in the video sequence to be identified. The image sequence with the state type of document flattened is a first type of image sequence, and the image sequence with the state type of document flipping forward is a second type of image sequence. Next, using a pre-trained post-recurrent neural network, identifying the action type (including complete page turning and incomplete page turning) of the second type of image sequence, deleting the second type of image sequence with the action type of incomplete page turning, and merging the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain a processed video sequence. Finally, page-based recognition can be performed based on the processed video sequence.

[0065] For example, such as Figure 1 As shown, the specific process of image frame comparison and extraction in the video sequence of document review can be as follows:

[0066] 101. Obtain the video sequence to be identified, wherein the video sequence to be identified records the process of flipping through physical documents.

[0067] For example, a video sequence can be obtained from a database to obtain the video sequence to be identified; or a video sequence input by a user can be received to obtain the video sequence to be identified; or a video sequence sent by a video acquisition device can be received to obtain the video sequence to be identified, and so on.

[0068] The video sequences stored in the database, as well as the video sequences input by the user, can both be obtained by a video capture device through the process of examining physical documents. This video capture device can include a video recorder, camera, or scanning equipment, etc., depending on the specific application requirements, and will not be elaborated upon here.

[0069] It should be noted that when flipping through physical documents, the actions of turning pages and unfolding the pages can be performed by a machine or by hand. After each page is unfolded, there can be a slight pause so that the video capture device can capture the content on that page, and then the action of turning the next page can be performed to unfold the content of the next page, and so on, until the entire physical document has been flipped through. The flipping process is recorded as video, and a video sequence recording the process of flipping through the physical document can be obtained.

[0070] As can be seen from the above process of flipping through physical documents, the images in this video sequence can be divided into two categories based on the state of the document: one is the state where the document is flattened, which is referred to as "document flattened" in this embodiment; the other is the state where the document is being flipped through, which is referred to as "document flipping". Each time a flipping action is completed, the document changes its page to display the content of the next page to the video capture device. Therefore, by capturing the image sequence of the flipping action, the image sequences representing the two pages can be separated. In other words, it is necessary to identify the state type of the document (hereinafter referred to as state type) in each frame of the video sequence and determine that the image sequence belongs to "document flipping". Then, the image sequences representing the two pages can be separated for subsequent page recognition.

[0071] 102. Using a pre-set trained convolutional neural network, the state type of each frame in the video sequence to be identified is recognized, where the state type includes "document flattened" and "document being flipped".

[0072] For example, a pre-trained convolutional neural network can be obtained. Then, images from the video sequence to be identified are sequentially input into the trained convolutional neural network. The trained convolutional neural network extracts features from the images and classifies the document status of the images based on the extracted features. The classification result indicates the status type of the image.

[0073] Optionally, to improve recognition accuracy, the images in the video sequence to be recognized can be preprocessed before training the convolutional neural network, such as denoising or normalization.

[0074] Optionally, in order to make the overall recognition process less time-consuming and more efficient, the trained convolutional neural network can be selected as a lightweight model as possible.

[0075] Optionally, the trained convolutional neural network can be obtained either by directly using a pre-trained convolutional neural network or by pre-setting it by maintenance personnel, or by being trained automatically by the image frame recognition device. That is, before the step "using a pre-set trained convolutional neural network to identify the state type of each frame in the video sequence to be recognized," the image frame comparison and extraction method in the document review video sequence may further include:

[0076] Multiple images are collected as training samples, and these training samples are labeled with state types, such as the document being flattened or the document being flipped open. The depth features of the training samples are extracted through a pre-set convolutional neural network. The state type of the training samples is predicted based on the extracted depth features. The convolutional neural network is then converged using a pre-set first loss function based on the labeled state type and the predicted state type of the training samples. This yields a model that can extract the depth features of the document and perceive the state type of the document, which is the trained convolutional neural network.

[0077] The first loss function can be set according to the needs of the actual scenario or application, and will not be elaborated here.

[0078] Of course, in order to improve the accuracy of recognition, the training samples can be preprocessed before being input into the trained convolutional neural network, such as denoising or normalization, etc., which will not be elaborated here.

[0079] 103. Based on the position and state type of each frame in the video sequence to be identified, divide the video sequence into multiple image sequences that alternate between two state types: "document flattened" and "document flipped open." For example, this could be done as follows:

[0080] Based on the position and state type of each frame in the video sequence to be identified, frames with the same state type, adjacent and consecutive positions are divided into the same image sequence. In this way, multiple image sequences can be obtained that alternate between the two state types of "document flattened" and "document flipped open".

[0081] It should be noted that, for ease of description, in the embodiments of this application, the image sequence with the state type of "document flattened" is referred to as the first type of image sequence, and the image sequence with the state type of "document flipped open" is referred to as the second type of image sequence; that is, the image sequence with the state type of "document flattened" can be set as the first type of image sequence, and the image sequence with the state type of "document flipped open" can be set as the second type of image sequence, thereby obtaining multiple image sequences that repeatedly alternate between the first type of image sequence and the second type of image sequence.

[0082] For example, see Figure 2 If the state types of frames i to i+4 and frames i+9 to i+13 in the video sequence to be identified are all "document flattened", while the state type of frames i+5 to i+8 is "document being flipped", then since the state types of frames i to i+4 are all "document flattened" and their positions are adjacent and continuous, frames i to i+4 can be classified into the same first type of image sequence; similarly, since the state types of frames i+5 to i+8 are all "document flattened" and their positions are adjacent and continuous, frames i to i+4 can be classified into the same first type of image sequence. Since the state types of all the images are "document being flipped through" and their positions are adjacent and continuous, the images from frame i+5 to frame i+8 can be divided into the same second type of image sequence. Similarly, since the state types of all the images from frame i+9 to frame i+13 are "document flattened" and their positions are adjacent and continuous, the images from frame i+9 to frame i+13 can be divided into the same first type of image sequence. By analogy, multiple image sequences that alternate between the first type of image sequence and the second type of image sequence can be obtained.

[0083] It should be noted that, for the sake of convenience, in Figure 2 In this document, the image of a frame with a status type of "document flattened" is labeled "A", and the image of a frame with a status type of "document being flipped through" is labeled "B". It should be understood that this labeling is not intended to limit the embodiments of this application.

[0084] 104. A pre-set recurrent neural network is used to identify the action type of the second type of image sequence, where the action type includes complete page turning and incomplete page turning.

[0085] For example, a pre-trained recurrent neural network can be used to extract the hidden layer coding features of the second type of image sequence, and then the action type of the second type of image sequence can be identified based on the hidden layer coding features.

[0086] The hidden layer coding feature reflects whether the page-turning action enables the document to display the next page of unfolded content. Specifically, the step "identifying the action type of the second type of image sequence based on the hidden layer coding feature" may include:

[0087] When the page-turning action, based on the hidden layer coding feature, enables the document to display the next page of flattened content, the action type of the second type of image sequence is determined to be a complete page turn; when the page-turning action, based on the hidden layer coding feature, fails to enable the document to display the next page of flattened content, the action type of the second type of image sequence is determined to be an incomplete page turn.

[0088] Optionally, to improve the accuracy of recognition, the second type of image sequence can be preprocessed before being input into the trained recurrent neural network, such as denoising or normalization.

[0089] Optionally, the selection and specific parameters of the recurrent neural network after training can be determined according to the needs of the actual application. For example, LSTM (Long Short-Term Memory) or a general RNN network can be used, etc.

[0090] Optionally, the trained recurrent neural network can be obtained either by directly using a pre-trained recurrent neural network or by pre-setting it by maintenance personnel, or by being trained automatically by the image frame comparison and extraction device in the document flipping video sequence. That is, before the step "using a pre-set trained recurrent neural network to identify the action type of the second type of image sequence," the image frame comparison and extraction method in the document flipping video sequence may further include:

[0091] Image sequences from multiple complete and incomplete page-turning processes are collected as image sequence samples. Each image sequence sample is labeled with an action type, such as complete or incomplete page turn. Then, a pre-defined recurrent neural network is used to encode the image sequence samples to extract their hidden layer coding features. Based on the extracted hidden layer coding features, the action type of the image sequence samples is predicted. According to the labeled action type and the predicted action type of the image sequence samples, a pre-defined second loss function is used to converge the recurrent neural network, thus obtaining a model that can extract the hidden layer coding features of image sequences and perceive their action types, which is the trained recurrent neural network.

[0092] The second loss function can be set according to the needs of the actual scenario or application, and will not be elaborated here.

[0093] Of course, in order to improve the accuracy of recognition, the image sequence samples can be preprocessed before being input into the recurrent neural network for training. For example, they can be denoised or normalized, etc., which will not be elaborated here.

[0094] It's important to note that action type recognition is necessary because the existence of image sequences performing page-turning actions (i.e., the second type of image sequence) does not necessarily mean a complete page-turning process. Different document materials vary, and for some stiffer paper documents, whether turned manually or by machine, page-turning often fails, meaning the page-turning action is incomplete. For example, force may be applied to a page of the document, turning it over. The captured image is in the state of being turned, but due to the paper material or other factors, the force may not complete the page turn, causing the page to return to its unturned state. In other words, the image sequence representing the same page content occurs before and after the page-turning action. Therefore, forcibly classifying images in this way would result in duplicate image sequences, leading to duplicate data collection. Thus, action type recognition to determine whether the page-turning action is complete (whether the page-turning was successful) is crucial to avoid generating duplicate image sequences and preventing duplicate data collection.

[0095] 105. Delete the second type of image sequence with the action type of incomplete page turning, and merge the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain the processed video sequence.

[0096] For example, still using Figure 2 Taking the video sequence to be identified as an example, if the action type of a certain second-type image sequence Y1 in the video sequence to be identified is incomplete page turning, then refer to... Figure 3 The second type of image sequence Y1 can be deleted (i.e., the images from frames i+5 to i+8 can be deleted). Then, the two first type of image sequences before and after the second type of image sequence Y1, namely the first type of image sequence X1 and the first type of image sequence X2, are merged to obtain a new first type of image sequence "X1+X2". This process is repeated for other second type of image sequences with incomplete page turning actions, and the processed video sequence can be obtained.

[0097] 106. Page recognition based on processed video sequences.

[0098] For example, images that meet preset conditions can be selected from the first type of image sequence in the processed video sequence as the target images for the current sequence. The target images are then merged according to their order in the processed video sequence, and page content recognition is performed on the merged target images to obtain the electronic document data corresponding to the video sequence to be recognized. For example, the specific process can be as follows:

[0099] S1. Determine the first type of image sequence to be identified from the processed video sequence to obtain the current sequence, and then execute step S2.

[0100] S2. Select images that meet the preset conditions from the current sequence as the target images corresponding to the current sequence, and return to execute step S1, that is, return to execute the step of "determining the first type of image sequence to be identified from the processed video sequence" until all first type of image sequences have been identified, and then execute step S3.

[0101] S3. Merge the target images according to their order in the processed video sequence, and perform page content recognition on the merged target images to obtain the electronic certificate data corresponding to the video sequence to be recognized.

[0102] These preset conditions can be set according to the needs of actual applications. For example, although there are no obvious images of documents being flipped through in the first type of image sequence, the flatness of the images before and after the second type of image sequence (i.e., the image sequence of documents being flipped through) will have a more obvious change. Since page curvature directly affects the recognition of information on the page, the flatness of the image (i.e., the captured page) can be taken into consideration when selecting target images. Similarly, since lighting conditions also directly affect document imaging, such as strong light or local reflection problems caused by the document material, the reflectivity of the image (i.e., the captured page) can also be used as a selection target image. One of the factors to consider is the image quality; in addition, the positional relationship between the document and the camera of the video capture device will also affect the image of the document. If part of the page is outside the camera's field of view, it will cause the document to be missing. In this case, it should be detected in time and an effective warning should be issued. In addition, the movement of flipping through the page can also cause the image to be blurred. Therefore, the boundary detection results of the page and the clarity can also be included in the selection of target images. So, for the above reasons, the best image can be selected from four dimensions: the flatness, reflectivity, clarity and boundary detection of the image (i.e., the captured page) to obtain the target image.

[0103] It should be noted that, for ease of description, in this embodiment, the detection of the flatness, reflectivity, sharpness, and boundaries of the image (i.e., the captured page) is referred to as image flatness, reflectivity, sharpness, and boundary detection. It should be understood that, in this embodiment, image flatness refers to the flatness (or flatness score) of the image, such as the captured document page; image reflectivity refers to the reflectivity (or reflectivity score) of the image, such as the captured document page; image sharpness refers to the sharpness (or sharpness score) of the image, such as the captured document page; and image boundary detection refers to the detection of the boundaries of the image, such as the captured document page.

[0104] Optionally, to improve filtering efficiency, a "picture quality assessment model" can be used to filter target images during the specific filtering process. That is, the step "filtering images that meet preset conditions from the current sequence as target images corresponding to the current sequence" may include:

[0105] An image quality assessment model is used to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence. Based on the detection results, a comprehensive quality score is calculated for each frame in the current sequence, and the image with the highest comprehensive quality score is selected as the target image corresponding to the current sequence.

[0106] Optionally, since the granularity of local features is finer than that of global features, when detecting the sharpness, smoothness, reflectivity, and boundaries of each frame of an image, each frame can be divided into multiple squares before detection and statistical analysis. That is, the step "using an image quality assessment model to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence" can include:

[0107] A1. Determine the frame to be detected in the current sequence, obtain the current frame, and divide the image of the current frame into multiple squares.

[0108] For example, the image of the current frame can be divided into N×M blocks.

[0109] A2. Use an image quality assessment model to identify the sharpness type of each block in the current frame. The sharpness type includes sharp and blurry. Based on the identification results, count the number of blocks with the sharpness type of sharp. Calculate the ratio of the number of blocks with the sharpness type of sharp to the total number of blocks in the current frame to obtain the image sharpness of the current frame.

[0110] For example, if the number of sharp squares is denoted as S1, then the sharpness (i.e., sharpness score) score1 of the current frame image is:

[0111]

[0112] Specifically, the sharpness type identification sub-model in the image quality assessment model can be used to identify the sharpness type of each block in the current frame.

[0113] A3. Use an image quality assessment model to identify the flatness type of each block in the current frame. The flatness type includes flat and uneven. Based on the identification results, count the number of blocks with the flatness type of flat. Calculate the ratio of the number of blocks with the flatness type of flat to the total number of blocks in the current frame to obtain the flatness of the image in the current frame.

[0114] For example, if the number of squares with the flatness type "flat" is denoted as S2, then the flatness score (i.e., flatness score) of the current frame image is:

[0115]

[0116] Specifically, the flatness type recognition sub-model in the image quality assessment model can be used to identify the flatness type of each block in the current frame.

[0117] A4. Use an image quality assessment model to identify the reflectivity type of each block in the current frame. The reflectivity type includes reflective and non-reflective. Based on the identification results, count the number of blocks with the reflectivity type of non-reflective. Calculate the ratio of the number of blocks with the reflectivity type of non-reflective to the total number of blocks in the current frame to obtain the reflectivity of the image in the current frame.

[0118] For example, if the number of non-reflective squares is denoted as S3, then the reflectivity (i.e., reflectivity score) of the current frame image is:

[0119]

[0120] Specifically, the reflection type recognition sub-model in the image quality assessment model can be used to identify the reflection type of each block in the current frame.

[0121] A5. Use an image quality assessment model to detect the coordinates of the four vertices of the document in the current frame, and determine whether the four vertices are in the current frame based on their coordinates. Count the number of vertices in the current frame, calculate the ratio of the number of vertices in the current frame to 4, and obtain the boundary detection result of the image in the current frame.

[0122] For example, if the number of vertices in the current frame is denoted as S4, then the boundary detection result (i.e., boundary condition score) score4 of the current frame image is:

[0123]

[0124] Specifically, the vertex detection sub-model in the image quality assessment model can be used to detect the coordinates of the four vertices of the document in the current frame.

[0125] The execution of steps A2 to A5 can be done in any order. Based on the scores of each dimension obtained from A2 to A5, the overall quality score of the current frame can be calculated, as follows:

[0126]

[0127] A6. Return to the step of determining the frame that needs to be detected in the current sequence, until all frames in the current sequence have been detected.

[0128] In this way, the overall quality score of all frames in the current sequence can be calculated based on the detection results of A2 to A5. The image with the highest overall quality score is the best image, and this best image is used as the target image corresponding to the current sequence.

[0129] Optionally, the sharpness type recognition sub-model, smoothness type recognition sub-model, reflectivity type recognition sub-model, and vertex detection sub-model in the image quality assessment model can be obtained either directly using pre-trained models or trained by the image frame recognition device itself. That is, before the step "using the image quality assessment model to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence", the image frame comparison and extraction method in the document review video sequence may also include:

[0130] Train the clear type recognition sub-model, the flat type recognition sub-model, the reflective type recognition sub-model, and the vertex detection sub-model; for example, as follows:

[0131] (1) Training of the clear type recognition sub-model;

[0132] Multiple image samples are collected and divided into multiple blocks, such as N×M blocks. These blocks are preprocessed, such as by normalization, to obtain block samples. Each block sample is labeled with a clarity type, such as "clear" or "unclear". Then, the block samples are input into a preset neural network, which extracts features from the block samples and predicts the clarity type of the block samples based on the extracted features. Based on the predicted clarity type and the labeled clarity type, a preset third loss function is used to converge the neural network, thus obtaining a model that can predict the clarity type of the blocks. This model is called the clarity type recognition sub-model in this embodiment of the application.

[0133] (2) Training of the flatness type recognition sub-model;

[0134] Multiple image samples are collected and divided into multiple squares, for example, N×M squares. These squares are preprocessed, such as by normalization, to obtain square samples. Each square sample is labeled with a flatness type, such as flat or uneven. Then, the square samples are input into a preset neural network, which extracts features from the square samples and predicts the flatness type of the square samples based on the extracted features. Based on the predicted flatness type and the labeled flatness type, a preset fourth loss function is used to converge the neural network, thus obtaining a model that can predict the flatness type of the squares. This model is called the flatness type recognition sub-model in this embodiment of the application.

[0135] (3) Training of the reflectivity type recognition sub-model;

[0136] Multiple image samples are collected and divided into multiple squares, for example, N×M squares. These squares are preprocessed, such as by normalization, to obtain square samples. Each square sample is labeled with a reflection type, such as "reflective" or "non-reflective". Then, the square samples are input into a preset neural network, which extracts features from the square samples and predicts the reflection type of the square samples based on the extracted features. Based on the predicted reflection type and the labeled reflection type, the neural network is converged using a preset fifth loss function to obtain a model that can predict the reflection type of the squares. This model is called the reflection type recognition sub-model in this embodiment of the application.

[0137] (4) Training of the vertex detection sub-model;

[0138] Multiple images containing page images are acquired, and these images are preprocessed, such as by normalization, to obtain image samples. Each image sample is labeled with the vertex coordinates of the page image, for example, labeled as reflective or non-reflective. Then, a key point detection model or a line segment detection model is used to extract features from the image samples, and the vertex coordinates of the image samples are predicted based on the extracted features. Based on the predicted vertex coordinates and the labeled vertex coordinates, a preset sixth loss function is used to converge the neural network, thus obtaining a model with the ability to locate vertex of page images. This model is called the vertex detection sub-model in this embodiment of the application.

[0139] The third, fourth, fifth, and sixth loss functions can be set according to the needs of the actual application. For example, the third, fourth, and fifth loss functions can all adopt the classification loss function, etc., which will not be elaborated here.

[0140] Optionally, since the image quality of the target image directly affects the subsequent recognition of document page information and the execution of electronic tasks such as key information extraction, in order to improve the accuracy of data during subsequent document digitization, the image quality of the target image can be checked before recognizing the page content to determine whether it meets the quality requirements. That is, before the step of "merging the target images according to their order in the processed video sequence," the page recognition method may further include:

[0141] Determine whether the sharpness, flatness, reflectivity, and boundary detection results of the target image all meet the preset quality requirements. If yes, then perform the step of merging the target images according to their order in the processed video sequence; otherwise, generate a prompt message indicating that the image does not meet the quality requirements.

[0142] When staff receive a notification that the image does not meet quality requirements, they can make timely adjustments to obtain a new image that meets the quality requirements.

[0143] Optionally, thresholds can be set for sharpness, smoothness, reflectivity, and boundary detection results to determine whether the sharpness, smoothness, reflectivity, and boundary detection results of the target image all meet the preset quality requirements.

[0144] For example, taking a sharpness threshold of T1, a smoothness threshold of T2, a reflectivity threshold of T3, and a boundary condition threshold of T4 as an example, the target image is determined to meet the quality requirements when sharpness score1, smoothness score2, reflectivity score3, and boundary detection result score4 simultaneously meet the following conditions:

[0145] score1>T1

[0146] score2>T2

[0147] score3>T3

[0148] score4>T4

[0149] Otherwise, if any one of the following criteria fails to meet the condition: sharpness score1, flatness score2, reflectivity score3, or boundary detection result score4, the target image is determined to be non-compliant with quality requirements. In this case, a prompt message indicating that the image does not meet quality requirements can be generated.

[0150] Optionally, since there may be pauses during the browsing process, if we can further determine whether the action is abnormal and issue effective prompts or alarms in a timely manner, we can also avoid damage to physical documents or errors in data collection.

[0151] For example, if image frames in a video sequence are constantly being flipped through, it indicates that the document flipping is causing a stutter. Therefore, a threshold can be set. If the number of image frames in the video sequence with the state type "document being flipped through" exceeds a preset threshold and has not yet returned to a flattened state, then the document flipping is considered to be stuttering. Optionally, before the step "performing page recognition based on the processed video sequence", the image frame comparison and extraction method in the document flipping video sequence may also include:

[0152] The number of image frames in each second-type image sequence in the processed video sequence is obtained. If the number of image frames is greater than a preset threshold, it is determined that the second-type image sequence has page-turning stutter, and an early warning is issued for the page-turning stutter.

[0153] Otherwise, if the number of image frames is less than or equal to the preset threshold, it is determined that the second type of image sequence is turning pages normally and there is no page turning stutter. Therefore, the operation of page recognition based on the processed video sequence can continue.

[0154] The preset threshold can be set according to the needs of actual applications, which will not be elaborated here.

[0155] As can be seen from the above, after obtaining the video sequence to be identified that records the process of flipping through physical documents, this embodiment can use a trained convolutional neural network to identify the state type of each frame in the video sequence to be identified. Based on the position and state type of each frame in the video sequence, the video sequence is divided into multiple image sequences that alternate between two state types: document flattened and document flipping. Then, a recurrent neural network is used to identify the action type of the second type of image sequence (i.e., the image sequence with the state type of document flipping). Based on the identification result, the second type of image sequence with the action type of incomplete page flipping is deleted, and the two preceding and following first type of image sequences (i.e., the image sequence with the state type of document flattened) are merged to obtain the processed video sequence. Then, based on this processed video... The sequence is used for page segmentation recognition. Because this scheme considers multiple factors, such as the document's flipping status and the completeness of the page-turning action, rather than solely relying on the average pixel difference between consecutive frames, it does not affect the page-turning detection effect for very similar page-turning images, such as consecutive blank pages. In other words, this scheme can not only effectively segment video sequences with ordinary page numbers, but also handle video sequences with consecutive blank page numbers, greatly improving page segmentation recognition and enhancing the accuracy of electronic document data. Furthermore, since this scheme also considers the completeness of the page-turning action as one of the criteria for page-turning judgment, it can accurately identify page-turning errors even when encountering external noise interference, such as incomplete page-turning due to paper quality, demonstrating good robustness.

[0156] Based on the methods described in the preceding embodiments, the following examples will provide further details.

[0157] In this embodiment, the image frame comparison and extraction device in the video sequence of the document review will be specifically integrated into an electronic device as an example for explanation.

[0158] like Figure 4 and Figure 5 As shown, the specific process of image frame comparison and extraction in the video sequence of document review can be as follows:

[0159] 201. An electronic device acquires a video sequence to be identified, wherein the video sequence to be identified records the process of flipping through physical documents.

[0160] 202. The electronic device uses a pre-trained convolutional neural network to identify the state type of each frame in the video sequence to be identified, wherein the state type includes "document flattened" and "document being flipped".

[0161] In addition to directly using a pre-trained convolutional neural network, such as a CNN network, the trained convolutional neural network can also be trained by the image frame comparison and extraction device in the document flipping video sequence. For specific training methods, please refer to the previous embodiments, which will not be repeated here.

[0162] 203. The electronic device divides the images of frames with the same state type, adjacent and consecutive positions into the same image sequence according to the position and state type of each frame in the video sequence to be identified. The image sequence with the state type of "document flattened" is set as the first type of image sequence, and the image sequence with the state type of "document flipped open" is set as the second type of image sequence. Thus, the video sequence to be identified is divided into multiple image sequences that alternate between the first type of image sequence and the second type of image sequence (i.e., the two state types alternate repeatedly).

[0163] 204. The electronic device uses a pre-trained recurrent neural network to extract the hidden layer coding features of the second type of image sequence, and identifies the action type of the second type of image sequence based on these hidden layer coding features. For example, it can be done as follows:

[0164] When the electronic device determines that the page-turning action can enable the document to display the next page of flattened content based on the hidden layer coding feature, the action type of the second type of image sequence is determined to be a complete page turn. When the electronic device determines that the page-turning action cannot enable the document to display the next page of flattened content based on the hidden layer coding feature, the action type of the second type of image sequence is determined to be an incomplete page turn.

[0165] In addition to directly using a post-trained recurrent neural network, such as an LSTM network, the trained recurrent neural network can also be trained by the image frame comparison and extraction device in the document flipping video sequence. For specific training methods, please refer to the previous embodiments, which will not be repeated here.

[0166] 205. The electronic device deletes the second type of image sequence with the action type of incomplete page turning, and merges the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain the processed video sequence. For example, see [link to video sequence]. Figure 5 .

[0167] 206. The electronic device obtains the number of image frames of each second type of image sequence in the processed video sequence. If the number of image frames is less than or equal to a preset threshold, then step 207 is executed. If the number of image frames is greater than the preset threshold, then it is determined that the second type of image sequence has page turning stutter, and an early warning is issued for the page turning stutter.

[0168] Once staff receive an alert, they can take timely action to prevent damage to physical documents or errors in data collection.

[0169] The preset threshold can be set according to the needs of actual applications, which will not be elaborated here.

[0170] 207. The electronic device selects images that meet preset conditions from the first type of image sequence of the processed video sequence as the target images corresponding to the current sequence.

[0171] The preset conditions can be set according to the needs of the actual application. For example, the best image can be selected from four dimensions: image clarity, flatness, reflectivity, and boundary detection, to obtain the target image. For example, it can be as follows:

[0172] The electronic device determines the frame to be detected in the current sequence, obtains the current frame, divides the image of the current frame into N×M blocks, and then calculates the sharpness, smoothness, and reflectivity of the current frame image, and detects the boundaries of the image, as follows:

[0173] A sharpness type recognition sub-model is used to identify the sharpness type of each square in the current frame. Based on the recognition results, the number of squares with the sharpness type "sharp" (S1) is counted. The ratio of S1 to the total number of squares N×M in the current frame is calculated to obtain the sharpness score1 of the current frame image. For example,

[0174] The flatness type recognition sub-model is used to identify the flatness type of each block in the current frame. Based on the recognition results, the number of blocks with the flatness type S2 is counted. The ratio of S2 to the total number of blocks N×M in the current frame is calculated to obtain the flatness score2 of the image in the current frame.

[0175] The reflection type recognition sub-model is used to identify the reflection type of each block in the current frame. Based on the recognition results, the number of blocks with the reflection type of non-reflective is counted as S3. The ratio of S3 to the total number of blocks in the current frame N×M is calculated to obtain the flatness score3 of the image in the current frame.

[0176] The vertex detection sub-model is used to detect the coordinates of the four vertices of the document in the current frame, and the coordinates of the four vertices are used to determine whether the four vertices are in the current frame. The number of vertices in the current frame, S4, is counted, and the ratio of S4 to 4 is calculated to obtain the boundary detection result of the image in the current frame, i.e., the boundary condition score4.

[0177] Adding scores1,2,3, and4 together yields the overall quality score for the current frame. Selecting the frame with the highest score from this first-class image sequence yields the target image. For example, see [link to relevant documentation]. Figure 3 If the image of frame i+9 in the first type of image sequence “X1+X2” has the highest overall quality score, then the image of frame i+9 can be selected as the target image of the first type of image sequence “X1+X2”, and so on.

[0178] Optionally, after selecting the target image, the electronic device can also determine whether the target image's score1, score2, score3, and score4 are all greater than their respective preset thresholds. If they are all greater than their respective preset thresholds, it indicates that the target image meets the image quality requirements, and step 208 can be continued. Otherwise, if any one of them is less than or equal to its respective preset threshold, it indicates that the target image does not meet the image quality requirements. At this time, a prompt message indicating that the image does not meet the quality requirements can be generated so that staff can take timely action.

[0179] For details, please refer to the previous embodiments, which will not be repeated here.

[0180] 208. The electronic device merges the target images according to their order in the processed video sequence, and performs page content recognition on the merged target images to obtain the electronic certificate data corresponding to the video sequence to be recognized.

[0181] As can be seen from the above, after acquiring the video sequence to be identified, which records the process of flipping through a physical document, the electronic device in this embodiment can use a trained convolutional neural network to identify the state type of each frame in the video sequence to be identified. Based on this, the video sequence to be identified is divided into multiple image sequences that alternate between two state types: the document is flattened and the document is being flipped. Then, it further determines whether the page-flipping action of the "document being flipped" image sequence is complete and whether there is any lag, and performs corresponding processing accordingly before performing page-by-page recognition on the processed video sequence. Since this solution mainly considers the flipping of the document when detecting page turning... The method considers multiple factors, including the reading status, the completeness of the page-turning action, and whether the page-turning is choppy, rather than solely relying on the average pixel difference of consecutive frames. Therefore, even if the page-turning images are very similar, such as consecutive blank pages, it will not affect the page-turning detection effect. This can improve the page-turning recognition effect, enhance the accuracy of electronic document data, and avoid damage to physical documents. In addition, since this solution also considers the completeness of the page-turning action as one of the criteria for page-turning judgment, even if there is external noise interference, such as page-turning failure due to paper quality or other reasons, this solution can still accurately identify the page-turning and has good robustness.

[0182] To better implement the above methods, this application also provides an image frame comparison and extraction device in a document review video sequence. This image frame comparison and extraction device in a document review video sequence can be integrated into an electronic device, such as a server or terminal. The terminal may include a mobile phone, tablet computer, laptop computer, and / or personal computer.

[0183] like Figure 6 As shown, the image frame comparison and extraction device in the document review video sequence may include an acquisition unit 301, a status recognition unit 302, a segmentation unit 303, an action recognition unit 304, a processing unit 305, and a page recognition unit 306, as detailed below:

[0184] (1) Obtain unit 301;

[0185] The acquisition unit 301 is used to acquire a video sequence to be identified, wherein the video sequence to be identified records the process of flipping through physical documents.

[0186] (2) Status recognition unit 302;

[0187] The state recognition unit 302 is used to identify the state type of each frame of the video sequence to be recognized by using a preset trained convolutional neural network. The state type includes "document flattened" and "document being flipped".

[0188] For example, the state recognition unit 302 can specifically be used to acquire a preset trained convolutional neural network, then sequentially input the images in the video sequence to be recognized into the trained convolutional neural network, which extracts features from the images and classifies the document status of the images based on the extracted features, wherein the classification result indicates the state type of the image.

[0189] (3) Divide the unit into 303;

[0190] The segmentation unit 303 is used to divide the video sequence to be identified into multiple image sequences that alternate between two state types: "document flattened" and "document flipped open," based on the position and state type of each frame image in the video sequence to be identified. The image sequence with the state type "document flattened" is the first type of image sequence, and the image sequence with the state type "document flipped open" is the second type of image sequence.

[0191] For example, the segmentation unit 303 can be used to segment images of frames with the same state type, adjacent and consecutive positions into the same image sequence according to the position and state type of each frame image in the video sequence to be identified, so that the video sequence to be identified is divided into multiple image sequences that alternate between two state types: the document is flattened and the document is being flipped.

[0192] (4) Action recognition unit 304;

[0193] The action recognition unit 304 is used to recognize the action type of the second type of image sequence using a preset trained recurrent neural network, which includes complete page turning and incomplete page turning.

[0194] For example, the action recognition unit 304 can be used to extract the hidden layer coding features of the second type of image sequence using a preset trained recurrent neural network, and then to recognize the action type of the second type of image sequence based on the hidden layer coding features.

[0195] The hidden layer coding feature reflects whether the page-turning action can enable the document to display the next page of flattened content. That is, when the action recognition unit 304 determines that the page-turning action can enable the document to display the next page of flattened content based on the hidden layer coding feature, the action type of the second type of image sequence can be determined as a complete page turn; and when the hidden layer coding feature determines that the page-turning action cannot enable the document to display the next page of flattened content, the action type of the second type of image sequence can be determined as an incomplete page turn.

[0196] (5) Processing unit 305;

[0197] The processing unit 305 is used to delete the second type of image sequence with the action type of incomplete page turning, and merge the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain the processed video sequence.

[0198] (6) Pagination recognition unit 306;

[0199] Page recognition unit 306 is used for page recognition based on the processed video sequence.

[0200] For example, the pagination recognition unit 306 can be used to select images that meet preset conditions from the first type of image sequence of the processed video sequence as the target images corresponding to the current sequence. For example, an image quality assessment model can be used to detect the sharpness, flatness, reflectivity, and boundaries of each frame image in the current sequence, and calculate the comprehensive quality score of each frame image in the current sequence based on the detection results. Then, the image with the highest comprehensive quality score is selected as the target image corresponding to the current sequence, and so on. After that, the target images can be merged according to the order of the target images in the processed video sequence, and page content recognition can be performed on the merged target images to obtain the electronic certificate data corresponding to the video sequence to be recognized.

[0201] Optionally, the trained convolutional neural network, trained recurrent neural network, and image quality assessment model can be obtained either directly from the pre-trained model or by the image frame comparison and extraction device in the document review video sequence. That is, the image frame comparison and extraction device in the document review video sequence can also include a training unit, as follows:

[0202] The training unit is used to train the convolutional neural network to obtain the trained convolutional neural network.

[0203] Optionally, this training unit can also be used to train a recurrent neural network to obtain a trained recurrent neural network.

[0204] Optionally, this training unit can also be used to train an image quality assessment model.

[0205] For details on the training methods of the above models, please refer to the previous method implementation examples, which will not be repeated here.

[0206] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0207] As can be seen from the above, since the image frame comparison and extraction device in the document flipping video sequence of this embodiment can integrate multiple factors such as the document flipping status and the completeness of the page-turning action as considerations for page-turning detection, rather than relying solely on the difference in the average pixel value of consecutive frames, it can accurately detect even cases where the flipped images are very similar, such as consecutive blank pages, greatly improving the pagination recognition effect and contributing to the accuracy of electronic document data. In addition, since this solution also considers the completeness of the page-turning action as one of the criteria for judging page-turning, even if there is external noise interference, such as the failure to complete page-turning due to paper quality or other reasons, this solution can accurately identify the issue. Therefore, this solution also has good robustness.

[0208] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0209] To this end, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute steps in any of the image frame comparison and extraction methods provided in the document review video sequence of this application. For example, the instructions can execute the following steps:

[0210] A video sequence to be identified is obtained, which records the process of flipping through a physical document. A pre-trained convolutional neural network is used to identify the state type of each frame in the video sequence, including "document flattened" and "document flipped forward." Based on the position and state type of each frame in the video sequence, the sequence is divided into multiple image sequences that alternate between the two state types: "document flattened" and "document flipped forward." The image sequence with the state type "document flattened" is designated as the first type, and the image sequence with the state type "document flipped forward" is designated as the second type. A pre-trained recurrent neural network is used to identify the action type of the second type of image sequence, including "complete page turning" and "incomplete page turning." Second type image sequences with the action type "incomplete page turning" are deleted, and the two first type image sequences preceding and following the second type of image sequence with the action type "incomplete page turning" are merged to obtain the processed video sequence. Page recognition is then performed based on the processed video sequence.

[0211] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0212] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0213] Since the instructions stored in the storage medium can execute the steps in any of the image frame comparison and extraction methods in the document review video sequence provided in the embodiments of this application, the beneficial effects that any of the image frame comparison and extraction methods in the document review video sequence provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0214] The foregoing has provided a detailed description of a method and apparatus for comparing and extracting image frames in a video sequence for document review, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for comparing and extracting image frames in a video sequence of document review, characterized in that, include: Obtain a video sequence to be identified, which records the process of flipping through physical documents; A pre-trained convolutional neural network is used to identify the state type of each frame in the video sequence to be identified, and the state type includes document flattened and document being flipped. Based on the position and state type of each frame in the video sequence to be identified, the video sequence to be identified is divided into multiple image sequences that alternate between two state types: document flattened and document flipped open. The image sequence with the state type of document flattened is the first type of image sequence, and the image sequence with the state type of document flipped open is the second type of image sequence. A pre-set recurrent neural network is used to identify the action types of the second type of image sequence, including complete page turning and incomplete page turning; Delete the second type of image sequence with incomplete page turning action type, and merge the two first type of image sequences before and after the second type of image sequence with incomplete page turning action type to obtain the processed video sequence; Page segmentation recognition is performed based on the processed video sequence.

2. The method according to claim 1, characterized in that, The step of dividing the video sequence to be identified into multiple image sequences that alternate between two state types—"document flattened" and "document flipped open"—based on the position and state type of each frame image in the video sequence to be identified, wherein the image sequence with the state type "document flattened" is the first type of image sequence, and the image sequence with the state type "document flipped open" is the second type of image sequence includes: Based on the position and state type of each frame in the video sequence to be identified, frames with the same state type, adjacent and consecutive positions are divided into the same image sequence. Image sequences with the status type "document flattened" are set as the first type of image sequences, and image sequences with the status type "document flipping" are set as the second type of image sequences, resulting in multiple image sequences that alternate between the two status types of "document flattened" and "document flipping".

3. The method according to claim 1, characterized in that, The step of using a pre-trained recurrent neural network to identify the action type of the second type of image sequence includes: The hidden layer coding features of the second type of image sequence are extracted using a pre-trained recurrent neural network. The action type of the second type of image sequence is identified based on the hidden layer coding features.

4. The method according to claim 3, characterized in that, The hidden layer coding feature reflects whether the page-turning action enables the document to display the next page of unfolded content. The step of identifying the action type of the second type of image sequence based on the hidden layer coding feature includes: When the page-turning action is determined to enable the document to display the next page of flattened content based on the hidden layer coding features, the action type of the second type of image sequence is determined to be a complete page turn. When it is determined from the hidden layer coding features that the page-turning action cannot display the next page of the document, the action type of the second type of image sequence is determined to be incomplete page turning.

5. The method according to any one of claims 1 to 4, characterized in that, Before performing pagination recognition based on the processed video sequence, the method further includes: Obtain the number of image frames for each second-type image sequence in the processed video sequence; If the number of image frames is greater than a preset threshold, it is determined that the second type of image sequence has page-turning lag, and an early warning is issued for the page-turning lag.

6. The method according to any one of claims 1 to 4, characterized in that, The pagination recognition based on the processed video sequence includes: The first type of image sequence to be identified is determined from the processed video sequence to obtain the current sequence; The image that meets the preset conditions is selected from the current sequence as the target image corresponding to the current sequence, and the process returns to the step of determining the first type of image sequence to be identified from the processed video sequence, until all first type of image sequences have been identified. The target images are merged according to their order in the processed video sequence, and page content recognition is performed on the merged target images to obtain the electronic certificate data corresponding to the video sequence to be recognized.

7. The method according to claim 6, characterized in that, The step of selecting images from the current sequence that meet preset conditions as target images corresponding to the current sequence includes: An image quality assessment model is used to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence. Calculate the overall quality score of each frame in the current sequence based on the detection results; The image with the highest overall quality score is selected as the target image for the current sequence.

8. The method according to claim 7, characterized in that, The step of using an image quality assessment model to detect the sharpness, smoothness, reflectivity, and boundaries of each frame in the current sequence includes: The current frame is determined from the current sequence to obtain the current frame; Divide the current frame's image into multiple squares; An image quality assessment model is used to identify the sharpness type of each block in the current frame. The sharpness type includes sharp and blurry. The number of blocks with the sharpness type is counted, and the ratio of the number of blocks with the sharpness type to the total number of blocks in the current frame is calculated to obtain the image sharpness of the current frame. An image quality assessment model is used to identify the flatness type of each block in the current frame. The flatness type includes flat and uneven. The number of blocks with the flatness type of flat is counted, and the ratio of the number of blocks with the flatness type of flat to the total number of blocks in the current frame is calculated to obtain the flatness of the image in the current frame. An image quality assessment model is used to identify the reflection type of each block in the current frame. The reflection type includes reflective and non-reflective. The number of blocks with the reflection type of non-reflective is counted, and the ratio of the number of blocks with the reflection type of non-reflective to the total number of blocks in the current frame is calculated to obtain the reflectivity of the image in the current frame. The image quality assessment model is used to detect the coordinates of the four vertices of the document in the current frame. Based on the coordinates of the four vertices, it is determined whether the four vertices are in the current frame. The number of vertices in the current frame is counted, and the ratio of the number of vertices in the current frame to 4 is calculated to obtain the boundary detection result of the image in the current frame. Return to the step of determining the frame that needs to be detected in the current sequence, until all frames in the current sequence have been detected.

9. The method according to claim 7, characterized in that, Before merging the target images according to their order in the processed video sequence, the method further includes: Determine whether the sharpness, smoothness, reflectivity, and boundary detection results of the target image all meet the preset quality requirements; If so, then proceed with the step of merging the target images according to their order in the processed video sequence; If not, a message indicating that the image does not meet quality requirements will be generated.

10. A device for comparing and extracting image frames in a video sequence of document browsing, characterized in that, include: An acquisition unit is used to acquire a video sequence to be identified, wherein the video sequence to be identified records the process of flipping through physical documents; The state recognition unit is used to identify the state type of each frame image in the video sequence to be recognized by using a preset trained convolutional neural network. The state type includes "document flattened" and "document being flipped". The segmentation unit is used to divide the video sequence to be identified into multiple image sequences that alternate between two state types: "document flattened" and "document flipped open," based on the position and state type of each frame image in the video sequence to be identified. The image sequence with the state type "document flattened" is the first type of image sequence, and the image sequence with the state type "document flipped open" is the second type of image sequence. An action recognition unit is used to identify the action type of the second type of image sequence using a preset trained recurrent neural network, wherein the action type includes complete page turning and incomplete page turning; The processing unit is used to delete the second type of image sequence with the action type of incomplete page turning, and merge the two first type of image sequences before and after the second type of image sequence with the action type of incomplete page turning to obtain the processed video sequence; The pagination recognition unit is used to perform pagination recognition based on the processed video sequence.

Images