Method and device for electronizing book through automatic shooting

Through optical flow estimation and corner point detection models, the page turning action and page stable state of the book are automatically identified, and the electronicization of books is realized, which solves the problem of low processing efficiency in the existing technology, and improves the operation efficiency of electronic books and Image quality.

CN119027460APending Publication Date: 2024-11-26SHANGHAI HEHE INFORMATION TECH DEV +3
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410962980.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing methods of electronicizing books require manual or mechanical photography to take page by page, and the processing efficiency is low and slow, so it is impossible to realize page recognition and shooting during the automated continuous page turn.

Method used

The optical flow estimation model and corner point detection model are used to automatically identify the book page turning action and page stable state through video frame extraction and corner point annotation. Automatic shooting is performed only in the stable state, and subsequent image processing is performed to keep the content aligned.

Benefits of technology

It realizes the electronicization of automated books without manually judging the shooting timing, improves operation efficiency and processing speed, and ensures clear and stable shooting of each page.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027460A_ABST
    Figure CN119027460A_ABST
Patent Text Reader

Abstract

The invention discloses a method for electronizing books through automatic shooting. The method comprises the following steps: acquiring continuous and uninterrupted page turning videos of a target paper book, and extracting continuous page turning pictures of the target book; reasoning a page turning action according to any two continuous target book page turning pictures by using a trained optical flow estimation model, and reasoning four corner points of each page of a target book according to any one target book page turning picture by using a trained corner point detection model, whether the target book is in a stable state or not is jointly judged according to the reasoning results of the optical flow estimation model and the angular point detection model, and only one photograph, used for electronization, of the target book is automatically shot only when the target book is in the stable state. According to the method, the shooting opportunity does not need to be manually judged, the page turning action is automatically recognized by using the optical flow estimation algorithm based on deep learning, the stable state suitable for shooting is automatically determined, the operation efficiency of book electronization is improved, and the processing time is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to a method for digitizing books, that is, a method for obtaining electronic books from paper books. Background Art

[0002] With the development of information technology, the digitization of books has become a trend. Digitizing physical paper books to obtain books in the form of electronic documents is conducive to better management, dissemination and utilization of information resources.

[0003] The existing method of digitizing books is to turn the pages of the book manually or mechanically, take a photo of each page of the book when it is in a stable state, and then process and merge the photos of each page of the book to obtain an electronic book. This method has low processing efficiency and slow speed. Summary of the invention

[0004] The technical problem to be solved by the present application is how to automatically identify the stable state of each page of a book in the process of continuous and uninterrupted book turning (turning pages of a book), and take a photo of each page of the book in the automatically identified stable state to digitize the book.

[0005] To solve the above technical problems, the present application proposes a method for automatically shooting and digitizing books, comprising the following steps: Step S1: Obtain a video of a paper book turning pages, extract video frames as book page turning pictures, and mark corner points on each book page turning picture; continuous book page turning pictures and corner point markings are used as training data sets. Step S2: Use any two continuous book page turning pictures in the training data set to train an optical flow estimation model, so that the optical flow estimation model infers the book turning action based on the direction and amplitude of the optical flow between any two continuous book page turning pictures. Step S3: Use any book page turning picture and corner point markings in the training data set to train a corner point detection model, so that the corner point detection model identifies the four corner points of each page of the book from each book page turning picture. The steps S2 and S3 may be performed in any order, simultaneously, or alternately. Step S4: Obtain a video of a target paper book turning pages continuously and uninterruptedly, and extract continuous target book page turning pictures from the video frames. Step S5: Use the trained optical flow estimation model to infer the book turning action based on any two consecutive target book page turning pictures, use the trained corner point detection model to infer the four corner points of each page of the target book based on any target book page turning picture, and jointly judge whether the target book is in a stable state based on the inference results of the optical flow estimation model and the corner point detection model, and automatically take only one target book for electronicization photo when the target book is in a stable state. Step S6: Post-process all the photos of the target book for electronicization, including one or more processing of perspective transformation, deformation correction, enhanced sharpening, and clarity improvement. Step S7: The processed electronic pictures of all pages of the target book are sent to the trained optical flow estimation model, and the trained optical flow estimation model performs optical flow estimation on any two consecutive electronic pictures of the target book, and maps the optical flow field of the previous picture to the next picture so that the book content position of the next picture is aligned with the book content position of the previous picture, that is, the blank space around the edges of the book content of the two pictures before and after is kept consistent, and finally the electronic pictures of all pages of the target book are output as electronic books.

[0006] Furthermore, in step S1, the shooting range of the video of the paper book turning pages is one page of the book, which is called a single-page video of the paper book turning pages; the video frame of the single-page video of the paper book turning pages is extracted as a single-page book turning page picture; four corner points are marked on the single-page book turning page picture, which respectively represent the four corner points of the one page of the book being shot. Alternatively, the shooting range of the video of the paper book turning pages is two pages of the book, which is called a double-page video of the paper book turning pages; the video frame of the double-page video of the paper book turning pages is extracted as a double-page book turning page picture; eight corner points are marked on the double-page book turning page picture, which respectively represent the eight corner points of the two pages of the book being shot, and each page has four corner points. Alternatively, step S1 is changed to step S1a; step S1a: obtain a picture of the paper book turning pages, and mark the corner points on each picture of the book turning pages; continuous pictures of the book turning pages and corner point markings are used as a training data set.

[0007] Furthermore, in step S1 or step S4, video frames are extracted from the book page turning video as the book page turning pictures, and a video frame is extracted every n video frames, where the value range of n is an integer between 0 and 5.

[0008] Furthermore, in step S2, the amplitude of the optical flow indicates the degree of turning the book; if the amplitude of the optical flow between any two consecutive pictures of turning the book pages is not 0, it indicates that the book is being turned, and at this time, the direction of the optical flow is combined to determine whether the book is being turned backward or forward; if the amplitude of the optical flow between any two consecutive pictures of turning the book pages is 0, it indicates that the book is not turned; if the direction of the optical flow between any two consecutive pictures of turning the book pages is to the left, it indicates that the book is being turned forward, and if the direction of the optical flow between any two consecutive pictures of turning the book pages is to the right, it indicates that the book is being turned backward.

[0009] Furthermore, in step S2, the optical flow estimation model inputs two consecutive book page turning pictures, called picture A and picture B, during the training stage, and the optical flow estimation model estimates the optical flow from picture A to picture B and outputs an optical flow field; when the optical flow estimation model calculates the loss, the optical flow field is applied to picture A for spatial transformation to obtain A', and the image similarity of A' and B is constrained to be as close as possible; the optical flow estimation model also maps the optical flow field of picture A to picture B, so that the position of the book content in the latter picture is aligned with the position of the book content in the former picture, so that the blank space around the edges of the book content in the former and latter two pictures remains consistent.

[0010] Furthermore, in step S2, the optical flow estimation model uses a weighted sum of SSIM loss, Grad loss and L1 loss as a training loss function during training, and calculates and back-propagates the loss on feature maps of multiple different resolutions.

[0011] Furthermore, in the step S2, the optical flow estimation model adopts any one of the FlowNet series network, the RAFT series network, and the FlowFormer series network; in the step S3, the corner detection model adopts any one of the YolO series network, the EAST series network, and the DETR series network.

[0012] Further, in step S5, when the book turning action inferred by the optical flow estimation model for two consecutive target book page turning pictures is forward turning or backward turning, it is determined that the target book is not in a stable state. When the corner point positions obtained by the corner point detection model for two consecutive target book page turning pictures change, it is determined that the target book is not in a stable state. When the book turning action inferred by the optical flow estimation model for two consecutive target book page turning pictures is not turning the book, and the corner point positions obtained by the corner point detection model for two consecutive target book page turning pictures do not change, it is determined that the target book is in a stable state.

[0013] Furthermore, in step S7, the trained optical flow estimation model is used to map the optical flow calculation coordinates. For two consecutive target book page turning pictures, referred to as pictures A and B, the optical flow is estimated using pictures A and B. According to the spatial transformation of A' calculated by the optical flow, A' and B correspond in spatial coordinates, so that the content positions of the two pictures before and after the processing correspond, that is, the surrounding blank space is consistent.

[0014] The present application also proposes a device for automatically photographing and digitizing books, including a training data construction unit, a first model training unit, a second model training unit, an input data processing unit, a judgment and shooting unit, an image post-processing unit, and an image blank alignment unit. The training data construction unit is used to obtain a video of a paper book turning pages, extract video frames as book page turning pictures, and mark corner points on each book page turning picture; continuous book page turning pictures and corner point markings are used as training data sets. The first model training unit is used to train an optical flow estimation model using any two continuous book page turning pictures in the training data set, so that the optical flow estimation model infers the book turning action based on the direction and amplitude of the optical flow between any two continuous book page turning pictures. The second model training unit is used to train a corner point detection model using any book page turning picture and corner point markings in the training data set, so that the corner point detection model identifies the four corner points of each page of the book from each book page turning picture. The input data processing unit is used to obtain a video of a target paper book turning pages continuously and uninterruptedly, and extract continuous target book page turning pictures from the video frames. The judging and shooting unit is used to use the trained optical flow estimation model to infer the book turning action based on any two consecutive target book page turning pictures, and use the trained corner point detection model to infer the four corner points of each page of the target book based on any target book page turning picture; based on the inference results of the optical flow estimation model and the corner point detection model, jointly judge whether the target book is in a stable state, and automatically shoot only one target book for electronicization photo when the target book is in a stable state. The image post-processing unit is used to perform one or more of perspective transformation, deformation correction, enhanced sharpening, and clarity improvement on all the photos of the target book for electronicization. The image blank alignment unit is used to use the trained optical flow estimation model to perform optical flow estimation on any two consecutive electronic pictures of the target book, map the optical flow field of the previous picture to the next picture so that the book content position of the next picture is aligned with the book content position of the previous picture, so that the blank margins around the book content of the two pictures before and after are consistent, and finally output the electronic pictures of all the pages of the target book as electronic books.

[0015] The technical effect achieved by the present application is: there is no need to manually judge the shooting timing, but to use the optical flow estimation algorithm based on deep learning to automatically identify the book turning action, automatically determine the stable state suitable for shooting, and automatically shoot the book in each stable state, thereby improving the operational efficiency of digitizing books and shortening the processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method for digitizing books by automatically photographing proposed in this application.

[0017] Figure 2It is a schematic diagram of the structure of the device for automatically photographing and digitizing books proposed in this application.

[0018] Explanation of the reference numerals in the figure: training data construction unit 1, first model training unit 2, second model training unit 3, input data processing unit 4, judgment and shooting unit 5, image post-processing unit 6, image blank alignment unit 7. DETAILED DESCRIPTION

[0019] See also Figure 1 The method for automatically photographing and digitizing books proposed in this application includes the following steps.

[0020] Step S1: Obtain a video of a paper book turning pages, extract video frames as book page turning pictures, and mark corner points on each book page turning picture. All book page turning pictures are arranged in the order of the video frames in the video stream, that is, in the order of the paper book turning process, which are called continuous book page turning pictures. The continuous book page turning pictures and corner point annotations (also called labels) are used as training data sets.

[0021] In one case, the shooting range of the video of turning pages of a paper book is a page of the book, which means that only the left half or the right half of a page of a flat book is shot, which is called a single-page video of turning pages of a paper book. The video frame of the single-page video of turning pages of a paper book is extracted as a single-page turning picture of the book. Four corner points are marked on the single-page turning picture of the book, respectively representing the four corner points of a page of the book shot.

[0022] In another case, the shooting range of the video of the paper book turning pages is two pages of the book, which refers to shooting the two pages of the left half and the right half of the flat book, which is called the double-page video of the paper book turning pages. The video frames of the double-page video of the paper book turning pages are extracted as the book double-page turning page picture. Eight corner points are marked on the book double-page turning page picture, which respectively represent the eight corner points of the two pages of the book shot, and each page has four corner points.

[0023] In this step, video frames are extracted from the book page turning video as book page turning pictures, and a video frame is extracted every n video frames, where the value range of n is an integer between 0 and 5. If n=0, it means frame-by-frame extraction. The book page turning pictures include pictures in the process of turning the book page, at which time each page of the book has not reached a stable state, and pictures after the book page turning is stabilized, at which time each page of the book has reached a stable state.

[0024] Alternatively, the step S1 is changed to step S1a. Step S1a: Obtain a picture of a paper book turning pages. For example, automatically take pictures of the page turning process of a paper book at fixed time intervals. Mark the corner points on each picture of the book turning pages. In one case, the shooting range of the book turning page picture is one page of the book, which is called a single-page book turning page picture. Four corner points are marked on the single-page book turning page picture. In another case, the shooting range of the book turning page picture is two pages of the book, which is called a double-page book turning page picture. Eight corner points are marked on the double-page book turning page picture. All the book turning page pictures are arranged in the order in which they are taken, that is, in the order in which the paper book turns pages, and are called continuous book turning page pictures. The continuous book turning page pictures and the corner point markings are used as training data sets.

[0025] Step S2: Use any two consecutive book page turning pictures in the training data set to train an optical flow estimation model, so that the optical flow estimation model can infer the book turning action based on the direction and amplitude of the optical flow (also called optical flow field) between any two consecutive book page turning pictures. There are three types of book turning actions: turning the book backward, turning the book forward, and not turning the book. Specifically, the amplitude of the optical flow indicates the degree of turning the book. If the amplitude of the optical flow between any two consecutive book page turning pictures is not 0, it means that the book is being turned. At this time, the direction of the optical flow is combined to determine whether the book is turned backward or forward. If the amplitude of the optical flow between any two consecutive book page turning pictures is 0, it means that the book is not turned. If the direction of the optical flow between any two consecutive book page turning pictures is to the left, it means that the book is turned forward, and if the direction of the optical flow between any two consecutive book page turning pictures is to the right, it means that the book is turned backward.

[0026] Classic optical flow estimation models such as FlowNet (optical flow network), PWCNet (CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume), RAFT (Recurrent All-Pairs Field Transforms for Optical Flow), etc. all input two pictures during the training phase and use the existing optical flow field as supervision to calculate the loss - for example, the endpoint error (EPE) loss.

[0027] The optical flow estimation model used in the present application can be any one of the FlowNet series network, the RAFT series network, and the FlowFormer (optical flow transformer) series network. The input of the optical flow estimation model of the present application is only two consecutive images, and it is impossible to obtain direct supervision of the real optical flow field. The optical flow estimation model of the present application inputs two consecutive book page turning pictures in the training stage-for example, called Figure A and Figure B, and the optical flow estimation model estimates the optical flow from Figure A to Figure B, and outputs the optical flow field. When the optical flow estimation model of the present application calculates the loss, the optical flow field is applied to Figure A for spatial transformation to obtain A', and the image similarity of A' and B is constrained to be as close as possible. In addition, the optical flow estimation model of the present application also maps the optical flow field of the previous picture (Figure A) to the next picture (Figure B), so that the position of the book content of the next picture corresponds (aligns) to the position of the book content of the previous picture, so that the blank space around the edges of the book content of the two pictures before and after is consistent, which is part of the intermediate operation process of the optical flow estimation model, and the subsequent steps will take out the results of the intermediate operation process separately.

[0028] Preferably, the optical flow estimation model of the present application uses, for example, the weighted sum of SSIM (structural similarity index) loss, Grad (gradient) loss and L1 (L1 norm) loss as the loss function of training during training to ensure the smoothness of the optical flow estimation, and calculates the loss and back-propagates on feature maps of multiple different resolutions. The SSIM loss is not affected by pixel values, can measure the structural similarity of images, and is very suitable for the tasks performed by the optical flow estimation model of the present application. The L1 loss is an auxiliary measure of image similarity. In addition, since the optical flow estimation model of the present application does not have direct supervision of the optical flow field, the directly estimated optical flow field may not be continuous and reversible, so an additional gradient loss is added to constrain the reversibility of the optical flow field.

[0029] Step S3: Use any one of the book page turning pictures and corner point annotations in the training data set to train a corner point detection model, so that the corner point detection model can identify the four corner points of each page of the book from each book page turning picture. If the input book page turning picture is a single-page book turning picture, the corner point detection model outputs the four corner points of the single-page book turning picture, which are as close to the four annotated corner points as possible. If the input book page turning picture is a double-page book turning picture, the corner point detection model outputs the eight corner points of the double-page book turning picture, which are as close to the eight annotated corner points as possible.

[0030] The corner detection model used in this application can be any one of the YolO (You Only Look Once) series network, the EAST (Efficient and Accurate Scene Text detection) series network, and the DETR (Detection Transformer) series network. During training, the corner detection model of this application uses, for example, the weighted IoU (Intersection over Union) loss and CE (Cross-Entropy) loss as the loss function of training, calculates the loss on the feature map and back-propagates.

[0031] There is no strict restriction on the order of step S2 and step S3, and they can be performed in any order, or performed simultaneously, or performed alternately.

[0032] Step S4: Aim the camera at the target paper book, and turn the pages of the target paper book continuously and uninterruptedly by manual or mechanical means. Obtain a video of the target paper book turning pages continuously and uninterruptedly, and extract continuous target book page turning pictures from the video frames. For example, extract a video frame every n video frames, where the value range of n is an integer between 0 and 5.

[0033] Step S5: Use the trained optical flow estimation model to infer the book turning action based on any two consecutive target book page turning pictures. There are three types of book turning actions inferred by the optical flow estimation model: turning the book backward, turning the book forward, and not turning the book. Use the trained corner point detection model to infer the four corner points of each page of the target book based on any target book page turning picture. Based on the inference results of the optical flow estimation model and the corner point detection model, jointly judge whether the target book is in a stable state. Specifically, the corner point change information of two consecutive target book page turning pictures is used to assist the inference results of the optical flow estimation model to judge, and only one photo of the target book for electronicization is automatically taken when the target book is in a stable state.

[0034] If the optical flow estimation model infers that the book-turning action for two consecutive target book page-turning images is forward turning or backward turning, no matter what the output of the corner detection model is, it is determined that the target book is not in a stable state.

[0035] If the corner positions obtained by the corner detection model for two consecutive target book page turning images change, no matter what the output of the optical flow estimation model is, it is determined that the target book is not in a stable state.

[0036] If the optical flow estimation model infers that the book turning action for two consecutive target book page turning pictures is not turning the book, and the corner point positions obtained by the corner point detection model for two consecutive target book page turning pictures do not change, the target book is determined to be in a stable state. The target book is in a stable state when two consecutive target book page turning pictures indicate that a certain page of the target book (including single page or double page situations) has been stably displayed in front of the photographic device. At this time, take a photo of the target book and take a photo for book digitization, so that a certain page of the target book (including single page or double page situations) can be clearly and completely obtained.

[0037] This step is verified by the optical flow estimation model and the corner detection model to ensure that a photo for book digitization is taken only when the target book is in a stable state without turning the book. In this way, stable and clear photos of each page of the target book are automatically taken when the target book is in a stable state without turning the book, and only once to avoid repeated photos under the same static conditions; after triggering the forward or backward book turning action, the target book is photographed again after it reaches a stable state again.

[0038] A careful analysis of step S4 and step S5 shows that the shooting of the target paper book in this application is divided into two types. The first shooting type is to shoot a video of the turning process of the target paper book in step S4, which is used to extract video frames as input data for the optical flow estimation model and the corner detection model, and is not used for book digitization. The second shooting type is to take clear and stable photos of each page of the target book after determining that the target book is in a stable state in step S5. It is used for book digitization and is not used as input for any deep learning model. In actual applications, the turning action of the target paper book is continuous and uninterrupted, and the first shooting type (video) is also continuous and uninterrupted, while the second shooting type (photo) is performed intermittently, and the optical flow estimation model and the corner detection model determine the shooting timing of the second shooting type (photo) according to the first shooting type (video).

[0039] Step S6: Perform perspective transformation on all the photos of the target book for digitization. The perspective transformation transforms the same batch of photos into photos of the same size, which is convenient for subsequent output into a book. Optionally, if it is a double-page photo of a book, it is separated into two single-page photos of the book. Then, the pictures of all the pages of the target book are respectively subjected to deformation correction, enhanced sharpness, clarity improvement and other processing to obtain the digitized pictures of all the pages of the target book.

[0040] Step S7: The processed electronic images of all pages of the target book may have different margins around them due to algorithms such as trimming correction, which affects the aesthetics and coordination of the electronic books. This step uses the trained optical flow estimation model to perform optical flow estimation on any two consecutive electronic images of the target book, and maps the optical flow field of the previous image to the next image so that the position of the book content of the next image corresponds (aligns) to the position of the book content of the previous image, so that the margins around the edges of the book content of the two images are consistent. The results of the intermediate calculation process of the optical flow estimation model are used here. Finally, the electronic images of all pages of the target book are aligned into volumes and output as digital books in the form of electronic documents.

[0041] The optical flow field learned by the optical flow estimation model of the present application is a vector of length × width × 2. This step uses the optical flow to calculate the coordinate mapping. For two consecutive target book page turning pictures - for example, called pictures A and B, the optical flow is estimated using pictures A and B, and A' is calculated according to the optical flow (spatial transformation, i.e., optical flow coordinate mapping). A' and B correspond in spatial coordinates, so that the content positions of the two pictures before and after the processing correspond, the surrounding blank space is consistent, and it is more beautiful and coordinated.

[0042] See also Figure 2 The device for automatically photographing and digitizing books proposed in this application includes a training data construction unit 1, a first model training unit 2, a second model training unit 3, an input data processing unit 4, a judgment and photographing unit 5, an image post-processing unit 6, and an image blank alignment unit 7. Figure 2 The device shown corresponds to Figure 1 The method shown.

[0043] The training data construction unit 1 is used to obtain a video of a paper book turning pages, extract video frames as book turning page pictures, and mark corner points on each book turning page picture. Continuous book turning page pictures and corner point markings are used as training data sets.

[0044] The first model training unit 2 is used to train an optical flow estimation model using any two consecutive book page turning pictures in the training data set, so that the optical flow estimation model can infer the book turning action based on the direction and amplitude of the optical flow between any two consecutive book page turning pictures.

[0045] The second model training unit 3 is used to train a corner point detection model using any book page turning picture and corner point annotations in the training data set, so that the corner point detection model can identify four corner points of each page of the book from each book page turning picture.

[0046] The input data processing unit 4 is used to aim the photographic device at the target paper book, obtain a video of the target paper book continuously and uninterruptedly turning pages, and extract continuous target book page turning pictures from the video frames.

[0047] The judging and photographing unit 5 is used to use the trained optical flow estimation model to infer the book turning action based on any two consecutive target book page turning pictures, and use the trained corner point detection model to infer the four corner points of each page of the target book based on any target book page turning picture. Based on the inference results of the optical flow estimation model and the corner point detection model, it is jointly judged whether the target book is in a stable state, and only one photo of the target book for digitization is automatically taken when the target book is in a stable state.

[0048] The image post-processing unit 6 is used to perform one or more of perspective transformation, deformation correction, sharpening enhancement, and definition enhancement on all the photos of the target book for digitization, so as to obtain processed digitized pictures of all pages of the target book.

[0049] The image margin alignment unit 7 is used to use the trained optical flow estimation model to perform optical flow estimation on any two consecutive electronic images of the target book, map the optical flow field of the previous image to the next image so that the book content position of the next image is aligned with the book content position of the previous image, so that the margins around the edges of the book content of the previous and next images are consistent, and finally output the electronic images of all pages of the target book as digital books in the form of electronic documents.

[0050] Compared with the prior art, the method of automatically photographing and digitizing books proposed in this application has the following technical innovations and beneficial effects.

[0051] First, the present application continuously shoots a video of the book flipping process, extracts continuous video frames (allowing to extract one video frame every few video frames) and feeds them into two models. The optical flow estimation model estimates the optical flow between two consecutive frames, and the corner point detection model obtains the corner points of each page of the book, and then judges whether the target book is in a stable state based on the direction and amplitude of the optical flow and the change of the corner points (i.e., the combination of the reasoning results of the two models). Only when the target book is in a stable state, a stable and clear photo of each page of the book is automatically taken for digitization, that is, the time point at which each page of the book can be clearly and stably photographed is automatically determined based on the book flipping video.

[0052] Second, the application can automatically identify single and double pages, without manually specifying whether to shoot a single page or a double page. The application automatically determines based on the number of corner points output by the corner point detection model. If 4 corner points belong to a single page, and 8 corner points belong to a double page, it is automatically cut into two single pages. The human-computer interaction interface of the corresponding processing software (APP) displays a frame surrounding a single-page book for a single page, and two frames surrounding each page of a double-page book for a double page. The automatic identification of single and double pages is more intelligent with the animation display.

[0053] Third, the present application aligns the document margin position based on optical flow mapping. The optical flow estimation model of the present application is used as a whole to obtain the book turning action represented by two consecutive book turning pictures, and the intermediate operation process of optical flow mapping is included in the calculation process. The present application reuses the optical flow estimation model, and separately extracts the results of the intermediate operation process of the optical flow mapping to ensure that all electronic pictures have consistent edge margins.

[0054] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A method for automatically photographing and digitizing books, characterized in that: The steps include: Step S1: Obtain a video of a paper book turning pages, extract video frames as book page turning pictures, and mark corner points on each book page turning picture; continuous book page turning pictures and corner point markings are used as a training data set; Step S2: using any two consecutive book page turning pictures in the training data set to train an optical flow estimation model, so that the optical flow estimation model can infer the book turning action according to the direction and amplitude of the optical flow between any two consecutive book page turning pictures; Step S3: using any book page turning picture and corner point annotation in the training data set to train a corner point detection model, so that the corner point detection model can identify four corner points of each page of the book from each book page turning picture; The step S2 and step S3 may be performed in sequence, simultaneously, or alternately; Step S4: obtaining a video of the target paper book continuously and uninterruptedly turning pages, and extracting continuous target book page turning images from the video frames; Step S5: using the trained optical flow estimation model to infer a book turning action based on any two consecutive target book page turning pictures, using the trained corner point detection model to infer four corner points of each page of the target book based on any target book page turning picture, and jointly judging whether the target book is in a stable state based on the inference results of the optical flow estimation model and the corner point detection model, and automatically taking only one photo of the target book for digitization only when the target book is in a stable state; Step S6: post-processing all the photos of the target book for digitization, including one or more processing of perspective transformation, deformation correction, enhanced sharpness, and clarity improvement; Step S7: The processed electronic images of all pages of the target book are sent to the trained optical flow estimation model. The trained optical flow estimation model performs optical flow estimation on any two consecutive electronic images of the target book, and maps the optical flow field of the previous image to the next image so that the position of the book content of the next image is aligned with the position of the book content of the previous image, that is, the blank space around the edges of the book content of the previous and next images is kept consistent, and finally the electronic images of all pages of the target book are output as electronic books.

2. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In the step S1, the shooting range of the video of turning pages of a paper book is one page of the book, which is called a single-page video of turning pages of a paper book; extracting a video frame of the single-page video of turning pages of a paper book as a book single-page turning picture; marking four corner points on the book single-page turning picture, which respectively represent four corner points of a page of the book that is shot; Alternatively, the shooting range of the video of the paper book turning pages is two pages of the book, which is called a double-page video of the paper book turning pages; extracting video frames of the double-page video of the paper book turning pages as a double-page turning picture of the book; marking eight corner points on the double-page turning picture of the book, respectively representing eight corner points of the two pages of the book shot, and four corner points on each page; Alternatively, the step S1 is changed to step S1a; Step S1a: obtaining a picture of a paper book turning pages, and marking a corner point on each picture of a book turning pages; Continuous book page turning pictures and corner annotations are used as training data sets.

3. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In the step S1 or step S4, video frames are extracted from the book page turning video as the book page turning pictures, and a video frame is extracted every n video frames, where the value range of n is an integer between 0 and 5.

4. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In step S2, the amplitude of the optical flow indicates the degree of turning the book; if the amplitude of the optical flow between any two consecutive pictures of turning the book pages is not 0, it means that the book is being turned, and at this time, the direction of the optical flow is combined to determine whether the book is turning backward or forward; if the amplitude of the optical flow between any two consecutive pictures of turning the book pages is 0, it means that the book is not turned; if the direction of the optical flow between any two consecutive pictures of turning the book pages is to the left, it means that the book is turning forward, and if the direction of the optical flow between any two consecutive pictures of turning the book pages is to the right, it means that the book is turning backward.

5. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In step S2, the optical flow estimation model inputs two consecutive book page turning pictures, referred to as picture A and picture B, during the training phase, and the optical flow estimation model estimates the optical flow from picture A to picture B and outputs an optical flow field; When the optical flow estimation model calculates the loss, the optical flow field is applied to the image A to perform spatial transformation to obtain A', and the image similarity between A' and B is constrained to be as close as possible; The optical flow estimation model also maps the optical flow field of Figure A to Figure B, so that the position of the book content in the latter picture is aligned with the position of the book content in the previous picture, so that the blank space around the edges of the book content in the front and back pictures remains consistent.

6. The method for automatically photographing and digitizing books according to claim 5, characterized in that: In step S2, the optical flow estimation model uses the weighted sum of SSIM loss, Grad loss and L1 loss as the training loss function during training, and calculates the loss on feature maps of multiple different resolutions and back-propagates.

7. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In the step S2, the optical flow estimation model adopts any one of the FlowNet series network, the RAFT series network, and the FlowFormer series network; in the step S3, the corner detection model adopts any one of the YolO series network, the EAST series network, and the DETR series network.

8. The method for automatically photographing and digitizing books according to claim 1, characterized in that: In the step S5, when the optical flow estimation model infers that the book turning action for two consecutive target book page turning pictures is forward turning or backward turning, it is determined that the target book is not in a stable state; When the corner point positions obtained by the corner point detection model for two consecutive target book page turning images change, it is determined that the target book is not in a stable state; When the optical flow estimation model infers that the book turning action for two consecutive target book page turning pictures is not turning the book, and the corner point positions obtained by the corner point detection model for two consecutive target book page turning pictures do not change, it is determined that the target book is in a stable state.

9. The method for automatically photographing and digitizing books according to claim 5, characterized in that: In step S7, the trained optical flow estimation model is used to map the optical flow calculation coordinates. For two consecutive target book page turning pictures, referred to as pictures A and B, the optical flow is estimated using pictures A and B. The spatial transformation of A' is calculated based on the optical flow, and A' and B correspond in spatial coordinates, so that the content positions of the two pictures after processing correspond, that is, the surrounding blank spaces are consistent.

10. A device for automatically photographing and converting books into electronic form, characterized in that: It includes a training data construction unit, a first model training unit, a second model training unit, an input data processing unit, a judgment and shooting unit, an image post-processing unit, and an image blank alignment unit; The training data construction unit is used to obtain a video of a paper book turning pages, extract video frames as book page turning pictures, and mark corner points on each book page turning picture; Continuous book page turning pictures and corner annotations are used as training data sets; The first model training unit is used to train an optical flow estimation model using any two consecutive book page turning pictures in the training data set, so that the optical flow estimation model can infer the book turning action according to the direction and amplitude of the optical flow between any two consecutive book page turning pictures; The second model training unit is used to train a corner point detection model using any book page turning picture and corner point annotations in the training data set, so that the corner point detection model can identify four corner points of each page of the book from each book page turning picture; The input data processing unit is used to obtain a video of a target paper book continuously and uninterruptedly turning pages, and extract continuous target book page turning pictures from the video frames; The judging and photographing unit is used to use the trained optical flow estimation model to infer a book turning action based on any two consecutive target book page turning pictures, and use the trained corner point detection model to infer four corner points of each page of the target book based on any target book page turning picture; jointly judge whether the target book is in a stable state based on the inference results of the optical flow estimation model and the corner point detection model, and automatically take only one photo of the target book for electronicization only when the target book is in a stable state; The image post-processing unit is used to perform one or more of perspective transformation, deformation correction, enhanced sharpness, and definition improvement on all photos of the target book for digitization; The image margin alignment unit is used to use the trained optical flow estimation model to perform optical flow estimation on any two consecutive electronic pictures of the target book, map the optical flow field of the previous picture to the next picture so that the book content position of the next picture is aligned with the book content position of the previous picture, so that the margins around the edges of the book content of the previous and next pictures are consistent, and finally output the electronic pictures of all pages of the target book as electronic books.

Citation Information

Patent Citations

  • Page turning detection method and device, electronic device and storage medium

    CN110443224A

  • Electronic book generation method and device, and medium

    CN111556251A

  • Multi-page scanning based on machine learning

    CN117082178A

  • Overhead scanner apparatus, image acquisition method, and program

    JP2011254366A

  • Method of shooting page

    TW201709105A