Book image processing method and related device

Abstract

The invention discloses a book image processing method and a related device, and relates to the technical field of image processing, and the method comprises the steps: determining a single-page book image region from a book image, and determining a left upper corner point, a right upper corner point, a right lower corner point and a left lower corner point of the single-page book image in the single-page book image region; and finally, determining quadrilateral regions corresponding to the left upper corner point, the right upper corner point, the right lower corner point and the left lower corner point of the single-page book image as effective regions of the single-page book image. As the four angular points are minimum and most stable geometric constraints of the page area, the quadrangle boundary of the page can be uniquely determined, and perspective distortion and complex backgrounds can be adapted. Compared with other features (such as edges, colors and textures) and corner detection, the method is more efficient and robust, so that in the scheme, the effective area of the single page of the book can be obtained by determining the image of the single page of the book and the four corners of the single page of the book, and the accuracy of a search result can be improved by using the effective area to search the book.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a book image processing method and related devices. Background Art

[0002] In the scenario of searching for books by taking photos, the images taken by users using the camera on devices (such as picture book reading robots, learning machines, etc.) often contain content unrelated to the book (such as background, other incomplete pages, etc.), which will affect the accuracy of the search results. Therefore, it is necessary to determine the single-page valid area of ​​the book from the captured image and search based on the single-page valid area of ​​the target object to improve the accuracy of the search results.

[0003] Current methods for determining the effective area include image segmentation algorithms and image prediction algorithms. These two algorithms rely on edge detection (such as Canny) and contour finding to determine the effective area. Since book images often contain multiple elements (text, charts, formulas, etc.), and book images may have some interference due to paper material and printing process (such as paper reflection, binding line shadow, ink penetration, etc.), the above two algorithms cannot determine the effective area of ​​a single page of the book.

[0004] Therefore, how to provide a book image processing method to obtain the effective area of ​​a single page of the book and thereby improve the accuracy of search results has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application provides a book image processing method and related devices to obtain the effective area of ​​a single page of the book, thereby improving the accuracy of search results. The specific solution is as follows:

[0006] A first aspect of the present application provides a book image processing method, comprising:

[0007] Get the book image to be processed;

[0008] Determining a single-page book image region from the book image and determining single-page key points; the single-page key points include an upper left corner point, an upper right corner point, a lower right corner point, and a lower left corner point of the single-page book image in the single-page book image region;

[0009] A quadrilateral area corresponding to the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image is determined as a valid area of ​​the single-page book image.

[0010] In a possible implementation, determining a single-page book image region from the book image and determining key points of the single page includes:

[0011] A pre-trained pose estimation model is used to determine a single-page book image region and single-page key points from the book image. The pose estimation model is trained using training image samples labeled with single-page key point labels and single-page book image region labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region.

[0012] In one possible implementation, the posture estimation model is trained in the following manner:

[0013] Get image samples;

[0014] Labeling each of the image samples to obtain labeled image samples; the labeled labels include single-page book image region labels and single-page key point labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region;

[0015] Performing data augmentation processing on the labeled image samples to obtain training image samples;

[0016] The pose estimation model is trained by taking the model prediction results close to the single-page book image region labels and single-page key point labels of the training image samples as the training target.

[0017] In a possible implementation, after determining the internal area corresponding to the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image as the single-page book image valid area, the method further includes:

[0018] Based on the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image, deformation correction and / or surface correction are performed on the effective area of ​​the single-page book image to obtain a corrected effective area of ​​the single-page book image.

[0019] In a possible implementation, obtaining the book image to be processed includes:

[0020] Acquire a book image taken by a user for the first time, and determine the book image taken by the user for the first time as the book image to be processed;

[0021] or,

[0022] Get the book image taken by the user for the first time;

[0023] Determining whether the book image taken by the user for the first time meets a preset condition;

[0024] If the book image taken by the user for the first time does not meet the preset conditions, outputting shooting guidance information to prompt the user to take the book image again based on the shooting guidance information;

[0025] The book image photographed again by the user based on the photographing guide information is obtained as the book image to be processed.

[0026] In a possible implementation, determining whether the book image taken by the user for the first time meets a preset condition includes:

[0027] Performing a single-page key point confidence detection on the book image taken by the user for the first time to obtain a single-page key point confidence detection result;

[0028] Determine whether the single-page key point confidence detection result meets the preset key point confidence condition;

[0029] If the single-page key point confidence detection result meets the preset key point confidence condition, it is determined that the book image taken by the user for the first time meets the preset condition.

[0030] A second aspect of the present application provides a book image processing device, comprising:

[0031] An acquisition unit, configured to acquire a book image to be processed;

[0032] a single-page book image region and single-page key point determination unit, configured to determine a single-page book image region from the book image and determine single-page key points; the single-page key points comprising the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image in the single-page book image region;

[0033] The single-page book image valid area determining unit is configured to determine a quadrilateral area corresponding to the upper left corner, upper right corner, lower right corner and lower left corner of the single-page book image as the single-page book image valid area.

[0034] In a possible implementation, the single-page book image area and single-page key point determination unit is specifically configured to:

[0035] A pre-trained pose estimation model is used to determine a single-page book image region and single-page key points from the book image. The pose estimation model is trained using training image samples labeled with single-page key point labels and single-page book image region labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region.

[0036] In one possible implementation, the apparatus further includes: a posture estimation model training unit;

[0037] The posture estimation model training unit is specifically used to:

[0038] Get image samples;

[0039] Labeling each of the image samples to obtain labeled image samples; the labeled labels include single-page book image region labels and single-page key point labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region;

[0040] Performing data augmentation processing on the labeled image samples to obtain training image samples;

[0041] The pose estimation model is trained by taking the model prediction results close to the single-page book image region labels and single-page key point labels of the training image samples as the training target.

[0042] In a possible implementation, the apparatus further includes:

[0043] A correction unit is configured to, after determining the internal area corresponding to the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image as the single-page book image valid area, perform deformation correction and / or surface correction on the single-page book image valid area based on the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image, to obtain a corrected single-page book image valid area.

[0044] In a possible implementation, the acquiring unit includes:

[0045] a first acquiring unit, configured to acquire a book image photographed by a user for the first time, and determine the book image photographed by the user for the first time as the book image to be processed;

[0046] or,

[0047] The second acquisition unit is used to acquire a book image taken by a user for the first time; determine whether the book image taken by the user for the first time meets a preset condition; if the book image taken by the user for the first time does not meet the preset condition, output shooting guidance information to prompt the user to take the book image again based on the shooting guidance information; and acquire the book image taken by the user again based on the shooting guidance information as the book image to be processed.

[0048] In a possible implementation, the second acquiring unit is specifically configured to:

[0049] Perform a single-page key point confidence detection on the book image taken by the user for the first time to obtain a single-page key point confidence detection result; determine whether the single-page key point confidence detection result meets a preset key point confidence condition; if the single-page key point confidence detection result meets the preset key point confidence condition, determine that the book image taken by the user for the first time meets the preset condition.

[0050] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the book image processing method of the first aspect or any implementation of the first aspect.

[0051] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0052] The memory is used to store computer programs;

[0053] The processor is configured to execute the computer program so that the electronic device can implement the book image processing method of the first aspect or any implementation manner of the first aspect.

[0054] In a fifth aspect, the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the book image processing method of the first aspect or any implementation of the first aspect.

[0055] By leveraging the above-mentioned technical solution, the present application provides a book image processing method and related apparatus. After acquiring a book image to be processed, the method determines a single-page book image region and single-page key points from the book image. The single-page key points include the top left corner, top right corner, bottom right corner, and bottom left corner of the single-page book image within the single-page book image region. Finally, the quadrilateral region corresponding to the top left corner, top right corner, bottom right corner, and bottom left corner of the single-page book image is determined as the single-page book image valid region. Because the four corner points are the smallest and most stable geometric constraints on the book page region, they can uniquely determine the quadrilateral boundary of the page while also adapting to perspective distortion and complex backgrounds. Compared to other features (such as edges, color, and texture), corner point detection is more efficient and robust. Therefore, in this solution, by determining the single-page book image and its four corner points, the single-page book valid region can be obtained. Using this region for book searches can improve the accuracy of search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0057] Figure 1 A flowchart of a book image processing method provided in an embodiment of the present application;

[0058] Figure 2A schematic diagram of a single-page book image region determined from a book image and key points of a single page provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the effective area of ​​a single-page book image provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a training process for a posture estimation model provided in an embodiment of the present application;

[0061] Figure 5 A schematic structural diagram of a book image processing device provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0064] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0065] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0066] In the scenario of searching for books by taking photos, the images taken by users using the camera on devices (such as picture book reading robots, learning machines, etc.) often contain content unrelated to the book (such as background, other incomplete pages, etc.), which will affect the accuracy of the search results. Therefore, it is necessary to determine the single-page valid area of ​​the book from the captured image and search based on the single-page valid area of ​​the target object to improve the accuracy of the search results.

[0067] Current methods for determining the effective area include image segmentation algorithms and image prediction algorithms. These two algorithms rely on edge detection (such as Canny) and contour finding to determine the effective area. Since book images often contain multiple elements (text, charts, formulas, etc.), and book images may have some interference due to paper material and printing process (such as paper reflection, binding line shadow, ink penetration, etc.), the above two algorithms cannot determine the effective area of ​​a single page of the book.

[0068] To address the aforementioned issues, embodiments of the present application provide a book image processing method. This method can be deployed on various embedded devices (CPU, GPU, or NPU devices), such as picture book reading robots and learning machines. It processes user-captured book images in real time, detects the valid area of ​​a single-page book image, and produces a complete single-page book image. This single-page book image can then be used for book searches, improving the accuracy of search results.

[0069] The book image processing method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0070] Reference Figure 1 , Figure 1 A flow chart of a book image processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, a book image processing method provided by an embodiment of the present application may include the following steps, which are described in detail below.

[0071] S101: Acquire a book image to be processed;

[0072] In this application, the book image to be processed can be an image of any book taken at any angle by a user using a camera on a device (such as a picture book reading robot, a learning machine, etc.). This application does not impose any restrictions on this.

[0073] S102: determining a single-page book image region from the book image, and determining single-page key points; the single-page key points include an upper left corner point, an upper right corner point, a lower right corner point, and a lower left corner point of the single-page book image in the single-page book image region;

[0074] For ease of understanding, refer to Figure 2 , Figure 2 The schematic diagram of the single-page book image area determined from the book image and the key points of the single page provided in the embodiment of the present application is as follows: Figure 2 As shown, the inner area corresponding to the red rectangular box is the single-page book image area, wherein the upper left corner, upper right corner, lower right corner and lower left corner of the single-page book image (i.e., the four red dots in the red rectangular box) are the single-page key points.

[0075] S103: Determine a quadrilateral area corresponding to the upper left corner, the upper right corner, the lower right corner, and the lower left corner of the single-page book image as a valid area of ​​the single-page book image.

[0076] In this application, the quadrilateral corresponding to the upper left corner point, upper right corner point, lower right corner point and lower left corner point of the single-page book image can be obtained by connecting the remaining corner points in sequence with one of the corner points as the origin and ending at the corner point. The quadrilateral area corresponding to the upper left corner point, upper right corner point, lower right corner point and lower left corner point of the single-page book image refers to the inner area of ​​the quadrilateral corresponding to the upper left corner point, upper right corner point, lower right corner point and lower left corner point of the single-page book image. For ease of understanding, refer to Figure 3 , Figure 3 A schematic diagram of the effective area of ​​a single-page book image provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in the figure, the quadrilateral area obtained by connecting the four red points in the red rectangular box (the quadrilateral corresponding to the black box) is the valid area of ​​the single-page book image.

[0077] This embodiment provides a book image processing method. After acquiring a book image to be processed, a single-page book image region and key points of the single page are determined from the image. The key points of the single page include the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image region. Finally, the quadrilateral region corresponding to the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image is determined as the valid region of the single-page book image. Because the four corner points are the smallest and most stable geometric constraints of the book page region, they can uniquely determine the quadrilateral boundary of the page while also adapting to perspective distortion and complex backgrounds. Compared to other features (such as edges, color, and texture), corner point detection is more efficient and robust. Therefore, in this solution, by determining the single-page book image and its four corner points, the valid region of the single page can be obtained. Using this region for book searches can improve the accuracy of search results.

[0078] Pose estimation is a core task in computer vision, aiming to detect and locate the spatial positions of key points of a person or object and infer their pose structure. It has wide applications in areas such as motion control, human-computer interaction, animation, and sports analysis. In this application, a pre-trained pose estimation model can be used to determine the image region of a single page and identify key points within the book image.

[0079] In a possible implementation, determining a single-page book image region from the book image and determining key points of the single page includes:

[0080] A pre-trained pose estimation model is used to determine a single-page book image region and single-page key points from the book image. The pose estimation model is trained using training image samples labeled with single-page key point labels and single-page book image region labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region.

[0081] In another embodiment of the present application, the training method of the posture estimation model is described, referring to Figure 4 , Figure 4 A schematic diagram of a training process for a posture estimation model provided in an embodiment of the present application may include the following steps:

[0082] S201: Obtain image samples;

[0083] In this application, image samples can be book image samples or image samples of other objects, and this application does not impose any restrictions on this. The book image samples can be obtained by collecting books in real scenes, and these books cover multiple publishers, multiple disciplines, etc., and this application does not impose any restrictions on this.

[0084] S202: Labeling each of the image samples to obtain labeled image samples; the labeled labels include single-page book image region labels and single-page key point labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region;

[0085] In this application, a labeling tool can be used to label each of the image samples to obtain the labeled image samples. In this application, the visualization diagram of the single-page book image area label and the single-page key point label can be referred to. Figure 2 , assuming Figure 2 The image in is an image sample, the red rectangle is the single-page book image area label, and the four red dots are the single-page key point labels.

[0086] S203: Performing data enhancement processing on the labeled image samples to obtain training image samples;

[0087] In this application, data enhancement processing is performed on the labeled image samples to increase the diversity of image samples and improve the model training effect.

[0088] In this application, data enhancement processing methods include any one or more of geometric transformation, blur enhancement, and color enhancement. For ease of understanding, in this application, the above data enhancement processing methods are explained separately, as follows:

[0089] Geometric transformation refers to modifying the shape and position of an image, including translation enhancement, scaling enhancement, and rotation enhancement. Scaling the image size, rotating the image angle, and translating the image position increase data diversity.

[0090] Blur enhancement involves applying image processing techniques such as median blur, motion blur, and Gaussian blur to an image. Median blur replaces the central pixel value with the median value of its neighboring pixels (nonlinear filtering), effectively removing salt-and-pepper noise (random black and white noise). Motion blur simulates linear blur (convolution along a specific direction) caused by object or camera movement, improving the model's robustness to motion blur. Gaussian blur applies a weighted average of the image based on a Gaussian function (normal distribution), assigning the highest weight to the central pixel and decreasing weight to surrounding pixels with distance. This simulates lens defocus or depth of field effects, improving the model's robustness.

[0091] Color enhancement uses image processing techniques to adjust image brightness, contrast, saturation, and grayscale. Brightness adjustment increases or decreases the brightness of all pixels (e.g., synchronously scaling the RGB channels). This simulates lighting changes (e.g., day / night scenes) to enable the model to adapt to both day and night. Contrast adjustment stretches or compresses the pixel value range, enhancing or weakening color differences, highlighting details or softening the image, improving the model's learning of details. Saturation adjustment adjusts color saturation (the S channel) in the HSV / HSL space. This simulates the color performance of different devices and enhances the model's robustness across devices. Grayscale adjustment converts images to grayscale, improving the model's ability to learn from different data types.

[0092] S204: Taking the model prediction results close to the single-page book image region labels and single-page key point labels of the training image samples as training targets, a posture estimation model is trained.

[0093] Considering that when users use the camera on a device (such as a picture book reading robot, a learning machine, etc.) to capture a book image, the camera normal is not completely perpendicular to the horizontal plane of the photographed book (for example, in the case of a picture book reading robot, there is a significant acute angle between the camera normal and the picture book). This causes deformation problems (such as perspective deformation, trapezoidal deformation, etc.) in the book image captured. This will cause deformation problems in the effective area of ​​the single-page book image determined above. Using such a deformed single-page book image for searching will also limit the search accuracy.

[0094] Therefore, in another implementation of the present application, after determining the internal area corresponding to the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image as the single-page book image valid area, the method further includes:

[0095] Based on the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image, deformation correction and / or surface correction are performed on the effective area of ​​the single-page book image to obtain a corrected effective area of ​​the single-page book image.

[0096] Considering that image deformation correction is primarily accomplished through perspective transformation, a projective transformation that converts an image from one perspective to another, when photographing documents, photos, etc., the image may appear tilted or distorted due to the shooting angle. Through perspective transformation, the image can be corrected to a front view, facilitating subsequent processing and analysis. Therefore, in this application, perspective transformation can be used to perform deformation correction on the effective area of ​​the single-page book image.

[0097] The perspective transformation can be represented by a 3×3 transformation matrix M:

[0098]

[0099] For the midpoint (x, y) of the original image, after perspective transformation, the point (x', y') on the target image will be obtained. The calculation relationship between them is as follows:

[0100]

[0101] According to the calculation deduction, two equations are obtained:

[0102]

[0103] In the actual calculation process, the M matrix has 8 unknowns. Therefore, to calculate the M matrix, at least four corresponding point coordinates are required. In the present application, the coordinates of the four points of the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image can be used to calculate the M matrix. Compared with the existing technology of calculating the M matrix using the coordinates of points obtained by other methods, the four points of the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image are more accurate, so that the effect of using perspective transformation to correct the deformation of the effective area of ​​the single-page book image is better.

[0104] In this application, a curved surface correction algorithm can be used to correct the curved surface of the effective area of ​​the single-page book image. Compared to the existing technology, which inaccurately determines the effective area and has poor curved surface correction effects, in this application, the effective area of ​​the single-page book image is more accurately determined by using the quadrilateral area corresponding to the four points of the single-page book image: the upper left corner, the upper right corner, the lower right corner, and the lower left corner. Therefore, the curved surface correction effect can be improved.

[0105] In another embodiment of the present application, a specific implementation of obtaining the book image to be processed is described. In one possible implementation, a book image taken by a user for the first time may be obtained, and the book image taken by the user for the first time may be determined as the book image to be processed.

[0106] In another possible implementation, a book image taken by a user for the first time can be obtained; it is determined whether the book image taken by the user for the first time meets a preset condition; if the book image taken by the user for the first time does not meet the preset condition, shooting guidance information is output to prompt the user to take the book image again based on the shooting guidance information; and the book image taken by the user again based on the shooting guidance information is obtained as the book image to be processed.

[0107] In this application, the shooting guidance information is used to prompt the user to capture a book image that meets preset conditions. This information can be in text or voice form, and this application does not impose any restrictions on this. The content of the information can be set based on the needs of the scenario, depending on the preset conditions. In one possible implementation, the prompt content may include prompting the user to adjust the shooting position and angle. If the detection result meets the preset conditions, the book image captured by the user for the first time is determined as the book image to be processed.

[0108] In a possible implementation, determining whether the book image taken by the user for the first time meets a preset condition includes:

[0109] Perform a single-page key point confidence detection on the book image taken by the user for the first time to obtain a single-page key point confidence detection result; determine whether the single-page key point confidence detection result meets a preset key point confidence condition; if the single-page key point confidence detection result meets the preset key point confidence condition, determine that the book image taken by the user for the first time meets the preset condition.

[0110] In the present application, the key points and confidence levels of a single page of a book image taken by the user for the first time can be determined to obtain a single page key point confidence detection result. The key points and confidence levels of a single page of a book image taken by the user for the first time can be determined using a posture estimation model pre-trained in the present application.

[0111] In one possible implementation, a confidence threshold can be set for each single-page key point. If the confidence detection results of each single-page key point in the single-page key point confidence detection results are not lower than the corresponding confidence threshold, it is determined that the single-page key point confidence detection results meet the preset key point confidence conditions.

[0112] A book image processing method provided by an embodiment of the present application has been described above. The following will introduce an apparatus for executing the above-described book image processing method.

[0113] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a book image processing device provided in an embodiment of the present application. Figure 5 As shown, the book image processing device includes:

[0114] An acquisition unit 11 is used to acquire a book image to be processed;

[0115] The single-page book image region and single-page key point determination unit 12 is configured to determine the single-page book image region from the book image and determine the single-page key points; the single-page key points include the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image in the single-page book image region;

[0116] The single-page book image valid area determining unit 13 is configured to determine a quadrilateral area corresponding to the upper left corner, upper right corner, lower right corner, and lower left corner of the single-page book image as the single-page book image valid area.

[0117] In a possible implementation, the single-page book image area and single-page key point determination unit is specifically configured to:

[0118] A pre-trained pose estimation model is used to determine a single-page book image region and single-page key points from the book image. The pose estimation model is trained using training image samples labeled with single-page key point labels and single-page book image region labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region.

[0119] In one possible implementation, the apparatus further includes: a posture estimation model training unit;

[0120] The posture estimation model training unit is specifically used to:

[0121] Get image samples;

[0122] Labeling each of the image samples to obtain labeled image samples; the labeled labels include single-page book image region labels and single-page key point labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region;

[0123] Performing data augmentation processing on the labeled image samples to obtain training image samples;

[0124] The pose estimation model is trained by taking the model prediction results close to the single-page book image region labels and single-page key point labels of the training image samples as the training target.

[0125] In a possible implementation, the apparatus further includes:

[0126] A correction unit is configured to, after determining the internal area corresponding to the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image as the single-page book image valid area, perform deformation correction and / or surface correction on the single-page book image valid area based on the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the single-page book image, to obtain a corrected single-page book image valid area.

[0127] In a possible implementation, the acquiring unit includes:

[0128] a first acquiring unit, configured to acquire a book image photographed by a user for the first time, and determine the book image photographed by the user for the first time as the book image to be processed;

[0129] or,

[0130] The second acquisition unit is used to acquire a book image taken by a user for the first time; determine whether the book image taken by the user for the first time meets a preset condition; if the book image taken by the user for the first time does not meet the preset condition, output shooting guidance information to prompt the user to take the book image again based on the shooting guidance information; and acquire the book image taken by the user again based on the shooting guidance information as the book image to be processed.

[0131] In a possible implementation, the second acquiring unit is specifically configured to:

[0132] Perform a single-page key point confidence detection on the book image taken by the user for the first time to obtain a single-page key point confidence detection result; determine whether the single-page key point confidence detection result meets a preset key point confidence condition; if the single-page key point confidence detection result meets the preset key point confidence condition, determine that the book image taken by the user for the first time meets the preset condition.

[0133] Each unit in the above-mentioned book image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above-mentioned units.

[0134] An electronic device is also provided in an embodiment of the present application. Figure 6, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0135] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0136] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0137] An embodiment of the present application further provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the book image processing methods provided in the embodiments of the present application.

[0138] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any book image processing method provided in the embodiment of the present application.

[0139] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0140] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0141] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0142] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A book image processing method, characterized in that: include: Get the book image to be processed; Determining a single-page book image region from the book image, and determining key points of the single page; The single-page key points include the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image in the single-page book image area; A quadrilateral area corresponding to the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image is determined as a valid area of ​​the single-page book image.

2. The method according to claim 1, characterized in that The step of determining a single-page book image region from the book image and determining key points of the single page includes: A pre-trained pose estimation model is used to determine a single-page book image region and single-page key points from the book image. The pose estimation model is trained using training image samples labeled with single-page key point labels and single-page book image region labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region.

3. The method according to claim 2, characterized in that The training method of the posture estimation model includes: Get image samples; Labeling each of the image samples to obtain labeled image samples; the labeled labels include single-page book image region labels and single-page key point labels; the single-page book image region labels are used to indicate whether the image sample contains a single-page book image region; Performing data augmentation processing on the labeled image samples to obtain training image samples; The pose estimation model is trained by taking the model prediction results close to the single-page book image region labels and single-page key point labels of the training image samples as the training target.

4. The method according to any one of claims 1 to 3, characterized in that After determining the inner areas corresponding to the upper left corner, the upper right corner, the lower right corner, and the lower left corner of the single-page book image as the single-page book image valid area, the method further includes: Based on the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image, deformation correction and / or surface correction are performed on the effective area of ​​the single-page book image to obtain a corrected effective area of ​​the single-page book image.

5. The method according to claim 1, wherein The step of obtaining the book image to be processed includes: Acquire a book image taken by a user for the first time, and determine the book image taken by the user for the first time as the book image to be processed; or, Get the book image taken by the user for the first time; Determining whether the book image taken by the user for the first time meets a preset condition; If the book image taken by the user for the first time does not meet the preset conditions, outputting shooting guidance information to prompt the user to take the book image again based on the shooting guidance information; The book image photographed again by the user based on the photographing guide information is obtained as the book image to be processed.

6. The method according to claim 5, characterized in that The determining whether the book image taken by the user for the first time meets a preset condition includes: Performing a single-page key point confidence detection on the book image taken by the user for the first time to obtain a single-page key point confidence detection result; Determine whether the single-page key point confidence detection result meets the preset key point confidence condition; If the single-page key point confidence detection result meets the preset key point confidence condition, it is determined that the book image taken by the user for the first time meets the preset condition.

7. A book image processing device, characterized in that: include: An acquisition unit, configured to acquire a book image to be processed; a single-page book image region and single-page key point determination unit, configured to determine a single-page book image region and single-page key points from the book image; The single-page key points include the upper left corner point, the upper right corner point, the lower right corner point and the lower left corner point of the single-page book image in the single-page book image area; The single-page book image valid area determining unit is configured to determine a quadrilateral area corresponding to the upper left corner, upper right corner, lower right corner and lower left corner of the single-page book image as the single-page book image valid area.

8. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the book image processing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the book image processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the book image processing method according to any one of claims 1 to 6.

Images