Electronic device, method of controlling electronic device, and computer-readable storage medium

Abstract

An electronic device (100) includes a camera module (10) that photographs a photo of an object to acquire a main camera image, a distance sensor module (20) that acquires distance depth information of the object by using light, and an image signal processor (30) that controls the camera module (10) and the distance sensor module (20) to acquire a camera image based on the main camera image and the distance depth information.

Description

Technical Field

[0001] The present invention relates to an electronic device, a method for controlling the electronic device, and a computer-readable storage medium. Background Technology

[0002] Recently, devices that capture images of books by pressing the book against a flat glass surface and scanning it from the bottom with a line sensor have become popular.

[0003] Some devices can photograph books from the top when they are open; however, these devices are more expensive and have a lower penetration rate than previous types.

[0004] Users who read a lot of books often use a format where the books are unfolded into sheets of paper and read them continuously.

[0005] On the other hand, there is also a traditional technique that uses a smartphone camera to read paper placed on a flat surface. This traditional technique is implemented through free apps and is widely used. However, in the existing technology, when the paper is bent, the surface of the paper also appears bent in the image read from the paper.

[0006] Therefore, it is necessary to use the camera of an electronic device such as a smartphone to obtain an image of the page surface extending in a plane from the image of the curved page surface that has unfolded as the book is opened.

[0007] Summary of the Invention - This disclosure aims to solve at least one of the aforementioned technical problems. Accordingly, this disclosure requires providing an electronic device and a method for controlling the electronic device.

[0008] According to this disclosure, electronic devices may include a camera module, a distance sensor module, and an image signal processor.

[0009] The camera module takes photos of the object to obtain images from the main camera. The distance sensor module uses light to obtain distance and depth information of an object.

[0010] The image signal processor controls the camera module and distance sensor module based on the main camera image and distance and depth information to acquire camera images.

[0011] The image signal processor estimates the curve corresponding to the position on the page surface based on the main camera image and distance and depth information, and then estimates the surface corresponding to the position on the page surface when the book is open.

[0012] Based on the main camera image, the image signal processor corrects the estimated surface.

[0013] Based on the corrected and estimated surface, the image signal processor acquires an image of the page surface projected onto a plane by transforming the surface projection of the page in the main camera image into a plane.

[0014] In some embodiments, the distance sensor module emits pulsed light toward the object and detects the reflected light of the pulsed light reflected from the object, thereby obtaining time-of-flight (ToF) depth information as distance-depth information.

[0015] In some embodiments, wherein The image signal processor controls the camera module and the distance sensor module to acquire main camera images of the curved page surface and distance and depth information of the curved page surface. The image signal processor estimates the curve corresponding to the position on the page surface based on the main camera image and distance / depth information, and then estimates the surface corresponding to the position on the page surface. The image signal processor uses image information extracted from the main camera image of the book's page surface to correct the estimated surface curvature.

[0016] In some embodiments, the image signal processor uses the contours of the book's page surface extracted from the main camera image to correct the estimated surface curvature.

[0017] In some embodiments, wherein, in the case of correcting the estimated surface, The image signal processor extracts the outline of the book's page surface from the main camera image; The image signal processor uses Time-of-Flight (ToF) depth information on the page surface to obtain candidate coordinates of the line segment connecting the upper and lower ends of the book's crease. The image signal processor obtains candidate line segments by projecting and transforming the candidate coordinates of the line segments into an image that includes the contours on the page surface. The image signal processor selects the crease lines of a book from the line segments obtained by searching the contour using candidate line segments; The image signal processor obtains the difference between the centroid position of the candidate line segment and the centroid position of the selected crease line as the line segment correction offset; The image signal processor projects and transforms the coordinates of the page surface into an image that includes the outline of the page surface by using ToF depth information, and obtains the edge search coordinates based on the coordinates of the edge ends of the page surface. It uses line segment correction offset to make the candidate line segments overlap with the crease lines of the book's outline. The image signal processor obtains the corrected edge coordinates of the book's edge end portion by searching the outline using edge search coordinates, and The image signal processor re-estimates the surface corresponding to the position on the page surface based on the corrected edge coordinates.

[0018] In some embodiments, the image signal processor obtains a main camera image including the curved page surface by using a camera module to capture a photograph of the opened and bent curved page surface; and The image signal processor obtains ToF depth information of the curved page surface through the distance sensor module.

[0019] In some embodiments, the image signal processor estimates a curve corresponding to the location of the curved page surface in a plane perpendicular to the crease direction at the location of the open crease of the page, based on the main camera image and ToF depth information.

[0020] In some embodiments, the image signal processor sets a crease location specified by the user on the open page in an image captured by the camera module while the book is open; and Specifically, by using a camera module to capture an image of the curved page surface of the opened and bent pages while the book is open, the image signal processor acquires a main camera image including the curved page surface, and sets a crease location specification box in the main camera image.

[0021] In some embodiments, the system may also include a display module, an input module, and a main processor.

[0022] The display module shows predefined information.

[0023] The input module receives the user's operation.

[0024] The main processor controls the display module and the input module.

[0025] The image signal processor displays the crease location specification box on the display module along with the main camera image captured by the camera module; The image signal processor responds to the operation input related to the instruction of the user to the input module regarding the crease position specification box, and sets the crease position specification box at the position specified by the user on the curved page surface of the main camera image.

[0026] In some embodiments, the detection resolution of the distance sensor module is lower than that of the camera module.

[0027] According to this disclosure, a method for controlling an electronic device includes: a camera module that captures a photograph of an object to obtain a main camera image; a distance sensor module that acquires distance and depth information of the object using light; and an image signal processor that controls the camera module and the distance sensor module to acquire a camera image based on the main camera image and the distance and depth information. The method includes: estimating a curve corresponding to the position of a page surface based on the main camera image and the distance and depth information using the processor, and further estimating a surface corresponding to the position of the page surface when the book is open; correcting the estimated surface based on the main camera image using the processor; and acquiring an image of the page surface projected onto a plane by transforming the surface projection of the page in the main camera image into a plane using the processor.

[0028] According to this disclosure, a computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program implements a method for controlling an electronic device, the electronic device comprising: a camera module that captures a photograph of an object to obtain a main camera image; a distance sensor module that acquires distance and depth information of the object by using light; and an image signal processor that controls the camera module and the distance sensor module to acquire a camera image based on the main camera image and the distance and depth information; the method comprising: estimating, by means of the processor, a curve corresponding to the position of the page surface based on the main camera image and the distance and depth information, and further estimating a surface corresponding to the position of the page surface when the book is open; correcting the estimated surface based on the main camera image by means of the processor; and acquiring an image of the page surface projected onto a plane by means of the processor, based on the corrected estimated surface, by transforming the surface projection of the page in the main camera image into a plane. Attached Figure Description

[0029] These and / or other aspects and advantages of the embodiments of this disclosure will become apparent and more readily understood from the following detailed description with reference to the accompanying drawings, in which: Figure 1 This is a diagram illustrating an example arrangement of an electronic device 100 and an object 101 according to an embodiment of the present invention; Figure 2 It is shown Figure 1 A diagram showing an example configuration of the electronic device 100; Figure 3 It is shown that it is used for Figure 1 and Figure 2The diagram illustrates an example of the overall process by which the electronic device 100 acquires an image by transforming the page surface of the book 101 in an open state into a flat plane. Figure 4 It is shown Figure 3 A diagram showing a specific example of step S1 for acquiring main camera images and ToF depth information; Figure 5 This is a diagram showing an example of what is displayed on the display module 45 of the electronic device 100 when a specified crease position is specified; Figure 6 It is shown Figure 3 A diagram showing a specific example of step S2 for performing top view processing; Figure 7A This is a diagram illustrating an example of reference point cloud data P, showing an image of a curved page surface imaged from an oblique direction when the book is open; Figure 7B This is an example diagram showing the first point cloud data P1 obtained by projection transformation from the depth direction (from the front); Figure 8A This is a diagram illustrating an example of scanning along multiple lines L1 relative to the first point cloud data P1 in a direction perpendicular to the longitudinal direction D1 of the box specifying the crease location. Figure 8B This is a diagram illustrating an example of second point cloud data P2 obtained by rotating the first point cloud data by a first rotation R such that the estimated crease position D2 is parallel to a preset reference direction (z-axis direction); Figure 9A This is a diagram illustrating an example of scanning along multiple lines L2 relative to the second point cloud data P2 in a direction perpendicular to a preset reference direction (z-axis direction); Figure 9B This is a diagram illustrating an example of the slope of ridge N in the longitudinal direction of the second point cloud data P2 during scanning; Figure 9C This is a diagram illustrating an example of obtaining third point cloud data P3 by rotating the second point cloud data P2 through a second rotation Q so that the calculated inclination of the ridge N is parallel to a preset reference plane (zy plane); Figure 10A This is a diagram illustrating an example of scanning along multiple lines L3 relative to the third point cloud data P3 in the reference direction (z-axis direction); Figure 10B This is a diagram showing an example of the average value A of the third point cloud data P3 in the depth direction (x-axis direction) near multiple lines L3; Figure 10CThis is a diagram showing an example of a curve E obtained by approximating the average value A using a fourth-order or higher-order polynomial in the directions perpendicular to the reference direction (y-axis direction) and the depth direction (x-axis direction); Figure 11 It shows the execution Figure 6 A diagram showing a specific example of step S241 in the processing of the corrected surface; Figure 12A This is an example diagram showing a main camera image obtained by capturing a book; Figure 12B It shows how to detect Figure 12A The image shown is a diagram of the processed image of the book edges from the main camera image; Figure 12C It shows that by from Figure 12B The image shown is a diagram obtained by extracting the outline of the book from the image shown. Figure 13A This is a diagram showing an example of candidate coordinates for extracting the crease lines of a book from the third point cloud data P3; Figure 13B It is by... Figure 13A The candidate line segments of the extracted crease lines are projected and transformed into a form including Figure 12C The image was obtained by showing the outline of the book. Figure 14A This is a diagram illustrating an example of using candidate line segments to search the outline of a book to obtain the crease lines of the book; Figure 14B It is a graph showing the offset corresponding to the difference between the crease line and the candidate line segment of the book selected in the outline; Figure 15 This is a diagram showing the situation where the top and bottom of the book need to be corrected in the third point cloud data P3; Figure 16A This is a diagram showing the third point cloud data P3 before the correction of the upper and lower ends; Figure 16B It is one of them Figure 16A The third point cloud data P3 shown is projected and transformed before the upper and lower ends are corrected, including... Figure 12C The image shown depicts the outline of a book, aligned with the creases. Figure 17 This is a diagram illustrating the edge coordinates of the edge portion of the outline of a book, obtained by searching for the outline using edge search coordinates. Figure 18A It involves projecting and transforming the third point cloud data P3, which is corrected at the top and bottom, into a format that includes... Figure 12C The image shown depicts the outline of a book, aligned with the creases. Figure 18BIt involves projecting and transforming the third point cloud data P3, which is corrected at the top and bottom, into a format that includes... Figure 12C The image shown depicts the outline of a book, aligned with the creases. Figure 19 It shows the execution Figure 6 A diagram showing a specific example of step S25 in the segmentation process; Figure 20A This is a diagram illustrating an example of dividing curve E into points such that the error between points of curve E approximated by a polynomial falls within an acceptable range. Figure 20B This illustrates dividing the curve E into points such that the error between points of the curve E approximated by the polynomial falls within an acceptable range, and... Figure 20A A graph of consecutive examples; Figure 20C This illustrates dividing the curve E into points such that the error between points of the curve E approximated by the polynomial falls within an acceptable range, and... Figure 20B A graph of consecutive examples; Figure 21 It shows the execution Figure 3 A diagram showing a specific example of step S3 in the image correction process; Figure 22A This is a diagram showing an example of how the third point cloud data P3 is divided into multiple rectangular regions for each part obtained by the segmentation curve E; Figure 22B This is a diagram illustrating an example of coordinates obtained by inversely transforming the third point cloud data P3 into the projection space of the point cloud data P1. Figure 23A This is a diagram illustrating an example of how the third point cloud data P3, obtained through the segmentation curve E, is divided into multiple rectangular regions J. Figure 23B This is a diagram illustrating an example of multiple rectangular regions J unfolded from multiple rectangular regions J in three-dimensional space (x, y, z) into multiple rectangular regions G in two-dimensional space (u, v); Figure 24 This is a diagram illustrating an example of the relationship between the coordinates of point cloud data when it is unfolded on a plane and the coordinates on the captured image; Figure 25 This is an example of an image obtained by projecting a transformation onto each corresponding rectangular region to make the curved page surface in the main camera image become a flat surface; Figure 26 This is an example diagram showing images of the surfaces of two pages 200 and 201 that have been projected and transformed into a plane. Detailed Implementation

[0030] Embodiments of this disclosure will be described in detail, and examples of embodiments will be shown in the accompanying drawings. Throughout the description, the same or similar elements and elements having the same or similar functions are denoted by the same reference numerals. The embodiments described herein with reference to the accompanying drawings are illustrative and intended to illustrate this disclosure, and should not be construed as limiting this disclosure.

[0031] Figure 1 This is a diagram illustrating an example arrangement of an electronic device 100 and an object 101 according to an embodiment of the present invention. Figure 2 It is shown Figure 1 A diagram showing an example configuration of the electronic device 100.

[0032] like Figure 1 and Figure 2 As shown, for example, electronic device 100 includes a camera module 10, a distance sensor module 20, and an image signal processor 30. The image signal processor 30 controls the camera module 10 and the distance sensor module 20, and processes camera image data acquired from the camera module 10. For example, Figure 1 The figure in Figure 101 describes a book object.

[0033] like Figure 2 As shown, the camera module 10 includes, for example, a main lens 10a capable of focusing on an object, a main image sensor 10b for detecting images input via the main lens 10a, and a main image sensor driver 10c for driving the main image sensor 10b.

[0034] In addition, such as Figure 2 As shown, the camera module 10 includes, for example, a gyroscope sensor 10d for detecting the angular velocity and acceleration of the camera module 10, a focusing and OIS actuator 10f for actuating the main lens 10a, and a focusing and OIS driver 10e for driving the focusing and OIS actuator 10f.

[0035] For example( Figure 1 ( ), camera module 10 acquires the main camera image of object 101.

[0036] like Figure 2 As shown, the distance sensor module 20 includes, for example, a ToF lens 20a, a distance sensor 20b that detects reflected light input via the ToF lens 20a, a distance sensor driver 20c that drives the distance sensor 20b, and a projector 20d that outputs pulsed light.

[0037] The distance sensor module 20 acquires distance and depth information of the object 101 by using light. Specifically, the distance sensor module 20 acquires time-of-flight (ToF) depth information (ToF depth value) as distance and depth information, for example, by emitting pulsed light toward the object 101 and detecting the reflected light from the object 101.

[0038] The detection resolution of the distance sensor module 20 is lower than that of the camera module 10.

[0039] Image signal processor 30 controls, for example, camera module 10 and distance sensor module 20, to acquire a camera image based on a main camera image obtained by means of camera module 10 and ToF depth information obtained by means of distance sensor module 20, the camera image being the main camera image.

[0040] In addition, such as Figure 2 As shown, for example, electronic device 100 includes a global navigation satellite system (GNSS) module 40, a wireless communication module 41, a codec 42, a speaker 43, a microphone 44, a display module 45, an input module 46, an inertial measurement unit (IMU) 47, a main processor 48, and a memory 49.

[0041] For example, GNSS module 40 measures the current position of electronic device 100.

[0042] For example, wireless communication module 41 performs wireless communication with the Internet.

[0043] For example, such as Figure 2 As shown, codec 42 performs encoding and decoding bidirectionally using predefined encoding / decoding methods.

[0044] For example, speaker 43 outputs sound based on sound data decoded by codec 42.

[0045] For example, microphone 44 outputs sound data to codec 42 based on the input sound.

[0046] Display module 45 displays predefined information.

[0047] Input module 46 receives user input (user operations).

[0048] The IMU 47 detects, for example, the angular velocity and acceleration of electronic device 100.

[0049] The main processor 48 controls the GNSS module 40, the wireless communication module 41, the codec 42, the speaker 43, the microphone 44, the display module 45, the input module 46, and the IMU 47.

[0050] The memory 49 stores the programs and data required by the image signal processor 30 to control the camera module 10 and the distance sensor module 20, the acquired image data, and the programs and data required by the main processor 48 to control the electronic device 100.

[0051] For example, memory 49 includes a computer-readable storage medium thereon storing a computer program that, when executed by main processor 48, implements a method for controlling electronic device 100. For example, the method includes: controlling camera module 10 and distance sensor module 20 by means of processor 30 to acquire a main camera image and distance-depth information of the curved page surface, the main camera image including the curved page surface of a page in an open book state; estimating a surface corresponding to the position of the page surface by means of processor 30, by estimating a curve corresponding to the position of the page surface based on the main camera image and the distance-depth information; correcting the estimated surface using image information of the page surface of the book extracted from the main camera image by means of processor 30; and acquiring an image of the page surface projected onto a plane by means of processor 30, based on the corrected estimated surface, by transforming the projection of the curved surface of the page in the main camera image into a plane.

[0052] In this embodiment, the electronic device 100 with the above configuration is a mobile phone such as a smartphone, or it can be other types of electronic devices (e.g., tablet computers and PDAs) that include a camera module 102.

[0053] Next, an example of a method for controlling the electronic device 100 having the above-described configuration and functions will be described. Specifically, an example of the process of the electronic device 100 for acquiring a camera image of the curved page surface in a planar state by using a camera to obtain a camera image of the curved page surface in a state where the book is open, based on a curved image of the curved page surface in a state where the book is open.

[0054] Figure 3 It is shown that it is used for Figure 1 and Figure 2 The diagram illustrates an example of the overall process by which an electronic device 100 acquires an image by transforming the curved page surface of a book 100 in an open state into a flat surface.

[0055] First, such as Figure 3As shown, the image signal processor 30 controls the camera module 10 and the distance sensor module 20 to capture the main camera image and the ToF depth information of the curved page surface (step S1). The main camera image includes the curved page surface in the state of the book being opened.

[0056] Next, as Figure 3 As shown, the image signal processor 30 performs top view processing (step S2). The top view processing estimates the crease position of the page based on the main camera image and ToF depth information, and estimates the curve corresponding to the position of the curved page surface perpendicular to the crease position.

[0057] That is, by estimating the curve corresponding to the position on the page surface, the surface corresponding to the position on the page surface is estimated.

[0058] In this top view processing (step S2), as described later, the estimated surface is corrected by using information from the surface image of the book's pages extracted from the main camera image (e.g., the outline of the book's page surface).

[0059] Next, as Figure 3 As shown, the image signal processor 30 performs image correction processing (step S3). This image correction processing projects and transforms the curved page surface in the main camera image into a plane based on the estimated curve (the corrected surface). Thus, the image signal processor 30 acquires an image of the page surface that has been projected and transformed into a plane.

[0060] Next, as Figure 3 As shown, the image signal processor 30 determines whether additional shooting of a curved page, such as another curved page, is needed while the book is open (step S4).

[0061] If additional shooting is required, the image signal processor 30 returns to step S1 and captures the main camera image and the ToF depth information of the curved page surface of the page, the main camera image including the curved page surface of another page in the state of the book being opened.

[0062] In step S2, the image signal processor 30 estimates another curve corresponding to the position of the curved page surface of the other page based on the main camera image and ToF depth information.

[0063] Then, in step S3, the image signal processor 30 performs a projection transformation on the curved page surface of the other page in the main camera image based on the estimated other curve, so as to make it planar. Thus, the image signal processor 30 acquires an image of the page surface transformed into a planar surface.

[0064] If the image signal processor 30 determines in step S4 that no additional shooting is required, the image signal processor 30 combines the acquired images of the two surface pages on the plane and opens the book. Thus, the image signal processor 30 acquires images of the surfaces of the two pages on the plane in this state (step S5).

[0065] The present invention has the following prerequisites (a) to (d): (a) The book opens vertically (along the direction of the crease) in almost the same curvature. (b) The binding margin (crease position) of the book is substantially above or below one of the center, left, and right edges of the screen. (c) The detection resolution of the distance sensor module 20 is lower than that of the camera module 10. (d) The distance sensor module 20 is capable of detecting the distance from the surface of the book.

[0066] This article, Figure 3 An example of the process for acquiring main camera images and ToF depth information will be described below.

[0067] Figure 4 It is shown Figure 3 The figure shows a specific example of step S1 for acquiring the main camera image and ToF depth information. Figure 5 This is a diagram showing an example of the display on the display module 45 of the electronic device 100 when a specified crease position is specified.

[0068] like Figure 4 As shown, the image signal processor 30 sets a crease position specification box 45a in the image captured by the camera module 10 when the book is open (step S11).

[0069] The image signal processor 30 displays the crease location specification box 45a together with the main camera image captured by the camera module 10 on the display module 45.

[0070] In response to an operation input from a user to the input module 46 relating to an instruction for a crease position specification box 45a, the image signal processor 30 sets the crease position specification box 45a at a position specified by the user on the curved page surface of the main camera image.

[0071] For example, such as Figure 5 As shown, the crease position specification box 45a is initially displayed in the center operating position or a previous operating position in the display module 45. When the user wants to specify another position, the user touches the position to be specified. Alternatively, the display module 45 may initially be empty, and then, when the user touches the display module 45, the crease position specification box 45a may be displayed at the touch position in the display module 45.

[0072] Furthermore, when the user operates the frame direction specification button B1, the crease position specification frame 45a rotates up and down or left and right within the display module 45. The frame direction specification button B1 is included in the input module 46.

[0073] In addition, for example, such as Figure 5 As shown, the crease location specification box 45a has a mark indicating the vertical direction (an arrow in the example).

[0074] For example, in display module 45, when the page display is upside down, the user can rotate the page display to the opposite direction by operating the box orientation specification button B1.

[0075] For example, when a user touches and drags the end of the crease position specification box 45a, the crease position specification box 45a can be rotated based on the center position.

[0076] For example, when a user touches the display module 45 with two fingers, the crease position specification box 45a is set in the direction of the touch point connected to the display module 45.

[0077] Then, as Figure 4 As shown, by taking a photo of the curved page surface of the opened and bent page with the camera module 10 while the book is open, the image signal processor 30 acquires a main camera image including the curved page surface of the page, and sets a crease position specification box 45a on the main camera image (step S12).

[0078] For example, such as Figure 5 As shown, when the user operates the shutter button 46 of the input module 46, the camera module 10 takes a picture of the curved page surface of the book that is opened and bent when the book is displayed on the display module 45.

[0079] When the book 101 is open, the image signal processor 30 acquires a main camera image including the curved page surface by capturing the curved page surface of the open and curved page with the camera module 10.

[0080] The image signal processor 30 acquires ToF depth information of the curved surface of the page by illuminating the curved surface of the page with pulsed light from the distance sensor module 20.

[0081] Next, it will be used for execution. Figure 3 An example of the workflow for processing the top view shown will be described below.

[0082] Figure 6 It is shown Figure 3 The figure shows a specific example of step S2 for performing top view processing.

[0083] First, such as Figure 6 As shown, the image signal processor 30 performs a projection transformation from point cloud data to data captured from the front (step S21).

[0084] Then, the image signal processor 30 performs estimation of the crease position of the point cloud data after projection transformation, and rotation of the point cloud data to data (step S22).

[0085] Then, as Figure 6 As shown, the image signal processor 30 performs the estimation of ridge tilt and the projection transformation of the rotated point cloud data (step S23).

[0086] Then, the image signal processor 30 estimates the curved page surface by estimating the curve corresponding to the position of the curved page surface of the page (step S24).

[0087] Thus, as Figure 6 As shown, in the top view processing, the image signal processor 30 estimates the curve corresponding to the position of the curved page surface in a plane (XY plane) perpendicular to the crease direction (z-axis direction) of the open crease position of the page based on the main camera image and ToF depth information (steps S22 to S24).

[0088] Figure 7A This is an example of reference point cloud data P showing an image of the curved page surface of a curved page imaged from an oblique direction when the book is open. Figure 7B This is an example diagram showing the first point cloud data P1 obtained by projection transformation from the depth direction (from the front).

[0089] like Figure 7A As shown, the image signal processor 30 acquires reference point cloud data of the curved surface of the page based on the main camera image and ToF depth information.

[0090] Then, the image signal processor 30 calculates the normal vector by applying principal component analysis to the reference point cloud data.

[0091] Then, as Figure 7B As shown, the image signal processor 30 will perform a projection transformation T on the reference point cloud data P to the first point cloud data P1 of the main camera image taken from the depth direction (front), based on the calculated normal vector.

[0092] Thus, principal component analysis is applied to the reference point cloud data P taken from an oblique angle to generate normal vectors, and a projection transformation T is performed on the data taken from the front (yz plane).

[0093] Figure 8AThis is a diagram illustrating an example of scanning along multiple lines L1 relative to the first point cloud data P1 in a direction perpendicular to the longitudinal direction D1 of the box specifying the crease location. Figure 8B This is a diagram illustrating an example of second point cloud data P2 obtained by rotating the first point cloud data by a first rotation R such that the estimated crease position D2 is parallel to a preset reference direction (z-axis direction). like Figure 8A As shown, the image signal processor 30 scans the first point cloud data P1 along multiple lines L1 in a direction (short direction) perpendicular to the longitudinal direction D1 of the crease location specification box 45a.

[0094] Then, the image signal processor 30 calculates the slope of the valley of the first point cloud data P1 by applying the least squares method to the scanned first point cloud data P1.

[0095] like Figure 8A As shown, the image signal processor 30 estimates the crease location (valley location) M based on the calculated slope of the valley.

[0096] Then, as Figure 8B As shown, the image signal processor 30 acquires the second point cloud data P2 by performing a first rotation R on the first point cloud data P1 so that the estimated crease position M is parallel to a preset reference direction (z-axis direction).

[0097] Furthermore, when the image signal processor 30 scans the first point cloud data P1 along multiple lines L1, the image signal processor 30 extracts data from the first point cloud data P1 within a predetermined range, which is the data used for the least squares method.

[0098] Figure 9A This is a diagram showing an example of scanning along multiple lines L2 relative to the second point cloud data P2 in a direction perpendicular to a preset reference direction (z-axis direction). Figure 9B This is a diagram showing an example of the slope of ridge N in the longitudinal direction of the second point cloud data P2 during scanning. Figure 9C This is a diagram illustrating an example of obtaining third point cloud data P3 by rotating the second point cloud data P2 by a second rotation Q such that the calculated inclination of the ridge N is parallel to a preset reference plane (zy plane).

[0099] For example, such as Figure 9A As shown, the image signal processor 30 scans the second point cloud data P2 along multiple lines L2 in a direction perpendicular to the reference direction (z-axis direction).

[0100] Then, for example, such as Figure 9B As shown, the image signal processor 30 calculates the slope of the ridge N along the reference direction of the scanned second point cloud data by applying the least squares method to the scanned second point cloud data.

[0101] Then, for example, such as Figure 9C As shown, the image signal processor 30 obtains the third point cloud data P3 by rotating the second point cloud data P2 by a second rotation Q, so that the slope of the calculated ridge N is parallel to a preset reference plane (zy plane).

[0102] The coordinate transformation from the original reference point cloud data P to the third point cloud data P3 is "QRT".

[0103] Figure 10A This is a diagram showing an example of scanning along multiple lines L3 relative to the third point cloud data P3 in the reference direction (z-axis direction). Figure 10B This is a diagram showing an example of the average value A of the third point cloud data P3 in the depth direction (x-axis direction) near multiple lines L3. Figure 10C This is a diagram showing an example of a curve E obtained by approximating the average value A using a fourth-order or higher-order polynomial in the directions perpendicular to the reference direction (y-axis direction) and the depth direction (x-axis direction).

[0104] For example, such as Figure 10A As shown, the image signal processor 30 scans the third point cloud data P3 along multiple lines L3 in the reference direction (z).

[0105] Then, for example, such as Figure 10B As shown, the image signal processor 30 calculates the average value A of the third point cloud data P3 near the depth direction (x-axis direction) of multiple lines L3.

[0106] Then, for example, such as Figure 10C As shown, the image signal processor 30 calculates the curve E, which is approximated by a fourth-order or higher-order polynomial expression (1) in the direction perpendicular to the reference direction (y-axis direction) and the depth direction (x-axis direction) based on the average value A.

[0107]

[0108] Thus, through Figure 6 In step S24, the curve corresponding to the position on the page surface is estimated to estimate the surface corresponding to the position on the page surface.

[0109] Next, according to Figure 6 In step 241, the image signal processor 30 corrects the estimated surface by using image information of the page surface of the book extracted from the main camera image.

[0110] In the following example, the information describing the surface image of a book's page extracted from the main camera image is the outline of the book's page surface, but it is not limited to this.

[0111] Next, it will be used for execution. Figure 6 An example of the process for correcting the surface shown (step S241) will be described below.

[0112] This article, Figure 11 It shows the execution Figure 6 A diagram showing a specific example of step S241 in the processing of the corrected surface. Figure 12A This is an example diagram showing a main camera image obtained by capturing a book. Figure 12B It shows how to detect Figure 12A The image shown is a diagram of the processed image of the book's edge from the main camera image. Figure 12C It shows that by from Figure 12B The image shown is a diagram obtained by extracting the outline of the book from the image shown.

[0113] When correcting the estimated surface ( Figure 6 In step S241), firstly, for example in Figure 11 In step S2411, the image signal processor 30 extracts the outline RX of the page surface of the book 101 from the main camera image G. Figures 12A to 12C ).

[0114] like Figures 12A to 12C As shown, after detecting the edges of the page surface of book 101 from the main camera image G ( Figure 12A The image signal processor 30 binarizes the detected edges and retains the edges with long-connected components. Figure 12B ), to obtain the outline RX ( Figure 12C ).

[0115] Next, in Figure 11 In step S2412, the image signal processor 30 performs alignment processing of the crease position of the book based on ToF depth information and the crease position of the book based on the main camera image.

[0116] This article, Figure 13A This is a diagram showing an example of extracting candidate coordinates for the crease lines of a book from the third point cloud data P3. Figure 13B It is by... Figure 13A The candidate line segments of the extracted crease lines are projected and transformed into a form including Figure 12C The image was obtained by showing the outline of the book. Figure 14A This is a diagram illustrating an example of using candidate line segments to search the outline of a book to obtain the crease lines of the book. Figure 14B This is a graph showing the offset corresponding to the difference between the crease line and the candidate line segment of the book selected in the outline.

[0117] More specifically, the image signal processor 30 obtains candidate coordinates of the line segment LZ connecting the upper Zmax and lower Zmin of the book's crease based on the ToF depth information about the page surface. Figure 13A ). Furthermore, the image signal processor 30 obtains candidate line segment LZA by projecting and transforming the candidate coordinates of line segment LZ into an image that includes the contours on the page surface. Figure 13B ).

[0118] Furthermore, the image signal processor 30 selects the book's crease line from the line segments obtained by performing a search for the contour RX using candidate line segments LZA (UZ). Figure 14A ).

[0119] For example, the image signal processor 30 searches for the contour RX by scanning multiple times at predetermined intervals in a direction perpendicular to the candidate crease line on the image, and sets the found points as crease location candidates.

[0120] Then, the image signal processor 30 selects a reasonable line segment as the crease line from multiple crease location candidates, for example, through Random Sample Consensus (RANSAC).

[0121] In addition, the image signal processor 30 obtains the difference between the centroid position of the candidate line segment LZA and the centroid position of the selected crease line, as the line segment correction offset OH. Figure 14B It should be noted that this difference can be caused by factors such as lens movement during filming.

[0122] Next, in Figure 11 In step S2413, the image signal processor 30 corrects the coordinates of the upper and lower ends of the page surface based on the ToF depth information.

[0123] This article, Figure 15 This is a diagram showing the situation where the top and bottom of the book need to be corrected in the third point cloud data P3. Figure 16A This is a diagram showing the third point cloud data P3 before the correction of the upper and lower ends. Figure 16B It is one of them Figure 16A The third point cloud data P3 shown is projected and transformed before the upper and lower ends are corrected, including... Figure 12C The image shown is an outline of a book, aligned with the creases.

[0124] For example, such as Figure 15 As shown, when the lower angle θMmin and the upper angle θMmax are less than a preset angle (e.g., 60°), the projection transformation error caused by the position measurement error is considered to be large. In this case, as... Figure 16A and Figure 16BAs shown, the paper edges do not match due to position measurement error.

[0125] As described above, when the position measurement error is large, the electronic device 100 performs corrections at the top and bottom of the page surface.

[0126] The lower angle θMmin is the angle between the line connecting the camera center M of the electronic device 100 and the lower end of the book (point cloud data P3) and the line connecting the peak of the book (point cloud data P3).

[0127] The upper angle θMmax is the angle between the line connecting the camera center M of the electronic device 100 and the upper end of the book (point cloud data P3) and the line connecting the peak of the book (point cloud data P3).

[0128] On the other hand, if the lower angle θMmin and the upper angle θMmax are greater than or equal to preset angles, the projection transformation error caused by the position error is considered to be small. In this case, the electronic device 100 does not perform corrections at the upper and lower edges of the page surface.

[0129] This article, in Figure 11 In step S2413, for example, the image signal processor 30 projects and transforms the coordinates of the page surface based on the ToF depth information to an image that includes the contour RX of the page surface, and obtains coordinates for edge search based on the coordinates of the edges of the page surface, so that the candidate line segments overlap with the crease lines of the contour RX by using line segment correction offset.

[0130] Furthermore, the image signal processor 30 uses edge search coordinates to search for the outline. Then, the image signal processor 30 obtains the corrected edge coordinates of the edge ends of the outline of the book obtained through the search.

[0131] This article, Figure 17 This is a diagram illustrating the edge coordinates of the edge portion of a book's outline, obtained by searching for the outline using edge search coordinates.

[0132] like Figure 17 As shown, the image signal processor 30 performs a search for the contour line RX between the upper and lower ends by adding or subtracting the offset from the coordinate values ​​of the edge ends of the book in the third point cloud data P3 in three-dimensional space and transforming them into image coordinates.

[0133] Next, in Figure 11 In step S2414, the image signal processor 30 re-estimates the surface corresponding to the position of the page surface based on the corrected edge coordinates (coordinates of the three-dimensional position obtained by back-projection correcting the edge coordinates).

[0134] Note that, as in step S24 above, in Figure 11In step S2414, the image signal processor 30 re-estimates the surface corresponding to the position of the page surface by reapplying a polynomial approximation to the corrected coordinate sequence.

[0135] This article, Figure 18A It involves projecting and transforming the third point cloud data P3, which is corrected at the top and bottom, into a format that includes... Figure 12C The image shown is an outline of a book, aligned with the creases. Figure 18B It involves projecting and transforming the third point cloud data P3, which is corrected at the top and bottom, into a format that includes... Figure 12C The image shown is an outline of a book, aligned with the creases.

[0136] like Figure 18A and 18B As shown, by performing corrections on the upper and lower ends, the edge HX obtained by projecting the corrected third point cloud data P3 onto the upper and lower ends coincides with the outline RX of the book.

[0137] According to steps S2411 to S2414 above, the image signal processor 30 uses the contour (image information) of the page surface of the book extracted from the main camera image to correct the estimated surface.

[0138] Then, for example, such as Figure 6 As shown in step S25, the image signal processor 30 acquires an image of the page surface that has been projected onto the plane by projecting and transforming the curved surface of the page in the main camera image to make it a plane, based on the corrected curved surface.

[0139] Next, it will be used for execution. Figure 6 An example of the flow of the segmentation process (step 25) shown will be described below.

[0140] Figure 19 It shows the execution Figure 6 A diagram showing a specific example of step S25 in the segmentation process. Figure 20A This is a diagram illustrating an example of dividing the curve E into points such that the error between points of the curve E approximated by the polynomial falls within an acceptable range. Figure 20B This illustrates dividing the curve E into points such that the error between points of the curve E approximated by the polynomial falls within an acceptable range, and... Figure 20A A series of examples. Figure 20C This illustrates dividing the curve E into points such that the error between points of the curve E approximated by the polynomial falls within an acceptable range, and... Figure 20B A series of examples.

[0141] For example, such as Figure 19 and Figure 20AAs shown, the image signal processor 30 calculates the error between points in the segmentation process (step S251).

[0142] Next, as Figure 19 As shown, the image signal processor 30 determines whether the error between points is within the allowable range (step S252).

[0143] Then, as Figure 19 , Figure 20B and Figure 20C As shown, when the error between points exceeds the allowable range, the image signal processor 30 segments the points that exceed the range (step S253).

[0144] On the other hand, when the error between points falls within the allowable range, the image signal processor 30 ends the segmentation process.

[0145] Next, it will be used for execution. Figure 3 An example of the image correction process shown will be described below.

[0146] Figure 21 It shows the execution Figure 3 A diagram showing a specific example of step S3 in the image correction process. Figure 22A This is a diagram showing an example of how the third point cloud data P3 is divided into multiple rectangular regions for each part obtained by the segmentation curve E. Figure 22B This is a diagram illustrating an example of coordinates obtained by inversely transforming the third point cloud data P3 into the projection space of the point cloud data P1. Figure 23A This is a diagram illustrating an example of how the third point cloud data P3, obtained through the segmentation curve E, is divided into multiple rectangular regions J for each part. Figure 23B This is a diagram illustrating an example of multiple rectangular regions J unfolded from multiple rectangular regions J in three-dimensional space (x, y, z) into multiple rectangular regions G in two-dimensional space (u, v). Figure 24 This is a diagram illustrating an example of the relationship between the coordinates of point cloud data when it is unfolded on a plane and the coordinates on the captured image.

[0147] First, such as Figure 21 As shown, in the image correction process, the image signal processor 30 sets the area to be processed (step S31).

[0148] For example, such as Figure 14A As shown, the image signal processor 30 divides the third point cloud data P3 into multiple rectangular regions for each part obtained through the segmentation curve E.

[0149] For example, through the inverse transform T as shown in expression (2) -1 R -1 Q -1The coordinates of the third point cloud data P3 are transformed into the projected spatial coordinates of the original reference point of the point cloud data P. Figure 14B ).

[0150]

[0151] Then, as Figure 21 As shown in step S32, the image signal processor 30 estimates the plane used to extend the three-dimensional space ( Figure 23A ) to a two-dimensional space plane ( Figure 23B The transformation matrix of ).

[0152] For example, such as Figure 23A As shown, the image signal processor 30 divides the third point cloud data P3 into multiple rectangular regions J for each part obtained by the segmentation curve.

[0153] Then, as Figure 23B As shown, the image signal processor 30 transforms multiple rectangular regions J obtained by dividing the third point cloud data P3 in the three-dimensional space (x, y, z) into a two-dimensional space (u, v) to obtain multiple rectangular regions G that are expanded in the two-dimensional space (u, v).

[0154] For example, Figure 23B The width of the rectangular region G shown is represented by expression (3). Furthermore, Figure 23B The length of the rectangular region G shown is represented by expression (4).

[0155] (3) (4) Then, the image signal processor 30 calculates the transformation matrix for each region based on the relationship between the coordinates when unfolded on the plane and the coordinates on the captured image.

[0156] As shown in expression (5), by calibration, the offset is added to the coordinates normalized according to the distance for magnification / reduction to obtain the offset values ​​Ou, Ov and magnification / reduction ratio k that match the distortion-corrected camera image.

[0157]

[0158] Two expressions can be created using a single associated point. Expression (7) contains nine unknowns.

[0159]

[0160] Expression (8) is derived from expressions (6) and (7).

[0161]

[0162] Then, in expression (8), if “c” is assumed (e.g., “1”), the number of unknowns becomes 8 (expression (9)).

[0163]

[0164]

[0165] Then, in expression (9), when establishing the relation of expression (10), expression (11) is obtained.

[0166]

[0167] Therefore, if the correspondence between the four points in the coordinates when unfolded on the plane and the coordinates on the captured image is known, then the unknowns can be obtained.

[0168] That is, as shown in expression (12), the transformation matrix is ​​calculated by solving the equations from the combination of known points.

[0169]

[0170] Next, the image signal processor 30 performs a projection transformation based on the coordinates of multiple rectangular regions and the coordinates of the projection space of the reference point cloud data P, so that the curved page surface in the main camera image becomes a flat surface. Figure 21 Step S33 in the process.

[0171] Figure 25 This is an example of an image obtained by projecting a transformation onto each corresponding rectangular region to make the curved page surface in the main camera image become flat.

[0172] For example, such as Figure 25 As shown, the image signal processor 30 performs a projection transformation on each corresponding rectangular region, so that the curved page surface in the main camera image becomes a flat surface.

[0173] Then, the image signal processor 30 acquires an image of the page surface that has been projected and transformed into a plane by the projection transformation.

[0174] Then, the image signal processor 30 determines whether the entire area of ​​the third point cloud data P3 has been processed. Figure 21 Step S34 in the process.

[0175] Then, when the image signal processor 30 has not processed the entire area of ​​the third point cloud data P3, processing returns to... Figure 21 Step S31 in the process.

[0176] On the other hand, such as Figure 23B As shown, the segmentation process ends when the image signal processor 30 has processed the entire area of ​​the third point cloud data P3.

[0177] Figure 26 This is an example diagram showing images of the surfaces of two pages 200 and 201 that have been projected and transformed into a plane.

[0178] like Figure 26 As shown, the image signal processor 30 synthesizes two images of the surfaces of the two pages of the acquired plane, and acquires images of the surfaces of the two pages of the plane when the book is open.

[0179] Therefore, by using the camera of an electronic device 100 such as a smartphone, it is possible to obtain a camera image of the surface of the page extending in a plane from a camera image of the surface of the curved page of an open book.

[0180] As described above, according to the present invention, an enlarged image can be obtained by simply photographing an enlarged book. This technology can be provided at low cost as a smartphone application. The present invention does not require large-scale equipment.

[0181] In the description of embodiments of this disclosure, it should be understood that terms such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “top,” “bottom,” “front,” “back,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” should be interpreted as referring to directions or positions as described or shown in the accompanying drawings discussed. These related terms are used merely to simplify the description of this disclosure and do not indicate or imply that the mentioned devices or elements must have a particular orientation, or must be constructed or operated in a particular orientation. Therefore, these terms should not constitute a limitation of this disclosure.

[0182] Furthermore, the terms such as “first” and “second” used herein for descriptive purposes are not intended to indicate or imply relative importance or significance, or to imply the number of technical features indicated. Therefore, a feature defined as “first” and “second” may include one or more of those features. In the description of this disclosure, unless otherwise stated, “a plurality” means “two or more”.

[0183] In the description of embodiments of this disclosure, the terms “mounted,” “connected,” “coupled,” etc., are used extensively and, unless otherwise specified or limited, can be, for example, a fixed connection, a detachable connection, or an integral connection; can also be a mechanical connection or an electrical connection; can also be a direct connection or an indirect connection via an intermediate structure; or can also be internal communication between two elements that can be understood by those skilled in the art based on the specific circumstances.

[0184] In embodiments of this disclosure, unless otherwise specified or limited, the structure of the first feature "on" or "below" the second feature may include embodiments in which the first feature and the second feature are in direct contact, or embodiments in which the first feature and the second feature are not in direct contact with each other, but are in contact through an additional feature formed between them. Furthermore, the term "on," "above," or "top" the second feature may include embodiments in which the first feature is orthogonally or obliquely positioned on, above, or "top" the second feature, or simply means that the first feature is at a height higher than the second feature; while the term "below," "below," or "bottom" the second feature may include embodiments in which the first feature is orthogonally or obliquely positioned below, below, or "bottom" the second feature, or simply means that the first feature is at a height lower than the second feature.

[0185] Various embodiments and examples have been provided in the foregoing description to implement different structures of this disclosure. To simplify this disclosure, certain elements and arrangements have been described above. However, these elements and arrangements are merely examples and are not intended to limit this disclosure. Furthermore, reference numerals and / or reference letters may be repeated in different examples of this disclosure. Such repetition is for simplification and clarity purposes and does not indicate a relationship between different embodiments and / or arrangements. In addition, examples of different processes and materials are provided in this disclosure. However, those skilled in the art should understand that other processes and / or materials may also be applied.

[0186] Throughout this specification, references to "embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" mean that a particular feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this disclosure. Therefore, the appearance of these phrases throughout this specification does not necessarily refer to the same embodiment or example of this disclosure. Furthermore, in one or more embodiments or examples, specific features, structures, materials, or characteristics may be combined in any suitable manner.

[0187] Any process or method described in the flowchart or otherwise described herein can be understood as including one or more modules, segments, or portions of code comprising executable instructions for implementing specific logical functions or steps in that process, and the scope of preferred embodiments of this disclosure includes other implementations, wherein those skilled in the art will understand that functionality may be implemented in an order different from the order shown or discussed, including in substantially the same order or in reverse order.

[0188] The logic and / or steps otherwise described herein or shown in flowcharts, such as a specific list of executable instructions for implementing that logic, may be embodied in any computer-readable medium that will be used by or in conjunction with an instruction execution system, apparatus, or device (e.g., a computer-based system, a processor-integrated system, or other system capable of obtaining instructions from or in conjunction with an instruction execution system, apparatus, or device that executes instructions). For the purposes of this specification, "computer-readable medium" can be any means that adaptively includes, stores, communicates, propagates, or transmits a program that will be used by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to: electronic connections (electronic devices) having one or more wires, portable computer casings (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because, for example, when a program needs to be obtained electronically, it can be optically scanned onto paper or other suitable media, then edited, decrypted or processed by other suitable methods, and then the program can be stored in computer memory.

[0189] It should be understood that each part of this disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in memory and executed by an appropriate instruction execution system. For example, if implemented in hardware, similarly in another embodiment, these steps or methods can be implemented by one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0190] Those skilled in the art will understand that all or part of the steps in the exemplary methods described above can be implemented by instructing related hardware using programs. These programs can be stored in a computer-readable storage medium, and when run on a computer, they include one or a combination of the steps in the method embodiments of this disclosure.

[0191] Furthermore, each functional unit in the embodiments of this disclosure can be integrated into a processing module, or these units can be separate physical entities, or two or more units can be integrated into a processing module. The integrated module can be implemented in hardware or as a software functional module. When the integrated module is implemented as a software functional module and sold or used as a standalone product, the integrated module can be stored in a computer-readable storage medium.

[0192] The aforementioned storage media can be read-only memory, disk, CD, etc.

[0193] Although embodiments of the present disclosure have been shown and described, those skilled in the art will understand that these embodiments are illustrative and should not be construed as limiting the present disclosure, and that changes, modifications, substitutions and variations may be made in the embodiments without departing from the scope of the present disclosure.

Claims

1. An electronic device, comprising: The electronic device includes: a camera module that photographs a subject to acquire a main camera image; a distance sensor module that acquires distance depth information of the subject by using light; and an image signal processor that controls the camera module and the distance sensor module to acquire a camera image based on the main camera image and the distance depth information, wherein the image signal processor estimates a curved surface corresponding to a position of a page surface in an open book state by estimating a curve corresponding to the position of the page surface based on the main camera image and the distance depth information, the image signal processor extracts an outline of the page surface of the book from the main camera image in correcting the estimated curved surface, the image signal processor extracts the outline of the page surface of the book from the main camera image; the image signal processor acquires candidate coordinates of a line segment connecting upper and lower ends of a fold of the book based on time-of-flight (ToF) depth information on the page surface; the image signal processor obtains a candidate line segment by projecting and transforming the candidate coordinates of the line segment into an image including the outline on the page surface; the image signal processor selects a fold line of the book from a line segment obtained by searching the outline using the candidate line segment; the image signal processor acquires a difference between a barycentric position of the candidate line segment and a barycentric position of the selected fold line as a line segment correction offset; the image signal processor acquires edge search coordinates based on coordinates of edge ends of the page surface by projecting and transforming coordinates of the page surface into an image including the outline of the page surface based on the ToF depth information, overlaps the candidate line segment with the fold line of the outline of the book using the line segment correction offset; and the image signal processor obtains corrected edge coordinates of an edge end portion of the outline of the book by searching the outline using the edge search coordinates, and the image signal processor re-estimates a curved surface corresponding to a position of the page surface based on the corrected edge coordinates; and the image signal processor acquires an image of the page surface that has been projected onto a plane by projecting and transforming a curved surface of a page in the main camera image into a plane based on the corrected estimated curved surface. 2.The electronic device of claim 1, wherein the distance sensor module emits pulsed light to the subject and detects reflected light of the pulsed light reflected from the subject, thereby acquiring time-of-flight (ToF) depth information as the distance depth information. ​ The image signal processor controls the camera module and the distance sensor module to acquire the main camera image including the curved page surface of the page and the distance depth information of the curved page surface; The image signal processor estimates a curve corresponding to the position of the page surface based on the main camera image and the distance depth information, and further estimates the curved surface corresponding to the position of the page surface.

4. The electronic device of claim 3, wherein, The image signal processor acquires the main camera image including the curved page surface by taking a photo of the curved page surface of the page that has been opened and curved using the camera module; And The image signal processor acquires the ToF depth information of the curved page surface through the distance sensor module.

5. The electronic device of claim 3, wherein, Based on the main camera image and the ToF depth information, the image signal processor estimates a curve corresponding to the position of the curved page surface in a plane perpendicular to the fold direction of the open fold position of the page.

6. The electronic device of claim 4, wherein, The image signal processor sets a fold position designation frame designated by a user on the opened page in an image taken by the camera module in the state of the opened book. And The image signal processor acquires the main camera image including the curved page surface by taking an image of the curved page surface of the page that has been opened and curved in the state of the opened book using the camera module, in which the fold position designation frame is set.

7. The electronic device of claim 6, wherein, The electronic device further comprises: a display module configured to display predefined information; an input module configured to receive the user's operation; and a main processor configured to control the display module and the input module, wherein the image signal processor displays the fold position designation frame on the display module together with the main camera image taken by the camera module; The image signal processor sets the fold position designation frame at a position designated by the user on the curved page surface of the main camera image in response to the user inputting an operation input related to an instruction of the fold position designation frame to the input module.

8. The electronic device of any of claims 1-7, wherein, The detection resolution of the distance sensor module is lower than the detection resolution of the camera module.

9. A method for controlling an electronic device, the electronic device comprising: a camera module that takes a photo of an object to acquire a main camera image; a distance sensor module that acquires distance depth information of the object by using light; and an image signal processor that controls the camera module and the distance sensor module to acquire a camera image based on the main camera image and the distance depth information; characterized in that The method comprises: by means of the processor, estimating a curve corresponding to the position of a page surface based on the main camera image and the distance depth information, and further estimating a curved surface corresponding to the position of the page surface in the state of an opened book; correcting the estimated curved surface by the processor using a contour of the page surface of the book extracted from the main camera image, in the case of correcting the estimated curved surface, extracting the contour of the page surface of the book from the main camera image by the processor; obtaining candidate coordinates of a line segment connecting upper and lower ends of a fold of the book based on time-of-flight (ToF) depth information on the page surface by the processor; obtaining a candidate line segment by the processor by projecting and transforming the candidate coordinates of the line segment into an image including the contour on the page surface; selecting a fold line of the book from line segments obtained by searching the contour using the candidate line segment by the processor; obtaining a line segment correction offset as a difference between a barycentric position of the candidate line segment and a barycentric position of the selected fold line by the processor; overlapping the candidate line segment with the fold line of the contour of the book using the line segment correction offset by the processor by obtaining edge search coordinates based on coordinates of edge ends of the page surface by projecting and transforming the coordinates of the page surface based on the ToF depth information into the image including the contour of the page surface, obtaining corrected edge coordinates of an edge end portion of the contour of the book by searching the contour using the edge search coordinates by the processor, and re-estimating a curved surface corresponding to a position of the page surface based on the corrected edge coordinates by the processor; and obtaining an image of the page surface that has been projected onto a plane by projecting and transforming a curved surface of a page in the main camera image into a plane based on the corrected estimated curved surface by the processor.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements a method for controlling an electronic device including a camera module that photographs a subject to obtain a main camera image, a distance sensor module that obtains distance depth information of the subject by using light, and an image signal processor that controls the camera module and the distance sensor module to obtain a camera image based on the main camera image and the distance depth information; The method includes: estimating a curved line corresponding to a position of a page surface by the processor based on the main camera image and the distance depth information, and further estimating a curved surface corresponding to the position of the page surface in an open book state; correcting the estimated curved surface by the processor based on the main camera image; wherein the estimated curved surface is corrected by the processor using a contour of the page surface of the book extracted from the main camera image, in the case of correcting the estimated curved surface, extracting the contour of the page surface of the book from the main camera image by the processor; based on time-of-flight (ToF) depth information on the page surface, obtaining, by the processor, candidate coordinates of a line segment connecting upper and lower ends of a fold of the book; obtaining, by the processor, a candidate line segment by projecting and transforming the candidate coordinates of the line segment into an image comprising the contour on the page surface; selecting, by the processor, a fold line of the book from line segments obtained by searching the contour using the candidate line segment; obtaining, by the processor, a barycentric position of the candidate line segment and a barycentric position of the selected fold line as a line segment correction offset; overlapping, by the processor, the candidate line segment with the fold line of the contour of the book using the line segment correction offset based on edge search coordinates obtained based on coordinates of edge ends of the page surface by projecting and transforming coordinates of the page surface into the image comprising the contour of the page surface based on the ToF depth information; obtaining, by the processor, corrected edge coordinates of an edge end portion of the contour of the book by searching the contour using the edge search coordinates, and re-estimating, by the processor, a curved surface corresponding to a position of the page surface based on the corrected edge coordinates; and based on the corrected estimated curved surface, obtaining, by the processor, an image of the page surface that has been projected onto a plane by projecting and transforming the curved surface of the page in the main camera image as a plane. based on time-of-flight (ToF) depth information on the page surface, obtaining, by the processor, candidate coordinates of a line segment connecting upper and lower ends of a fold of the book; obtaining, by the processor, a candidate line segment by projecting and transforming the candidate coordinates of the line segment into an image comprising the contour on the page surface; selecting, by the processor, a fold line of the book from line segments obtained by searching the contour using the candidate line segment; obtaining, by the processor, a barycentric position of the candidate line segment and a barycentric position of the selected fold line as a line segment correction offset; overlapping, by the processor, the candidate line segment with the fold line of the contour of the book using the line segment correction offset based on edge search coordinates obtained based on coordinates of edge ends of the page surface by projecting and transforming coordinates of the page surface into the image comprising the contour of the page surface based on the ToF depth information; obtaining, by the processor, corrected edge coordinates of an edge end portion of the contour of the book by searching the contour using the edge search coordinates, and re-estimating, by the processor, a curved surface corresponding to a position of the page surface based on the corrected edge coordinates; and based on the corrected estimated curved surface, obtaining, by the processor, an image of the page surface that has been projected onto a plane by projecting and transforming the curved surface of the page in the main camera image as a plane.

Images