Electronic devices, methods for controlling electronic devices, and recording media

Abstract

The present invention relates to an electronic device for accurately performing character recognition, a method for controlling the electronic device, and a recording medium having a program recorded thereon for causing a computer to execute the method. In an electronic device for capturing images, characters in an image can be accurately recognized. The electronic device includes an imaging unit, a distance measuring unit, a shape estimation unit, and a coordinate conversion unit. The imaging unit captures an object and captures image data. The distance measuring unit measures the distance from the imaging unit to each of a plurality of measurement points on the surface of the object. The shape estimation unit estimates the shape of the object based on the measured distance. The coordinate conversion unit performs coordinate conversion on the image data, and the coordinate conversion converts the three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

Description

Technical Field

[0001] The present invention relates to an electronic device, a method for controlling the electronic device, and a program for causing a computer to execute the method. In particular, the present invention relates to an electronic device for executing character recognition, a method for controlling the electronic device, and a program for causing a computer to execute the method. Background Art

[0002] In the past, a technology called optical character recognition (OCR) was used to extract characters written on printed materials using an information processing device and process the characters. For example, a system has been proposed that uses an on-vehicle camera to image a sign on the road and reads the characters written on the sign by OCR for translation (see Patent Document 1).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2003-323693 Summary of the invention

[0006] Technical issues

[0007] In the above conventional technology, the meaning of the characters recorded in the sign is recognized by optical character recognition. However, in the above conventional technology, there is a problem that when characters are recorded on a curved surface or an inclined surface when viewed from a vehicle-mounted camera, the shape of the characters is distorted, and optical character recognition cannot be accurately performed.

[0008] The present invention has been made in view of the above circumstances, and an object of the present invention is to accurately recognize characters in an image in an electronic device for capturing an image.

[0009] Technical Solution

[0010] The present invention is proposed to solve the above-mentioned problems. In a first aspect of the present invention, an electronic device, a method for controlling the electronic device, and a program for causing a computer to execute the method are provided, wherein the electronic device includes: an imaging unit that images an object and captures image data; a distance measuring unit that measures the distance from the imaging unit to each of a plurality of measurement points on the surface of the object; a shape estimating unit that estimates the shape of the object based on the measured distances; and a coordinate conversion unit that performs coordinate conversion on the image data, wherein the coordinate conversion converts three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

[0011] In addition, in the first aspect, the electronic device may further include a character recognition unit configured to recognize characters on the surface of the object in the image data after the coordinate conversion, thereby producing an effect of recognizing characters.

[0012] In addition, in the first aspect, the electronic device may further include an irradiation unit that performs irradiation of irradiation light, wherein the distance measurement unit measures the distance based on a phase difference between reflected light of the irradiation light and the irradiation light, thereby producing an effect of measuring the distance based on the phase difference between the reflected light and the irradiation light.

[0013] In addition, in the first aspect, the camera unit may be used for the process of capturing the image data and for the process of receiving the irradiation light, thereby producing an effect of capturing the image data and receiving the irradiation light in the same camera unit.

[0014] In addition, in the first aspect, the irradiation section may perform irradiation of pulsed light as the irradiation light, the pulsed light being synchronized with a predetermined periodic signal, thereby producing an effect of performing irradiation of pulsed light.

[0015] In addition, in the first aspect, the irradiation unit can select either the spot light or the diffuse light according to a predetermined operation and perform irradiation of the light as the irradiation light, and when performing the irradiation of the diffuse light, the distance measurement unit measures the distance. Thus, an effect of performing irradiation of the spot light or the diffuse light is produced.

[0016] In addition, in the first aspect, when a predetermined button is pressed halfway, the irradiation unit may start irradiation of the irradiation light, and when the predetermined button is pressed all the way, the imaging unit may capture the image data. Thus, irradiation of the irradiation light and imaging are performed according to button operation.

[0017] In addition, in the first aspect, the electronic device may be a camera unit mounted on the wearable terminal. Thus, an effect of converting three-dimensional coordinates on the surface of the object into planar coordinates on the reference plane is produced in the camera unit.

[0018] In the first aspect, the shape estimating unit estimates any one of the plurality of candidate shapes as the shape of the object based on the distance. This produces an effect of estimating any one of the plurality of candidate shapes as the shape of the object.

[0019] In addition, in the first aspect, the shape estimation section may include: a coordinate detection section that detects the coordinates of the plurality of measurement points based on the distance as measurement coordinates; a function acquisition section that acquires, for each of the plurality of candidate shapes, a function representing the relationship between the coordinates of the candidate shape and the coordinates of a predetermined reference coordinate system by using the measurement coordinates; an error calculation section that calculates an error when assuming the shape of the object for each of the plurality of candidate shapes based on the acquired function and the measurement coordinates; and an estimation processing section that estimates a shape having the smallest error among the plurality of candidate shapes as the shape of the object. Thus, an effect of estimating a shape having the smallest error among the plurality of candidate shapes as the shape of the object is produced.

[0020] In addition, in the first aspect, the image data includes a plurality of pixel data, the imaging unit includes phase difference detection pixels and normal pixels, the phase difference detection pixels detect the phase difference between two pupil segmentation images, the normal pixels perform photoelectric conversion on light and generate any one of the plurality of pixel data, and the distance measuring unit measures the distance based on the phase difference detected by the phase difference detection pixels. Thus, an effect of measuring the distance based on the phase difference detected by the phase difference detection pixels is produced.

[0021] Beneficial Effects

[0022] According to the present invention, the best effect of being able to accurately recognize characters in an image in an electronic device that captures an image can be produced. Note that the effects described here are not necessarily restrictive, and any of the effects described herein can be produced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram showing a configuration example of an electronic device in a first embodiment of the present invention.

[0024] Figure 2 : is a block diagram showing a configuration example of a three-dimensional shape estimating section in the first embodiment of the present invention.

[0025] Figure 3 is a diagram showing an example of distance information in the first embodiment of the present invention.

[0026] Figure 4 is a diagram showing an example of measurement coordinates in the first embodiment of the present invention.

[0027] Figure 5 is a diagram showing an example of parameters in the first embodiment of the present invention.

[0028] Figure 6 is a block diagram showing a configuration example of an image pickup device in a first embodiment of the present invention.

[0029] Figure 7 is a block diagram showing a configuration example of a pixel circuit in the first embodiment of the present invention.

[0030] Figure 8 is a block diagram showing a configuration example of a coordinate conversion section in the first embodiment of the present invention.

[0031] Fig. 9 : is a diagram showing an example of the shape of laser light in the first embodiment of the present invention.

[0032] Fig.10 : is a timing chart showing an example of exposure control of the pixel circuit in the Q1Q2 detection period in the first embodiment of the present invention.

[0033] Fig.11 : is a timing chart showing an example of exposure control of the pixel circuit in the Q3Q4 detection period in the first embodiment of the present invention.

[0034] Fig.12 : is a timing chart showing an example of exposure control of the pixel circuit within the image capturing period in the first embodiment of the present invention.

[0035] Fig.13 is a diagram showing an example of the relationship between the reference coordinate system and the plane coordinate system in the first embodiment of the present invention.

[0036] Fig.14 is a diagram showing an example of the relationship between the reference coordinate system and the cylindrical coordinate system in the first embodiment of the present invention.

[0037] Fig.15 1 is a diagram showing an example of a usage scenario of the electronic device in the first embodiment of the present invention when photographing an inclined plane.

[0038] Fig.16 : are diagrams showing examples of a plane shape, images before and after conversion, and a translation result in the first embodiment of the present invention.

[0039] Fig.17 1 is a diagram showing an example of a usage scenario of the electronic device in the first embodiment of the present invention when photographing a cylinder.

[0040] Fig.18 is a diagram showing an example of a column shape, images before and after conversion, and a translation result in the first embodiment of the present invention.

[0041] Fig.19 is a flowchart showing an example of the operation of the electronic device in the first embodiment of the present invention.

[0042] Fig. 20: is a block diagram showing a configuration example of an image pickup system in a modified example of the first embodiment of the present invention.

[0043] Fig.21 is an example showing the appearance of an image pickup system in a modified example of the first embodiment of the present invention.

[0044] Fig. 22 is a block diagram showing a configuration example of an electronic device in a second embodiment of the present invention.

[0045] Fig.23 is a block diagram showing a configuration example of an image pickup device in a second embodiment of the present invention.

[0046] Fig.24 is a flowchart showing an example of the operation of the electronic device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will describe the methods for implementing the present invention (hereinafter referred to as embodiments). The description will be made in the following order:

[0048] 1. First Embodiment (Example of Estimating Shape and Performing Coordinate Conversion)

[0049] 2. Second Embodiment (Example of Estimating Shape and Performing Coordinate Conversion Based on Results of Distance Measurement Using Phase Difference Detection Pixels)

[0050] <1. First Embodiment>

[0051] [Configuration Example of Electronic Device]

[0052] Figure 1 1 is a block diagram showing a configuration example of the electronic device 100 in the first embodiment. The electronic device 100 includes an operation section 111, a control section 112, a laser irradiation section 113, an insertion and removal section 114, a diffusion plate 115, a three-dimensional shape estimation section 200, a distance measurement section 116, a switch 117, and an imaging device 300. In addition, the electronic device 100 also includes an imaging lens 118, a coordinate conversion section 400, an optical character recognition section 119, a translation processing section 120, and a sound output section 121.

[0053] The operation unit 111 generates an operation signal according to the user's operation of a button or a switch. For example, the electronic device 100 is provided with a button that can be pressed in two stages. The operation unit 111 generates an operation signal for indicating any one of a state where the button is not pressed, a state where the button is pressed halfway, and a state where the button is pressed fully, and provides the operation signal to the control unit 112.

[0054] The laser irradiation section 113 irradiates in a predetermined direction using visible light (red light, etc.) having directivity as laser light under the control of the control section 112. For example, the laser light is applied in a direction substantially parallel to the optical axis direction of the imaging lens 118. Note that the laser irradiation section 113 is an example of an irradiation section described in the claims.

[0055] The inserting and removing section 114 performs a process of inserting the diffuser plate 115 into the optical path of the laser light or a process of removing the diffuser plate 115 from the optical path under the control of the control section 112. The inserting and removing section 114 is realized by incorporating an actuator such as a motor.

[0056] The diffuser plate 115 diffuses the laser light. Before the diffuser plate 115 is inserted, the shape of the laser light is dot-shaped. On the other hand, after the diffuser plate 115 is inserted, the laser light is diffused and becomes, for example, circular. Note that the shape of the diffused laser light is not limited to circular and may be linear or may be triangular.

[0057] In addition, switching of the shape of the laser light can be achieved by using, for example, a laser pointer like LP-RD312BKN produced by SANWA SUPPLY INC. or a laser pointer ELP-G20 produced by KOKUYO Co., Ltd. as a condensing pointer of the laser irradiation section 113 .

[0058] The imaging lens 118 gathers light and guides the light to the imaging device 300. Under the control of the control unit 112, the imaging device 300 captures an object and generates image data. A plurality of pixels arranged in a two-dimensional grid are provided in the imaging device 300. Each pixel generates pixel data having a level corresponding to the amount of light received. The imaging device 300 provides image data including these pixel data to the switch 117. Note that the imaging lens 118 and the imaging device 300 are examples of the imaging unit described in the claims.

[0059] The switch 117 switches the output destination of the image data from the image pickup device 300 under the control of the control section 112. The switch 117 outputs the image data to any one of the distance measurement section 116 and the coordinate conversion section 400.

[0060] Under the control of the control unit 112, the distance measuring unit 116 measures the distance from the imaging device 300 to a plurality of measurement points on the object. Here, the measurement point is a point irradiated with the diffused laser. The distance measuring unit 116 measures the distance based on the phase difference between the applied laser and the reflected light of the applied laser. This distance measurement method is called the time of flight (ToF) method. The distance measuring unit 116 provides distance information indicating the distance of each measurement point to the three-dimensional shape estimation unit 200.

[0061] The three-dimensional shape estimation unit 200 estimates the shape of the object based on the distance information. The three-dimensional shape estimation unit 200 provides parameters related to the estimated shape of the object to the coordinate conversion unit 400. These parameters will be described in detail below. Note that the three-dimensional shape estimation unit 200 is an example of a shape estimation unit described in the claims.

[0062] The coordinate conversion section 400 performs a predetermined coordinate conversion on the image data by using the parameters from the three-dimensional shape estimation section 200. The coordinate conversion is a process of converting the coordinates on the surface of the object irradiated with laser light into the coordinates on a predetermined reference plane. For example, a plane parallel to the imaging plane of the imaging device 300 is used as a reference plane. The coordinate conversion section 400 provides the image data after the coordinate conversion to the optical character recognition section 119.

[0063] The control section 112 controls the entire electronic device 100 . When a predetermined button is pressed halfway, the control section 112 controls the laser irradiation section 113 to start continuous irradiation with laser light. In addition, the control section 112 causes the insertion and removal section 114 to remove the diffusion plate 115 .

[0064] Then, when the button is fully pressed, the control section 112 controls the laser irradiation section 113 to intermittently irradiate with laser. For example, the control section 112 provides a rectangular wave or sine wave light emission control signal CLKp to the laser irradiation section 113, and the laser irradiation section 113 emits light synchronized with the signal. Pulse light having a frequency of, for example, 20 megahertz (MHz) is applied by this control.

[0065] In addition, when the button is fully pressed, the control section 112 causes the insertion and removal section 114 to insert the diffusion plate 115. In addition, the control section 112 controls the imaging device 300 to capture image data. Then, the control section 112 controls the switch 117 to provide the image data to the distance measuring section 116, and causes the distance measuring section 116 to measure the distance within a fixed distance measurement period.

[0066] After the distance measurement period has elapsed, the control section 112 controls the image pickup device 300 to pick up image data again. In addition, the control section 112 controls the switch 117 to supply the image data to the coordinate conversion section 400 .

[0067] The optical character recognition section 119 performs OCR on the image data and supplies the OCR result to the translation processing section 120 .

[0068] The translation processing unit 120 performs a translation process of replacing the language (Japanese, etc.) of the text including characters recognized by the OCR with another predetermined language (English, etc.). The translation processing unit 120 provides the translation processing result to the sound output unit 121. The sound output unit 121 outputs the translation result of the translation processing unit 120 using sound.

[0069] Note that although components such as the laser irradiation section 113 and the imaging element 300 are provided in one device (electronic device 100), these components may be dispersedly provided in a plurality of devices. For example, the laser irradiation section 113, the insertion and removal section 114, and the diffusion plate 115 may be provided in a laser pointer unit outside the electronic device 100. In addition, the optical character recognition section 119 and the translation processing section 120 may be provided on an information processing device outside the electronic device 100. In this case, the electronic device 100 only needs to send the image data and the character information to the information processing device and receive the OCR result or the translation result.

[0070] In addition, although the electronic device 100 performs OCR, the electronic device 100 may also perform processing other than OCR of the image data. For example, the electronic device 100 may perform processing for recognizing a specific object (face, etc.) or a two-dimensional barcode instead of OCR. In addition, although the electronic device 100 outputs the translation result using sound, the translation result may also be displayed on a display unit such as a liquid crystal monitor.

[0071] [Structural Example of Three-Dimensional Shape Estimation Unit]

[0072] Figure 2 1 is a block diagram showing a configuration example of the three-dimensional shape estimation unit 200 in the first embodiment. The three-dimensional shape estimation unit 200 includes a measurement coordinate storage unit 210, a measurement coordinate detection unit 220, a distance information storage unit 230, a least squares calculation unit 240, a parameter storage unit 250, an error calculation unit 260, and a parameter supply unit 270.

[0073] The distance information storage unit 230 stores distance information indicating the distance of each measurement point measured by the distance measurement unit 116 .

[0074] The measurement coordinate detection unit 220 detects the three-dimensional coordinates of each measurement point in a predetermined reference coordinate system based on the distance information as the measurement coordinates. For the reference coordinate system, for example, a coordinate system including the X0 axis and the Y0 axis parallel to the imaging plane of the imaging device 300 and the Z0 axis perpendicular to the imaging plane is used. The measurement coordinate detection unit 220 acquires in advance the angle defined by the direction of the diffused laser and the predetermined reference axes (such as the X0, Y0 and Z0 axes), and calculates the measurement coordinates of each measurement point using trigonometric functions based on the angle and the measurement distance. The measurement coordinate detection unit 220 enables the measurement coordinate storage unit 210 to store the measurement coordinates of each measurement point. In the following, the number of measurement points is represented by N (N is an integer greater than 2), and the measurement coordinates of the i-th (i is an integer from 0 to N-1) measurement point are represented by (x 0i ,y 0i ,z 0i )express.

[0075] The least squares calculation unit 240 calculates a function that best fits the measurement coordinate set for each of the plurality of candidate shapes by using the least squares method. The plurality of candidate shapes include, for example, a columnar shape and a planar shape inclined relative to a reference plane. In addition, here, the function obtained by calculation is a function for expressing the relationship between the coordinates in the coordinate system of the candidate shape and the coordinates in the reference coordinate system.

[0076] Here, assuming that a coordinate system including Xp, Yp, and Zp axes orthogonal to each other is a plane coordinate system on an inclined plane, the relationship between coordinates in the plane coordinate system and coordinates in the reference coordinate system is represented, for example, by the following expression.

[0077]

[0078] In the above expression, (x pi ,y pi ,z pi ) represents the coordinates in the plane coordinate system. In addition, Rp represents a rotation matrix and is represented by the following expression.

[0079] Rp=Rxp·Ryp·Rzp

[0080] In the above expression, Rxp represents the rotation angle tr around the x0 axis. xp In addition, Ryp represents the rotation angle tr around the y0 axis yp The rotation matrix when , and Rzp is the rotation angle tr around the z0 axis zp The rotation matrix when .

[0081] In addition, Tp represents a translation vector including an x0 component, a y0 component, and a z0 component. The values ​​of these x0 component, y0 component, and z0 component are represented by A, B, and C.

[0082] When the measurement point is located on the Xp-Yp plane, the Zp component of the coordinates of the measurement point should be 0. Therefore, when the Zp component is not 0, the component is regarded as an error. Rp and Tp when the sum of square errors Ep is minimized are obtained by the following expressions.

[0083]

[0084] To solve the above expression, for example, we use tr xp , tr yp , tr zp , A, B, and C perform partial differentiation on both sides of the above expression.

[0085] Below, assuming that a coordinate system including Xc, Yc, and Zc axes orthogonal to each other is a cylindrical coordinate system as a cylinder, the relationship between coordinates in the cylindrical coordinate system and coordinates in the reference coordinate system is represented by, for example, the following expression.

[0086]

[0087] In the above expression, (x ci ,y ci ,z ci ) represents the coordinates in the cylindrical coordinate system. In addition, Rc represents the rotation matrix and is represented by the following expression.

[0088] Rc=Rxc·Ryc·Rzc

[0089] In the above expression, Rxc represents the rotation angle tr around the x0 axis. xc In addition, Ryc represents the rotation angle tr around the y0 axis yc , and Rzc is the rotation angle tr around the z0 axis zc The rotation matrix when . r is the radius of the cylinder.

[0090] In addition, Tc represents a translation vector including an x0 component, a y0 component, and a z0 component. The values ​​of these x0 components, y0 components, and z0 components are represented by D, E, and F.

[0091] Assuming that r is the radius of the cylinder, when the measuring point is located on the surface of the cylinder, the sum of the squares of the Xc and Zc coordinates of the measuring point should be equal to the square of r. Thus, when the difference between them is not 0, the difference is regarded as an error. Rc, Tc, and r when the sum of squared errors Ec is minimized are obtained by the following expressions.

[0092]

[0093] To solve the above expression, for example, we use tr xc , tr yc , tr zc , D, E, and F perform partial differentiation on both sides of the above expression.

[0094] The least squares calculation section 240 causes the parameter storage section 250 to store the obtained parameter set of Rp and Tp and the parameter set of Rc, Tc, and r.

[0095] For each candidate shape, the error calculation section 260 calculates the error when assuming the shape by using the parameters corresponding to the candidate shape. For example, the error calculation section 260 calculates the sum Ep of square errors corresponding to the inclined plane shape and the sum Ec of square errors corresponding to the columnar shape by using Expressions 2 and 4, and provides the sums Ep and Ec to the parameter providing section 270.

[0096] The parameter providing section 270 estimates any one of the plurality of candidate shapes as the shape of the actual object based on the error. The parameter providing section 270 compares the sum of square errors Ep and the sum of square errors Ec with each other, and estimates the shape corresponding to the error having the smaller value as the shape of the object. Then, the parameter providing section 270 reads the parameter group corresponding to the shape from the parameter holding section 250, and provides the parameter group together with the identification information of the estimated shape to the coordinate conversion section 400. Note that the parameter providing section 270 is an example of the estimation processing section described in the claims.

[0097] Note that, in order to estimate the shape, the three-dimensional shape estimation unit 200 assumes two shapes, namely, a plane shape and a columnar shape, but the types of candidate shapes are not limited thereto. The candidate shape may be, for example, a spherical shape or a cubic shape. In addition, the number of candidate shapes is not limited to two and may be three or more. When the number of candidate shapes is three or more, the parameter providing unit 270 estimates the candidate shape with the smallest error as the shape of the object.

[0098] Figure 3 1 is a diagram showing an example of distance information in the first embodiment. For each measurement point, the distance information includes the distance from the imaging lens 118 to the measurement point. For example, when the distance to the measurement point P1 is 100 meters and the distance to the measurement point P2 is 102 meters, the three-dimensional shape estimation unit 200 stores "100" corresponding to P1 and "102" corresponding to P2 in the distance information storage unit 230.

[0099] Figure 4is a diagram showing an example of measurement coordinates in the first embodiment. For each measurement point, the measurement coordinates on the X0 axis, the Y0 axis, and the Z0 axis are calculated based on the distance to the measurement point and the laser irradiation angle. It is assumed that the X0 coordinate, the Y0 coordinate, and the Z0 coordinate at the measurement point P1 are calculated as 50, 50, and 51, respectively, and the X0 coordinate, the Y0 coordinate, and the Z0 coordinate at the measurement point P2 are calculated as 50, 51, and 51, respectively. In this case, the three-dimensional shape estimation unit 200 saves the measurement coordinates (50, 50, 51) corresponding to the measurement point P1 and the measurement coordinates (50, 51, 51) corresponding to the measurement point P2 into the measurement coordinate storage unit 210.

[0100] Figure 5 1 is a diagram showing an embodiment of the parameters of the first embodiment. The function of expression 1 corresponding to the plane shape includes parameters of the rotation matrix Rp and the translation vector Tp. The function of expression 2 corresponding to the column shape includes parameters of the rotation matrix Rc, the translation vector Tc, and the radius r. The three-dimensional estimation unit 200 calculates these parameters by the least squares method and saves them in the parameter storage unit 250.

[0101] [Configuration Example of Image Pickup Device]

[0102] Figure 6 3 is a block diagram showing an example of the configuration of the imaging device 300 in the first embodiment. The imaging device 300 includes a row scanning circuit 310, a pixel array section 320, a timing control section 340, a plurality of AD (analog to digital) conversion sections 350, a column scanning circuit 360, and a signal processing section 370. The pixel array section 320 includes a plurality of pixel circuits 330 arranged in a two-dimensional grid. Hereinafter, a set of pixel circuits 330 arranged in a predetermined direction is referred to as a "row", and a set of pixel circuits 330 arranged in a direction perpendicular to the row is referred to as a "column". The above-mentioned AD conversion section 350 is provided for each column.

[0103] The timing control section 340 controls the row scanning circuit 310 , the AD conversion section 350 , and the column scanning circuit 360 in synchronization with the vertical synchronization signal.

[0104] The row scanning circuit 310 exposes all rows simultaneously and sequentially selects rows after exposure to cause the rows to output pixel signals. Under the control of the row scanning circuit 310, the pixel circuits 330 output pixel signals each having a level corresponding to the amount of light received.

[0105] The AD conversion sections 350 each perform AD conversion on a pixel signal from a row corresponding thereto. Under the control of the column scanning circuit 360, the AD conversion section 350 outputs the AD-converted pixel signal as pixel data to the signal processing section 370. The column scanning circuit 360 sequentially selects the AD conversion section 350 and causes the AD conversion section 350 to output pixel data.

[0106] The signal processing section 370 performs signal processing such as Correlated Double Sampling (CDS) processing on the image data including the pixel data and supplies the image data after the signal processing to the switch 117 .

[0107] [Configuration Example of Pixel Circuit]

[0108] Figure 7 330 is a block diagram showing a configuration example of the pixel circuit 330 in the first embodiment. The pixel circuit 330 includes a light receiving element 331 , a transfer switch 332 , charge storage sections 333 and 334 , and selector switches 335 and 336 .

[0109] The light receiving element 331 performs photoelectric conversion on light and generates electric charge. For example, a photodiode is used for the light receiving element 331.

[0110] Under the control of the row scanning circuit 310, the transmission switch 332 connects the light receiving element 331 to any one of the charge storage section 333, the charge storage section 334, and the reset power supply Vrst. The transmission switch 332 is implemented by, for example, a plurality of metal oxide semiconductor (MOS) transistors. When the light receiving element 331 is connected to the reset power supply Vrst, the charge output from the drain of the MOS transistor is eliminated, and the charge of the light receiving element 331 is initialized.

[0111] The charge storage portions 333 and 334 store charges and generate a voltage corresponding to the amount of the stored charges. For example, for the charge storage portions 333 and 334, a floating diffusion layer is used.

[0112] Under the control of the row scanning circuit 310, the selector switch 335 opens and closes the path between the charge storage section 333 and the AD conversion section 350. Under the control of the row scanning circuit 310, the selector switch 336 opens or closes the path between the charge storage section 334 and the AD conversion section 350. For example, when the FD read signal RD_FD1 is supplied by the row scanning circuit 310, the selector switch 335 becomes a closed state, and when the FD read signal RD_FD2 is supplied by the row scanning circuit 310, the selector switch 336 becomes a closed state. Each of the selector switches 335 and 336 is implemented by, for example, a MOS transistor.

[0113] [Configuration Example of Coordinate Transformation Section]

[0114] Figure 8 4 is a block diagram showing a configuration example of the coordinate conversion section 400 in the first embodiment. The coordinate conversion section 400 includes a clipping processing section 420 , a frame memory 410 , and an address conversion section 430 .

[0115] The clipping processing unit 420 clips out a region having a predetermined shape (which is, for example, a rectangle and includes a portion surrounded by a circular laser) from the image data. The clipping processing unit 420 causes the frame memory 410 to store the clipped region as a clipping region. The frame memory 410 stores the clipping region.

[0116] For each pixel in the clipping area, the address conversion unit 430 converts the coordinates of the pixel into coordinates in the reference plane. The coordinate conversion unit 400 receives the identification information of the shape estimated by the three-dimensional shape estimation unit 200 and the parameter group of the shape. When the shape is estimated as a planar shape, the rotation matrix Rp and the translation vector Tp of Expression 1 are provided as parameters to the coordinate conversion unit 400. At the same time, when the shape is estimated as a three-dimensional shape, the rotation matrix Rc, the translation vector Tc and the radius r of Expression 2 are provided as parameters to the coordinate conversion unit 400.

[0117] When the shape is estimated to be a planar shape, the coordinates (u ) on the imaging device 300 are expressed by the following expression using parameters Rp and Tp: i ,v i ) and the coordinates on the inclined plane (x ei ,y ei )’s positional relationship.

[0118]

[0119] In the above expression, f represents the focal length. By using the above expression, the address conversion unit 430 outputs the coordinates (u i ,v i ) as the pixel value of the reference plane coordinate (x ei ,y ei ) The address conversion unit 430 can thereby generate an image facing the imaging plane.

[0120] Meanwhile, when the shape is estimated to be a columnar shape, the coordinates (u ) on the imaging device 300 are expressed by the following expression using parameters Rp, Tp, and r. i ,v i ) and the coordinates on the cylindrical surface (x ei ,y ei )’s positional relationship.

[0121]

[0122] By using the above expression, the address conversion unit 430 outputs the coordinates (u i ,v i ) as the pixel value of the reference plane coordinate (x ei ,y ei ) The address conversion unit 430 can thereby generate an image facing the imaging plane.

[0123] Note that the coordinate conversion section 400 cuts out only the periphery of the portion surrounded by the laser light and performs coordinate conversion thereon, but may perform coordinate conversion on the entire image without performing cutout.

[0124] Fig. 9 1 is a diagram showing an example of the shape of the laser in the first embodiment. Part a of the figure is a diagram showing an example of the shape of the laser when the button is half pressed. As shown in part a of the figure, when the button is half pressed, the dot laser 500 is applied with the diffuser plate 115 removed.

[0125] in addition, Fig. 9 Part b of FIG. 1 is a diagram showing an example of the shape of the laser when the button is fully pressed. As shown in part b of the figure, when the button is fully pressed, the diffuser 115 is inserted and a circular laser is applied. The user adjusts the position irradiated with the laser so that the characters to be processed by OCR (such as characters on a price tag, etc.) fall into the circular shape of the laser.

[0126] Note that in the ToF method, surface irradiation is generally performed in which the entire surface on which distance measurement is to be performed is irradiated with laser. However, if distance measurement is performed only on a portion to be processed by OCR, the electronic device 100 does not need to irradiate the entire surface with laser, and it is sufficient to irradiate only the portion with laser. When the range of laser irradiation is narrowed, it is possible to suppress the electronic device 100 from consuming more energy than when the entire surface is irradiated with laser.

[0127] In addition, in the ToF method, non-visible light (such as infrared light) is generally used to perform irradiation. However, in the assumed usage scenario of the electronic device 100, the user needs to visually identify the laser irradiation position. Therefore, the electronic device 100 performs irradiation using visible light such as red light. Also in the case of using visible light, the principle of distance measurement by ToF is similar to the principle of distance measurement in the case of infrared light.

[0128] Fig.103 is a timing diagram showing an example of exposure control of the pixel circuit within the Q1Q2 detection period in the first embodiment. When the button is fully pressed, the pixel circuit 330 alternately repeats the detection of the received light quantities Q1 and Q2 and alternately repeats the detection of the received light quantities Q3 and Q4. Hereinafter, the detection period for receiving the light quantities Q1 and Q2 is referred to as the "Q1Q2 detection period", and the detection period for receiving the light quantities Q3 and Q4 is referred to as the "Q3Q4 detection period". The length of each of the Q1Q2 detection period and the Q3Q4 detection period is the period of the vertical synchronization signal VSYNC (for example, 1 / 60 seconds).

[0129] Here, when a specific phase (e.g., rising edge) of the laser light emission control signal CLKp is set to 0 degrees, the received light quantity Q1 is the accumulated amount of light quantity q1 received from 0 degrees to 180 degrees in the Q1Q2 detection period. The frequency of the light emission control signal CLKp is as high as 20 megahertz (MHz), and therefore the amount of light q1 received per cycle (1 / 20 microseconds) is small and difficult to detect. Thus, the pixel circuit 330 accumulates each q1 in the Q1Q2 detection period of, for example, 1 / 60 seconds (longer than the cycle (1 / 20 microseconds) of the light emission control signal CLKp), and detects the total amount of q1 as the received light quantity Q1. In addition, the received light quantity Q2 is the accumulated amount of light quantity q2 received from 180 degrees to 360 degrees in the Q1Q2 detection period.

[0130] In addition, the received light amount Q3 is the accumulated amount of the reflected light amount q3 received from 90 degrees to 270 degrees in the Q3Q4 detection period. In addition, the received light amount Q4 is the accumulated amount of the reflected light amount q4 received from 270 degrees to 90 degrees in the Q3Q4 detection period.

[0131] Substituting these received light amounts Q1, Q2, Q3, and Q4 into the following expression, the distance measuring section 116 can thus calculate the distance d to the measurement point. For example, Larry Li of Texas Instruments Incorporated describes a method for deriving this expression in "Time-of-Flight Camera—An Introduction," a technical white paper SLOA190B published in January 2014 (revised in May 2014).

[0132] d=(c / 4πf)*tan -1 {(Q3-Q4) / (Q1-Q2)}

[0133] In the above expressions, for example, d represents a distance, and its unit is meter (m). For example, c represents the speed of light, and its unit is meter per second (m / s). tan -1 () represents the inverse function of the tangent function.

[0134] For example, in the Q1Q2 detection period from time T1 to time T2, the received light amounts Q1 and Q2 in the period are detected. First, the row scanning circuit 310 provides a reset signal RST to all rows within a predetermined pulse period starting from time T1. The amount of charge stored in the charge storage portions 333 and 334 in all rows is initialized by the reset signal RST. In addition, the row scanning circuit 310 initializes the charge of the light receiving element 331 in all rows using the FD selection signal SEL_FD.

[0135] Then, within the cycle of the light emission control signal CLKp in the Q1Q2 detection period, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 333 from 0 degrees to 180 degrees for all rows using the FD selection signal SEL_FD. The received light quantity q1 is stored in the charge storage section 333 by this control.

[0136] In addition, within the cycle of the emission control signal CLKp in the Q1Q2 detection period, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 334 from 180 degrees to 360 degrees for all rows using the FD selection signal SEL_FD. The received light quantity q2 is stored in the charge storage section 334 by this control.

[0137] At time T11 immediately before time T2, the row scanning circuit sequentially provides FD read signals RD_FD1 and RD_FD2 to the first row. The pixel signal corresponding to the amount of received light of the first row is read by this control. Next, the row scanning circuit 310 sequentially provides FD read signals RD_FD1 and RD_FD2 to the second row and reads the pixel signal. Thereafter, similarly, the row scanning circuit 310 sequentially selects rows and reads pixel signals.

[0138] In this way, in the Q1Q2 detection period, each pixel circuit 330 detects the received light amount Q1 from 0 degrees to 180 degrees and the received light amount Q2 from 180 degrees to 360 degrees.

[0139] Fig.11 3 is a timing diagram showing an example of exposure control of the pixel circuit 330 in the Q3Q4 detection period in the first embodiment. For example, in the Q3Q4 detection period from time T2 to time T3, the received light amounts Q3 and Q4 of the period are detected. First, the row scanning circuit 310 supplies the reset signal RST to all rows within a predetermined pulse period starting from time T2, and initializes the charge amounts stored in the charge storage portions 333 and 334 in all rows. In addition, the row scanning circuit 310 initializes the charge of the light receiving element 331 in all rows using the FD selection signal SEL_FD.

[0140] Then, for all rows, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 334 from the initial 0 degrees to 90 degrees using the FD selection signal SEL_FD. By this control, the received light amount q4 is stored in the charge storage section 334. Thereafter, within the cycle of the light emission control signal CLKp, for all rows, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 333 from 90 degrees to 270 degrees using the FD selection signal SEL_FD. By this control, the received light amount q3 is stored in the charge storage section 333.

[0141] In addition, within the cycle of the light emission control signal CLKp in the Q3Q4 detection period, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 334 from 270 degrees to 90 degrees for all rows using the FD selection signal SEL_FD. By this control, the received light amount q4 is stored in the charge storage section 334.

[0142] At time T2 immediately before time T3, the row scanning circuit 310 sequentially provides FD read signals RD_FD1 and RD_FD2 to the first row. Through this control, the pixel signals corresponding to the received light amounts Q3 and Q4 of the first row are read. Thereafter, similarly, the row scanning circuit 310 sequentially selects rows and reads pixel signals.

[0143] In this way, in the detection period of Q3Q4, each pixel circuit 330 detects the received light amount Q3 from 90 degrees to 270 degrees and the received light amount Q4 from 270 degrees to 90 degrees.

[0144] Fig.12 1 is a timing chart showing an example of exposure control of the pixel circuit 330 in the imaging period in the first embodiment. First, the row scanning circuit 310 supplies the reset signal RST to all rows within a predetermined pulse period starting from time T3, and initializes the amount of charge stored in all rows. In addition, the row scanning circuit 310 initializes the charge of the light receiving element 331 in each row using the FD selection signal SEL_FD.

[0145] Then, for all rows, the row scanning circuit 310 transfers the charge generated by the light receiving element 331 to the charge storage section 333 using the FD selection signal SEL_FD. Thereafter, at time T31 immediately before time T4, the row scanning circuit 310 provides the FD read signal RD_FD1 to the first row and reads the pixel signal. Next, the row scanning circuit 310 provides the FD read signal RD_FD1 to the second row and reads the pixel signal. Thereafter, similarly, the row scanning circuit 310 sequentially selects rows and reads pixel signals. Thus, image data is captured.

[0146] Fig.13 1 is a diagram showing an example of the relationship between the reference coordinate system and the plane coordinate system in the first embodiment. As described above, the following assumption is made: the reference coordinate system including the X0 axis, the Y0 axis, and the Z0 axis is rotated and offset in parallel by Rp and Tp, thereby obtaining a plane coordinate system including the Xp axis, the Yp axis, and the Zp axis. Based on the above assumption, the electronic device 100 calculates the sum Ep of square errors assuming that N measurement points (e.g., measurement point 502 and measurement point 503) are located on the Xp-Yp plane 510.

[0147] Fig.14 is a diagram showing an example of the relationship between the reference coordinate system and the cylindrical coordinate system in the first embodiment. As described above, the following assumption is made: the reference coordinate system including the X0 axis, the Y0 axis, and the Z0 axis is rotated and offset in parallel by Rc and Tc, thereby obtaining a cylindrical coordinate system including the Xc axis, the Yc axis, and the Zc axis. Based on the above assumption, the electronic device 100 calculates the sum Ec of square errors assuming that N measurement points (e.g., measurement point 502 and measurement point 503) are located on the surface of the cylinder 511.

[0148] Fig.15 is a diagram showing an example of a usage scenario of the electronic device 100 in the first embodiment when photographing an inclined plane. Part a of the figure is a diagram showing an example of the shape of the laser when the button is pressed halfway. As shown in part a of the figure, when the button is pressed halfway, a point-shaped laser 500 is applied. Here, the electronic device 100 is, for example, a stick-shaped device. The user grasps the electronic device 100 with his hand and can change the direction of the laser. The user moves the point irradiated by the laser 500 and fully presses the button at the position of the character to be processed by OCR. For example, when a boneless fish fillet is provided with a flat price tag and the characters of the price tag are read by OCR, the user moves the point irradiated by the laser to the price tag. The plane of the price tag corresponds to Fig.13 plane 510.

[0149] Fig.15 Part b of FIG. 5 is a diagram showing an example of the shape of the laser when the button is fully pressed. As shown in part b of the figure, when the button is fully pressed, a circular laser 501 is applied to the price tag (plane 510).

[0150] Fig.16 1 is a diagram showing an example of a plane shape, images before and after conversion, and a translation result in the first embodiment. Part a of the figure is a diagram showing an example of a reference plane shape. The color scale represents depth. For example, a plane inclined relative to the imaging plane of the imaging device 300 is assumed.

[0151] Fig.16Part b shows an example of a portion of the captured image data that is irradiated with laser light. Although the character strings of "Salmon" and "100 yen" are written in the price tag, the characters are distorted because the price tag plane does not face the camera plane. Therefore, according to this situation, there is a risk that the characters cannot be accurately read by OCR.

[0152] In this regard, the electronic device 100 converts coordinates on the inclined plane into coordinates on the reference plane. Fig.16 Part c of FIG. 1 is an example of an image after coordinate conversion. As shown in part c of the figure, the plane not facing the imaging plane is converted into a reference plane facing the imaging plane, and the distortion of the characters disappears. The electronic device 100 reads the characters in the converted image through OCR and performs translation.

[0153] Fig.16 Part d of FIG. 1 is a diagram showing an example of a translation result. Japanese "Salmon" and "100 yen" are translated into English "Salmon" and "100 yen", and are outputted by voice.

[0154] Fig.17 1 is a diagram showing an example of a usage scenario of the electronic device 100 of the first embodiment when photographing an object having a columnar shape. Part a of the diagram is a diagram showing an example of the shape of the laser when the button is pressed halfway. For example, in the case where a label is attached to a columnar portion of a wine bottle, and the user intends to read the characters of the label by OCR, the user moves the point irradiated by the laser 500 to the label. The columnar portion of the wine bottle corresponds to Fig.14 Cylinder 511.

[0155] Fig.17 Part b of FIG. 5 is a diagram showing an example of the shape of the laser when the button is fully pressed. As shown in part b of the figure, when the button is fully pressed, a columnar laser 501 is applied to the label (column 511).

[0156] Fig.18 1 is a diagram showing an example of a column shape, images before and after conversion, and a translation result in the first embodiment. Parts a and b of the figure are diagrams showing examples of candidate column shapes, respectively.

[0157] Fig.18 Part b shows an example of a portion irradiated with laser light in the captured image data. Although the character string "Antioxidant free nice wine" is recorded in the label, the character is distorted because the surface of the label is curved. Therefore, in this case, there is a risk that the character cannot be accurately read by OCR.

[0158] In this regard, the electronic device 100 converts coordinates on the cylinder into coordinates on the reference plane. Fig.18 Part c of FIG. 1 is an example of an image after coordinate conversion. As shown in part c of the figure, the curved surface of the cylinder is converted to a reference plane, and the distortion of the character disappears. The electronic device 100 reads the characters in the converted image through OCR and performs translation.

[0159] Fig.18 Part d of FIG. 1 is a diagram showing an example of a translation result. The Japanese word "Antioxidant free nice wine" is translated into the English word "Antioxidant free nice wine" and is outputted by sound.

[0160] [Operation examples of electronic devices]

[0161] Fig.19 1 is a flowchart showing an example of the operation of the electronic device 100 in the first embodiment. For example, when the electronic device 100 is powered on, the operation starts. The electronic device 100 determines whether the user presses the button halfway (step S901). When the button is pressed halfway (step S901: yes), the electronic device 100 removes the diffuser plate (step S902), and performs spot laser irradiation (step S903). Then, the electronic device 100 determines whether the user presses the button all the way (step S904).

[0162] When the button is not pressed halfway (step S901 : No) or when the button is not pressed fully (step S904 : No), the electronic device 100 repeats step S901 and subsequent steps.

[0163] When the button is fully pressed (step S904: Yes), the electronic device 100 inserts the diffuser (step S905), and performs circular laser irradiation and reception of its reflected light (step S906). The electronic device 100 performs distance measurement by the ToF method for each measurement point (step S907), and estimates the three-dimensional shape of the object based on the measured distance (step S908).

[0164] Then, the electronic device 100 performs photography of the object (step S909), and performs coordinate conversion on the image data (step S910). The electronic device 100 performs optical character recognition on the coordinate-converted image (step 911), and performs translation and sound output (step S912). After step S912, the electronic device 100 ends the operation for OCR.

[0165] In this way, according to the first embodiment of the present invention, the electronic device 100 estimates the shape of the object based on the distance to the measurement point and performs coordinate conversion based on the shape. Thus, it is possible to eliminate the distortion of characters and increase the accuracy of OCR.

[0166] [Deformation example]

[0167] In the above-mentioned first embodiment, the stick-shaped electronic device 100 includes an operating unit 111, an imaging device 300, etc. However, a camera unit mounted on a wearable terminal may be provided with an operating unit 111, etc. The imaging system in this modified example of the first embodiment is different from that of the first embodiment in that a camera unit mounted on a wearable terminal is provided with an operating unit 111, etc.

[0168] Fig. 20 1 is a block diagram showing a configuration example of a camera system. The camera system includes a wearable terminal 150 and a camera unit 101. In addition, the wearable terminal 150 includes an operation section 151 and a terminal control section 152. In addition, the configuration of the camera unit 101 is similar to that of the electronic device 100 of the first embodiment, except that the camera unit 101 includes a camera unit control section 122 and a switch 123 instead of the control section 112 and the switch 117. Note that the camera unit 101 is an example of an electronic device described in the claims.

[0169] The operation unit 151 generates an operation signal according to the user's operation of a switch or a button. For example, the operation unit 151 performs an operation for capturing image data (for example, pressing a shutter button). The operation unit 151 provides the generated operation signal to the terminal control unit 152.

[0170] The terminal control section 152 controls the entire wearable terminal 150. The terminal control section 152 supplies an operation signal to the camera unit control section 122, and receives image data and the like from the camera unit control section 122.

[0171] The camera unit control section 122 controls the entire camera unit 101. When the camera unit 101 is not installed on the wearable terminal 150, the camera unit control section 122 performs control similar to the control in the first embodiment. At the same time, when the camera unit 101 is installed on the wearable terminal 150, the camera unit control section 122 determines whether to perform an operation for capturing image data (e.g., pressing a shutter button). When the shutter button is pressed, the camera unit control section 122 controls the switch 123 to output image data to the wearable terminal 150.

[0172] Under the control of the camera unit control section 122 , the switch 123 switches the output destination of the image data.

[0173] Fig.21 1 is an example of the appearance of the camera system in the modified example of the first embodiment. The wearable terminal 150 is a glasses-type terminal. The camera unit 101 can be connected to the side of the wearable terminal 150 via the connection tool 155. The lens portion of the terminal is omitted in the figure. For the connection tool 155, for example, the unexamined Japanese patent application publication No. 2015-515638 can be used. Figure 1 or Figure 2 The connection tool shown in .

[0174] The camera unit 101 also operates when detached from the wearable terminal 150. The function of the camera unit 101 is similar to that of the electronic device 100 of the first embodiment.

[0175] In addition, the wearable terminal 150 is provided with a blocking member 156. When the camera unit 101 is mounted, the blocking member 156 blocks the laser light from the camera unit 101. Note that light incident toward the imaging element 300 is not blocked.

[0176] For example, it is assumed that the laser irradiation section 113 is provided at the lower part of the camera unit 101, and the image pickup device 300 is provided at the upper part of the image pickup element 101. In this case, the blocking member 156 blocks only the lower part of the camera unit 101. Thus, only the laser light is blocked. The laser light is blocked in a case where distance measurement is unnecessary, and thus the laser light can be prevented from appearing in the captured image data. This can improve the convenience of the image pickup system.

[0177] When the camera unit 101 is installed on the wearable terminal 150, the camera unit 101 does not perform distance measurement, but captures image data, etc. The wearable terminal 150 analyzes the image data and displays predetermined information on the head-up display of the terminal. For example, current location information, an image to be synthesized with the recognized object, etc. are displayed.

[0178] In this way, according to the modified example of the first embodiment of the present invention, when the camera unit 101 is mounted on the wearable terminal 150, the blocking member 156 blocks the laser light. Therefore, it is possible to prevent the laser light from appearing in the captured image data.

[0179] <2. Second Embodiment>

[0180] Although the first embodiment described above performs distance measurement by the ToF method, the ToF method requires components for performing laser irradiation, and thus it is difficult to reduce costs and sizes accordingly. The electronic device 100 of the second embodiment is different from the electronic device 100 in the first embodiment in that it achieves reductions in size and cost.

[0181] Fig. 22 1 is a block diagram showing a configuration example of the electronic device 100 in the second embodiment. For the electronic device 100 of the second embodiment, a digital camera such as a digital single-lens reflex camera is assumed. In addition, compared with the electronic device 100 of the first embodiment, the electronic device 100 of the second embodiment is different in that a display section 123 is further provided instead of the laser irradiation section 113.

[0182] In addition, in the image pickup device 300 of the second embodiment, ordinary pixels and phase difference detection pixels for detecting the phase difference of a pair of pupil-split images are provided.

[0183] When a button such as a shutter button is halfway pressed, the control section 112 controls the image pickup device 300 to shoot an image with a relatively low resolution as a live view image in synchronization with a vertical synchronization signal, and then controls the switch 117 to provide the live view image to the display section 123.

[0184] When the button is fully pressed, the control unit 112 causes the image sensor 300 to capture image data having a higher resolution than the live view image. Then, the control unit 112 controls the switch 117 to supply the image data to the distance measurement unit 116 .

[0185] The display unit 123 displays image data and is implemented by, for example, a liquid crystal display.

[0186] The distance measuring section 116 measures the distance based on the pixel signal of the phase difference pixel, and provides the image data and the distance information to the three-dimensional shape estimating section 200. In the image data, the pixel value of the position of the pixel signal of the phase difference pixel is interpolated from the surrounding pixels. The processing of the coordinate conversion section 400 and the like is similar to that of the first embodiment.

[0187] Note that the electronic device 100 uses the distance measurement information to estimate the three-dimensional shape, but the distance measurement information is also used for auto focus (AF). Fig. 22 In the embodiment, the AF control unit for controlling the position of the focus lens based on the distance measurement information is omitted.

[0188] Fig.23 1 is a block diagram showing a configuration example of an imaging device 300 of the second embodiment. The imaging device 300 of the second embodiment is different from the imaging device 300 of the first embodiment in that a normal pixel circuit 380 and a phase difference detection pixel circuit 390 are provided instead of the pixel circuit 330.

[0189] The normal pixel circuit 380 is a pixel that performs photoelectric conversion on visible light such as R (red), G (green) and B (blue), and generates a pixel signal. The phase difference detection pixel circuit 390 is a pixel for detecting the phase difference of a pair of pupil segmentation images. In this way, the following method is called an image plane phase difference method: pixels for pixel difference detection are set on the image plane, and the distance is measured based on the signal of the pixel. For example, the structure of the imaging device 300 in this image plane phase difference method is described in Japanese Patent Application Laid-Open No. 2012-124791.

[0190] Fig.24 1 is a flowchart showing an example of the operation of the electronic device 100 in the second embodiment. The electronic device 100 determines whether the user presses the button halfway (step S901). When the button is pressed halfway (step S901: yes), the electronic device 100 captures and displays a live view image (step S921). The user adjusts the orientation of the electronic device 100 while viewing the live view image so that the characters to be processed by OCR fall into the monitor (display unit 123). After the adjustment, the user presses the button all the way.

[0191] Then, the electronic device 100 determines whether the user has fully pressed the button (step S904). When the button is fully pressed, the electronic device 100 captures image data, and obtains the distance of each measurement point based on the phase difference detected in the phase difference detection pixel circuit 390 (step S922). In addition, the electronic device 100 estimates the three-dimensional shape (step S908), and interpolates the phase difference pixels in the image data (step S923). Then, the electronic device 100 executes step S910 and subsequent steps.

[0192] In this way, according to the second embodiment of the present invention, since the distance is measured based on the phase difference detected by the phase difference detection pixel, it is not necessary to provide the laser irradiation part 113 and the diffusion plate 115, and the number or components can be reduced. This is conducive to reducing size or cost.

[0193] Note that the above-described embodiments are examples for implementing the present invention, and the matters in the embodiments and the matters for defining the invention in the claims have a corresponding correspondence. Similarly, the matters for defining the invention in the claims and the matters represented by the same names as the matters for defining the invention in the embodiments of the present invention have a corresponding correspondence. However, the present invention is not limited to these embodiments, and the present invention can be implemented by modifying these embodiments in various ways without departing from the gist of the present technology.

[0194] In addition, the processing steps described in the above embodiments may be understood as a method including a series of these steps. Alternatively, the processing steps described in the above embodiments may be understood as a program for causing a computer to execute a series of these steps or a recording medium storing the program. For example, a CD (Compact Disc), an MD (Mini Disc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray (registered trademark) disc, etc. may be used as a recording medium.

[0195] Note that the effects described here are not necessarily limiting, and any effects described in the present invention can be produced.

[0196] Note that the present technology may have the following configurations.

[0197] (1) An electronic device comprising:

[0198] An imaging unit that captures an object and captures image data;

[0199] a distance measuring unit that measures a distance from the imaging unit to each of a plurality of measurement points on the surface of the object;

[0200] a shape estimating unit that estimates the shape of the object based on the measured distance; and

[0201] A coordinate conversion section performs coordinate conversion on the image data, the coordinate conversion converting three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

[0202] (2) The electronic device according to (1), further comprising:

[0203] A character recognition unit recognizes characters on the surface of the object in the image data after the coordinate conversion.

[0204] (3) The electronic device according to (1), further comprising

[0205] an irradiation section that performs irradiation with irradiation light,

[0206] The distance measuring unit measures the distance based on a phase difference between the reflected light of the irradiation light and the irradiation light.

[0207] (4) The electronic device according to (3), wherein:

[0208] The imaging unit performs a process for capturing the image data and a process for receiving the irradiation light.

[0209] (5) The electronic device according to (3) or (4), wherein:

[0210] The irradiation section performs irradiation of pulse light as the irradiation light, and the pulse light is synchronized with a predetermined periodic signal.

[0211] (6) The electronic device according to any one of (3) to (5), wherein:

[0212] The irradiation section selects one of the spot light and the diffuse light according to a predetermined operation and performs irradiation of the light as the irradiation light, and

[0213] The distance measuring unit measures the distance while the irradiation of the diffuse light is performed.

[0214] (7) The electronic device according to any one of (3) to (6), wherein:

[0215] When a predetermined button is pressed halfway, the irradiation unit starts irradiating the irradiation light, and

[0216] When the predetermined button is fully pressed, the imaging unit captures the image data.

[0217] (8) The electronic device according to any one of (1) to (7), wherein:

[0218] The electronic device is a camera unit mounted to a wearable terminal.

[0219] (9) The electronic device according to any one of (1) to (8), wherein:

[0220] The shape estimating unit estimates any one of a plurality of candidate shapes as the shape of the object based on the distance.

[0221] (10) The electronic device according to (9), wherein the shape estimating unit includes:

[0222] a coordinate detection unit that detects the coordinates of each of the plurality of measurement points based on the distance as measurement coordinates;

[0223] a function acquisition section that acquires, for each of the plurality of candidate shapes, a function for expressing a relationship between coordinates of the candidate shape and coordinates of a predetermined reference coordinate system by using the measured coordinates;

[0224] an error calculation section that calculates an error when a shape of the object is assumed for each of the plurality of candidate shapes based on the acquired function and the measurement coordinates; and

[0225] An estimation processing unit estimates a shape having a minimum error among the plurality of candidate shapes as a shape of the object.

[0226] (11) The electronic device according to (1) or (2), wherein:

[0227] The image data includes a plurality of pixel data.

[0228] The imaging section includes a phase difference detection pixel that detects a phase difference between two pupil division images and a normal pixel that performs photoelectric conversion on light and generates any one of the plurality of pixel data, and

[0229] The distance measurement section measures the distance based on the phase difference detected by the phase difference detection pixels.

[0230] (12) A method for controlling an electronic device, comprising:

[0231] an imaging step, in which an imaging unit is used to image the object and capture image data;

[0232] a distance measuring step of measuring a distance from the imaging unit to each of a plurality of measuring points on a surface of the object;

[0233] a shape estimating step of estimating the shape of the object based on the measured distance; and

[0234] A coordinate conversion step of performing coordinate conversion on the image data, the coordinate conversion converting three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

[0235] (13) A program for causing a computer to execute the following steps:

[0236] an imaging step, in which an imaging unit is used to image the object and capture image data;

[0237] a distance measuring step of measuring a distance from the imaging unit to each of a plurality of measuring points on a surface of the object;

[0238] a shape estimating step of estimating the shape of the object based on the measured distance; and

[0239] A coordinate conversion step of performing coordinate conversion on the image data, the coordinate conversion converting three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

[0240] Reference numerals list

[0241] 100 electronic device; 101 camera unit; 111, 151 operation unit; 112 control unit; 113 laser irradiation unit; 114 insertion and removal unit; 115 diffusion plate; 116 distance measurement unit; 117, 123 switch; 118 camera lens; 119 optical character recognition unit; 120 translation processing unit; 121 sound output unit; 122 camera unit control unit; 123 display unit; 150 wearable terminal; 152 terminal control unit; 155 connection tool; 156 blocking member; 200 three-dimensional shape estimation unit; 210 measurement coordinate storage unit; 220 measurement coordinate detection unit; 230 distance information storage 1. a least squares calculation unit; 2. a parameter storage unit; 2. a error calculation unit; 2. a parameter providing unit; 3. a camera device; 3. a row scanning circuit; 3. a pixel array unit; 3. a pixel circuit; 3. a light receiving element; 3. a transmission switch; 3. a charge storage unit; 3. a selector switch; 3. a timing control unit; 3. a AD conversion unit; 3. a column scanning circuit; 3. a signal processing unit; 3. a normal pixel circuit; 3. a phase difference detection pixel circuit; 4. a coordinate conversion unit; 4. a frame memory; 4. a shearing processing unit; 4. a address conversion unit.

Claims

1. An electronic device, comprising: An imaging unit that captures an object and captures image data; a distance measuring unit that measures a distance from the imaging unit to each of a plurality of measurement points on the surface of the object; a shape estimating unit for estimating the shape of the object based on the measured distance; and A coordinate conversion unit that performs coordinate conversion on the surface of the object in the image data, the coordinate conversion converting the three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape, wherein the surface of the object in the image data before the coordinate conversion is not parallel to the imaging plane of the imaging unit, and the surface of the object in the image data after the coordinate conversion is converted into the reference plane parallel to the imaging plane of the imaging unit.

2. The electronic device according to claim 1, further comprising: A character recognition unit recognizes characters on the surface of the object in the image data after the coordinate conversion.

3. The electronic device according to claim 1, further comprising: an irradiation section that performs irradiation with irradiation light, The distance measuring unit measures the distance based on a phase difference between the reflected light of the irradiation light and the irradiation light.

4. The electronic device according to claim 3, wherein: The imaging unit performs a process for capturing the image data and a process for receiving the irradiation light.

5. The electronic device according to claim 3, wherein: The irradiation section performs irradiation of pulse light as the irradiation light, and the pulse light is synchronized with a predetermined periodic signal.

6. The electronic device according to claim 3, wherein: The irradiation section selects one of the spot light and the diffuse light according to a predetermined operation and performs irradiation of the light as the irradiation light, and The distance measuring unit measures the distance while the irradiation of the diffuse light is performed.

7. The electronic device according to claim 3, wherein: When a predetermined button is pressed halfway, the irradiation unit starts irradiating the irradiation light, and When the predetermined button is fully pressed, the imaging unit captures the image data.

8. The electronic device according to any one of claims 1 to 7, wherein: The electronic device is a camera unit mounted to a wearable terminal.

9. The electronic device according to any one of claims 1 to 7, wherein: The shape estimating unit estimates any one of a plurality of candidate shapes as the shape of the object based on the distance.

10. The electronic device according to claim 9, wherein: The shape estimating unit comprises: a coordinate detection unit that detects the coordinates of each of the plurality of measurement points based on the distance as measurement coordinates; a function acquisition section that acquires, for each of the plurality of candidate shapes, a function for expressing a relationship between coordinates of the candidate shape and coordinates of a predetermined reference coordinate system by using the measured coordinates; an error calculation section that calculates an error when a shape of the object is assumed for each of the plurality of candidate shapes based on the acquired function and the measurement coordinates; and An estimation processing unit estimates a shape having a minimum error among the plurality of candidate shapes as a shape of the object.

11. The electronic device according to claim 1 or 2, wherein: The image data includes a plurality of pixel data. The imaging section includes a phase difference detection pixel that detects a phase difference between two pupil division images and a normal pixel that performs photoelectric conversion on light and generates any one of the plurality of pixel data, and The distance measurement section measures the distance based on the phase difference detected by the phase difference detection pixels.

12. A method for controlling an electronic device according to any one of claims 1 to 11, comprising: an imaging step, in which an imaging unit is used to image the object and capture image data; a distance measuring step of measuring a distance from the imaging unit to each of a plurality of measuring points on a surface of the object; a shape estimating step, in which the shape of the object is estimated based on the measured distance; and A coordinate conversion step of performing coordinate conversion on the image data, the coordinate conversion converting three-dimensional coordinates on the surface of the object into plane coordinates on a predetermined reference plane based on the estimated shape.

13. A recording medium storing a program for causing a computer to execute the method according to claim 12.

Images