Camera module, photographing method, and electronic device

Abstract

The present technology relates to a camera module, a photographing method, and an electronic apparatus with which it is possible to reduce the memory capacity required for electronic camera-shake correction. The camera module comprises an imaging unit for outputting a photographic image for each image block of a prescribed number of horizontal lines, an image block storage unit for storing the image block, and an image correction unit for performing camera-shake correction for the each image block. The present technology can be applied, for example, to a digital video camera provided with an electronic camera-shake correction function.

Description

[Technical Field]

[0001] The present technology relates to a camera module, an imaging method, and an electronic device, and more particularly to a camera module, an imaging method, and an electronic device that perform electronic image stabilization. [Background technology]

[0002] Representative methods for correcting camera shake in an imaging device include an optical image stabilizer (OIS) and an electronic image stabilization (EIS).

[0003] One type of electronic image stabilization method performs image stabilization based on the amount of motion calculated from a captured image. However, this method can result in a decrease in the accuracy of image stabilization due to complex calculations, reduced accuracy in measuring the amount of motion in low light, and errors in estimating the amount of camera shake for moving subjects.

[0004] In response to this, electronic image stabilization using motion sensor information acquired by an angular velocity sensor, an acceleration sensor, etc. has been proposed (see, for example, Patent Document 1). In the invention described in Patent Document 1, the movement of the camera module is detected using motion sensor information acquired by an angular velocity sensor, an acceleration sensor, etc., and image stabilization of the captured image is performed for each frame. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2017 / 014071 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the invention described in Patent Document 1, image stabilization is performed on a captured image for each frame, so a memory capable of storing at least one frame of captured image is required. This increases the memory capacity, which results in, for example, higher costs, an increase in the area of ​​the LSI (Large Scale Integration), and an increase in power consumption. Furthermore, the increased power consumption may require the installation of large cooling fins or cooling fans.

[0007] The present technology has been made in view of such circumstances, and makes it possible to reduce the memory capacity required for electronic image stabilization. [Means for solving the problem]

[0008] A camera module according to one aspect of the present technology includes an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines, an image block memory unit that stores the image blocks, and an image correction unit that performs image stabilization for each image block.

[0009] An image capturing method according to one aspect of the present technology outputs a captured image for each image block of a predetermined number of horizontal lines, stores the image blocks, and performs image stabilization for each image block.

[0010] An electronic device according to one aspect of the present technology includes an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines, an image block memory unit that stores the image blocks, and an image correction unit that performs image stabilization for each image block.

[0011] In one aspect of the present technology, a captured image is output for each image block of a predetermined number of horizontal lines, the image blocks are stored, and image stabilization is performed for each image block. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a camera module to which the present technology is applied. [Figure 2] FIG. 2 is a block diagram for explaining a camera shake correction process. [Figure 3] FIG. 10 is a diagram illustrating an example of drive timing of an image sensor and a motion sensor. [Figure 4] FIG. 10 is a diagram illustrating an example of output timing of an image sensor. [Figure 5] FIG. 2 is a diagram illustrating an example of an image block. [Figure 6] FIG. 10 is a diagram illustrating a method for generating an extended image block. [Figure 7] FIG. 10 is a diagram illustrating an example of an extended image block. [Figure 8] FIG. 10 is a diagram for explaining a method for extracting motion data. [Figure 9] FIG. 10 is a diagram illustrating an example of a method for extracting motion data. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for extracting motion data. [Figure 11] FIG. 10 is a diagram for explaining a method for calculating a rotational movement amount. [Figure 12] FIG. 2 is a diagram illustrating an example of an arrangement of pixels in a captured image. [Figure 13] FIG. 10 is a diagram for explaining a method for generating a captured image frame. [Figure 14] FIG. 2 is a diagram showing an example of a captured image frame. [Figure 15] 10A and 10B are diagrams for explaining a process of transforming a captured image frame. [Figure 16] 10A and 10B are diagrams for explaining a process of transforming a captured image frame. [Figure 17] FIG. 10 is a diagram illustrating an example of an output image frame. [Figure 18] 10A and 10B are diagrams illustrating an example of a method for setting a cropping position of an output image. [Figure 19] FIG. 10 is a diagram showing an output image frame divided into frame blocks. [Figure 20] 10A and 10B are diagrams illustrating examples of overlapping between an output image frame and frame blocks. [Figure 21] FIG. 10 is an enlarged view showing an example of how an output image frame and frame blocks overlap. [Figure 22] FIG. 10 is a diagram for explaining coordinate transformation of an output image. [Figure 23] FIG. 10 is a diagram for explaining coordinate transformation of an output image. [Figure 24] 10A and 10B are diagrams for explaining a method for extracting and arranging pixel data of an output image. [Figure 25] 10A and 10B are diagrams for explaining a method for extracting and arranging pixel data of an output image. [Figure 26] 10A and 10B are diagrams for explaining a method for extracting and arranging pixel data of an output image. [Figure 27] FIG. 10 is a diagram illustrating an example of an output format of an output image. [Figure 28] FIG. 10 is a diagram for explaining a method for arranging pixel data of an output image. [Figure 29] FIG. 10 is a diagram illustrating an example of an output image. [Figure 30] 1 is a block diagram showing a configuration example of an embodiment of an electronic device to which the present technology is applied. [Figure 31] FIG. 1 is a diagram illustrating an example of use of an image sensor. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present technology will be described in the following order. 1. Embodiment 2. Variations 3.Other

[0014] <<1. Embodiment>> An embodiment of the present technology will be described with reference to FIGS.

[0015] <Configuration example of camera module 1> FIG. 1 shows an embodiment of a camera module 1 to which the present technology is applied.

[0016] The camera module 1 includes a mode switching unit 11, a synchronization processing unit 12, an image sensor 13, an image block memory unit 14, an image block extension unit 15, an extended image block memory unit 16, a motion sensor 17, a motion data memory unit 18, a motion data extraction unit 19, a filter 20, a rotational movement amount detection unit 21, an image correction unit 22, an output image memory unit 23, and an output control unit 24.

[0017] The mode switching unit 11 switches the drive mode of the camera module 1. There are two drive modes for the camera module 1: a frame blanking mode and a shooting mode. In the frame blanking mode, the image sensor 13 is not driven between frames, and only the motion sensor 17 is driven. In the shooting mode, both the image sensor 13 and the motion sensor 17 are driven.

[0018] The synchronization processing unit 12 controls the synchronization between the operation of the image sensor 13 and the operation of the motion sensor 17 .

[0019] The image sensor 13 is configured by, for example, a CMOS image sensor etc. The image sensor 13 includes an imaging control unit 31 and an imaging unit 32.

[0020] The imaging control unit 31 controls the imaging by the imaging unit 32 under the control of the synchronization processing unit 12 .

[0021] The imaging unit 32 has a pixel region in which a plurality of pixels are arranged two-dimensionally. Under the control of the imaging control unit 31, the imaging unit 32 performs exposure and output for each block (hereinafter referred to as a pixel block) of a predetermined number of horizontal lines in the pixel region. The imaging unit 32 also generates an image block including pixel data of the pixels in the pixel block, and stores the image block data, with a header added to the beginning of the image block, in the image block storage unit 14. As a result, one frame of a captured image obtained by imaging is output for each image block and stored in the image block storage unit 14.

[0022] The image block extension unit 15 extends an image block by adding a portion of pixel data of adjacent image block data to the image block in the image block data stored in the image block storage unit 14. The image block extension unit 15 stores the extended image block (hereinafter referred to as an extended image block) in the extended image block storage unit 16.

[0023] The motion sensor 17 is configured, for example, by a 6-axis sensor capable of measuring 3-axis acceleration and 3-axis angular velocity. Note that the motion sensor 17 may also be configured, for example, by a 9-axis sensor capable of measuring 3 additional axes of the earth. The motion sensor 17 generates sensor data (hereinafter referred to as motion data) indicating the measurement results and stores the sensor data in the motion data storage unit 18.

[0024] The motion data extraction unit 19 extracts motion data used to detect the amount of rotational movement of the captured image from the motion data stored in the motion data storage unit 18 and supplies the extracted motion data to the filter 20 .

[0025] The filter 20 is configured by a digital filter such as a moving average filter, an IIR (Infinite Impulse Response) filter, an FIR (Finite Impulse Response) filter, etc. The filter 20 filters the motion data and supplies the filtered motion data to a rotational movement amount detection unit 21.

[0026] The rotational movement amount detection unit 21 detects the amount of rotational movement of the captured image based on the filtered motion data. The rotational movement amount detection unit 21 supplies data indicating the detected amount of rotational movement to the deformation unit 42 of the image correction unit 22.

[0027] The image correction unit 22 performs camera shake correction on the captured image for each image block. More specifically, the image correction unit 22 performs rotation correction on the rotational movement of the captured image for each image block. The image correction unit 22 also performs distortion correction on the lens distortion of the camera module 1 for each image block. The image correction unit 22 includes a captured image frame generation unit 41, a deformation unit 42, an output image frame generation unit 43, a crop position setting unit 44, a coordinate conversion unit 45, and an output image generation unit 46.

[0028] The photographed image frame generating unit 41 generates a photographed image frame that indicates the shape of the photographed image, and supplies it to the transforming unit 42 .

[0029] The transformation unit 42 transforms the captured image frame by performing distortion correction on the captured image frame and further performing rotation correction based on the rotational movement amount detected by the rotational movement amount detection unit 21. This calculates the captured image transformed by the lens distortion and rotational movement and the shape of each image block included in the captured image. The transformation unit 42 supplies the transformed captured image frame to the crop position setting unit 44.

[0030] The output image frame generation unit 43 generates an output image frame indicating the shape and pixel positions of the output image, and supplies it to the cutout position setting unit 44 and the coordinate conversion unit 45 .

[0031] The cropping position setting unit 44 sets the output image frame at a position in the transformed captured image frame where the output image is to be cropped. This sets the position from which the output image is to be cropped in the captured image having the shape calculated by the deformation unit 42. The cropping position setting unit 44 supplies the captured image frame and data indicating the cropping position to the coordinate conversion unit 45.

[0032] The coordinate conversion unit 45 converts the coordinates of each pixel of the output image into coordinates in the captured image transformed by distortion and rotational movement based on the captured image frame, the output image frame, and the cropping position. The coordinate conversion unit 45 supplies data indicating the coordinates before and after conversion of each pixel of the output image to the output image generation unit 46.

[0033] The output image generation unit 46 acquires the extended image block from the extended image block storage unit 16. The output image generation unit 46 generates pixel data for each pixel of the output image based on pixel data for pixels in the extended image block that correspond to the post-conversion coordinates of each pixel of the output image. The output image generation unit 46 generates an output image by arranging the generated pixel data in the output image storage unit 23 according to the pre-conversion coordinates of each pixel of the output image.

[0034] The output control unit 24 controls the external output of the output image stored in the output image storage unit 23. The output control unit 24 notifies the mode switching unit 11 that the output image has been output.

[0035] <Image stabilization processing> Next, the image stabilization process executed by the camera module 1 will be described with reference to the flowchart of FIG.

[0036] In step S1, the camera module 1 starts driving the motion sensor 17. As a result, the motion sensor 17 starts a process of measuring the acceleration and angular velocity of the camera module 1 at a predetermined driving frequency (sampling frequency) and storing motion data indicating the measurement results in the motion data storage unit 18.

[0037] For example, if the drive frequency of the motion sensor 17 is 4 kHz, the motion sensor 17 measures acceleration and angular velocity and stores the motion data every 0.25 ms.

[0038] In step S2, the image sensor 13 starts capturing the next frame.

[0039] Specifically, the mode switching unit 11 instructs the synchronization processing unit 12 to switch from the frame blanking mode to the shooting mode.

[0040] The synchronization processing unit 12 starts synchronizing the operation of the image sensor 13 with the operation of the motion sensor 17. For example, the synchronization processing unit 12 synchronizes the horizontal synchronization signal of the image sensor 13 with the drive signal of the motion sensor 17. This synchronizes the exposure timing of each pixel block of the image sensor 13 with the measurement timing of the motion sensor 17.

[0041] Moreover, under the control of the imaging control unit 31, the imaging unit 32 starts exposing each pixel block in the pixel area in order from the first pixel block.

[0042] 3 shows an example of the drive timing of the image sensor 13 and the motion sensor 17. The horizontal axis represents time, and the vertical axis represents the pixel block numbers of the image sensor 13. Note that serial numbers starting from 0 are assigned sequentially to the pixel blocks at the beginning of the pixel area.

[0043] Period T1 in the figure indicates the frame blanking period of each pixel block of the image sensor 13, i.e., the period when each pixel block is not driven. Period T2 indicates the exposure period of each pixel block of the image sensor 13. Period T3 indicates the output period (readout period) of each pixel block of the image sensor 13. The white circles in the figure indicate the measurement timing (sampling timing) of the motion sensor 17.

[0044] For example, if the frame rate of image sensor 13 is 30 fps (frames per second) and the drive frequency (sampling frequency) of motion sensor 17 is 4 kHz, the number of motion data samples per frame is 4000 kHz / 30 fps = 133.33... In other words, the number of motion data samples per frame is 133 or 134.

[0045] Furthermore, for example, if the pixel area of ​​the image sensor 13 is 4000 pixels vertically by 4000 pixels horizontally, and the number of horizontal lines in each pixel block is 40, the pixel area is divided into 100 pixel blocks.

[0046] For example, 33 or 34 pieces of motion data are assigned to the frame blanking period of the image sensor 13, and 100 pieces of motion data are assigned to the exposure period + output period of the image sensor 13. As a result, for example, one piece of motion data is assigned to the exposure period + data output period of each pixel block.

[0047] In step S3, the camera module 1 starts outputting image blocks.

[0048] 4, the imaging unit 32 starts a process of reading pixel data of each pixel in pixel block units, starting from the first pixel block in the pixel area, and generating an image block including the read pixel data, under the control of the imaging control unit 31. The imaging unit 32 also starts a process of generating image block data shown in FIG.

[0049] Here, the image block data includes a header and an image block.

[0050] The header includes, for example, a frame number, an image block number, exposure conditions, pixel size, and the like.

[0051] An image block contains pixel data for each pixel in the corresponding pixel block.

[0052] Furthermore, the image block extension unit 15 starts a process of generating an extended image block based on each image block data stored in the image block storage unit 14. The image block extension unit 15 starts a process of storing the generated extended image block in the extended image block storage unit 16.

[0053] Here, a method for generating an extended image block will be described with reference to FIGS.

[0054] Fig. 6 shows image block data including the (n-1)th to (n+1)th image blocks, respectively, and Fig. 7 shows an example of an extended image block obtained by extending the nth image block.

[0055] The image block extension unit 15 removes the header from the nth image block data. The image block extension unit 15 also adds pixel data contained in a predetermined number of horizontal lines (e.g., two lines) at the end of the previous (n-1th) image block to the beginning of the nth image block. The image block extension unit 15 also adds pixel data contained in a predetermined number of horizontal lines (e.g., two lines) at the beginning of the next (n+1th) image block to the end of the nth image block.

[0056] In this way, an extended image block is generated by extending the first and last horizontal lines of the n-th image block, as shown in FIG.

[0057] The pixel data of the expanded portion of the expanded image block is used, for example, for color interpolation of the pixel data in the image block before expansion.

[0058] Furthermore, the image block extension unit 15 first generates an extended image block corresponding to the first image block of the captured image and stores it in the extended image block storage unit 16. Thereafter, every time an extended image block stored in the extended image block storage unit 16 is read, the image block extension unit 15 generates an extended image block corresponding to the next image block and stores it in the extended image block storage unit 16.

[0059] In step S4, the camera module 1 calculates the amount of rotational movement. Specifically, the motion data extraction unit 19 reads out motion data corresponding to the image blocks to be subjected to image stabilization from the motion data storage unit 18. Note that the image blocks are set as the targets for image stabilization in order from the first image block of the captured image.

[0060] For example, when the nth image block is the target of image stabilization, the motion data extraction unit 19 sets the reference time to the center time of the exposure period of the pixel block corresponding to the nth image block. For example, as shown in Fig. 8, the motion data extraction unit 19 reads out from the motion data storage unit 18 a predetermined number of pieces of motion data obtained at the time closest to the reference time (hereinafter referred to as reference motion data) and before and after that.

[0061] 9 and 10 show examples of extracted motion data corresponding to pixel block number 0 (hereinafter referred to as pixel block 0). Note that the motion data indicated by black circles in Fig. 9 and Fig. 10 indicate reference motion data.

[0062] Fig. 9 shows an example in which a total of 11 pieces of motion data are extracted, including the reference motion data and five pieces before and after the reference motion data. Fig. 10 shows an example in which a total of seven pieces of motion data are extracted, including the reference motion data and three pieces before and three pieces after the reference motion data.

[0063] The motion data extraction unit 19 supplies the extracted motion data to the filter 20 .

[0064] The filter 20 filters the extracted motion data using a predetermined method, and supplies the filtered motion data to a rotational movement amount detection unit 21 .

[0065] The motion data storage unit 18 is provided with a memory capacity equal to or greater than the number of motion data used by the filter 20.

[0066] The rotational movement amount detection unit 21 calculates the rotational movement amount of the captured image (image sensor 13) based on the filtered motion data. The method for calculating the rotational movement amount is not particularly limited, but for example, the Euler method, the Quaternion method, etc. may be used.

[0067] For example, as shown in FIG. 11, a rotation matrix R, a projection transformation matrix K, and a projection transformation matrix K are used to convert the coordinates of each pixel in the captured image P before rotational movement into the coordinates of each pixel in the image P′ after rotational movement. -1 Here, the rotation matrix R, the projection transformation matrix K, and the projection transformation matrix K are calculated. -1 is expressed by the following equation (1).

[0068]

number

[0069] θ pitch is the rotation angle of the image sensor 13 in the pitch direction in the camera coordinate system, and θ roll is the rotation angle of the image sensor 13 in the roll direction in the camera coordinate system, and θ yaw indicates the rotation angle of the image sensor 13 in the yaw direction in the camera coordinate system. x is the focal length in the x-axis direction (horizontal direction) of the camera coordinate system, and f y is the focal length in the y-axis direction (vertical direction) of the camera coordinate system, and x c is the optical center in the x-axis direction of the camera coordinate system, and y c indicates the optical center in the y-axis direction of the camera coordinate system.

[0070] The rotational movement amount detection unit 21 supplies data indicating the calculated rotational movement amount to the transformation unit 42.

[0071] In step S5, the camera module 1 calculates the amount of deformation of the captured image.

[0072] First, the photographed image frame generating unit 41 generates a photographed image frame and supplies the generated photographed image frame to the transforming unit 42 .

[0073] Specifically, Fig. 12 shows the arrangement of pixels in a captured image. For ease of explanation, the captured image is shown here with a smaller number of pixels (25 pixels vertically × 37 pixels horizontally) than the actual number.

[0074] The photographed image frame generation unit 41 sets frame points that constitute the photographed image frame Fa between pixels of the photographed image at predetermined intervals, as shown in Fig. 13. In this example, the frame points are set at intervals of six pixels vertically and seven pixels horizontally. Then, the photographed image frame generation unit 41 generates a mesh-like photographed image frame Fa by connecting adjacent frame points with straight lines, as shown in Fig. 14.

[0075] The captured image frame Fa is divided in the same way as the image blocks of the captured image. For ease of explanation, it is assumed below that the captured image is divided into four image blocks, and the captured image frame Fa is divided into four frame blocks BF0 to BF3 corresponding to the image blocks.

[0076] In the following description, when there is no need to distinguish between the frame blocks BF0 to BF3, they will simply be referred to as frame blocks BF.

[0077] The coordinates of each frame point are expressed by the coordinates of the image coordinate system of the captured image. For example, the coordinates of each frame point are expressed by the coordinates of the pixel in the upper left corner of the captured image when the coordinates of the pixel in the upper left corner of the captured image are set as the origin of the image coordinate system.

[0078] The deformation unit 42 deforms the captured image frame Fa. Specifically, the deformation unit 42 reflects the distortion of the lens (not shown) of the camera module 1 in the captured image frame Fa. For example, distortion correction parameters of OpenCV (Open Source Computer Vision Library) are used in this deformation process.

[0079] As a result, for example, the captured image frame Fa shown in A of Fig. 15 is transformed into the captured image frame Fa shown in B of Fig. 15 by reflecting the distortion of the lens of the camera module 1. The transformed captured image frame Fa shows the shape of the captured image when the distortion is reflected in the captured image. Furthermore, each transformed frame block BF shows the shape of the image block when the distortion is reflected in each image block of the captured image.

[0080] Next, the transformation unit 42 transforms the captured image frame Fa by rotating and moving the captured image frame Fa. Specifically, the transformation unit 42 transforms the captured image frame Fa by using the rotation matrix R, the projective transformation matrix K, and the projective transformation matrix K. -1 The captured image frame Fa is rotated by the rotational movement amount calculated by the rotational movement amount detection unit 21 using the above formula.

[0081] As a result, for example, the captured image frame Fa in Fig. 16A is transformed into the captured image frame Fa shown in Fig. 16B by reflecting the movement and deformation due to the rotational movement of the image sensor 13. The transformed captured image frame Fa shows the shape and position of the captured image when the distortion and rotational movement are reflected on the captured image. Furthermore, each transformed frame block BF shows the shape and position of the image block when the distortion and rotational movement are reflected on each image block of the captured image.

[0082] The transformation unit 42 supplies the transformed photographed image frame Fa to the cut-out position setting unit 44 .

[0083] In step S6, the cut-out position setting unit 44 sets the cut-out position of the output image.

[0084] Specifically, first, the output image frame generating unit 43 generates an output image frame and supplies it to the cropping position setting unit 44 and the coordinate conversion unit 45 .

[0085] 17 shows an example of an output image frame Fb. As described above, the output image frame Fb is a frame that indicates the shape and pixel positions of the output image. Here, for ease of explanation, the cropped image is represented with fewer pixels (8 pixels vertically × 15 pixels horizontally) than the actual number.

[0086] The coordinates of each pixel in the output image frame Fb are set independently of the captured image frame Fa. For example, the coordinates of the pixel in the upper left corner of the output image frame Fb are set as the origin.

[0087] Next, as shown in FIG. 18, the cut-out position setting unit 44 sets an output image frame Fb in the post-transformation captured image frame Fa at a position from which the output image is cut out.

[0088] The output image frame Fb is set, for example, at a predetermined position of the output image frame Fb before deformation (for example, the center of the output image frame Fb before deformation). As a result, the crop position of the output image is set to a predetermined position in the image coordinate system of the captured image.

[0089] In this way, the position from which the output image is cut out is set in the captured image that has been transformed by distortion and rotational movement.

[0090] The cut-out position setting unit 44 supplies the photographed image frame Fa and data indicating the set cut-out position to the coordinate conversion unit 45.

[0091] In step S7, the coordinate conversion unit 45 performs coordinate conversion. Specifically, the coordinate conversion unit 45 converts the coordinates of pixels in the output image frame into coordinates in the transformed captured image frame. More specifically, the coordinate conversion unit 45 converts the coordinates of pixels in the output image frame included in a frame block corresponding to an image block to be subjected to image stabilization into coordinates in the transformed frame block.

[0092] Fig. 19 is a diagram in which the output image frame Fb is divided into regions included in each frame block BF of the captured image frame Fa. Fig. 20 is a diagram showing the overlap between frame block BF0 and the output image frame Fb. Fig. 21 is an enlarged view of the overlapping portion of frame block BF0 and output image frame Fb in Fig. 20. As shown in Fig. 21, in this example, pixels Pc1 to Pc15 of the output image frame Fb are included in frame block BF0.

[0093] For example, if the image block corresponding to frame block BF0 is the target of image stabilization, the coordinate conversion unit 45 converts the coordinates of pixels Pc1 to Pc15 of the output image frame Fb included in frame block BF0 into coordinates in frame block BF0.

[0094] For example, first, the coordinate conversion unit 45 converts the coordinates of the intersections Pb1 to Pb4, which are obtained by appropriately thinning out the pixel intersections in the output image frame Fb shown in FIGS. 20 and 21, into coordinates in the frame block BF0.

[0095] 22A, the intersection point Pb2 in the output image frame Fb is included in the area surrounded by frame points Pa1 to Pa4 in the frame block BF0. The coordinate conversion unit 45 then calculates the coordinate of the intersection point Pb2 in the frame block BF0 based on the coordinates of the frame points Pa1 to Pa4 and the distances between the frame points Pa1 to Pa4 and the intersection point Pb2.

[0096] 22B, the intersection point Pb1 in the output image frame Fb is included in the area surrounded by frame points Pa1 to Pa4 in the frame block BF0. The coordinate conversion unit 45 then calculates the coordinate of the intersection point Pb1 in the frame block BF0 based on the coordinates of the frame points Pa1 to Pa4 and the distances between the frame points Pa1 to Pa4 and the intersection point Pb1.

[0097] The coordinates of the frame points Pa1 to Pa4 are the coordinates in the photographed image frame Fa before deformation, that is, the coordinates in the photographed image before deformation.

[0098] Next, the coordinate transformation unit 45 calculates the coordinates of pixels Pc1 to Pc15 in the frame block BF0 based on the transformed coordinates of intersections Pb1 to Pb4. For example, as shown in Fig. 23, the coordinates of pixels Pc1 to Pc7 in the frame block BF0 are calculated based on the transformed coordinates of intersections Pb1 to Pb3.

[0099] Here, the positional relationship between intersections Pb1 to Pb4 and pixels Pc1 to Pc15 is known. Therefore, the amount of calculation required is reduced by calculating the coordinates of pixels Pc1 to Pc15 based on the transformed coordinates of intersections Pb1 to Pb4, rather than directly converting the coordinates of pixels Pc1 to Pc15. This reduction in calculation amount becomes greater as the number of pixels in the output image frame Fb increases.

[0100] For example, the coordinate conversion unit 45 may directly convert the coordinates of the pixels Pc1 to Pc15 without converting the coordinates of the intersections Pb1 to Pb4.

[0101] In this way, the coordinates in the frame block BF0 of each pixel of the output image frame Fb included in the transformed frame block BF0 are calculated. That is, the coordinates of pixels included in the image block corresponding to the transformed frame block BF0 among the pixels of the output image are converted to coordinates in that image block.

[0102] The coordinate conversion unit 45 supplies the output image generation unit 46 with data indicating the coordinates before and after conversion of each pixel in the output image frame that is the conversion target.

[0103] In step S8, the output image generation unit 46 outputs pixel data. For example, the output image generation unit 46 reads from the extended image block storage unit 16 an extended image block corresponding to the image block to be subjected to image stabilization.

[0104] The output image generation unit 46 generates pixel data for each pixel of the output image frame based on the pixel data of the pixels of the extended image block corresponding to the transformed coordinates of each pixel of the output image frame that was the target of transformation in the processing of step S7.

[0105] For example, Fig. 24 shows an example in which intersections Pb1 to Pb4 of the output image frame Fb are arranged in an extended image block BP0 that includes an image block corresponding to the frame block BF0 based on their transformed coordinates. Fig. 25 is an enlarged view of the periphery of intersections Pb1 to Pb4 of Fig. 24, and shows an example in which pixels Pc1 to Pc15 of the output image frame Fb are arranged in the extended image block BP0 based on their transformed coordinates.

[0106] For example, the pixel data of the pixel at the position where pixel Pc1 of extended image block BP0 is arranged is extracted as the pixel data of pixel Pc1. Similarly, pixel data of the other pixels of output image frame Fb are extracted from extended image block BP0.

[0107] Furthermore, the output image generation unit 46 performs color interpolation on the pixel data of the pixels of the output image frame Fb as necessary.

[0108] For example, if the pixels of the output image are arranged according to a Bayer array, the extracted pixel data contains information for only one color among R (red), G (green), and B (blue). Furthermore, for example, the coordinates of each pixel in the output image frame after transformation may not match the coordinates of the pixels in the extended image block. In other words, each pixel in the output image frame after coordinate transformation may be located between pixels in the extended image block.

[0109] In response to this, for example, the output image generation unit 46 interpolates color information of pixel data of each pixel based on pixel data of pixels surrounding the position where each pixel of the output image frame is arranged in the extended image block.

[0110] For example, as shown in A of Fig. 26, color information of the pixel data of pixel Pc1 is interpolated based on pixel data of pixels surrounding the position where pixel Pc1 is arranged in extended image block BP0. For example, as shown in B of Fig. 26, color information of the pixel data of pixel Pc2 is interpolated based on pixel data of pixels surrounding the position where pixel Pc2 is arranged in extended image block BP0.

[0111] Furthermore, the output image generating unit 46 arranges the pixel data of each pixel of the output image frame in the output image storage unit 23 according to the coordinates before conversion.

[0112] 27, the output image generation unit 46 supplies pixel information including pixel data (e.g., color information, etc.) and pre-conversion coordinates of pixels Pc1 to Pc15 of the output image frame to the output image storage unit 23. Then, the output image generation unit 46 arranges the pixel data of pixels Pc1 to Pc15 in the output image storage unit 23 in accordance with the pre-conversion coordinates.

[0113] 28 shows an example of an output image stored in the output image storage unit 23. For example, pixel data of the output image frame generated by the output image generation unit 46 is arranged within an area A0 of the output image according to the coordinates before conversion.

[0114] In step S9, the output image generation unit 46 determines whether or not all image blocks have been processed. If there are image blocks in the captured image that are the target of image stabilization that have not yet been subjected to image stabilization processing, the output image generation unit 46 determines that all image blocks have not yet been processed, and the process returns to step S4.

[0115] Thereafter, in step S9, the processes of steps S4 to S9 are repeatedly executed until it is determined that all image blocks have been processed.

[0116] This allows image stabilization for each image block. That is, the amount of rotational movement is detected for each image block, and the coordinates of each pixel in the output image are converted to coordinates in the image block that has been transformed by the distortion and rotational movement. Furthermore, pixel data for pixels at the coordinates of the transformed image block is extracted, color interpolation is performed on the extracted pixel data, and the pixel data is arranged according to the coordinates of the output image before the conversion.

[0117] An example of an output image is shown in Fig. 29. In this way, pixel data extracted from extended image blocks BP0 to BP3 are arranged in areas A0 to A3 of the output image, which are shown in different patterns.

[0118] In this way, an output image is obtained in which the distortion and rotational movement of the captured image have been corrected.

[0119] On the other hand, if it is determined in step S9 that all image blocks have been processed, the process proceeds to step S10.

[0120] In step S10, the output control unit 24 outputs the output image. Specifically, the output control unit 24 reads the output image from the output image storage unit 23 and outputs it to the outside. The output control unit 24 also notifies the mode switching unit 11 that the output of the output image has been completed.

[0121] In step S11, the camera module 1 determines whether or not to end the image capture. If it is determined that the image capture is not yet to end, the process returns to step S2, and the processes of steps S2 to S11 are repeatedly executed until it is determined in step S11 that the image capture is to end.

[0122] On the other hand, in step S11, if an operation to end photographing is performed on an operation unit (not shown), the camera module 1 determines that photographing is to be ended, and the photographing process ends.

[0123] In this way, distortion and rotational movement correction is performed for each image block, thereby obtaining an output image in which distortion and rotational movement have been corrected.

[0124] Furthermore, since distortion and rotational movement correction is performed in pixel block units, the capacity of the image block storage unit 14 can be reduced, for example, compared to when correction is performed in frame units. This allows, for example, the size of the LSI used in the camera module 1 to be reduced. Furthermore, power consumption is reduced, and heat generation is reduced. This allows the size or elimination of cooling fins and fans. As a result, the camera module 1 can be made smaller. Furthermore, the cost of the camera module 1 can be reduced.

[0125] <<2. Modifications>> Hereinafter, modifications of the above-described embodiment of the present technology will be described.

[0126] For example, the flowchart in Figure 2 shows an example in which the amount of rotational movement of a captured image is detected for each image block, and the detected rotational movement is corrected. On the other hand, for example, the amount of rotational movement of a captured image may be detected for each frame, and the detected rotational movement may be corrected. That is, the amount of rotational movement may be detected once for image blocks in the same frame, and rotational correction may be performed for each image block based on the same amount of rotational movement.

[0127] For example, distortion correction may be omitted and only rotation correction may be performed.

[0128] For example, the output image frame generating unit 43 may generate an output image frame that reflects the lens distortion in advance.

[0129] 27 to the output control unit 24, and the output control unit 24 may output the pixel information of each pixel of the output image to the outside, without storing it in the output image storage unit 23. In this case, outside the camera module 1, the pixel data of each pixel is arranged based on the coordinate data in the pixel information of each pixel of the output image, and the output image is generated.

[0130] <<3.Others>> <Example of electronic device configuration> The camera module 1 of the above-described embodiment can be applied to various electronic devices, such as imaging systems such as digital still cameras and digital video cameras, mobile phones with imaging functions, or other devices with imaging functions.

[0131] FIG. 30 is a block diagram showing an example of the configuration of an imaging device mounted on an electronic device.

[0132] As shown in FIG. 30, an imaging device 101 includes an optical system 102, an imaging element 103, a signal processing circuit 104, a monitor 105, and a memory 106, and is capable of capturing still images and moving images.

[0133] The optical system 102 is configured to have one or more lenses, and guides image light (incident light) from a subject to the image sensor 103, forming an image on the light receiving surface (sensor section) of the image sensor 103.

[0134] The camera module 1 according to the above-described embodiment is applied as the image sensor 103. Electrons are accumulated in the image sensor 103 for a certain period of time in accordance with an image formed on the light receiving surface via the optical system 102. A signal corresponding to the electrons accumulated in the image sensor 103 is then supplied to the signal processing circuit 104.

[0135] The signal processing circuit 104 performs various types of signal processing on the pixel signals output from the image sensor 103. The image (image data) obtained by the signal processing performed by the signal processing circuit 104 is supplied to a monitor 105 to be displayed, or supplied to a memory 106 to be stored (recorded).

[0136] In the imaging device 101 configured in this manner, by applying the camera module 1 according to the above-described embodiment, it is possible to capture an image in which, for example, camera shake and lens distortion have been more accurately corrected.

[0137] <Examples of using image sensors> FIG. 31 is a diagram showing an example of using the image sensor 13 of the camera module 1 described above.

[0138] The image sensor 13 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0139] ·Digital cameras, mobile devices with camera functions, and other devices that take images for viewing purposes - Devices used for traffic purposes, such as in-vehicle sensors that take pictures of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognizing the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliances according to those gestures. -Medical and healthcare equipment, such as endoscopes and devices that take blood vessel images using infrared light - Security devices such as surveillance cameras for crime prevention and cameras for person authentication Cosmetic devices such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp Sports equipment such as action cameras and wearable cameras for sports purposes Agricultural equipment such as cameras for monitoring the condition of fields and crops

[0140] <Configuration combination example> The present technology can also be configured as follows.

[0141] (1) an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; an image correction unit that performs image stabilization for each image block; A camera module comprising: (2) a rotational movement amount detection unit that detects the rotational movement amount of the captured image; Further preparation, The image correction unit performs rotation correction for each of the image blocks based on the detected amount of rotational movement. The camera module according to (1) above. (3) The image correction unit a deformation unit that calculates the shapes of the captured image and the image block that have been deformed by the rotational movement based on the detected amount of rotational movement; a cutout position setting unit that sets a position for cutting out an output image from the photographed image having the calculated shape; a coordinate transformation unit that transforms the coordinates of pixels in the output image into coordinates in the image block of the calculated shape; an output image generation unit that generates pixel data of the output image based on pixel data of pixels in the image block that correspond to the transformed coordinates; The camera module according to (2) above, (4) The output image generation unit arranges pixel data of the output image according to coordinates before conversion. The camera module according to (3) above. (5) an output control unit that controls output of pixel information including pixel data of each pixel of the output image and coordinates before conversion; The camera module according to (3) above is further provided. (6) The deformation unit calculates the shape of the captured image and the shape of the image block deformed by the lens distortion and rotational movement of the camera module. The camera module according to any one of (3) to (5). (7) The output image generation unit performs color interpolation on pixel data of the output image based on pixel data of pixels surrounding the pixel of the image block corresponding to the transformed coordinates. The camera module according to any one of (3) to (6). (8) The image correction unit further corrects the lens distortion of the camera module for each image block. The camera module according to (2) above. (9) Motion sensors that detect acceleration and angular velocity Further preparation, The rotational movement amount detection unit detects the rotational movement amount of the captured image based on sensor data from the motion sensor. The camera module according to any one of (2) to (8). (10) the rotational movement amount detection unit detects the rotational movement amount for each of the image blocks; The image correction unit performs rotation correction on each of the image blocks based on the rotational movement amount detected for each of the image blocks. The camera module according to (9) above. (11) The rotational movement amount detection unit detects the rotational movement amount based on a plurality of pieces of sensor data acquired by the motion sensor before and after the center of an exposure period of the image block. The camera module according to (10) above. (12) the rotational movement amount detection unit detects the rotational movement amount for each frame, The image correction unit performs rotation correction on the image block based on the rotational movement amount detected for each frame. The camera module according to (9) above. (13) The imaging unit performs exposure and output for each pixel block of the predetermined number of horizontal lines in the pixel area. The camera module according to any one of (1) to (8). (14) a motion sensor for detecting acceleration and angular velocity; a synchronization processing unit that synchronizes the measurement timing of the motion sensor with the exposure timing of each of the pixel blocks; The camera module according to (13) above, further comprising: (15) outputting the captured image for each image block of a predetermined number of horizontal lines; storing the image block; Image stabilization is performed for each image block. Shooting method. (16) an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; an image correction unit that performs image stabilization for each image block; An electronic device comprising:

[0142] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0143] 1 camera module, 12 synchronization processing unit, 13 image sensor, 14 image block storage unit, 16 image block expansion unit, 17 motion sensor, 19 motion data extraction unit, 21 rotation movement amount detection unit, 22 image correction unit, 23 output image storage unit, 41 captured image frame generation unit, 42 transformation unit, 43 output image frame generation unit, 44 crop position setting unit, 45 coordinate conversion unit, 46 output image generation unit, 101 imaging device, 102 optical system, 103 imaging element

Claims

1. an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; a rotational movement amount detection unit that detects a rotational movement amount of the captured image; an image correction unit that performs image stabilization for each of the image blocks, a deformation unit that calculates the shapes of the captured image and the image block that have been deformed by the rotational movement based on the detected amount of rotational movement; a cutout position setting unit that sets a position for cutting out an output image from the photographed image having the calculated shape; a coordinate transformation unit that transforms the coordinates of pixels in the output image into coordinates in the image block of the calculated shape; an output image generation unit that generates pixel data of the output image based on pixel data of pixels in the image block that correspond to the transformed coordinates; an image correction unit comprising: an output control unit that controls output of pixel information including pixel data and coordinates before conversion for each pixel of the output image; A camera module comprising:

2. An imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; a rotational movement amount detection unit that detects a rotational movement amount of the captured image; an image correction unit that performs image stabilization for each of the image blocks, a deformation unit that calculates the shape of the captured image and the shape of the image block that have been deformed by lens distortion and rotational movement based on the detected amount of rotational movement; a cutout position setting unit that sets a position for cutting out an output image from the photographed image having the calculated shape; a coordinate transformation unit that transforms the coordinates of pixels in the output image into coordinates in the image block of the calculated shape; an output image generation unit that generates pixel data of the output image based on pixel data of pixels in the image block that correspond to the transformed coordinates; an image correction unit comprising: A camera module comprising:

3. The output image generation unit arranges pixel data of the output image according to coordinates before conversion.

3. The camera module according to claim 1.

4. The output image generation unit performs color interpolation on pixel data of the output image based on pixel data of pixels surrounding the pixel of the image block corresponding to the transformed coordinates.

3. The camera module according to claim 1.

5. The image correction unit further corrects the lens distortion of the camera module for each image block. The camera module of claim 1 .

6. Motion sensors that detect acceleration and angular velocity Further preparation, The rotational movement amount detection unit detects the rotational movement amount based on sensor data from the motion sensor.

3. The camera module according to claim 1.

7. the rotational movement amount detection unit detects the rotational movement amount for each of the image blocks; The image correction unit performs rotation correction on each of the image blocks based on the rotational movement amount detected for each of the image blocks. The camera module according to claim 6 .

8. The rotational movement amount detection unit detects the rotational movement amount based on a plurality of pieces of sensor data acquired by the motion sensor before and after the center of an exposure period of the image block. The camera module according to claim 7 .

9. the rotational movement amount detection unit detects the rotational movement amount for each frame, The image correction unit performs rotation correction on the image block based on the rotational movement amount detected for each frame. The camera module according to claim 6 .

10. The imaging unit performs exposure and output for each pixel block of the predetermined number of horizontal lines in the pixel area.

3. The camera module according to claim 1.

11. a motion sensor for detecting acceleration and angular velocity; a synchronization processing unit that synchronizes the measurement timing of the motion sensor with the exposure timing of each of the pixel blocks; The camera module of claim 10 further comprising:

12. A camera module comprising: outputting the captured image for each image block of a predetermined number of horizontal lines; storing the image blocks; Detecting a rotational movement amount of the captured image; calculating shapes of the captured image and the image block transformed by the rotational movement based on the detected amount of rotational movement; setting a position for cutting out an output image in the captured image of the calculated shape; Transforming pixel coordinates of the output image into coordinates in the image block of the calculated shape; generating pixel data of the output image based on pixel data of pixels of the image block corresponding to the transformed coordinates; Controlling the output of pixel information including pixel data and coordinates before conversion for each pixel of the output image; Shooting methods including.

13. A camera module comprising: outputting the captured image for each image block of a predetermined number of horizontal lines; storing the image blocks; Detecting a rotational movement amount of the captured image; calculating a shape of the captured image and a shape of the image block that have been deformed by lens distortion and rotational movement based on the detected amount of rotational movement; setting a position for cutting out an output image in the captured image of the calculated shape; Transforming pixel coordinates of the output image into coordinates in the image block of the calculated shape; generating pixel data of the output image based on pixel data of pixels of the image block corresponding to the transformed coordinates; Controlling the output of pixel information including pixel data and coordinates before conversion for each pixel of the output image; Shooting methods including.

14. an imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; a rotational movement amount detection unit that detects a rotational movement amount of the captured image; an image correction unit that performs image stabilization for each of the image blocks, a deformation unit that calculates the shapes of the captured image and the image block that have been deformed by the rotational movement based on the detected amount of rotational movement; a cutout position setting unit that sets a position for cutting out an output image from the photographed image having the calculated shape; a coordinate transformation unit that transforms the coordinates of pixels in the output image into coordinates in the image block of the calculated shape; an output image generation unit that generates pixel data of the output image based on pixel data of pixels in the image block that correspond to the transformed coordinates; an image correction unit comprising: an output control unit that controls output of pixel information including pixel data and coordinates before conversion for each pixel of the output image; An electronic device comprising:

15. An imaging unit that outputs a captured image for each image block of a predetermined number of horizontal lines; an image block storage unit that stores the image blocks; a rotational movement amount detection unit that detects a rotational movement amount of the captured image; an image correction unit that performs image stabilization for each of the image blocks, a deformation unit that calculates the shape of the captured image and the shape of the image block that have been deformed by lens distortion and rotational movement based on the detected amount of rotational movement; a cutout position setting unit that sets a position for cutting out an output image from the photographed image having the calculated shape; a coordinate transformation unit that transforms the coordinates of pixels in the output image into coordinates in the image block of the calculated shape; an output image generation unit that generates pixel data of the output image based on pixel data of pixels in the image block that correspond to the transformed coordinates; an image correction unit comprising: An electronic device comprising:

Images