Camera module and using method thereof

Abstract

A camera module includes a lens, an image sensor, a driver, a motion sensor, and a processor. The image sensor senses light passing through the lens. The driver is connected to the lens and moves the lens. The motion sensor senses the motion state of the camera module and outputs a sensor gain value. The processor is electrically connected to the image sensor, the driver, and the motion sensor. The processor calculates the amount of lens movement corresponding to the sensor gain value based on a motion model, and the driver, upon receiving the lens movement, moves the lens to compensate for image blur caused by camera shake. A method of using the camera module is also proposed.

Description

[Technical Field]

[0001] This invention relates to a camera module and its usage method. [Previous Technology]

[0002] Since camera modules were introduced into smartphones and mobile devices, blurring caused by involuntary shaking (such as hand shaking) has significantly affected image quality. To reduce this impact, image stabilization (IS) technology was developed, and various anti-shake technologies have been developed on the market, such as optical image stabilization (OIS) and electronic image stabilization (EIS).

[0003] Optical Image Stabilization (OIS) uses internal hardware structures within the lens to move the lens element, compensating for image blur caused by camera shake. Technically, OIS utilizes a gyroscope as a reference for determining the lens's direction of motion. A gyroscope, based on the theory of conservation of angular momentum, is a device used to sense and maintain orientation. For example, a gyroscope senses movement along the X and Y axes, providing feedback on the overall platform's shaking. By inputting the correct gyroscope gain, the offset caused by vibration can be effectively reduced. The aforementioned gyroscope gain refers to converting the angular velocity signal into a compensation control signal.

[0004] However, camera modules will have some errors during the assembly process, so the anti-shake correction model provided by the manufacturer often cannot be well applied to every camera module. [Summary of the Invention]

[0005] The present invention provides a camera module and a method for using the same, which can provide a corresponding motion model for each camera module.

[0006] One embodiment of the present invention provides a camera module, which includes a lens, an image sensor, a driver, a motion sensor, and a processor. The image sensor is used to sense light passing through the lens. The driver is connected to the lens and is used to move the lens. The motion sensor is used to sense the motion state of the camera module and output a sensor gain value. The processor is electrically connected to the image sensor, the driver, and the motion sensor. The processor is used to calculate the amount of lens movement corresponding to the sensor gain value according to a motion model, so that the driver receives the amount of lens movement and moves the lens to compensate for image blur caused by shaking.

[0007] An embodiment of the present invention provides a method for using a camera module, which includes the following steps: A motion sensor is used to sense the motion state of the camera module to output a sensor gain value. A processor is used to calculate the lens movement amount corresponding to the sensor gain value based on a motion model, and a driver receives the lens movement amount and moves the lens to compensate for image blur caused by shaking.

[0008] Based on the above, in one embodiment of the present invention, the camera module and its usage method utilize a motion sensor to sense the motion state of the camera module and output a sensor gain value. Then, a processor calculates the lens movement amount corresponding to the sensor gain value based on the motion model, causing the driver to receive the lens movement amount and move the lens to compensate for image blur caused by shaking. Therefore, the camera module and its usage method of the present invention can provide a corresponding motion model for each camera module, thereby providing better image stabilization effects.

Implementation Method

[0010] FIG1 is a block diagram of a camera module according to an embodiment of the present invention. Referring to FIG1, an embodiment of the present invention provides a camera module 10, which includes a lens 100, an image sensor 200, a driver 300, a motion sensor 400, and a processor 500.

[0011] In detail, the lens 100 may include, for example, a combination of one or more optical lenses having refractive power. The optical lenses may include, for example, various combinations of non-planar lenses such as biconcave lenses, biconvex lenses, concave-convex lenses, convex-concave lenses, plano-convex lenses, and plano-concave lenses. The image sensor 200 may be a photosensor such as a complementary metal-oxide-semiconductor (CMOS), a charge-coupled device (CCD), or a photodiode. However, the present invention is not limited to the elements and types that the lens 100 and image sensor 200 may include.

[0012] The driver 300 is, for example, an actuator, a motor, a stepper motor, or a device using piezoelectric materials. The motion sensor 400 is, for example, a gyroscope, an accelerometer, a gravity sensor, etc. However, the present invention is not limited to the types of driver 300 and motion sensor 400.

[0013] Furthermore, the processor 500 described above may include, for example, a microcontroller unit (MCU), a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a programmable controller, a programmable logic device (PLD), or other similar devices or combinations thereof, and the present invention is not limited thereto. In addition, in one embodiment, the functions of the processor 500 may be implemented as multiple code snippets. These code snippets are stored in a memory and executed by the processor 500. Alternatively, in one embodiment, the functions of the processor 500 may be implemented as one or more circuits. The present invention does not limit the implementation of the functions of the processor 500 in software or hardware.

[0014] In this embodiment, the image sensor 200 is used to sense the light L passing through the lens 100. The driver 300 is connected to the lens 100 and is used to move the lens 100. The motion sensor 400 is used to sense the motion state of the camera module 10 and output a sensor gain value GA. Taking the motion sensor 400 as a gyroscope as an example, the sensor gain value GA can be a gain value corresponding to the 3-axis (X, Y, Z) angular momentum or a gain value of the 2-axis (X, Y) angular momentum.

[0015] In this embodiment, the processor 500 is electrically connected to the image sensor 200, the driver 300, and the motion sensor 400. The processor 500 is used to calculate the lens movement amount LM corresponding to the sensor gain value GA according to the motion model, so that the driver 300 receives the lens movement amount LM and drives the lens 100 to move to compensate for the image blur caused by shaking.

[0016] Figure 2 illustrates a schematic diagram of capturing multiple images using a camera module according to an embodiment of the present invention. Referring to Figures 1 and 2, in this embodiment, the processor 500 receives multiple images IM captured by the image sensor 200 of an object O during camera module 10 shaking, and multiple sensor gain values ​​GA sensed by the motion sensor 400 corresponding to these images IM during capture. The processor 500 then analyzes the images IM and the sensor gain values ​​GA to establish a motion model. The aforementioned object O can be a specific object or image, but the present invention is not limited to the type of object O.

[0017] In this embodiment, the processor 500 uses an artificial neural network to establish the motion model. Specifically, the processor 500 first uses the input layer of the artificial neural network to receive and process the images (IM) and sensor gain values ​​(GA). Then, it uses the convolutional layer of the artificial neural network to perform filter (kernel) matrix convolution processing on the processed images (IM) and sensor gain values ​​(GA) to generate multiple data sets. The aforementioned receiving and processing of the images (IM) using the input layer of the artificial neural network includes, for example, extracting a region of interest (ROI) at the center of the captured images (IM). Next, after performing filter matrix convolution processing, the processor 500 first uses a flattening layer in the artificial neural network to pool and select suitable data from this data, and then uses a connection layer in the artificial neural network to convert the suitable data into weights in the motion model.

[0018] Wherein, the aforementioned motion model may be a polynomial or other mathematical function, and the aforementioned weights may be coefficients in the polynomial or coefficients in other mathematical functions.

[0019] Furthermore, in this embodiment, after establishing the aforementioned motion model, the processor 500 can simultaneously use the motion model to calculate the lens movement amount LM' corresponding to the sensor gain value GA' when the image sensor 200 captures a test image IM' of the object O, and use the lens movement amount LM' to drive the driver 300 to move the lens 100, so as to detect whether the test image IM' captured by the image sensor 200 remains stable.

[0020] Figure 3 is a flowchart of a method for using a camera module according to an embodiment of the present invention. Referring to Figure 3, an embodiment of the present invention provides a method for using a camera module 10, which includes the following steps. Step S10: The motion sensor 400 senses the motion state of the camera module 10 and outputs a sensor gain value GA. Step S20: The processor 500 calculates the lens movement amount LM corresponding to the sensor gain value GA according to the motion model, and the driver 300 receives the lens movement amount LM and drives the lens 100 to move to compensate for image blur caused by shaking.

[0021] Figure 4 is a flowchart of establishing a motion model in a method of using a camera module according to an embodiment of the present invention. Referring to Figure 4, in this embodiment, the above-mentioned step S20 includes the following steps. Step S200: During the shaking process of the camera module 10, the processor 500 receives multiple images IM captured by the image sensor 200 of the object O and multiple sensor gain values ​​GA sensed by the motion sensor 400 corresponding to these images IM during the shooting, and analyzes these images IM and these sensor gain values ​​GA to establish a motion model.

[0022] Figure 5 is a detailed flowchart of the method for establishing a motion model in the use of a camera module according to an embodiment of the present invention. Referring to Figures 4 and 5, in this embodiment, the above-mentioned step S200 includes the following steps: establishing a motion model using an artificial neural network. The step of establishing a motion model using an artificial neural network includes the following steps. Steps S201 and S202: First, the input layer in the artificial neural network receives and processes these images IM and these sensor gain values ​​GA, and then the convolutional layer in the artificial neural network performs filter matrix convolution processing on the processed images IM and these sensor gain values ​​GA to generate multiple data. Steps S203 and S204: After performing filter matrix convolution processing, the flattening layer in the artificial neural network first uses pooling to select matching data from these data, and then the connection layer in the artificial neural network converts the matching data into weights in the motion model in the neural network.

[0023] Please refer to Figures 2 and 4 again. In this embodiment, the above-mentioned step S20 further includes the following steps. Step S210: After establishing the motion model, when the image sensor 200 captures a test image IM' on the object O, the processor 500 simultaneously calculates the lens movement amount LM' corresponding to the sensor gain value GA' when capturing the test image IM' using the motion model, and uses the lens movement amount LM' to drive the driver 300 to move the lens 100, so as to detect whether the test image IM' captured by the image sensor 200 remains stable.

[0024] In summary, in one embodiment of the present invention, the camera module and its usage method utilize a motion sensor to sense the motion state of the camera module and output a sensor gain value. Then, a processor calculates the lens movement amount corresponding to the sensor gain value based on the motion model, causing the driver to receive the lens movement amount and move the lens to compensate for image blur caused by shaking. That is, the camera module and its usage method of the present invention can provide a corresponding motion model for each camera module, thus providing better image stabilization effects.

[0025] Furthermore, the camera module and its usage method utilize a processor to receive multiple images of an object captured by an image sensor and multiple sensor gain values ​​sensed by a motion sensor corresponding to these images during the camera module's shaking process. The processor then analyzes these images and sensor gain values ​​to establish a motion model. The processor utilizes an artificial neural network to establish this motion model. Therefore, compared to traditional methods for establishing motion models using optical image stabilization, the camera module and its usage method of this invention can reduce the time required to establish a motion model, thereby improving production efficiency and increasing the uptime of production equipment. [Simplified Explanation of the Diagram]

[0009] Figure 1 is a block diagram of a camera module according to an embodiment of the present invention. Figure 2 is a schematic diagram illustrating the capture of multiple images using a camera module according to an embodiment of the present invention. Figure 3 is a flowchart of a method of using a camera module according to an embodiment of the present invention. Figure 4 is a flowchart of establishing a motion model in the method of using a camera module according to an embodiment of the present invention. Figure 5 is a detailed flowchart of establishing a motion model in the method of using a camera module according to an embodiment of the present invention.

Claims

1. A camera module, comprising: One shot; An image sensor is used to sense the light passing through the lens; A driver, connected to the lens, is used to move the lens; A motion sensor is used to sense the motion state of the camera module and output a sensor gain value. The system also includes a processor electrically connected to the image sensor, the driver, and the motion sensor. The processor is used to receive multiple images of an object captured by the image sensor and multiple sensor gain values ​​sensed by the motion sensor at the time of capturing the images during camera module shake. The processor analyzes the images and sensor gain values ​​to establish a motion model. The processor is used to calculate the lens movement amount corresponding to the sensor gain value based on the motion model. The driver then receives the lens movement amount and moves the lens to compensate for image blur caused by shake.

2. The camera module as claimed in claim 1, wherein after establishing the motion model, the processor is used to calculate the amount of lens movement corresponding to the sensor gain value when the image sensor captures a test image of the object, using the motion model, and using the amount of lens movement to drive the driver to move the lens, so as to detect whether the test image captured by the image sensor remains stable.

3. The camera module as claimed in claim 1, wherein the processor utilizes an artificial neural network to build the motion model.

4. The camera module as described in claim 3, wherein: The processor first uses the input layer in the artificial neural network to receive and process the images and sensor gain values, and then uses the convolutional layer in the artificial neural network to perform filter matrix convolution processing on the processed images and sensor gain values ​​to generate multiple data. After performing filter matrix convolution processing, the processor first uses the flattening layer in the artificial neural network to pool and select matching data, and then uses the connection layer in the artificial neural network to convert the matching data into weights in the motion model.

5. A method of using a camera module, comprising: A motion sensor is used to sense the motion state of the camera module, and a sensor gain value is output. The system utilizes a processor to calculate the lens movement amount corresponding to the sensor gain value based on a motion model. A driver receives the lens movement amount and drives the lens to move. During the camera module shake, the processor receives multiple images of an object captured by an image sensor and multiple sensor gain values ​​sensed by the motion sensor at the time of capturing these images. It then analyzes these images and sensor gain values ​​to establish the motion model and compensate for image blur caused by shaking.

6. The method of use as described in claim 5, wherein the step of using the processor to calculate the lens movement amount corresponding to the sensor gain value based on the motion model, and causing the driver to receive the lens movement amount and drive the lens to move, in order to compensate for image blur caused by shaking, further includes the following steps: After establishing the motion model, when the image sensor takes a test image of the object again, the processor simultaneously uses the motion model to calculate the lens movement amount corresponding to the sensor gain value when taking the test image, and uses the lens movement amount to cause the driver to drive the lens to move, so as to detect whether the test image taken by the image sensor remains stable.

7. The method of use as described in claim 5, wherein the step of using the processor to receive, during the shaking of the camera module, the images captured by the image sensor of the object and the sensor gain values ​​sensed by the motion sensor corresponding to the images at the time of capture, and analyzing the images and the sensor gain values ​​to establish the motion model includes the following steps: establishing the motion model using an artificial neural network.

8. The method of use as described in claim 7, wherein the step of establishing the motion model using the artificial neural network includes the following steps: first, receiving and processing the images and sensor gain values ​​using the input layer in the artificial neural network; then, performing filter matrix convolution processing on the processed images and sensor gain values ​​using the convolutional layer in the artificial neural network to generate multiple data; and after performing filter matrix convolution processing, first using the flattening layer in the artificial neural network to pool and select matching data from the data; and then using the connection layer in the artificial neural network to transform the matching data into weights in the motion model within the neural network.

Images