Artificial Intelligence-Based Image Stabilization Method and Its Camera Module

Abstract

This invention relates to an artificial intelligence-based image stabilization method and its camera module. The method includes: acquiring vibration detection data about an image from two or more sensors; outputting stabilization data using an artificial neural network model trained to output stabilization data for compensating for image jitter based on the vibration detection data; and using the stabilization data to compensate for image jitter.

Description

Technical Field

[0001] This invention relates to an image stabilization method and its camera module, and more particularly, to an artificial intelligence-based image stabilization method and its camera module. Background Technology

[0002] Image stabilizers (IS) are applied to cameras to compensate for camera shake, thereby solving the problem of blurry images caused by hand tremors.

[0003] Image stabilization techniques include optical image stabilizers (OIS) and stabilization techniques that utilize image sensors.

[0004] The camera module employing OIS technology detects vibrations using a navigation sensor such as a gyroscope and generates a compensation signal using a Hall sensor that detects changes in the separation between the coil and the magnet. Based on the compensation signal, image jitter is stabilized.

[0005] Specifically, the camera module can compensate for vibration by using the following methods: a lens movement method that fixes the image sensor and shifts the lens to compensate for vibration, and a camera rotation method that tilts both the image sensor and the lens to compensate for vibration. Summary of the Invention

[0006] [Technical Solution]

[0007] Typically, an OIS system can be defined as either an OIS system with both a lens and an image sensor fixed to it, or an OIS system with only an image sensor fixed to it.

[0008] Specifically, the camera tilt architecture type OIS system, in which the lens and image sensor are fixed together, may include photoelectric sensors and actuators, and can stabilize image shake by tilting the structure on which the lens and image sensor are simultaneously fixed.

[0009] Specifically, the barrel-shift architecture type OIS system with fixed sensors may include actuators, Hall sensors and magnets, and image jitter can be stabilized by using the magnets to shift the lens.

[0010] The OIS system uses a simple feedback algorithm to generate a stabilization signal based on the detection signal from a sensor that detects image jitter.

[0011] The inventors of this disclosure have recognized that the performance of the algorithm may be limited if such a simple feedback algorithm is used to perform stabilization.

[0012] Therefore, one object of this disclosure is to provide an AI-based image stabilization method and its camera module, which can maximize the processing speed of operations used to compensate for image jitter and reduce power consumption.

[0013] Therefore, another objective of this disclosure is to provide an AI-based image stabilization method and processor thereof, which can maximize the processing speed of operations used to compensate for image jitter and reduce power consumption.

[0014] Therefore, another object of this disclosure is to provide an artificial intelligence-based image stabilization method and processor thereof, which can compensate for multiple vibration modes in various usage environments through artificial intelligence reinforcement learning.

[0015] Therefore, another object of this disclosure is to provide an artificial intelligence-based image stabilization method and apparatus that can maximize the processing speed of operations used to compensate for image jitter and reduce power consumption through artificial intelligence training.

[0016] Therefore, another object of this disclosure is to provide an artificial intelligence-based image stabilization method and apparatus that can maximize the processing speed of operations used to compensate for image jitter and reduce power consumption by utilizing artificial intelligence technology to train specific patterns.

[0017] However, this disclosure is not limited thereto, and other issues will become clear to those skilled in the art from the following description.

[0018] An artificial intelligence-based image stabilization method can be provided. This method may include: acquiring vibration detection data about an image from two or more sensors; outputting stabilization data using an artificial neural network model trained to output stabilization data for compensating for image jitter based on the vibration detection data; and using the stabilization data to compensate for image jitter.

[0019] Artificial neural network models can be based on reinforcement learning.

[0020] Vibration detection data can include signals used to detect changes in the position of the camera module and lens.

[0021] Two or more sensors may include gyroscope sensors and Hall sensors.

[0022] Artificial neural network models can be trained models based on training data used for learning, so that the error value caused by image jitter converges to a predetermined value.

[0023] Artificial neural network models can receive vibration detection data as input and output control signals to control the movement of the lens included in the camera module to compensate for image shake.

[0024] Artificial neural network models can receive vibration detection data as input and output control signals to control the movement of image sensors included in the camera module to compensate for image jitter.

[0025] Artificial neural network models can receive vibration detection data as input to simultaneously perform training operations for inferring stable data and inference operations for stable data.

[0026] An artificial neural network model may include an input node that takes vibration detection data as input, a hidden layer that performs artificial intelligence (AI) operations on the input node, and an output node that outputs stabilized data.

[0027] A camera module may be provided. The camera module may include: a lens; an image sensor configured to output an image captured through the lens; two or more sensors configured to output vibration detection data; a controller configured to output stabilized data based on the vibration detection data; and a stabilization unit configured to use the stabilized data to compensate for image jitter, wherein the controller is configured to output the stabilized data using an artificial neural network model trained to output stabilized data based on the vibration detection data.

[0028] Two or more sensors may include at least two of gyroscope sensors, Hall effect sensors, and photoelectric sensors.

[0029] Vibration detection data can include signals detected by rotational movement along the x and y axes via a gyroscope sensor and by rotational movement along the x and y axes via a Hall sensor.

[0030] An artificial neural network model can be trained based on vibration detection data, so that the error value caused by image jitter converges to a predetermined value.

[0031] The error value can be based on the difference between the x-axis movement of the gyroscope sensor and the x-axis movement of the Hall sensor, and the difference between the y-axis movement of the gyroscope sensor and the y-axis movement of the Hall sensor.

[0032] The trained model may include a first model and a second model. The first model is trained to infer stable data based on vibration detection data, where the error value converges to a predetermined value. The second model is trained to evaluate the results of the stable data.

[0033] The controller can be configured to collect error values ​​during training and use the collected error values ​​to update the artificial neural network model.

[0034] The camera module may include a temperature sensor for sensing temperature.

[0035] The controller can be configured to output stable data using an artificial neural network model based on vibration detection data and temperature data acquired through a temperature sensor.

[0036] Vibration detection data may include signals detected by rotational movement along the x and y axes of a gyroscope sensor and rotational movement along the x, y, and z axes of a Hall sensor.

[0037] The controller can be configured to obtain defocus data based on the frequency components of the image, and output stable data using an artificial neural network model based on the vibration detection data and the defocus data.

[0038] Artificial neural network models can use the modulation transfer function (MTF) data of images as training data for training.

[0039] [Beneficial Effects]

[0040] According to this disclosure, by using an artificial neural network model to infer the compensation signal for compensating image jitter, the stabilization speed can be maximized and power consumption reduced.

[0041] Furthermore, according to this disclosure, an image stabilization method with improved accuracy can be provided by using an artificial neural network model to compensate for image jitter by considering various variables (such as temperature, defocus, image MTF, etc.) and sensor detection signals.

[0042] Furthermore, according to this disclosure, an image stabilization method with improved efficiency can be provided by performing focus adjustment and image stabilization.

[0043] Furthermore, according to this disclosure, image stabilization can be rapidly provided by a processor that processes a trained artificial neural network model.

[0044] Furthermore, according to this disclosure, image stabilization can be rapidly provided while reducing power consumption by using a dedicated inference processor that processes trained artificial neural network models.

[0045] Furthermore, according to this disclosure, image stabilization can be performed using multiple vibration modes of various usage environments trained by artificial intelligence reinforcement learning. Attached Figure Description

[0046] Figure 1 This is a schematic concept diagram illustrating a device including a camera module according to an example of this disclosure.

[0047] Figure 2 This is a schematic concept diagram illustrating a camera module according to an example of this disclosure.

[0048] Figure 3This is a schematic conceptual diagram illustrating an example of an artificial neural network model according to the present disclosure.

[0049] Figure 4 This is a schematic conceptual diagram illustrating a training method for an example artificial neural network model according to the present disclosure.

[0050] Figure 5 This is a schematic conceptual diagram illustrating a specific training operation of an artificial neural network model according to an example of this disclosure.

[0051] Figure 6 This is a schematic concept diagram illustrating an example of a method for compensating for image jitter using a trained artificial neural network model according to this disclosure.

[0052] Figure 7 This is a schematic conceptual diagram illustrating another example of an artificial neural network model according to this disclosure.

[0053] Figure 8 This is a schematic concept diagram illustrating a training method for an artificial neural network model according to another example of this disclosure.

[0054] Figure 9 This is a schematic concept diagram illustrating another example of a method for compensating for image jitter using a trained artificial neural network model according to this disclosure.

[0055] Figure 10 This is a schematic concept diagram illustrating another example of a camera module according to this disclosure.

[0056] Figure 11 This is a schematic conceptual diagram illustrating another example of an artificial neural network model according to this disclosure.

[0057] Figure 12 This is a schematic concept diagram illustrating a training method for an artificial neural network model according to another example of this disclosure.

[0058] Figure 13 This is a schematic conceptual diagram illustrating a specific training operation of an artificial neural network model according to another example of this disclosure.

[0059] Figure 14 This is a schematic concept diagram illustrating another example of a method for compensating for image jitter using a trained artificial neural network model according to this disclosure.

[0060] Figure 15 This is a flowchart illustrating an image stabilization method for a camera module in one example of this disclosure.

[0061] Figure 16 This is a schematic conceptual diagram illustrating a neural processing unit according to the present disclosure. Detailed Implementation

[0062] The specific structural descriptions or step-by-step descriptions of the examples of the concept disclosed in this disclosure or this application are merely illustrative for the purpose of explaining the examples of the concept disclosed.

[0063] Examples of the concepts based on this disclosure may be implemented in various forms and should not be construed as limited to the examples described in this disclosure or this application.

[0064] Because examples of the concept according to this disclosure can have various modifications and can take various forms, specific examples will be shown in the accompanying drawings and described in detail in this disclosure or application. However, this is not intended to limit the examples of the concept according to this disclosure to a particular form of disclosure, and should be understood to include all modifications, equivalents, and substitutions falling within the spirit and scope of this disclosure.

[0065] Terms such as first and / or second may be used to describe various elements, but these elements should not be limited by these terms.

[0066] The terms used above are merely for the purpose of distinguishing one element from another, and without departing from the scope of the conception according to this disclosure, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0067] When a component is described as being "connected" or "in contact" with another component, it should be understood that the component can be directly connected or in contact with that other component, but other components may also be present in between. On the other hand, when it is said that a component is "directly connected" or "directly in contact" with another component, it should be understood that there are no other components in between.

[0068] Other expressions describing the relationship between elements (e.g., "between" and "immediately between" or "adjacent to" and "directly adjacent to") should be interpreted similarly.

[0069] In this document, expressions such as “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B”, “at least one of A and B”, or “at least one of A or B” can refer to (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B in all of the following cases.

[0070] As used herein, expressions such as “first,” “second,” or “first or second” can modify various elements regardless of order and / or importance. Furthermore, such expressions are used only to distinguish one component from others, not to limit those components.

[0071] For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the claims described in this document, the first component may be named the second component, and similarly, the second component may be renamed the first component.

[0072] The terminology used in this disclosure is only for describing specific examples and is not intended to limit the scope of other examples.

[0073] Singular expressions may include plural expressions unless the context clearly specifies otherwise. The terms used herein (including technical or scientific terms) may have the same meaning as commonly understood by one of ordinary skill in the art as described herein.

[0074] In this document, terms defined in general dictionaries may be interpreted as having the same or similar meaning as in the context of the relevant art. Furthermore, unless explicitly defined in this document, these terms should not be interpreted in an ideal or overly formal sense. In some cases, even terms defined in this document should not be interpreted as excluding examples from this document.

[0075] The terminology used herein is for the purpose of describing specific examples only and is not intended to limit this disclosure.

[0076] Singular expressions include plural expressions unless the context clearly specifies otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the described features, quantities, steps, actions, components, parts, or combinations thereof. Therefore, it should be understood that the presence or addition of one or more other features or quantities, steps, actions, components, parts, or combinations thereof is not excluded.

[0077] Unless otherwise defined, all terms used herein (including technical or scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0078] Terms (e.g., terms defined in commonly used dictionaries) should be interpreted as having a meaning consistent with the meaning in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning, unless expressly defined in this disclosure.

[0079] As those skilled in the art will fully understand, each feature of the various examples of this disclosure can be combined, either partially or entirely, or with each other, and can be technically interlocked and driven in various ways. Furthermore, each example can be implemented independently of each other, or can be implemented together in a related relationship.

[0080] In describing examples, descriptions of technical content that is well-known in the technical field to which this disclosure pertains and is not directly related to this disclosure may be omitted. This is to more clearly convey the key points of this disclosure without obscuring them by omitting unnecessary descriptions.

[0081] The present disclosure will be described in detail below with reference to the accompanying drawings, illustrating preferred embodiments thereof.

[0082] Figure 1 This is a schematic concept diagram illustrating an apparatus including a camera module according to an example of this disclosure.

[0083] Reference Figure 1 Device A may include camera module 1000.

[0084] Device A may include a camera module 1000, a processor 140, and a first memory 3000. According to various examples, device A may be configured to also include a second memory 4000 separate from the first memory 3000. That is, device A may be configured to selectively include or exclude the second memory 4000.

[0085] The camera module 1000 is a unit for capturing images and may include an OIS system for detecting and compensating for image shake. For example, the OIS system may include at least two or more sensors for detecting image shake and compensation units for compensating for image shake.

[0086] Specifically, the OIS system may include a processor 140 as a processing unit. The processor 140 can be implemented with various modifications.

[0087] For example, processor 140 may be a computing device such as a central processing unit (CPU) or an application processor (AP).

[0088] For example, processor 140 may be a computing device such as a microprocessor unit (MPU).

[0089] For example, processor 140 may be a computing device such as a microcontroller unit (MCU).

[0090] For example, processor 140 may be a computing device such as a neural processing unit (NPU).

[0091] For example, processor 140 may be a computing device such as an image signal processor (ISP).

[0092] For example, processor 140 may be a system-on-a-chip (SoC) that integrates various computing devices such as CPU and NPU.

[0093] For example, processor 140 may be implemented as an integrated chip (IC) incorporating the aforementioned computing device. However, the examples in this disclosure are not limited to processor 140.

[0094] The processor 140 is operatively communicable with the camera module 1000 and the first memory 3000. Furthermore, the processor 140 is operatively communicable with the second memory 4000.

[0095] Processor 140 may correspond to a computing device such as a CPU or an application processor (AP). Furthermore, processor 140 may be implemented as an integrated chip, such as a System-on-a-Chip (SoC) integrating various computing devices (e.g., CPU, GPU, and NPU). In various examples, processor 140 may operate as a computing device within an OIS system.

[0096] The processor 140 can be implemented as an integrated chip, which integrates various computing devices such as an ISP and a CPU. These computing devices receive the Bayer pattern as the output image of the camera module and output data converted into RGB or YUV images.

[0097] In other words, processor 140 according to the examples of this disclosure may refer to a dedicated processor and / or an integrated heterogeneous processor.

[0098] The camera module 1000 can be implemented as an integrated chip integrated with the processor 140.

[0099] The first memory 3000 is a memory mounted on a semiconductor die and can be a memory used for caching or storing data processed in an on-chip area. The first memory 3000 may include one of the following: ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, and high-bandwidth memory (HBM). The first memory 3000 may include at least one memory cell. The first memory 3000 can be configured as homogeneous or heterogeneous memory cells.

[0100] The first memory 3000 can be configured as internal memory or on-chip memory.

[0101] The second memory 4000 may include one of the following: ROM, SRAM, DRAM, resistive RAM, magnetoresistive RAM, phase-change RAM, ferroelectric RAM, flash memory, and HBM. The second memory 4000 may include at least one memory cell. The second memory 4000 may be configured as homogeneous or heterogeneous memory cells.

[0102] The second memory 4000 can be configured as main memory or off-chip memory.

[0103] In the following text, reference will be made to Figure 2 Detailed description of camera module 1000.

[0104] Figure 2 This is a schematic concept diagram illustrating a camera module according to an example of this disclosure.

[0105] Reference Figure 2 The camera module 1000 may include a lens 100, an image sensor 110, a first sensor 120, a VCM driver (e.g., a voice coil motor driver) 130, a processor 140, and a VCM actuator 200.

[0106] However, as Figure 1 As shown, the processor 140 may be included in the camera module 1000 or configured to be located outside the camera module 1000. That is, Figure 2 The processor 140 can be excluded from the camera module 1000, and the processor 140 can be configured to communicate with the camera module 1000 from the outside.

[0107] Lens 100 is configured to form an optical image when collecting or distributing light from an object. Image sensor 110 is a component that converts light received through the lens into a digital signal and may include a charge-coupled device (CCD) sensor, a complementary metal-oxide-semiconductor (CMOS) sensor, etc.

[0108] The first sensor 120 is a sensor used to detect jitter in the camera module 1000 or device A. The first sensor 120 may be an angular velocity sensor that measures the rotational speed of an object, such as a gyroscope sensor. In other words, the motion signal obtained through the first sensor 120 may include information about the x-axis and y-axis movement of the first sensor 120.

[0109] VCM driver 130 can be configured to send control signals to VCM actuator 200 to properly adjust the position of lens 100.

[0110] VCM actuator 200 may include lens holder 210, magnet 220, second sensor 230, coil holder 240 and coil 250.

[0111] Specifically, the lens holder 210 can be configured to mount the lens 100, and the magnet 220 can be configured to use magnetic force to adjust the position of the lens 100.

[0112] The second sensor 230 is a sensor used to detect jitter in the VCM actuator 200, and may be a Hall sensor used to detect changes in the spacing between the OIS coil 250 and the magnet 220.

[0113] The coil holder 240 is configured such that the OIS coil 250 is mounted, and the OIS coil 250 is positioned opposite the magnet 220 to control the magnetic force of the magnet 220.

[0114] The processor 140 can be configured to take into account the lens 100, image sensor 110, first sensor 120, VCM driver (e.g., voice coil motor driver) 130 and VCM actuator 200 to handle the overall operation of the camera module 1000.

[0115] Specifically, the processor 140 can be configured to perform operations for compensating for image jitter via an artificial neural network.

[0116] Specifically, processor 140 can acquire motion signals (i.e., tremor detection signals) from at least two or more sensors (i.e., first sensor 120 and second sensor 230) included in camera module 1000. Processor 140 can use an artificial neural network model to generate inferred compensation signals, which are trained to infer compensation signals for compensating image jitter based on each acquired motion signal. Image jitter can be compensated based on the compensation signals inferred by processor 140.

[0117] Here, the motion signal may include the x-axis and y-axis values ​​of the Hall sensor and the x-axis and y-axis values ​​of the gyroscope sensor. Furthermore, the processor 140 may also include internal memory (not shown) and store various data for compensating for image jitter.

[0118] According to various examples of this disclosure, the camera module 1000 can be implemented as an edge device including an NPU and an NPU memory.

[0119] In the following text, reference will be made to Figures 3 to 6 The method of using an artificial neural network model by processor 140 to compensate for image jitter is described in detail.

[0120] Figure 3 This is a schematic conceptual diagram illustrating an example of an artificial neural network model according to the present disclosure.

[0121] Reference Figure 3The artificial neural network model 300 can receive input data as vibration detection data 310, which includes the movement signal of the camera module 1000 obtained by the first sensor 120 and the movement signal of the lens 100 obtained by the second sensor 230.

[0122] In detail, since the image sensor 110 is fixed to the camera module 1000, the vibration of the image sensor 110 can be substantially detected when the vibration of the camera module 1000 is sensed by the first sensor 120. Therefore, the vibration of the image sensor 110 can be substantially sensed by detecting the vibration of the camera module 1000.

[0123] Specifically, since the lens 100 is fixed to the lens mount 210, when the second sensor 230 senses vibration of the lens mount 210, the vibration of the lens 100 can be substantially detected. Therefore, the vibration of the lens 100 can be substantially sensed by detecting the shaking of the lens mount 210.

[0124] Here, vibration detection data may include, for example, the x-axis rotation angle and y-axis rotation angle for the camera module 1000, and the x-axis rotation angle and y-axis rotation angle for the VCM actuator 200.

[0125] In this sense, the vibration detection data of the VCM actuator 200 actually represents the vibration detection data of the lens 100.

[0126] Furthermore, despite Figure 2 The image shows that lens 100 is included in camera module 1000, but this only considers the function of lens 100 in camera module 1000, and it should be understood that lens 100 is fixed to lens holder 210.

[0127] In the example disclosed herein, the artificial neural network model 300 can receive vibration detection data 310, infer stabilization data for compensating for image jitter, and output the inferred stabilization data 320.

[0128] The VCM driver 130 can receive the inferred stabilization data 320 and output a control signal for stabilizing the shake of the lens 100. For example, the artificial neural network model 300 may also include a conversion unit for converting the stabilization data 320 into a control signal, and the stabilization data 320 can be converted into a control signal by the conversion unit and output.

[0129] According to various examples of this disclosure, the stabilized data 320 can be output as a control signal for controlling the VCM driver 130 to compensate for the position of the image sensor 110 based on the inferred stabilized data 320. In examples of this disclosure, the stabilized data 320 can be output as a control signal for compensating for the positions of the lens 100 and the image sensor 110.

[0130] In the examples disclosed herein, the artificial neural network model 300 may be a model to be trained through reinforcement learning.

[0131] For example, the artificial neural network model 300 can be a model such as RNN, CNN, DNN, MDP, DP, MC, TD (SARSA), QL, DQN, PG, AC, A2C, A3C, etc.

[0132] However, this disclosure is not limited to this, and can be various artificial neural network models trained to infer position compensation data by taking vibration detection data as input.

[0133] The artificial neural network model 300 according to various examples of this disclosure may be a model selected from a plurality of artificial neural network models. The plurality of artificial neural network models may represent actor models, which will be described later.

[0134] In the following text, reference will be made to Figure 4 Describe in detail the methods for training artificial neural network models.

[0135] Figure 4 This is a schematic conceptual diagram illustrating a training method for an artificial neural network model according to an example of this disclosure. In the presented example, it is assumed that the artificial neural network model 400 is a reinforcement learning-based model, such as an actor-critic model. Specifically, the operations described below can be performed by processor 140.

[0136] Reference Figure 4 It is possible to train an artificial neural network model 400 to select the optimal action in a given environment or state.

[0137] Here, the artificial neural network model 400 may be based on an actor-evaluator algorithm. The artificial neural network model 400 may include a first model (e.g., an actor model) that determines an action when given a state S in a given environment, and a second model (e.g., an evaluator model) that evaluates the value of the state S.

[0138] A given environment may include the current state St, the next possible state St+1, the action a that can be taken under any condition, the reward r for an action taken in a certain state, and a policy that determines the probability of taking a specific action in a given state.

[0139] Specifically, the first motion signals θYaw and θPitch of the camera module 1000 and the second motion signals θHSx and θHSy of the lens 100 are obtained by the sensor 410. Here, the first motion signals θYaw and θPitch can be obtained from the first sensor 120, and the second motion signals θHSx and θHSy can be obtained from the second sensor 230.

[0140] For example, the first motion signals θYaw and θPitch may include the x-axis rotation angle θPitch and the y-axis rotation angle θYaw for the camera module 1000.

[0141] For example, the second motion signals θHSx and θHSy may include the x-axis rotation angle θHSx and the y-axis rotation angle θHSy relative to the lens 100.

[0142] Here, the first motion signals θYaw and θPitch are signals used to substantially detect the jitter of the image sensor 110. The second motion signals θHSx and θHSy are signals used to substantially sense the jitter of the lens 100.

[0143] The processor 140 can determine environmental data, including the current state St, the next state St+1, the action at, the reward rt+1, and the policy, by using the obtained vibration detection data (θYaw, θPitch, θHSx, θHSy), and store it in the batch memory 420.

[0144] Here, the batch memory 420 may be a memory used to store environmental data for training the artificial neural network model 400 into batch data.

[0145] The data for the current state St represents the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy), and the data for the next state St+1 can refer to the vibration detection data obtained after taking an action for the current state. The data for the action at represents the stabilized data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) of the lens 100 that can be inferred based on the vibration detection data (θYaw, θPitch, θHSx, θHSy), and the data for the reward rt+1 represents the error value based on the difference between the x-axis rotation angle θPitch of the camera module 1000 and the x-axis rotation angle θHSx of the lens 100, and the difference between the y-axis rotation angle θYaw of the camera module 1000 and the y-axis rotation angle θHSy of the lens 100.

[0146] Subsequently, data about the current state St and the next state St+1 from the batch memory 420 are input as training data into each of the first model (e.g., actor model) and the second model (e.g., evaluator model) of the artificial neural network model.

[0147] Processor 140 trains artificial neural network model 400 to determine a policy (i.e., action) that maximizes reward rt+1. Here, the policy (i.e., action) that maximizes reward rt+1 can refer to stable data where the error value converges to a predetermined value. For example, the predetermined value can be zero. In this disclosure, when the error value converges to zero, it can be determined that there is no oscillation.

[0148] Specifically, the first model (e.g., the actor model) infers stable data based on the input vibration detection data (θYaw, θPitch, θHSx, θHSy). The first model (e.g., the actor model) determines the probability that the error value converges to a predetermined value when the position of lens 100 is compensated based on the inferred stable data.

[0149] The second model (e.g., an evaluator model) evaluates the value of the stabilized data inferred from the vibration detection data ((θYaw, θPitch, θHSx, θHSy)) input to the first model (e.g., an actor model). The evaluation result of this value is sent to the first model (e.g., the actor model) so that the first model (e.g., the actor model) can use it to determine subsequent actions.

[0150] During the training phase, vibration detection data can be provided in various forms.

[0151] For example, a camera module 1000 used for reinforcement learning can be fixed to a specific fixture and programmed to vibrate in a specific pattern.

[0152] For example, at least one user can hold the camera module 1000 used for reinforcement learning and make it vibrate for a specific period of time.

[0153] For example, for reinforcement learning, virtual vibration detection data can be provided.

[0154] Here, the specific mode can be sitting, walking, running, boating, caring, motorcycleing, etc.

[0155] During the reinforcement learning period, at least one specific pattern can be applied.

[0156] During reinforcement learning, at least one specific pattern can be applied sequentially or randomly.

[0157] The stabilization data inferred from the first model (e.g., the actor model) is sent to the VCM driver 130, and the VCM driver 130 sends a control signal to the VCM actuator 200 to compensate for the vibration of the VCM actuator 200.

[0158] In this disclosure, the stabilized data can be converted into voltages (vx, vy) that can be input to the VCM driver 130 to enable the VCM driver 130 to control the voltages of the VCM actuator 200, and then transmitted.

[0159] The VCM driver 130, which receives these voltages (vx, vy), sends control signals to the VCM actuator 200 for controlling the VCM actuator 200 on the x and y axes. Here, the control signals may include a current cx for controlling the VCM actuator 200 on the x-axis and a current cy for controlling the VCM actuator 200 on the y-axis. By compensating for the position of the lens 100 by the VCM actuator 200 receiving the currents cx and cy, image jitter can be compensated.

[0160] In the next step, vibration detection data can be acquired via sensor 410, and the above training operations can be repeated. These training operations can be performed until a maximum number of success criteria or epochs are reached, but are not limited to this. For example, a success criterion could include a criterion in which the error value determined based on the inferred stabilization data converges to zero.

[0161] According to various examples of this disclosure, processor 140 can update artificial neural network model 400 by collecting error values ​​obtained during training operations and using the collected error values ​​to further train artificial neural network model 400.

[0162] According to various examples of this disclosure, processor 140 can acquire modulation transfer function (MTF) data of an image and train a second model (e.g., an evaluator model) based on the acquired MTF data.

[0163] The first model (e.g., the actor model) can be customized according to various examples of this disclosure through reinforcement learning.

[0164] In the following text, reference will be made to Figure 5 Describe in detail the specific learning operations of the artificial neural network model 400.

[0165] Figure 5 This is a schematic conceptual diagram illustrating a specific training operation of an artificial neural network model according to an example of this disclosure.

[0166] Reference Figure 5Each of the first model 500 and the second model 510 receives the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy)st and the vibration detection data st+1 obtained after taking an action on the current state as training data.

[0167] The first model 500 receives vibration detection data (st, st+1) as input and outputs actions and policies. Here, actions represent inferred stable data, and policies include the probability πθ(at|St) of taking action at in the current state St, and the probability πθold(at|St) of taking action at in the current state St of the first model 500 before being updated to a batch.

[0168] The second model 510 receives vibration detection data (st, st+1) as input and outputs the values ​​of the vibration detection data (st, st+1) Vυ(St) and Vυ(St+1), as well as the expected benefits (e.g., advantages) for the output values ​​Vυ(St) and Vυ(St+1). Here, the value Vυ(St) represents the value of the vibration detection data St in the current state, and the value Vυ(St+1) represents the value of the vibration detection data St+1 in the next state.

[0169] The first model 500 can be based on the outputs of πθ(at|St), πθold(at|St), and the second model 510. Determine the loss L CLIP (θ). Here, θ can represent a parameter vector indicating all weights and biases of the first model 500. Loss L CLIP (θ) can be represented by the following <Equation 1>.

[0170] <Formula 1>

[0171]

[0172] Here, f t (θ) can represent and It can be represented as λ{(r+γV)} u (s t ))-V u (s t+1 )}. Then, r represents the reward, and γ represents the discount factor.

[0173] The first model 500 can use the aforementioned loss L CLIP (θ) is used to update the parameter vector θ of the first model 500.

[0174] The second model 510 can use the values ​​Vυ(St) and Vυ(St+1) to determine the loss L. V (u). Here, υ can represent the parameter vector of the second model 510. Loss L V (u) can be represented by the following <Equation 2>.

[0175] <Formula 2>

[0176] L V (u)=V u (s t+1 )-(r+γV u (s t ))

[0177] The second model 510 can be achieved by using the aforementioned loss L. V (u) is used to update the parameter vector υ of the second model 510.

[0178] The stabilized data, inferred from the action transmitted from the first model 500, is transformed by the conversion unit 520 into voltages vx and vy that can be input from the VCM driver 130. For example, the stabilized data can be converted into analog voltages or digital signals for controlling the VCM driver 130. When the voltage output from the conversion unit 520 switches, overshoot or ringing may occur. To address this issue, the conversion unit 520 can use an artificial neural network model trained to cause voltage tilting (e.g., downscaling or upscaling) to compensate for overshoot or ringing and output the correct model.

[0179] VCM driver 130 sends control signals cx and cy, causing VCM actuator 200 to compensate for the lens position. Afterward, it can perform actions such as... Figure 4 The training operation is shown.

[0180] In the following text, reference will be made to Figure 6 Detailed description of usage is as follows (see reference) Figure 4 and Figure 5 The method described above uses a trained artificial neural network model to compensate for image jitter.

[0181] In a trained artificial neural network model, once the weight data has been trained, the weight data is no longer updated.

[0182] More specifically, processor 140 can be multiple processors, so that training and inference can be performed separately using different processors.

[0183] For example, the processor 140 performing training according to the examples of this disclosure can be a GPU, and the processor 140 performing inference can be an NPU. That is, the processor used during machine learning and the processor used during inference after machine learning is complete can be different from each other.

[0184] For example, the processor 140 implemented as an NPU could be a dedicated processor for inference operations, which has relatively fast processing speed and reduced power consumption, but does not support machine learning algorithms.

[0185] The processor 140, implemented as an NPU, is not used for training but is instead implemented as a high-speed, low-power processor 140, and has the advantage of being implementable in various edge devices. Furthermore, by utilizing the processor 140 capable of machine learning to perform additional machine learning, and then providing the updated weight data to the processor 140 implemented as an NPU, the problem of unsupported machine learning algorithms can be addressed.

[0186] More specifically, the first model (e.g., the actor model) according to the example of this disclosure can be configured to be processed by a processor 140 implemented as an NPU built into the camera module 1000.

[0187] More specifically, the second model (e.g., the evaluator model) according to the examples of this disclosure can be configured to be processed by a training-only processor 140. Here, the training-only processor 140 can be a separate processor located outside the camera module 1000. However, the examples of this disclosure are not limited thereto.

[0188] Figure 6 This is a schematic conceptual diagram illustrating a method for compensating for image jitter using a trained artificial neural network model, according to an example of this disclosure. The operations described below can be performed by a processor 140 implemented as an NPU. However, the processor 140 of the example according to this disclosure is not limited to an NPU.

[0189] exist Figure 6 In the example, the second model (e.g., the evaluator model) can be excluded. Therefore, inference can be performed using only the first model (e.g., the actor model). In this case, the training of the weights of the first model (e.g., the actor model) can be completed. Furthermore, when only the first model (e.g., the actor model) is used, the training step can be excluded, and the second model (e.g., the evaluator model) can also be excluded, thereby improving power consumption, computational cost, and processing speed. Additionally, the high-speed and low-power camera module 1000 can be implemented by applying a low-power processor 140 implemented as an NPU.

[0190] Reference Figure 6The sensor 600 obtains vibration detection data St corresponding to the environment and stores the obtained vibration detection data St in the batch memory 610.

[0191] The vibration detection data St stored in the batch memory 610 is used as input data to the stabilization signal generator 620. For example, the stabilization signal generator 620 may include a first model (e.g., an actor model) with four input nodes, multiple hidden layers, and four output nodes. However, the structure of the first model (e.g., the actor model) is not limited to this.

[0192] When input from each of the four input nodes (θYaw, θPitch, θHSx, θHSy), the first model (e.g., an actor model) infers stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) that causes the error value r to converge to a predetermined value. Therefore, the first model (e.g., an actor model) can output the inferred stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis).

[0193] Here, the error value r can be the sum of the difference (errx) between the x-axis rotation angle θPitch of the camera module 1000 and the x-axis rotation angle θHSx of the lens 100, and the difference (erry) between the y-axis rotation angle θYaw of the camera module 1000 and the y-axis rotation angle θHSy of the lens 100. For example, the error value can be derived from, for example, -(|errx / r). x +err y The equation is expressed as |).

[0194] The Transpose unit can transform the stabilized data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) inferred from the first model (e.g., the actor model) into voltages that can be used as inputs to the VCM driver 130. The Transpose unit uses a conversion formula or lookup table to convert the stabilized data into voltages vx and vy, and outputs the converted voltages vx and vy as control signals. For example, these voltages (vx, vy) are in the range of 0V to 5V, and the corresponding range can correspond to the input standard of the VCM driver 130.

[0195] For example, when the input is +Xaxis, the voltage of the stabilization voltage vx can increase by one order (e.g., 0.001V). For example, when the input is -Xaxis, the voltage of the stabilization voltage vx can decrease by one order (e.g., 0.001V).

[0196] For example, when the input is +Yaxis, the stabilized voltage vy can increase by one order (e.g., 0.001V). Conversely, when the input is -Yaxis, the stabilized voltage vy can decrease by one order (e.g., 0.001V). In other words, the stabilized voltage can vary in units of a preset voltage step. The stabilization rate can be determined based on the unit of the voltage step and the interval between voltage step updates.

[0197] However, the examples in this disclosure are not limited to voltage levels and time intervals.

[0198] When the input voltages vx and vy are set, the VCM driver 130 can output a control signal to compensate for vibrations in the VCM actuator 200 based on stabilization data. This control signal can be used to control the current (cx, cy) of the VCM actuator 200. For example, the current (cx, cy) can be from -100mA to 100mA, and the corresponding range can correspond to the input standard of the VCM actuator 200.

[0199] When an input current (cx, cy) is applied, the VCM actuator 200 operates to compensate for the position of the lens 100 based on the input current, thereby performing image stabilization.

[0200] In the following text, reference will be made to Figures 7 to 9 The device A also includes a sensor for measuring temperature and a method for compensating for image jitter by further taking the temperature data into account.

[0201] Figure 7 This is a schematic conceptual diagram illustrating another example of an artificial neural network model according to this disclosure.

[0202] Reference Figure 7 The artificial neural network model 700 includes motion signals from the camera module 1000 obtained through the first sensor 120 and motion signals from the lens 100 obtained through the second sensor 230. Vibration detection data 710 and temperature data 720 obtained through a temperature sensor are input as input data.

[0203] More specifically, since the image sensor 110 is fixed to the camera module 1000, the vibration of the image sensor 110 can be substantially detected when the vibration of the camera module 1000 is sensed by the first sensor 120. Therefore, the vibration of the image sensor 110 can be substantially sensed by detecting the vibration of the camera module 1000.

[0204] More specifically, since the lens 100 is fixed to the lens mount 210, when the vibration of the lens mount 210 is sensed by the second sensor 230, the vibration of the lens 100 can be substantially sensed. Therefore, the vibration of the lens 100 can be substantially sensed by detecting the vibration of the lens mount 210.

[0205] Here, the vibration detection data can refer to the above reference. Figure 3 The vibration detection data described. The temperature sensor used to measure the temperature data may be a sensor included in the camera module 1000, a sensor included in the processor 140, or a dedicated sensor, but is not limited thereto.

[0206] In the examples disclosed herein, the artificial neural network model 700 can infer stabilization data for compensating for image jitter by taking vibration detection data 710 and temperature data 720 as inputs, and the artificial neural network model can output the inferred stabilization data 730.

[0207] In the following text, reference will be made to Figure 8 Detailed description Figure 7 Training method for artificial neural network model 700.

[0208] Figure 8 This is a schematic conceptual diagram illustrating a training method for an artificial neural network model according to another example of this disclosure. In the presented example, it is assumed that the artificial neural network model 800 is based on a model such as actor-evaluator reinforcement learning. In particular, the operations described below can be performed by processor 140.

[0209] Reference Figure 8 The sensor 810 obtains the first motion signals θYaw and θPitch of the camera module 1000, the second motion signals θHSx and θHSy of the lens 100, and the temperature data Temp.

[0210] Here, first motion signals θYaw and θPitch are obtained from the first sensor 120, second motion signals θHSx and θHSy are obtained from the second sensor 230, and temperature signal Temp is obtained from the temperature sensor.

[0211] Here, the first motion signals θYaw and θPitch are signals used to substantially detect the jitter of the image sensor 110. The second motion signals θHSx and θHSy are signals used to substantially sense the jitter of the lens 100.

[0212] The processor 140 can determine environmental data by using the obtained vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp and store it in the batch memory 820. The environmental data includes the current state St, the next state St+1, the action at, the reward rt+1, and the policy.

[0213] Here, the data for the current state St can represent the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp. The data for the next state St+1 can refer to the vibration detection data and temperature data obtained after taking an action in response to the current state.

[0214] The data representation of motion at can be based on the stabilized data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) of lens 100 inferred from vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp.

[0215] The reward rt+1 data represents the difference between the x-axis rotation angle θPitch for camera module 1000 and the x-axis rotation angle θHSx for lens 100, the difference between the y-axis rotation angle θYaw for camera module 1000 and the y-axis rotation angle θHSy for lens 100, and the error value of the temperature data Temp.

[0216] Subsequently, data about the current state St and the next state St+1 from the batch memory 820 are input as training data into each of the first model (e.g., actor model) and the second model (e.g., evaluator model) of the artificial neural network model 400.

[0217] Processor 140 trains artificial neural network model 400 to determine a strategy (i.e., action) that maximizes reward rt+1. Specifically, a first model (e.g., an actor model) infers stabilization data based on input vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp. The first model (e.g., the actor model) determines the probability that the error value converges to a predetermined value when the position of lens 100 is compensated based on the inferred stabilization data.

[0218] The second model (e.g., the evaluator model) evaluates the values ​​of the stabilized data inferred from the vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp, which are input to the first model (e.g., the actor model).

[0219] The evaluation results from the values ​​of the second model are sent to the first model (e.g., the actor model) so that the first model (e.g., the actor model) can be used to determine subsequent actions.

[0220] During the training phase, vibration detection data can be provided in various forms.

[0221] For example, a camera module 1000 used for reinforcement learning can be fixed to a specific fixture and programmed to move in a specific pattern.

[0222] For example, at least one user can hold the camera module 1000 used for reinforcement learning and move it for a specific period of time.

[0223] For example, for reinforcement learning, virtual vibration detection data can be provided.

[0224] Here, the specific mode can be sitting, walking, running, boating, caring, motorcycleing, etc.

[0225] During the reinforcement learning period, at least one specific pattern can be applied.

[0226] During reinforcement learning, at least one specific pattern can be applied sequentially or randomly.

[0227] Stabilized data inferred from the first model (e.g., the actor model) is sent to the VCM driver 130, and the VCM driver 130 sends control signals to the VCM actuator 200 to compensate for vibrations of the VCM actuator 200. Receiving these voltages vx and vy, the VCM driver 130 sends control signals to the VCM actuator 200 to control the VCM actuator 200 on the x and y axes. Image jitter can be compensated by compensating for the position of the lens 100 by the VCM actuator 200 receiving the control signals.

[0228] In the next step, vibration detection data and temperature data can be acquired via sensor 810, and the above training operations can be repeated. These training operations can be performed until a maximum number of success criteria or epochs are reached, but are not limited to this. For example, a success criterion could include a criterion in which the error value determined based on the inferred stabilization data converges to zero.

[0229] In the following text, reference will be made to Figure 5 This section provides a detailed description of the specific training operations for Artificial Neural Network Model 800. Artificial Neural Network Model 800 can be referenced... Figure 5 The training is performed in the same manner as described.

[0230] Reference Figure 5Each of the first model 500 and the second model 510 receives the current state St and the next state St+1 as training data. The current state St includes the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp. The next state St+1 includes the vibration detection data and temperature data obtained after taking an action on the current state.

[0231] The first model 500 outputs stable data inferred from the input vibration detection data and temperature data (st, st+1), as well as the probability πθ(at|St) of taking an action in the current state St and the probability πθold(at|St) of taking an action in the current state St of the first model 500 before being updated to the batch.

[0232] The second model 510 receives vibration detection data and temperature data (st, st+1) as input, and outputs the values ​​of vibration detection data (st, st+1) Vυ(St) and Vυ(St+1) as well as the expected benefits (e.g., advantages).

[0233] The first model 500 can be based on the outputs of πθ(at|St), πθold(at|St), and the second model 510. Determine the loss L CLIP (θ), and by using a determined loss L CLIP (θ) can update the parameter vector θ of the first model 500. The loss L can be updated using the above Equation 1. CLIP (θ).

[0234] The second model 510 can use the values ​​Vυ(St) and Vυ(St+1) to determine the loss L. V (u), and by using a determined loss L V (u) can update the parameter vector υ of the second model 510. The loss can be calculated using the above Equation 2.

[0235] The stabilized data sent from the first model 500 is converted into voltages (vx, vy) that can be input to the VCM driver 130 via the conversion unit 520 and output. The VCM driver 130 sends control signals (cx, cy) to cause the VCM actuator 200 to compensate for the position of the lens. Afterwards, it can be... Figure 8 The training operations are performed as described.

[0236] In the following text, reference will be made to Figure 9 Detailed description of usage is as follows (see reference) Figure 5 and Figure 8 The method described above uses an artificial neural network model to compensate for image jitter.

[0237] Figure 9 This is a schematic conceptual diagram illustrating a method for compensating for image jitter using a trained artificial neural network model, according to another example of this disclosure. The operations, which will be described later, can be performed by processor 140.

[0238] exist Figure 9 In the example, the second model (e.g., the evaluator model) can be excluded. Therefore, inference can be performed using only the first model (e.g., the actor model). In this case, the weights of the first model (e.g., the actor model) can be trained.

[0239] Furthermore, when only the first model (e.g., the actor model) is used, the training step can be eliminated, and the second model (e.g., the evaluator model) can also be eliminated, thereby improving power consumption, computational cost, and processing speed. Additionally, a high-speed, low-power camera module 1000 can be implemented by applying a low-power processor 140 implemented as an NPU.

[0240] Reference Figure 9 The sensor 900 obtains vibration detection data and temperature data St corresponding to the environment, and stores the obtained vibration detection data and temperature data St in the batch memory 910.

[0241] Vibration detection data and temperature data St stored in batch memory 910 are input as input data to stabilization signal generator 920. Stabilization signal generator 920 may include a first model (e.g., an actor model) having five input nodes, multiple hidden layers, and four output nodes. However, the structure of the first model (e.g., an actor model) is not limited thereto.

[0242] When inputting θYaw, θPitch, θHSx, θHSy, and Temp from each of the five input nodes, the first model (e.g., an actor model) infers stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) that converges the error value r to a predetermined value, and outputs the inferred stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis). Here, the error value r is the sum of the difference errx between the x-axis rotation angle θPitch of camera module 1000 and the x-axis rotation angle θHSx of lens 100, and the difference erry between the y-axis rotation angle θYaw of camera module 1000 and the y-axis rotation angle θHSy of lens 100. For example, the error value can be expressed as -(|errx||Hy ... x +err y The equation is expressed as |).

[0243] The stabilization signal generator 920 may further include a conversion unit Transpose, which is configured to convert the inferred stabilization data (+Xaxis, -Xaxis, +Yaxis, -Yaxis) into voltages that can be used as inputs to the VCM driver 130. The conversion unit Transpose uses a conversion formula or lookup table to convert the stabilization data into voltages (vx, vy) and outputs the converted voltages (vx, vy) as control signals.

[0244] When the input voltage (vx, vy) is applied, the VCM driver 130 can output control signals to compensate for vibrations in the VCM actuator 200 based on stabilization data. These control signals can be used to control the current (cx, cy) of the VCM actuator 200.

[0245] When an input current (cx, cy) is applied, the VCM actuator 200 operates to compensate for the position of the lens 100 based on the input current, thereby performing image stabilization.

[0246] In the following text, device A may also include a coil for autofocus (AF), and reference will be made to... Figures 10 to 14 Describe in detail the method of controlling AF and OIS based on artificial intelligence neural networks.

[0247] Figure 10 This is a schematic concept diagram illustrating another example of a camera module according to the present disclosure. In the presented example, for ease of description, the description of redundant elements as described above may be omitted.

[0248] Reference Figure 10 The camera module 1000 may include a lens 100, an image sensor 110, a first sensor 120, a VCM driver 130, a processor 140, and a VCM actuator 200. The VCM actuator 200 may also include an AF coil 260.

[0249] The AF coil 260 can be configured to correspond to an AF magnet (not shown), and a voltage can be applied to control the position of the AF magnet.

[0250] Processor 140 can acquire motion signals from at least two or more sensors (i.e., first sensor 120 and second sensor 230) included in camera module 1000, determine the amount of defocus based on the frequency components of the image, and infer the vibration compensation signal using an artificial neural network model trained to infer a vibration compensation signal for compensating for image jitter and focusing. Processor 140 can then compensate for image jitter and focusing based on the inferred vibration compensation signal. Here, the motion signals may also include the z-axis value of a Hall sensor used for focus adjustment.

[0251] In the following text, reference will be made to Figures 11 to 14 The method of compensating for image jitter and adjusting focus by using an artificial neural network model in processor 140 is described in detail.

[0252] Figure 11 This is a schematic conceptual diagram illustrating another example of an artificial neural network model according to this disclosure.

[0253] Reference Figure 11 In the artificial neural network model 1100, vibration detection data 1110 and defocus amount data 1120 are input as input data. The vibration detection data 1110 includes the movement signal of the camera module 1000 obtained by the first sensor 120 and the movement signal of the lens 100 obtained by the second sensor 230. The defocus amount 1120 is determined based on the frequency components of the image. Here, the vibration detection data may also include the z-axis rotation angle obtained by the Hall sensor.

[0254] In the examples disclosed herein, the artificial neural network model 1100 can compensate for image jitter by receiving vibration detection data 1110 and defocus data 1120 as inputs, infer stabilization data for adjusting focus, and output the inferred stabilization data 1130.

[0255] In the following text, reference will be made to Figure 12 Detailed description Figure 11 Training method of artificial neural network model 1100.

[0256] Figure 12 This is a schematic conceptual diagram illustrating a training method for an artificial neural network model according to another example of this disclosure. In the presented example, it is assumed that the artificial neural network model 1200 is based on a model such as actor-evaluator reinforcement learning. Specifically, the operations described below can be executed by processor 140.

[0257] Reference Figure 12 The sensor 1210 obtains the first motion signal (θYaw, θPitch) of the camera module 1000 and the second motion signal (θHSx, θHSy, θHSz) of the lens 100, and obtains defocus amount data determined based on the frequency components of the image. The defocus amount data can be acquired by the processor 140, the image sensor, or the image signal processor (ISP).

[0258] The processor 140 can determine environmental data, including the current state St, the next state St+1, the action at, the reward rt+1, and the policy, by using the acquired vibration detection data (θYaw, θPitch, θHSx, θHSy), lens focus data θHSz, and defocus amount data defocus_amount. The determined environmental data can then be stored in the batch memory 1220. Here, the data for the current state St can represent the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy), lens focus data θHSz, and defocus amount data defocus_amount. The data for the next state St+1 can represent the vibration detection data and defocus amount data acquired after taking an action on the current state. The data for the action at can represent the stabilized data (+Xaxis, -Xaxis, +Yaxis, -Yaxis, +Zaxis, -Zaxis) of the lens 100 that can be inferred based on the vibration detection data (θYaw, θPitch, θHSx, θHSy), the lens focus data θHSz, and the defocus amount data defocus_amount. The data for the reward rt+1 can represent the difference between the x-axis rotation angle θPitch of the camera module 1000 and the x-axis rotation angle θHSx of the lens 100, the difference between the y-axis rotation angle θYaw of the camera module 1000 and the y-axis rotation angle θHSy of the lens 100, and the error value based on the defocus amount data.

[0259] Subsequently, the data of the current state St and the next state St+1 from the batch memory 420 are input as training data into each of the first model (e.g., actor model) and the second model (e.g., evaluator model) of the artificial neural network model 1200.

[0260] The processor 140 trains an artificial neural network model 1200 on this basis to infer stable data where the error values ​​converge to predetermined values.

[0261] Specifically, the first model (e.g., the actor model) infers stabilization data based on the input vibration detection data (θYaw, θPitch, θHSx, θHSy), lens focus data θHSz, and defocus amount data defocus_amount. The first model (e.g., the actor model) determines the probability that the error value converges to a predetermined value when the position of lens 100 is compensated based on the inferred stabilization data.

[0262] The second model (e.g., an evaluator model) evaluates the values ​​of the stabilized data inferred from the input of the first model (e.g., an actor model) based on vibration detection data (θYaw, θPitch, θHSx, θHSy), lens focus data (θHSz), and defocus amount data (defocus_amount). The evaluation result of this value is sent to the first model (e.g., the actor model) so that the first model (e.g., the actor model) can use it to determine subsequent actions.

[0263] During the training phase, vibration detection data can be provided in various forms.

[0264] For example, a camera module 1000 used for reinforcement learning can be fixed to a specific fixture and programmed to vibrate in a specific pattern.

[0265] For example, at least one user can hold the camera module 1000 used for reinforcement learning and make it vibrate for a specific period of time.

[0266] For example, for reinforcement learning, virtual vibration detection data can be provided.

[0267] Here, the specific mode can be sitting, walking, running, boating, caring, motorcycleing, etc.

[0268] During the reinforcement learning period, at least one specific mode can be applied.

[0269] During reinforcement learning, at least one specific pattern can be applied sequentially or randomly.

[0270] Stabilized data inferred from a first model (e.g., an actor model) is sent to the VCM driver 130, and the VCM driver 130 sends control signals to the VCM actuator 200 to compensate for vibrations of the VCM actuator 200. In this disclosure, the stabilized data can be converted into voltages (vx, vy, vz) that can be input to the VCM driver 130 to cause the VCM driver 130 to control the VCM actuator 200 and then sent. The VCM driver 130, receiving these voltages (vx, vy, vz), sends control signals to the VCM actuator 200 to control the VCM actuator 200 on the x-axis, y-axis, and z-axis. Here, the control signals may include a current cx for controlling the VCM actuator 200 on the x-axis, a current cy for controlling the VCM actuator 200 on the y-axis, and a current cz for controlling the VCM actuator 200 on the z-axis.

[0271] The VCM actuator 200, which receives these currents cx, cy, and cz, operates to compensate for the position of the lens 100, thereby compensating for image jitter.

[0272] In the next step, after acquiring vibration detection data (θYaw, θPitch, θHSx, θHSy) and lens focus data θHSz via sensor 1210, and obtaining defocus amount data defocus_amount based on the frequency components of the image, the above training operations can be repeated. These training operations can be performed until a maximum number of success criteria or epochs are reached, but are not limited thereto. For example, a success criterion could include a criterion in which the error value determined based on the inferred stabilization data converges to zero.

[0273] In the following text, reference will be made to Figure 13 Describe in detail the specific training operations of the artificial neural network model 1200.

[0274] Figure 13 This is a schematic conceptual diagram illustrating a specific training operation of an artificial neural network model according to another example of this disclosure.

[0275] Reference Figure 13 Each of the first model 1300 and the second model 1310 receives the current state St and the next state St+1 as training data. The current state St includes the currently acquired vibration detection data (θYaw, θPitch, θHSx, θHSy), lens focus data θHSz, and defocus amount data defocus_amount. The next state St+1 includes the vibration detection data and defocus amount data obtained after taking an action on the current state.

[0276] The first model 1300 outputs stable data inferred from the input vibration detection data and defocus data (st, st+1), as well as the probability πθ(at|St) of taking an action in the current state St and the probability πθold(at|St) of taking an action in the current state St of the first model 1300 before being updated to the batch.

[0277] The second model 1310 receives vibration detection data and defocusing data (st, st+1) as input, and outputs the values ​​of vibration detection data (st, st+1) Vυ(St) and Vυ(St+1) as well as the expected benefits (e.g., advantages).

[0278] The first model 1300 can be based on πθ(at|St), πθold(at|St), and the output of the second model 1310. Determine the loss L CLIP (θ), and by using the determined loss L CLIP (θ) can update the parameter vector θ of the first model 500. The loss L can be updated using Equation 1 above. CLIP (θ).

[0279] The second model 1310 can use the values ​​Vυ(St) and Vυ(St+1) to determine the loss L. V (u), and by using the determined loss L V (u) can update the parameter vector υ of the second model 1310. The loss can be calculated using Equation 2 above.

[0280] The stabilized data sent from the first model 1300 is converted into voltages (vx, vy, vz) that can be input to the VCM driver 130 via the conversion unit 520 and output. The VCM driver 130 sends control signals (cx, cy, cz) to cause the VCM actuator 200 to compensate for the position of the lens. Afterwards, it can be... Figure 12 The training operation is performed on the site.

[0281] In the following text, reference will be made to Figure 14 Detailed description of usage is as follows (see reference) Figure 12 and Figure 13 The method described above uses a trained artificial neural network model to compensate for image jitter.

[0282] Figure 14 This is a schematic conceptual diagram illustrating a method for compensating for image jitter using a trained artificial neural network model, according to another example of this disclosure. The operations, which will be described later, can be performed by processor 140.

[0283] Reference Figure 14 Vibration detection data is obtained through sensor 1400, and after obtaining defocus data St based on the frequency components of the image, the obtained vibration detection data and defocus data St are stored in batch memory 1410.

[0284] exist Figure 14 In the example, the second model (e.g., the evaluator model) can be excluded. Therefore, inference can be performed using only the first model (e.g., the actor model). In this case, the training of the weights of the first model (e.g., the actor model) can be completed. Furthermore, when only the first model (e.g., the actor model) is used, the training step can be excluded, and the second model (e.g., the evaluator model) can also be excluded, thereby reducing power consumption, reducing computational load, and improving processing speed. Additionally, the high-speed, low-power camera module 1000 can be implemented by applying a low-power processor 140 implemented as an NPU.

[0285] The vibration detection data and defocusing data St stored in the batch memory 1410 are input as input data to the stabilization signal generator 1420. The stabilization signal generator 1420 may include a first model (e.g., an actor model) having six input nodes, multiple hidden layers, and six output nodes. However, the structure of the first model (e.g., the actor model) is not limited thereto.

[0286] When inputting θYaw, θPitch, θHSx, θHSy, θHSz, and defocus_amount from each of the six input nodes, the first model (e.g., the actor model) infers stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis, +Zaxis, -Zaxis) that converges the error value r to a predetermined value, and outputs the inferred stable data (+Xaxis, -Xaxis, +Yaxis, -Yaxis, +Zaxis, -Zaxis). Here, the error value r can be a value obtained by adding the sum of the difference errx between the x-axis rotation angle θPitch of the camera module 1000 and the x-axis rotation angle θHSx of the lens 100, and the difference erry between the y-axis rotation angle θYaw of the camera module 1000 and the y-axis rotation angle θHSy of the lens 100, to the defocus amount (defocus_amount). For example, the error value can be obtained by such as -(|errx||Hy ... x +err y The equation is expressed as |).

[0287] The defocus_amount can be adjusted based on the voltage vz value. The AF coil 260 can adjust its z-axis based on the cz value corresponding to the voltage vz. Therefore, the z-axis of the lens 100, which is fixed to the lens mount 210 to which the AF coil 260 is mounted, can be moved. Thus, the defocus_amount can be adjusted.

[0288] The stabilization signal generator 1420 may further include a transpose unit configured to convert the inferred stabilization data (+Xaxis, -Xaxis, +Yaxis, -Yaxis, +Zaxis, -Zaxis) into voltages usable as inputs to the VCM driver 130. The transpose unit uses a conversion formula or lookup table to convert the stabilization data into voltages (vx, vy, vz) and outputs the converted voltages (vx, vy, vz) as control signals.

[0289] When the input voltage (vx, vy, vz) is applied, the VCM driver 130 can output control signals to compensate for the vibration of the VCM actuator 200 based on the stabilization data. These control signals can be used to control the current (cx, cy, cz) of the VCM actuator 200.

[0290] When an input current (cx, cy, cz) is applied, the VCM actuator 200 operates to compensate for the position of the lens 100 according to the input current, thereby stabilizing the image and adjusting the focus.

[0291] According to various examples of this disclosure, the camera module 1000 can obtain vibration detection data (θYaw, θPitch, θHSx, θHSy) and temperature data Temp through sensors, and obtain defocus amount data defocus_amount based on lens focus data θHSz and the frequency components of the image.

[0292] The camera module 1000 can output stabilization data using an artificial neural network model trained to infer stabilization data for image stabilization and focus adjustment based on the obtained vibration detection data, temperature data, lens focus data, and defocus data.

[0293] In the following text, reference will be made to Figure 15 Describe the image stabilization method for the camera module.

[0294] Figure 15 This is a flowchart illustrating an image stabilization method of a camera module in one example of this disclosure. The operations described later in the presented example can be performed by the processor 140 of the camera module 1000.

[0295] Reference Figure 15 The processor 140 acquires image-related vibration detection data 1500 from two or more sensors. The two or more sensors according to examples of this disclosure may include at least two or more of a gyroscope sensor, a Hall sensor, and a photoelectric sensor. Here, the vibration detection data may include signals detected by rotational movement along the x and y axes of the gyroscope sensor and the Hall sensor.

[0296] The processor 140 uses an artificial neural network model to output stabilization data, which is trained to output stabilization data for compensating for image jitter based on the obtained vibration detection data (S1510).

[0297] Next, the processor 140 uses the output stabilization data to compensate for image jitter S1520.

[0298] Figure 16This is a schematic conceptual diagram illustrating a neural processing unit according to the present disclosure.

[0299] Figure 16 The neural processing unit (NPU) shown is a processor dedicated to performing operations on artificial neural networks.

[0300] An artificial neural network is a network of artificial neurons that, when receiving multiple inputs or stimuli, multiplies and adds weights, and transforms and transmits values ​​with added bias through an activation function. A trained artificial neural network can be used to output inference results based on input data.

[0301] An NPU can be a semiconductor implemented as an electrical / electronic circuit. This electrical / electronic circuit can include various electronic devices (e.g., transistors and capacitors).

[0302] The NPU may include a processing element (PE) array 11000, an NPU internal memory 12000, an NPU scheduler 13000, and an NPU interface 14000. Each of the multiple processing elements 11000, NPU internal memory 12000, NPU scheduler 13000, and NPU interface 14000 may be a semiconductor circuit with multiple transistors connected. Therefore, some of them may be difficult to identify and distinguish with the naked eye, and may only be identifiable through operation.

[0303] The NPU can be configured to infer the first model (e.g., the actor model).

[0304] For example, a particular circuit may operate as multiple processing elements 11000, or it may operate as an NPU scheduler 13000. The NPU scheduler 13000 may be configured to perform the functions of a controller configured to control the artificial neural network inference operations of the NPU.

[0305] The NPU may include: multiple processing elements 11000; an NPU internal memory 12000 configured to store artificial neural network models that can be inferred by the multiple processing elements 11000; and an NPU scheduler 13000 configured to control the multiple processing elements 11000 and the NPU internal memory 12000 based on data location information or structure-related information of the artificial neural network models. Here, the artificial neural network model may include information about the data location information or structure of the artificial neural network model. The artificial neural network model may refer to an AI recognition model trained to perform a specific inference function.

[0306] Multiple processing elements 11000 can perform operations on artificial neural networks.

[0307] The NPU Interface 14000 can communicate with various components (e.g., memory) connected to the NPU via the system bus.

[0308] The NPU scheduler 13000 can be configured to control the operation of multiple processing elements 11000 for inference operations of the neural processing unit, as well as the order of read and write operations of the NPU internal memory 12000.

[0309] The NPU scheduler 13000 can be configured to control multiple processing elements 11000 and NPU internal memory 12000 based on data location information or structure-related information from an artificial neural network model.

[0310] The NPU scheduler 13000 can analyze the structure of an artificial neural network model to be operated in multiple processing elements 11000, or can receive pre-analyzed information. For example, data of the artificial neural network that can be included in the model may include at least a portion of node data (i.e., feature maps) for each layer, layer layout data, location or structural information, and weight data (i.e., weight kernels) of the respective connection networks connecting the nodes of each layer. The data of the artificial neural network can be stored in memory provided within the NPU scheduler 13000 or the NPU internal memory 12000.

[0311] The NPU scheduler 13000 can schedule the order of operations of an artificial neural network model to be executed by the NPU based on the data location information or structural information of the artificial neural network model.

[0312] The NPU scheduler 13000 can obtain the memory address values ​​where the feature maps and weight data of the layers of the artificial neural network model are stored, based on the data location information or structural information of the artificial neural network model. For example, the NPU scheduler 13000 can obtain the memory address values ​​where the feature maps and weight data of the layers of the artificial neural network model stored in memory are stored. Therefore, the NPU scheduler 13000 can send the feature maps and weight data of the layers of the artificial neural network model to be driven from memory and store them in the NPU internal memory 12000.

[0313] Each layer's feature map can have a corresponding memory address value.

[0314] Each weighted data point can have a corresponding memory address value.

[0315] The NPU scheduler 13000 can schedule the operation sequence of multiple processing elements 11000 based on data location information or structure-related information of the artificial neural network model (e.g., data location information of the layer layout of the artificial neural network model or structure-related information of the artificial neural network model).

[0316] The NPU scheduler 13000 can perform scheduling based on data location information or structure-related information from artificial neural network models, allowing it to operate in a way that differs from conventional CPU scheduling concepts. Considering fairness, efficiency, stability, and response time, conventional CPU scheduling aims to provide the highest efficiency. That is, considering priority and operation time, conventional CPUs schedule to execute the most processing within the same amount of time.

[0317] Traditional CPUs use algorithms that schedule tasks based on data such as the priority of each process or the processing time of an operation.

[0318] Conversely, the NPU scheduler 13000 can determine the processing order based on data location information or structure-related information of the artificial neural network model.

[0319] In addition, the NPU scheduler 13000 can operate the NPU according to the data location information or structure-related information based on the artificial neural network model and / or the determined processing order of the NPU's data location information or information.

[0320] However, this disclosure is not limited to data location information or structure-related information of the NPU.

[0321] The NPU scheduler 13000 can be configured to store information about the location or structure of data in an artificial neural network model.

[0322] In other words, even if only information about the location or structure of the artificial neural network model is provided, the NPU scheduler 13000 can determine the processing order.

[0323] Furthermore, the NPU scheduler 13000 can determine the NPU processing order by considering information about the data location or structure of the artificial neural network model and information about the data location or structure of the NPU. Additionally, it can optimize NPU processing within the determined processing order.

[0324] The plurality of processing elements 11000 may refer to a configuration in which a plurality of processing elements PE1 to PE12 are arranged, the plurality of processing elements PE1 to PE12 being configured to compute feature maps and weight data of an artificial neural network. Each processing element may include a multiply-accumulate (MAC) arithmetic unit and / or an arithmetic logic unit (ALU). However, the examples according to this disclosure are not limited thereto.

[0325] Each processing element may be configured to optionally include an additional special function unit for processing additional special functions.

[0326] For example, the processing element PE can also be modified and implemented to further include batch normalization units, activation function units, interpolation units, etc.

[0327] Although Figure 16 Multiple processing elements are schematically shown, but an arithmetic unit implemented as a tree of multiple multipliers and adders can also be configured to be arranged in parallel by replacing the MAC in one of the processing elements. In this case, the multiple processing elements 11000 can be referred to as at least one processing element including multiple arithmetic units.

[0328] The plurality of processing elements 11000 are configured to include a plurality of processing elements PE1 to PE12. Figure 16 The multiple processing elements PE1 to PE12 shown are merely examples for ease of description, and the number of processing elements PE1 to PE12 is not limited thereto. The size or number of processing element arrays can be determined by the number of processing elements PE1 to PE12. The size of the processing element array can be implemented in the form of an N×M matrix. Here, N and M are integers greater than zero. The processing element array can include N×M processing elements. That is, at least one processing element can exist.

[0329] The characteristics of the artificial neural network model in which the NPU operates can be considered when designing the size of multiple processing elements 11000.

[0330] Multiple processing elements 11000 can be configured to perform functions required for artificial neural network operations, such as addition, multiplication, and accumulation. In other words, multiple processing elements 11000 can be configured to perform multiplication and accumulation (MAC) operations.

[0331] Based on various examples of this disclosure, artificial neural network models can be trained using reinforcement learning techniques.

[0332] According to various examples of this disclosure, vibration detection data may include signals used to detect changes in the position of the camera module and lens.

[0333] According to various examples of this disclosure, an artificial neural network model can be trained based on vibration detection data, such that the error value caused by image jitter converges to a predetermined value. Here, the error value can be based on the difference between the x-axis movement of the gyroscope sensor and the x-axis movement of the Hall sensor, and the difference between the y-axis movement of the gyroscope sensor and the y-axis movement of the Hall sensor.

[0334] According to various examples of this disclosure, an artificial neural network model can receive vibration detection data as input to output control signals for controlling the movement of lenses included in a camera module to compensate for image jitter.

[0335] According to various examples of this disclosure, an artificial neural network model can receive vibration detection data as input and output control signals for controlling the movement of an image sensor included in a camera module to compensate for image jitter.

[0336] According to various examples of this disclosure, the trained model may include a first model and a second model, wherein the first model is trained to infer stable data based on vibration detection data, wherein the error value converges to a predetermined value, and the second model is trained to evaluate the results of the stable data.

[0337] According to various examples of this disclosure, an artificial neural network model can simultaneously perform training and inference of stable data by taking vibration detection data as input.

[0338] According to various examples of this disclosure, an artificial neural network model may include an input node that takes vibration detection data as input, a hidden layer that performs AI operations (e.g., convolution operations), and an output node that outputs stabilized data.

[0339] According to various examples, processor 140 can be configured to collect error values ​​during training and use the collected error values ​​to update the artificial neural network model.

[0340] According to various examples of this disclosure, the camera module may include a temperature sensor for sensing temperature, and the processor 140 may be configured to output stable data using an artificial neural network model based on vibration detection data and temperature data acquired by the temperature sensor.

[0341] According to various examples of this disclosure, vibration detection data may include signals detected by rotational movement along the x and y axes of a gyroscope sensor and rotational movement along the x, y, and z axes of a Hall sensor, and processor 140 may be configured to obtain defocus data from the frequency components of an image and output stabilized data based on the vibration detection data and the defocus data using an artificial neural network model.

[0342] According to various examples of this disclosure, artificial neural network models can use modulation transfer function (MTF) data of images as training data for training.

[0343] MTF data can be obtained by quantizing the amount of defocus.

[0344] The examples shown in the specification and accompanying drawings are provided merely to facilitate the description of the subject matter of this disclosure and to provide specific examples to aid in understanding it, and are not intended to limit the scope of this disclosure. It will be apparent to those skilled in the art to which this disclosure pertains that other modifications based on the technical spirit of this disclosure can be made in addition to the examples disclosed herein.

[0345] Cross-reference to related applications

[0346] This application claims priority to Korean Patent Application No. 10-2021-0106909, filed with the Korean Intellectual Property Office on August 12, 2021, and Korean Patent Application No. 10-2022-0064967, filed with the Korean Intellectual Property Office on May 26, 2022, the disclosures of which are incorporated herein by reference.

Claims

1. A method for stabilizing images based on artificial intelligence (AI), the method comprising the following steps: Acquire vibration detection data about the image, the vibration detection data being obtained from two or more sensors; Output stabilized data for compensating for image jitter, said stabilized data being output using an artificial neural network (ANN) model trained to output said stabilized data based on said jitter detection data; and The image jitter is compensated using the stabilized data. The vibration detection data includes the x-axis rotation angle θPitch and y-axis rotation angle θYaw for the camera module, and the x-axis rotation angle θHSx and y-axis rotation angle θHSy for the lens. The stabilization data includes control signals used to compensate for the positions of the image sensor and the lens of the camera module, thereby reducing the power consumption of the camera module while improving the stabilization speed. The artificial neural network (ANN) model is a trained model based on training data used for learning, which makes the error value caused by image jitter close to a predetermined value, and The error value is the sum of the differences between the x-axis rotation angle θPitch for the camera module and the x-axis rotation angle θHSx for the lens, and the differences between the y-axis rotation angle θYaw for the camera module and the y-axis rotation angle θHSy for the lens.

2. The method according to claim 1, in, The artificial neural network (ANN) model is based on reinforcement learning.

3. The method according to claim 1, in, The two or more sensors include gyroscope sensors and Hall sensors.

4. The method according to claim 1, in, The artificial neural network (ANN) model is configured to receive the vibration detection data as input to simultaneously perform training operations for inferring the stabilized data and inference operations for the stabilized data.

5. The method according to claim 1, in, The artificial neural network (ANN) model includes an input node that receives the vibration detection data, a hidden layer that performs artificial intelligence (AI) operations on the input node, and an output node that outputs the stabilized data.

6. A camera module, the camera module comprising: Lens; An image sensor configured to output an image captured through the lens; Two or more sensors, the two or more sensors being configured to output vibration detection data about the image; A controller configured to output stabilized data based on the vibration detection data using an artificial neural network (ANN) model; as well as A stabilization unit, configured to use the stabilization data to compensate for image jitter. The artificial neural network (ANN) model is trained to output the stabilized data based on the vibration detection data. The vibration detection data includes the x-axis rotation angle θPitch and y-axis rotation angle θYaw for the camera module, and the x-axis rotation angle θHSx and y-axis rotation angle θHSy for the lens. The stabilization data includes control signals used to compensate for the positions of the image sensor and the lens of the camera module, thereby reducing the power consumption of the camera module while improving the stabilization speed. Specifically, the artificial neural network (ANN) model is trained based on the vibration detection data, so that the error value caused by image jitter is close to a predetermined value, and The error value is the sum of the differences between the x-axis rotation angle θPitch for the camera module and the x-axis rotation angle θHSx for the lens, and the differences between the y-axis rotation angle θYaw for the camera module and the y-axis rotation angle θHSy for the lens.

7. The camera module according to claim 6, in, The two or more sensors include at least two of a gyroscope sensor, a Hall sensor, and a photoelectric sensor.

8. The camera module according to claim 6, in, The vibration detection data includes signals detected by rotational movement along the x and y axes of a gyroscope sensor and rotational movement along the x and y axes of a Hall sensor.

9. The camera module according to claim 7, in, The error value is based on the difference between the x-axis movement of the gyroscope sensor and the x-axis movement of the Hall sensor, and the difference between the y-axis movement of the gyroscope sensor and the y-axis movement of the Hall sensor.

10. The camera module according to claim 7, in, The trained model includes a first model and a second model, wherein the first model is trained to infer stable data in which the error value is close to the predetermined value based on the vibration detection data, and the second model is trained to evaluate the results of the stable data.

11. The camera module according to claim 7, in, The controller is also configured to collect the error values ​​through training and to update the artificial neural network (ANN) model using the collected error values.

12. The camera module according to claim 6, in, The two or more sensors include temperature sensors for sensing temperature, and The controller is further configured to output the stabilization data based on the vibration detection data and the temperature data acquired by the temperature sensor.

13. The camera module according to claim 6, in, The vibration detection data includes: Signals detected by rotational movement along the x and y axes using a gyroscope sensor and rotational movement along the x, y, and z axes using a Hall sensor.

14. The camera module according to claim 6, in, The artificial neural network (ANN) model is configured to use the modulation transfer function (MTF) data of the image as training data for training.

15. The camera module according to claim 6, in, The two or more sensors are also configured to obtain defocus data from the frequency components of the image, and The controller is further configured to output the stabilization data based on the vibration detection data and the defocusing data.

Images