Apparatus and method for controlling optical image stabilization of camera using low power processor

Abstract

The present disclosure relates to an apparatus and a method for providing image stabilization by using a low power processor, and the method for providing image stabilization includes: loading an image stabilization control instruction group onto a memory in a general purpose processor when a camera is operated; executing initial setting based on the image stabilization control instruction group; driving a polling timer in dependence on the low power processor such that image stabilization can be performed at a preset period in the set of image stabilization control instructions; and performing, by the low-power processor, image stabilization with a set of image stabilization control instructions.

Description

Technical Field

[0001] The present disclosure relates to techniques for controlling optical image stabilization (OIS) of a camera using a low-power processor that controls the sensor rather than a separate image stabilization (IS) processor. Background Art

[0002] The development of electronic technology has accelerated the development and distribution of various electronic products. In particular, the spread of electronic devices having various functions, such as smartphones and tablet personal computers (PCs), has recently expanded.

[0003] When a user captures an image using an electronic device, image quality degradation such as image blur may occur due to slight shaking or other small vibrations (e.g., in transportation traveling on uneven roads). Electronic devices with image stabilization (IS) functions are being developed to prevent degradation in the quality of images captured using cameras.

[0004] The IS function applied to the electronic device detects motion caused by shaking using a motion sensor, and performs IS by moving the lens opposite to the motion caused by the shaking.

[0005] For such an IS, the electronic device includes a separate IS processor for controlling the IS.

[0006] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention

[0007] Aspects of the present disclosure address at least the above-mentioned problems and / or disadvantages and provide at least the advantages described below.Accordingly, one aspect of the present disclosure is to provide an apparatus and method for providing image stabilization (IS) using a low-power processor for controlling a sensor instead of a separate IS processor.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.

[0009] According to one aspect of the present disclosure, an optical image stabilization (OIS) method is provided. The OIS method includes: when an anti-shake start event occurs, using a general processor to store an IS control instruction group in a memory; using a low-power processor to drive a polling timer to perform IS at preset intervals by executing a polling timer instruction included in the IS control instruction group; and using the low-power processor to correct camera shake based on the IS control instruction group.

[0010] According to another aspect of the present disclosure, an OIS device is provided. The OIS device includes a camera, the camera including an IS driver configured to correct shake according to an IS result, a motion sensor configured to sense motion and output user interface (UI) motion information and IS motion information, a memory, and a mobile processor including a general processor and a low-power processor, wherein the general processor can be configured to store an IS control instruction group in the memory when an anti-shake start event occurs, and the low-power processor can be configured to drive a polling timer to perform IS at a preset interval by executing instructions included in the IS control instruction group stored in the memory, obtain IS motion information, generate an IS result according to the IS motion information, and provide the IS result to the IS driver.

[0011] According to an embodiment, an OIS device includes a camera, the camera including an IS driver configured to correct shake according to an IS result, a motion sensor configured to sense motion and output motion information, a memory, and a mobile processor including a general processor and a low-power processor, the general processor can be configured to store an IS control instruction group in the memory when an anti-shake start event occurs, and the low-power processor can be configured to drive a polling timer to perform IS at preset intervals by executing instructions included in the IS control instruction group stored in the memory, obtain motion information, generate an IS result according to the motion information, and provide the IS result to the IS driver.

[0012] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description which discloses various embodiments of the present disclosure in conjunction with the accompanying drawings.

[0013] The present disclosure relates to a device and method for providing IS using a low-power processor for controlling a sensor instead of a separate IS processor, and can eliminate the IS processor provided separately in the related art, thereby ensuring physical space for the IS processor in the electronic device, reducing the power waste required to drive the IS processor, and reducing the cost associated with providing the IS processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a diagram showing a configuration of an electronic device for controlling optical image stabilization (OIS) of a camera using a 2-channel motion sensor and an image stabilization (IS) control instruction group according to an embodiment of the present disclosure;

[0016] Figure 2 is a diagram showing a configuration of an electronic device for controlling an OIS of a camera using a 1-channel motion sensor and an IS control instruction group according to an embodiment of the present disclosure;

[0017] Figure 3 is a flowchart illustrating a process in which an electronic device controls an OIS of a camera according to an embodiment of the present disclosure;

[0018] Figure 4 is a flowchart illustrating a process in which an electronic device receives an IS adjustment value according to an embodiment of the present disclosure;

[0019] Figure 5 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a photo capturing mode when a motion sensor unit has two channels according to an embodiment of the present disclosure;

[0020] Figure 6 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a video shooting mode when a motion sensor unit has two channels according to an embodiment of the present disclosure;

[0021] Figure 7 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a photo capturing mode when a motion sensor unit has one channel according to an embodiment of the present disclosure;

[0022] Figure 8 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a video shooting mode when a motion sensor unit has one channel according to an embodiment of the present disclosure; and

[0023] Fig. 9 is a block diagram illustrating electronic devices in a network environment according to an embodiment of the present disclosure.

[0024] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION

[0025] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. The description includes various specific details to assist in understanding, but these details should be considered as merely exemplary. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0026] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it is clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0027] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0028] The terms used herein are only used for the purpose of describing specific embodiments, rather than limiting the embodiments. It will also be understood that when used herein, the terms "include" and / or "comprise" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as those commonly understood by those skilled in the art to which the embodiments belong. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0030] When describing the embodiments with reference to the accompanying drawings, the same reference numerals represent the same components, and repeated descriptions related thereto will be omitted. In the description of the embodiments, detailed descriptions of well-known related structures or functions will be omitted when it is considered that such descriptions will lead to an ambiguous interpretation of the present disclosure.

[0031] In addition, in the description of components, when describing the components of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used herein. These terms are only used for the purpose of distinguishing one constituent element from another constituent element, and the nature, order or sequence of the constituent elements are not limited by these terms. When a constituent element is described as being "connected", "coupled" or "attached" to another constituent element, it should be understood that one constituent element may be directly connected or attached to another constituent element, and intermediate constituent elements may also be "connected", "coupled" or "attached" to these constituent elements.

[0032] The same names may be used to describe the elements included in the above-mentioned embodiments and elements having common functions. Unless otherwise specified, the description of the examples may be applicable to the following examples, and therefore, for the sake of brevity, repeated descriptions will be omitted.

[0033] The following will refer to Figures 1 to 8 An apparatus and method for providing image stabilization (IS) using a low-power processor according to an embodiment of the present disclosure is described in detail.

[0034] Figure 1 is a diagram illustrating a configuration of an electronic device for controlling optical image stabilization (OIS) of a camera using a 2-channel motion sensor and an IS control instruction group according to an embodiment of the present disclosure.

[0035] refer to Figure 1 , the electronic device 100 may include a mobile processor 110 , a memory 120 , a camera unit 150 , and a motion sensor unit 160 .

[0036] The mobile processor 110 may include a general processor 111 , a low power processor 112 , an image data interface circuit unit 114 , and a general interface circuit unit 115 .

[0037] The general processor 111 may correspond to a central processing unit or an application processor, and may execute software to control at least one of other components of the electronic device 100 and perform various data processing or operations. The general processor 111 may include a plurality of cores.

[0038] When an anti-shake start event occurs, the general processor 111 may store the IS control instruction group 130 in the memory 120. When an anti-shake end event occurs, the general processor 111 may delete the IS control instruction group 130 from the memory 120.

[0039] In this case, the anti-shake start event may occur immediately before the camera unit 150 operates when the camera application is executed. However, the embodiment is not limited thereto, and the anti-shake start event may occur when the camera application is executed or when the camera unit 150 starts operating. In addition, the anti-shake end event may occur immediately before the operation of the camera unit 150 when the camera application ends stops. However, the embodiment is not limited thereto, and the anti-shake end event may occur when the camera application ends or when the operation of the camera unit 150 stops.

[0040] The low-power processor 112 may use less power than the general-purpose processor 111, or may be configured to be dedicated to a specified function. Figure 1 In the embodiment, the low power processor 112 can operate as a sensor hub processor for controlling at least one sensor. The low power processor 112 can include one core. The low power processor 112 can provide computing power to the IS control instruction group 130.

[0041] The image data interface circuit unit 114 may provide communication with the camera unit 150 , and the general interface circuit unit 115 may provide communication with peripheral devices of the electronic device 100 .

[0042] The camera unit 150 may include a camera image sensor 151 , a camera memory 152 , and an IS driver 153 .

[0043] The camera image sensor 151 may acquire an image by receiving light.

[0044] The camera image sensor 151 may be connected to the image data interface circuit unit 114 via a camera serial interface (CSI) channel to provide an acquired image to the general processor 111 through the image data interface circuit unit 114 .

[0045] The camera memory 152 may store IS adjustment values ​​for calibrating the IS driver 153 .

[0046] Upon receiving the IS result, the IS driver 153 may correct the shaking of the camera unit 150 by physically adjusting at least one lens (not shown).

[0047] The motion sensor unit 160 may sense the motion of the electronic device 100 and output user interface (UI) motion information and IS motion information.

[0048] The motion sensor unit 160 may output the UI motion information and the IS motion information through different communication channels.

[0049] More specifically, the motion sensor unit 160 may include a motion sensor 161 , a UI sensor data module 162 , and an IS sensor data module 163 .

[0050] The motion sensor 161 may be an element configured to sense motion of the electronic device 100 and output the sensed motion information as sensor data, and may include a 6-axis sensor having a 3-axis gyroscope and a 3-axis accelerometer.

[0051] The UI sensor data module 162 may process sensor data corresponding to the UI among the sensor data sensed by the motion sensor 161 into UI motion information, and provide the UI motion information to the sensor interface circuit unit 113 of the low power processor 112. In this case, the UI motion information may be transmitted through an I3C interface connected between the UI sensor data module 162 and the sensor interface circuit unit 113.

[0052] The IS sensor data module 163 may process sensor data corresponding to the jitter among the sensor data sensed by the motion sensor 161 into IS motion information, and provide the IS motion information to the sensor interface circuit unit 113 of the low power processor 112. In this case, the IS motion information may be transmitted through a serial peripheral interface (SPI) connected between the IS sensor data module 163 and the sensor interface circuit unit 113.

[0053] The memory 120 may store an operating system, application programs, and data to be stored (e.g., phone numbers, short message service (SMS) messages, compressed image files, videos, etc.) for controlling the overall operation of the electronic device 100. In addition, the memory 120 may store an IS control instruction group 130 and a UI motion sensor instruction 140 using the general processor 111 according to the present disclosure.

[0054] The low power processor 112 may drive a polling timer to perform IS at preset intervals by executing instructions included in the IS control instruction group 130 , obtain IS motion information, generate an IS result according to the IS motion information, and provide the IS result to the IS driver 153 .

[0055] More specifically, the IS control instruction group 130 may include a polling timer instruction 131 , a communication interface control instruction 132 , an IS driving control instruction 133 , and an IS motion sensor control instruction 134 .

[0056] The polling timer instruction 131 may be an element configured to perform IS at a preset interval, and may be an instruction for verifying whether the preset interval expires if an anti-shake end event does not occur after correcting the shake.

[0057] The polling timer instruction 131 may be an instruction for restarting a preset interval if the preset interval expires.

[0058] The polling timer instruction 131 may be an instruction for waiting for a preset time until the preset interval expires if the preset interval has not expired.

[0059] The polling timer using the polling timer instruction 131 of the present disclosure may be a software timer rather than a physical timer.

[0060] Traditional software timers are mainly used to handle daily user-based operations and therefore can be designed and used at the millisecond (msec) level.

[0061] However, a software timer designed at the millisecond level has a large error range and thus may not be suitable for performing time-sensitive operations such as an OIS algorithm, such as a device for controlling an OIS.

[0062] The present disclosure may provide a polling timer in nanoseconds, which is the minimum unit of a hardware (HW) tick counter of the low power processor 112 , thereby maximizing accuracy.

[0063] However, running the timer while polling the polling timer every nanosecond may place a burden on the low power processor 112, and therefore, a scaling factor may be used dynamically.

[0064] For example, in the case where the IS algorithm ends before 1 millisecond, the polling timer is made to expire quickly with an intentional delay of tens of microseconds (μsec), and when the expiration point is almost reached, the polling timer maintains the previous accuracy by converting to nanosecond units.

[0065] The communication interface control instruction 132 may be an instruction for connecting a first virtual communication interface between the memory 120 and the general processor 111 to receive the IS adjustment value. In this case, the first virtual communication interface may be a virtual communication interface simulating the operation of an inter-integrated circuit (I2C) interface.

[0066] In addition, the communication interface control instruction 132 may be an instruction for connecting a second virtual communication interface between the memory 120 and the general processor 111 using the general processor 111 in a case where a video exclusive image quality correction mode is executed in a video shooting mode, to obtain information for setting the IS intensity according to the ambient lighting, verify the IS intensity based on the information for setting the IS intensity, and provide the IS intensity to the low power processor 112 that executes the IS driving control instruction 133. In this case, the second virtual communication interface may be a virtual communication interface that simulates the operation of the I2C interface.

[0067] The IS motion sensor control instruction 134 may be an instruction for obtaining IS motion information from the motion sensor unit 160 through the sensor interface circuit unit 113 of the low power processor 112 using the low power processor 112 .

[0068] The IS drive control instruction 133 may be an instruction for receiving an IS adjustment value stored in the camera unit 150 through the first virtual communication interface using the general processor 111, initializing the IS based on the IS adjustment value using the low power processor 112, generating an IS result corresponding to the IS motion information by executing the IS algorithm using the low power processor 112, and providing the IS result to the IS driver 153 through the sensor interface circuit unit 113 using the low power processor 112. The initialization of the IS drive control instruction 133 may be to perform calibration using the IS adjustment value when executing the IS algorithm.

[0069] The IS driving control instruction 133 may be an instruction for generating an IS by taking the IS intensity into consideration when generating an IS result when a video-specific image quality correction mode is executed in a video shooting mode.

[0070] Meanwhile, the IS control instruction group 130 is shown to be divided into a polling timer instruction 131 , a communication interface control instruction 132 , an IS driving control instruction 133 , and an IS motion sensor control instruction 134 to describe their functions, respectively.

[0071] Figure 2 is a diagram illustrating a configuration of an electronic device for controlling an OIS of a camera using a 1-channel motion sensor and an IS control instruction group according to an embodiment of the present disclosure.

[0072] refer to Figure 2 , the electronic device 200 may include a mobile processor 210 , a memory 220 , a camera unit 250 , and a motion sensor unit 260 .

[0073] The mobile processor 210 may include a general processor 211 , a low power processor 212 , an image data interface circuit unit 214 , and a general interface circuit unit 215 .

[0074] The general processor 211 may correspond to a central processing unit or an application processor, and may execute software to control at least one of other components of the electronic device 200 and perform various data processing or operations. The general processor 211 may include a plurality of cores.

[0075] When an anti-shake start event occurs, the general processor 211 may store the IS control instruction group 230 in the memory 220. When an anti-shake end event occurs, the general processor 211 may delete the IS control instruction group 230 from the memory 220.

[0076] In this case, the anti-shake start event may occur immediately before the camera unit 250 operates when the camera application is executed. However, the embodiment is not limited thereto, and the anti-shake start event may occur when the camera application is executed or when the camera unit 250 starts operating. In addition, the anti-shake end event may occur immediately before the operation of the camera unit 250 when the camera application ends stops. However, the embodiment is not limited thereto, and the anti-shake end event may occur when the camera application ends or when the operation of the camera unit 250 stops.

[0077] The low-power processor 212 may use less power than the general-purpose processor 211, or may be configured to be dedicated to a specified function. Figure 2 In the embodiment of the present invention, the low power processor 212 can operate as a sensor hub processor for controlling the sensor. The low power processor 212 can include one core. The low power processor 212 can provide computing power to the IS control instruction group 230.

[0078] The image data interface circuit unit 214 may provide communication with the camera unit 250 , and the general interface circuit unit 215 may provide communication with peripheral devices of the electronic device 200 .

[0079] The camera unit 250 may include a camera image sensor 251 , a camera memory 252 , and an IS driver 253 .

[0080] The camera image sensor 251 may acquire an image by receiving light.

[0081] The camera image sensor 251 may be connected to the image data interface circuit unit 214 through a camera serial interface (CSI) channel to provide the acquired image to the general processor 211 through the image data interface circuit unit 214 .

[0082] The camera memory 252 may store IS adjustment values ​​for calibrating the IS driver 253 .

[0083] Upon receiving the IS result, the IS driver 253 may correct the shake of the camera unit 250 by physically adjusting at least one lens (not shown).

[0084] The motion sensor unit 260 may sense motion and output motion information.

[0085] The motion sensor unit 260 may output motion information through a communication channel (I3C interface).

[0086] More specifically, the motion sensor unit 260 may include a motion sensor 261 and a sensor data module 262 .

[0087] The motion sensor 261 may be an element configured to sense motion of the electronic device 200 and output the sensed motion information as sensor data, and may include a 6-axis sensor having a 3-axis gyroscope and a 3-axis accelerometer.

[0088] The sensor data module 262 may process the sensor data sensed by the motion sensor 261 into motion information and provide the motion information to the sensor interface circuit unit 213 of the low power processor 212 .

[0089] In this case, the motion information may be transmitted through an I3C interface connected between the sensor data module 262 and the sensor interface circuit unit 213. The motion information may be used as UI motion information or IS motion information, and may need to be further processed to be used as IS motion information.

[0090] The memory 220 may store an operating system, application programs, and data to be stored (e.g., phone numbers, SMS messages, compressed image files, videos, etc.) for controlling the overall operation of the electronic device 200. In addition, the memory 220 may store an IS control instruction group 230 and a UI motion sensor instruction 240 using the general processor 211 according to the present disclosure.

[0091] The low power processor 212 may drive the polling timer to perform IS at preset intervals by executing instructions included in the IS control instruction group 230 , obtain motion information, generate an IS result according to the motion information, and provide the IS result to the IS driver 253 .

[0092] More specifically, the IS control instruction group 230 may include a polling timer instruction 231 , a communication interface control instruction 232 , an IS drive control instruction 233 , an IS motion sensor control instruction 234 , an IS sensor data processing instruction 235 , and a sensor input / output data control instruction 236 .

[0093] The polling timer instruction 231 may be an element configured to perform IS at preset intervals, and may be an instruction for verifying whether a preset interval expires in the polling timer if the camera is continuously turned on after the shake is corrected.

[0094] The polling timer instruction 231 may be an instruction for restarting the preset interval when the preset interval expires so that the IS control instruction group 230 can repeat the correction of the jitter.

[0095] The polling timer instruction 231 may be an instruction for waiting for a preset time until the preset interval expires if the preset interval has not expired.

[0096] The communication interface control instruction 232 may be an instruction for connecting a first virtual communication interface between the memory 220 and the general processor 211 to receive the IS adjustment value. In this case, the first virtual communication interface may be a virtual communication interface simulating the operation of an I2C interface.

[0097] In addition, the communication interface control instruction 232 may be an instruction for connecting a second virtual communication interface between the memory 220 and the general processor 211 using the general processor 211, in a case where the video-specific image quality correction mode is executed in the video shooting mode, to obtain information for setting the IS intensity according to the ambient lighting, verify the IS intensity based on the information for setting the IS intensity, and provide the IS intensity to the low-power processor 212 that executes the IS drive control instruction 233. In this case, the second virtual communication interface may be a virtual communication interface that simulates the operation of the I2C interface.

[0098] The IS motion sensor control instruction 234 may be an instruction for obtaining motion information from the motion sensor unit 260 through the sensor interface circuit unit 213 of the low power processor 212 using the low power processor 212 .

[0099] The IS sensor data processing instructions 235 may be instructions for using the low power processor 212 to receive IS motion attribute setting information required to convert motion information into IS motion information to be used for IS from the general processor 211, and to process the motion information into IS motion information based on the IS motion attribute setting information.

[0100] The sensor input / output data control instruction 236 may be an instruction for controlling to postpone obtaining motion information with a low priority until the transmission of the motion information according to the priority is completed, in the case where a data conflict is detected in obtaining motion information based on the IS control instruction group using the low power processor 212. In this case, in the case where the camera is driven at the same time while processing a UI with motion information as in augmented reality, or in the case where IS of the camera is executed at the same time, there may be a high possibility of data conflict in the motion information.

[0101] That is, the sensor input / output data control instruction 236 may be an instruction for controlling to postpone obtaining motion information with a low priority when the IS motion sensor control instruction 234 and the UI motion sensor instruction 240 are intended to obtain motion information at a similar time point (before another data request ends).

[0102] In this case, the priority can be set so that the element that is intended to obtain (read) the motion information first can have a higher priority. Since the IS motion sensor control instruction 234 requires motion information faster than the UI motion sensor instruction 240, in the event of a data conflict, the motion information requested by the IS motion sensor control instruction 234 can be set to have a higher priority.

[0103] The IS sensor data processing instructions 235 may be instructions for using the low power processor 212 to receive IS motion attribute setting information required to convert motion information into IS motion information to be used for IS from the general processor 211, and to process the motion information into IS motion information based on the IS motion attribute setting information.

[0104] The IS drive control instruction 233 may be an instruction for receiving an IS adjustment value stored in the camera unit 250 through the first virtual communication interface using the general processor 211, initializing the IS based on the IS adjustment value using the low power processor 212, generating an IS result corresponding to the IS motion information by executing the IS algorithm using the low power processor 212, and providing the IS result to the IS driver 253 through the sensor interface circuit unit 213 using the low power processor 212. The IS drive control instruction 233 may be an instruction for performing calibration using the IS adjustment value while executing the IS algorithm.

[0105] The IS driving control instruction 233 may be an instruction for generating an IS by taking into account the IS intensity when generating an IS result when a video-specific image quality correction mode is executed in a video shooting mode.

[0106] At the same time, the IS control instruction group 230 is shown to be divided into polling timer instructions 231, communication interface control instructions 232, IS drive control instructions 233, IS motion sensor control instructions 234, IS sensor data processing instructions 235 and sensor input / output data control instructions 236 to describe their functions respectively.

[0107] Hereinafter, the method according to the present disclosure configured as described above will be described with reference to the accompanying drawings.

[0108] Figure 3 is a flowchart illustrating a process in which an electronic device controls an OIS of a camera according to an embodiment of the present disclosure.

[0109] refer to Figure 3 In the case where the anti-shake start event occurs in operation 310, the electronic device 100, 200 may store the IS control instruction group 130, 230 in the memory 120, 220 using the general processor 111, 211 in operation 312. In this case, the anti-shake start event may occur immediately before the camera unit 150, 250 operates when the camera application is executed. However, the embodiment is not limited thereto, and the anti-shake start event may occur when the camera application is executed or when the camera unit 150, 250 starts operating.

[0110] In operation 314, the electronic device 100, 200 may receive the IS adjustment value using the general purpose processor 111, 211. Figure 4 The operation 314 of receiving an IS adjustment value is further described.

[0111] In operation 316 , the electronic device 100 , 200 may use the low power processor 112 , 212 to initialize the IS control instruction group 130 , 230 based on the IS adjustment value.

[0112] Regarding the initialization in operation 316, Figure 1 The electronic device 100 and Figure 2 The electronic device 200 may operate differently.

[0113] In operation 316, Figure 1 The electronic device 100 may use the low power processor 112 to initialize the IS control instruction group based on the IS adjustment value.

[0114] In operation 316, Figure 2 The electronic device 200 may use the low power processor 212 to initialize the IS control instruction group based on the IS adjustment value and the IS motion attribute setting information.

[0115] In this case, the IS motion attribute setting information may be information required to convert motion information sensed by the motion sensor unit 260 into IS motion information to be used for IS, and may be received using the general processor 211 .

[0116] If initialization by calibration using the IS adjustment value is not required, then it can be obtained from Figure 3 The process omits operations 314 and 316.

[0117] In operation 318 , the electronic device 100 , 200 may use the low power processor 112 , 212 to drive the polling timer to perform IS at preset intervals by executing the polling timer instructions 131 , 231 included in the IS control instruction group 130 , 230 .

[0118] In operation 320 , the electronic device 100 , 200 may correct the shake of the camera unit 150 , 250 based on the IS control instruction group 130 , 230 using the low power processor 112 or 212 .

[0119] In operation 322 , the electronic device 100 , 200 may verify whether an anti-shake end event occurs.

[0120] In this case, the anti-shake end event may occur immediately before the operation of the camera unit 150, 250 stops as the camera application ends. However, the embodiment is not limited thereto, and the anti-shake end event may occur when the camera application ends or when the operation of the camera unit 150, 250 stops.

[0121] When an anti-shake end event occurs according to the result of verification in operation 322 , the electronic device 100 , 200 may delete the IS control instruction group 130 , 230 from the memory 120 , 220 using the general processor 111 , 211 in operation 324 .

[0122] In a case where the anti-shake end event does not occur according to the result of the verification in operation 322 , in operation 326 , the electronic device 100 , 200 may verify whether a preset interval of the polling timer expires using the low power processor 112 , 212 .

[0123] In case the preset interval expires according to the result of verification in operation 326 , the electronic device 100 , 200 may return to operation 320 and repeat a series of operations of correcting the shake of the camera unit 150 , 250 using the low power processor 112 , 212 .

[0124] In a case where the preset interval has not expired according to the result of verification in operation 326 , in operation 328 , the electronic device 100 , 200 may use the low power processor 112 , 212 to wait for a preset time until the preset interval of the polling timer expires.

[0125] Figure 4 is a flowchart illustrating a process in which an electronic device receives an IS adjustment value according to an embodiment of the present disclosure.

[0126] refer to Figure 4 In operation 410 , the electronic device 100 , 200 may connect the first virtual communication interface to the general processor 111 , 211 to obtain an IS adjustment value to be used for calibration based on the IS control instruction group 130 , 230 using the general processor 111 , 211 .

[0127] In operation 412 , the electronic device 100 , 200 may receive the IS adjustment value stored in the camera unit 150 , 250 using the general processor 111 , 211 , and store the received IS adjustment value in the memory 120 , 220 via the first virtual communication interface.

[0128] In this case, the first virtual communication interface may be a virtual communication interface that simulates the operation of an I2C interface.

[0129] About Figure 3 operation 320 of correcting the shaking of the camera units 150, 250 during the provision of IS, Figure 1 The electronic device 100 and Figure 2 The electronic device 200 may operate differently.

[0130] The following will refer to Figure 5 and Figure 6 describe Figure 1 The electronic device 100 performs the operation 320, and the following will refer to Figure 7 and Figure 8 describe Figure 2 An embodiment of the electronic device 200 performing operation 320.

[0131] Figure 5 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a photo capturing mode when a motion sensor unit has two channels according to an embodiment of the present disclosure.

[0132] refer to Figure 5 , in operation 510, the electronic device 100 may verify whether a mode of the camera is a photo capturing mode.

[0133] In the case that the current mode of the camera is the photo capturing mode according to the result verified in operation 510, in operation 520, the electronic device 100 may use the low power processor 112 to obtain IS motion information from the motion sensor unit 160 through the sensor interface circuit unit 113 of the low power processor 112 based on the IS control instruction group 130. The obtaining of the IS motion information in operation 520 may be performed in a manner that the low power processor 112 reads the IS motion information from the motion sensor unit 160 through the sensor interface circuit unit 113 at preset intervals (e.g., 1 / 6.6K seconds).

[0134] In operation 522 , the electronic device 100 may generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group 130 using the low power processor 112 .

[0135] In operation 524 , the electronic device 100 may provide the IS result to the IS driver 153 included in the camera unit 150 through the sensor interface circuit unit 113 based on the IS control instruction group 130 using the low power processor 112 so that the IS driver 153 may perform IS of the camera unit 150 .

[0136] Figure 6 is a flowchart illustrating a detailed process in which an electronic device controls an OIS of a camera in a video shooting mode when a motion sensor unit has two channels according to an embodiment of the present disclosure.

[0137] refer to Figure 6 , in operation 610, the electronic device 100 may verify whether a mode of the camera is a video shooting mode.

[0138] In a case where the current mode of the camera is the video photographing mode according to the result of the verification in operation 610 , the electronic device 100 may verify whether the current video photographing mode is a video-exclusive image quality correction mode in operation 612 .

[0139] When the current video shooting mode is the video-specific image quality correction mode according to the result verified in operation 612 , in operation 614 , the electronic device 100 may use the general processor 111 to connect a second virtual communication interface between the memory 120 and the general processor 111 based on the IS control instruction group 130 .

[0140] In operation 616 , the electronic device 100 may use the low power processor 112 to obtain information for setting the IS intensity according to the ambient lighting from the general processor 111 through the second virtual communication interface based on the IS control instruction group 130 .

[0141] In operation 618 , the electronic device 100 may verify the IS strength applied by the information for setting the IS strength based on the IS control instruction group 130 using the low power processor 112 .

[0142] In operation 620, the electronic device 100 may use the low power processor 112 to obtain IS motion information from the motion sensor unit 160 through the sensor interface circuit unit 113 of the low power processor 112 based on the IS control instruction group 130. The obtaining of the IS motion information in operation 620 may be performed in a manner that the low power processor 112 reads the IS motion information from the motion sensor unit 160 through the sensor interface circuit unit 113 at preset intervals (e.g., 1 / 6.6K seconds).

[0143] In operation 622 , the electronic device 100 may generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group 130 using the low power processor 112 .

[0144] In operation 624 , the electronic device 100 may provide the IS result to the IS driver 153 included in the camera unit 150 through the sensor interface circuit unit 113 based on the IS control instruction group 130 using the low power processor 112 so that the IS driver 153 may perform IS of the camera unit 150 .

[0145] When executing the IS algorithm in operation 624 , the electronic device 100 may execute the IS algorithm in consideration of the IS strength verified in operation 618 .

[0146] Meanwhile, in a case where the current video shooting mode is not the video-exclusive image quality correction mode according to the result verified in operation 612 , the electronic device 100 may not consider the IS strength but may perform operations 620 to 624 .

[0147] Figure 7 is a flowchart illustrating a detailed process of an electronic device controlling an OIS of a camera in a photo capturing mode when a motion sensor unit has one channel according to an embodiment of the present disclosure.

[0148] refer to Figure 7 , in operation 710, the electronic device 200 may verify whether a mode of the camera is a photo capturing mode.

[0149] In the case that the current mode of the camera is the photo capturing mode according to the result verified in operation 710, in operation 720, the electronic device 200 may use the low power processor 212 to obtain motion information from the motion sensor unit 260 through the sensor interface circuit unit 213 of the low power processor 212 based on the IS control instruction group 230. The obtaining of the IS motion information in operation 720 may be performed in a manner that the low power processor 212 reads the IS motion information from the motion sensor unit 260 through the sensor interface circuit unit 213 at preset intervals (e.g., 1 / 6.6K seconds).

[0150] In case the motion information is received in operation 720 , the electronic device 200 may detect whether a data conflict occurs in the motion information due to another element (eg, UI motion sensor instruction 240 ) based on the IS control instruction group 230 using the low power processor 212 in operation 722 .

[0151] When a data conflict is detected according to the result verified in operation 722, in operation 724, the electronic device 200 may use the low power processor 212 to control based on the IS control instruction group 230 to postpone obtaining motion information with a low priority until the transmission of the motion information according to the priority is completed.

[0152] In a case where no data conflict is detected according to the result of verification in operation 722 , the electronic device 200 may process the motion information into IS motion information to be used for IS based on the IS control instruction group 230 using the low power processor 212 in operation 726 .

[0153] In operation 728 , the electronic device 200 may generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group 230 using the low power processor 212 .

[0154] In operation 730 , the electronic device 200 may provide the IS result to the IS driver 253 included in the camera unit 250 through the sensor interface circuit unit 213 based on the IS control instruction group 230 using the low power processor 212 so that the IS driver 253 may perform IS of the camera unit 250 .

[0155] Figure 8 is a flowchart illustrating a detailed process in which an electronic device controls an OIS of a camera in a video shooting mode when a motion sensor unit has one channel according to an embodiment of the present disclosure.

[0156] refer to Figure 8 , in operation 810, the electronic device 200 may verify whether a mode of the camera is a video shooting mode.

[0157] In a case where the current mode of the camera is the video photographing mode according to the result of the verification in operation 810 , the electronic device 200 may verify whether the current video photographing mode is a video-exclusive image quality correction mode in operation 812 .

[0158] When the current video shooting mode is the video-specific image quality correction mode according to the result verified in operation 812 , in operation 814 , the electronic device 200 may use the general processor 211 to connect a second virtual communication interface between the memory 220 and the general processor 211 based on the IS control instruction group 230 .

[0159] In operation 816 , the electronic device 200 may use the low power processor 212 to obtain information for setting the IS intensity according to the ambient lighting from the general processor 211 through the second virtual communication interface based on the IS control instruction group 230 .

[0160] In operation 818 , the electronic device 200 may verify the IS strength applied by the information for setting the IS strength based on the IS control instruction group 230 using the low power processor 212 .

[0161] In operation 820, the electronic device 200 may use the low power processor 212 to obtain motion information from the motion sensor unit 260 through the sensor interface circuit unit 213 of the low power processor 212 based on the IS control instruction group 230. The obtaining of the IS motion information in operation 820 may be performed in a manner that the low power processor 212 reads the IS motion information from the motion sensor unit 260 through the sensor interface circuit unit 213 at preset intervals (e.g., 1 / 6.6K seconds).

[0162] In case the motion information is received in operation 820 , the electronic device 200 may detect whether a data conflict occurs in the motion information due to another element (eg, UI motion sensor instruction 240 ) based on the IS control instruction group 230 using the low power processor 212 in operation 822 .

[0163] When a data conflict is detected according to the result verified in operation 822, in operation 824, the electronic device 200 may use the low power processor 212 to control based on the IS control instruction group 230 to postpone obtaining motion information with a low priority until the transmission of the motion information according to the priority is completed.

[0164] In a case where no data conflict is detected according to the result of verification in operation 822 , the electronic device 200 may process the motion information into IS motion information to be used for IS based on the IS control instruction group 230 using the low power processor 212 in operation 826 .

[0165] In operation 828 , the electronic device 200 may generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group 230 using the low power processor 212 .

[0166] In operation 830 , the electronic device 200 may provide the IS result to the IS driver 253 included in the camera unit 250 through the sensor interface circuit unit 213 based on the IS control instruction group 230 using the low power processor 212 so that the IS driver 253 may perform IS of the camera unit 250 .

[0167] When executing the IS algorithm in operation 828 , the electronic device 200 may execute the IS algorithm in consideration of the IS strength verified in operation 818 .

[0168] Meanwhile, in a case where the current video shooting mode is not the video-exclusive image quality correction mode according to the result of verification in operation 812 , the electronic device 200 may not consider the IS strength but may perform operations 820 to 830 .

[0169] at the same time, Figure 1 The electronic device 100 or Figure 2 The electronic device 200 can be Fig. 9 The invention is realized in the form of an electronic device.

[0170] Fig. 9 is a block diagram illustrating an electronic device 901 in a network environment 900 according to an embodiment of the present disclosure.

[0171] refer to Fig. 9, the electronic device 901 in the network environment 900 may communicate with the electronic device 902 via a first network 998 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 904 or the server 908 via a second network 999 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 901 may communicate with the electronic device 904 via the server 908. According to another embodiment, the electronic device 901 may include a processor 920, a memory 930, an input module 950, a sound output module 955, a display module 960, an audio module 970, a sensor module 976, an interface 977, a connection terminal 978, a haptic module 979, a camera module 980, a power management module 988, a battery 989, a communication module 990, a subscriber identification module (SIM) 996, or an antenna module 997. In some embodiments, at least one component (e.g., the connection terminal 978) may be omitted from the electronic device 901, or one or more other components may be added to the electronic device 901. In some embodiments, some components (eg, sensor module 976, camera module 980, or antenna module 997) may be integrated into a single component (eg, display module 960).

[0172] The electronic device 901 may correspond to Figure 1 The electronic device 100 and Figure 2 electronic device 200.

[0173] The processor 920 may correspond to Figure 1 The mobile processor 110 or Figure 2 The mobile processor 210 is configured to:

[0174] The processor 920 may execute, for example, software (e.g., program 940) to control at least one other component (e.g., hardware or software component) of the electronic device 901 connected to the processor 920, and perform various data processing or calculations. According to another embodiment, as at least part of the data processing or calculation, the processor 920 may store a command or data received from another component (e.g., sensor module 976 or communication module 990) in the volatile memory 932, process the command or data stored in the volatile memory 932, and store the obtained data in the non-volatile memory 934. According to another embodiment, the processor 920 may include a main processor 921 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 923 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)), and the auxiliary processor 923 may operate independently of the main processor 921 or in conjunction with the main processor 921. For example, when the electronic device 901 includes a main processor 921 and an auxiliary processor 923, the auxiliary processor 923 is adapted to consume less power or be dedicated to a specified function than the main processor 921. The auxiliary processor 923 may be implemented separately from the main processor 921 or as a part of the main processor 921.

[0175] The main processor 921 may correspond to Figure 1 A general purpose processor 111 or Figure 1 and Figure 2 The general purpose processor 211, and the auxiliary processor 923 may correspond to Figure 1 A low power processor 112 or Figure 2 A low power processor 212 is provided.

[0176] The auxiliary processor 923 may replace the main processor 921 when the main processor 921 is in an inactive (e.g., sleep) state or control at least some of the functions or states related to at least one of the components of the electronic device 901 (e.g., display module 960, sensor module 976, or communication module 990) together with the main processor 921 when the main processor 921 is in an active state (e.g., executing an application). According to another embodiment, the auxiliary processor 923 (e.g., ISP or CP) may be implemented as a part of another component (e.g., camera module 980 or communication module 990) that is functionally related to the auxiliary processor 923. According to another embodiment, the auxiliary processor 923 (e.g., NPU) may include a hardware structure designated for artificial intelligence model processing. The artificial intelligence model may be generated by machine learning. Such learning may be performed by, for example, the electronic device 901 in which the artificial intelligence model is executed, or via a separate server (e.g., server 908). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks may include, for example, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), and bidirectional recurrent deep neural networks (BRDNNs), deep Q networks, or a combination of two or more thereof, but are not limited thereto. In addition to hardware structures, AI models may additionally or alternatively include software structures.

[0177] The memory 930 may store various data used by at least one component of the electronic device 901 (e.g., the processor 920 or the sensor module 976). The various data may include, for example, software (e.g., the program 940) and input data or output data for commands related thereto. The memory 930 may include a volatile memory 932 or a nonvolatile memory 934. The nonvolatile memory 934 may include an internal memory 936 and an external memory 938. The memory 930 may correspond to Figure 1 Memory 120 or Figure 2 Memory 220.

[0178] The program 940 may be stored as software in the memory 930 , and may include, for example, an operating system (OS) 942 , middleware 944 , or an application 946 .

[0179] The input module 950 may receive a command or data to be used by another component (e.g., the processor 920) of the electronic device 901 from outside (e.g., a user) of the electronic device 901. The input module 950 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0180] The sound output module 955 can output sound signals to the outside of the electronic device 901. The sound output module 955 may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing records. The receiver may be used to receive incoming calls. According to another embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0181] The display module 960 may visually provide information to the outside of the electronic device 901 (e.g., a user). The display module 960 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to another embodiment, the display module 960 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0182] The audio module 970 can convert sound into an electrical signal, and vice versa. According to another embodiment, the audio module 970 can obtain sound via the input module 950, or output sound via the sound output module 955 or an external electronic device (e.g., an electronic device 902 such as a speaker or earphone) directly or wirelessly connected to the electronic device 901.

[0183] The sensor module 976 may detect an operating state (e.g., power or temperature) of the electronic device 901 or an environmental state (e.g., a state of a user) outside the electronic device 901, and generate an electrical signal or data value corresponding to the detected state. According to another embodiment, the sensor module 976 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor. The sensor module 976 may be configured to include Figure 1 The motion sensor unit 160 or Figure 2 The motion sensor unit 260 is provided.

[0184] The interface 977 may support one or more designated protocols to be used for the electronic device 901 to be directly (e.g., wired) or wirelessly coupled to an external electronic device (e.g., the electronic device 902). According to another embodiment, the interface 977 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0185] The connection terminal 978 may include a connector, and the electronic device 901 may be physically connected to an external electronic device (e.g., the electronic device 902) via the connector. According to another embodiment, the connection terminal 978 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0186] The haptic module 979 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via his or her tactile or kinesthetic sense. According to another embodiment, the haptic module 979 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0187] The camera module 980 can capture still images and videos. According to another embodiment, the camera module 980 may include one or more lenses, image sensors, image signal processors, or flashes. The camera module 980 may be configured to include Figure 1 Camera unit 150 or Figure 2 The camera unit 250.

[0188] The power management module 988 may manage power supplied to the electronic device 901. According to another embodiment, the power management module 988 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0189] The battery 989 may supply power to at least one component of the electronic device 901. According to another embodiment, the battery 989 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0190] The communication module 990 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 901 and an external electronic device (e.g., electronic device 902, electronic device 904, or server 908), and perform communication via the established communication channel. The communication module 990 may include one or more communication processors that operate independently of the processor 920 (e.g., an application processor) and support direct (e.g., wired) communication or wireless communication. According to another embodiment, the communication module 990 may include a wireless communication module 992 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 994 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 998 (e.g., a short-range communication network such as Bluetooth). TMThe wireless communication module 992 may communicate with an external electronic device 904 using a wireless communication network (e.g., wireless fidelity (Wi-Fi) direct connection, or infrared data association (IrDA)) or a second network 999 (e.g., a long-range communication network such as a traditional cellular network, a fifth generation (5G) network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 992 may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the SIM 996 to identify and authenticate the electronic device 901 in a communication network (such as the first network 998 or the second network 999).

[0191] The wireless communication module 992 can support 5G networks and next-generation communication technologies after the fourth generation (4G) network, such as new radio (NR) access technology. NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC) or ultra-reliable and low-latency communication (URLLC). The wireless communication module 992 can support high-frequency bands (e.g., millimeter wave (mmWave) bands) to achieve, for example, high data transmission rates. The wireless communication module 992 can support various technologies for ensuring performance on high-frequency bands, such as beamforming, massive multiple-input multiple-output (MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming or massive antennas. The wireless communication module 992 can support various requirements specified in the electronic device 901, an external electronic device (e.g., electronic device 904) or a network system (e.g., a second network 999). According to an embodiment, the wireless communication module 992 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or higher), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0192] The antenna module 997 may send a signal or power to or receive a signal or power from the outside of the electronic device 901 (e.g., an external electronic device). According to another embodiment, the antenna module 997 may include an antenna including a radiating element including a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to another embodiment, the antenna module 997 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network such as the first network 998 or the second network 999 may be selected from a plurality of antennas by, for example, the communication module 990. A signal or power may be sent or received between the communication module 990 and the external electronic device via at least one selected antenna. According to another embodiment, another component other than the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 997.

[0193] According to another embodiment, the antenna module 997 may form a mmWave antenna module. According to another embodiment, the mmWave antenna module may include a PCB, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the PCB and capable of supporting a specified high frequency band (e.g., mmWave frequency band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the PCB and capable of transmitting or receiving signals in the specified high frequency band.

[0194] At least some of the above components can be coupled to each other and transmit signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0195] According to another embodiment, a command or data may be sent or received between the electronic device 901 and the external electronic device 904 via a server 908 coupled to the second network 999. Each of the external electronic devices 902 and 904 may be a device of the same type or a different type as the electronic device 901. According to another embodiment, all or some of the operations to be performed by the electronic device 901 may be performed at one or more of the external electronic devices 902 and 904 and the server 908. For example, if the electronic device 901 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 901 may request one or more external electronic devices to perform at least a portion of the function or service instead of performing the function or service, or in addition to performing the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or an additional function or additional service related to the request, and may transmit the result of the execution to the electronic device 901. The electronic device 901 may provide the result, with or without further processing of the result, as at least a part of the reply to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC) or client-server computing technology can be used. The electronic device 901 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 904 may include an Internet of Things (IoT) device. The server 908 may be an intelligent server using machine learning and / or neural networks. According to another embodiment, the external electronic device 904 or the server 908 may be included in the second network 999. The electronic device 901 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or health care) based on 5G communication technology or IoT-related technologies.

[0196] The method according to the above-described embodiment may be recorded in a non-transitory computer-readable medium, which includes program instructions for implementing various operations of the above-described embodiment. The medium may also include data files, data structures, etc., either alone or in combination with program instructions. The program instructions recorded on the medium may be program instructions specially designed and constructed for the purpose of the embodiment, or they may be types known and available to technicians in the field of computer software. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as compact disk read-only memory (CD-ROM) disks, digital versatile disks (DVDs) and / or Blu-ray discs; magneto-optical media, such as optical disks; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drive, memory card, memory stick, etc.). Examples of program instructions include both machine codes such as those generated by a compiler and files containing higher-level codes that can be executed by a computer using an interpreter. The above-mentioned device may be configured to be used as one or more software modules to perform the operations of the above-described embodiments, and vice versa.

[0197] Software may include computer programs, code segments, instructions, or some combination thereof to independently or consistently instruct or configure a processing device to operate as desired. Software and data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, computer storage medium or device, or in propagating signal waves capable of providing instructions or data to or interpreted by a processing device. Software may also be distributed on network-coupled computer systems so that the software is stored and executed in a distributed manner. Software and data may be stored by one or more non-transitory computer-readable recording media.

[0198] A number of embodiments have been described above. However, it should be understood that various modifications may be made to these embodiments. For example, suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0199] Accordingly, other implementations, other embodiments, and equivalents of the claims are within the scope of the appended claims.

[0200] According to another embodiment, an OIS method may include: when an anti-shake start event occurs, using a general processor to store an IS control instruction group in a memory; using a low-power processor to drive a polling timer to perform IS at a preset interval by executing a polling timer instruction included in the IS control instruction group; and using the low-power processor to correct the shake of the camera unit based on the IS control instruction group.

[0201] According to another embodiment, the OIS method may further include: when an anti-shake end event does not occur after correcting the shake of the camera unit, verifying whether a preset interval of a polling timer has expired using a low power processor; when the preset interval of the polling timer expires, repeatedly correcting the shake of the camera unit using the low power processor; and when the preset interval of the polling timer has not expired, waiting for a preset time using the low power processor until the preset interval of the polling timer expires.

[0202] According to another embodiment, the OIS method may further include deleting the IS control instruction group from the memory using a general processor when an anti-shake end event occurs.

[0203] According to another embodiment, the OIS method may further include: before driving the polling timer, using a general processor to connect a first virtual communication interface between the memory and the general processor to obtain an IS adjustment value; using the general processor to receive the IS adjustment value stored in the camera unit and storing the received IS adjustment value in the memory through the first virtual communication interface; and using the low-power processor to initialize the IS control instruction group based on the IS adjustment value.

[0204] According to another embodiment, in the OIS method, correcting the jitter of the camera unit based on the IS control instruction group may include: using a low-power processor to obtain IS motion information from the motion sensor unit through a sensor interface circuit unit of the low-power processor based on the IS control instruction group; using the low-power processor to generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group; and using the low-power processor to provide the IS result to an IS driver included in the camera unit through the sensor interface circuit unit based on the IS control instruction group.

[0205] According to another embodiment, in the OIS method, correcting the shake of the camera unit based on the IS control instruction group may also include: when a video-specific image quality correction mode is executed in the video shooting mode, using a general-purpose processor, connecting a second virtual communication interface between the memory and the general-purpose processor to obtain information for setting the IS intensity according to the ambient lighting; using a low-power processor, based on the IS control instruction group, obtaining information for setting the IS intensity from the general-purpose processor through the second virtual communication interface; and using the low-power processor, based on the IS control instruction group, verifying the IS intensity applied by the information for setting the IS intensity, wherein generating the IS result may include generating the IS result considering the IS intensity.

[0206] According to another embodiment, the OIS method may further include: before driving the polling timer, using a general processor, connecting a first virtual communication interface between the memory and the general processor to obtain an IS adjustment value; using the general processor, receiving the IS adjustment value stored in the camera unit and storing the received IS adjustment value in the memory through the first virtual communication interface; using the general processor, receiving IS motion attribute setting information required to convert motion information sensed by the motion sensor unit into IS motion information to be used for IS; and using a low-power processor, initializing the IS control instruction group based on the IS adjustment value and the IS motion attribute setting information.

[0207] According to another embodiment, in the OIS method, correcting the shake of the camera unit based on the IS control instruction group may include: using a low-power processor to obtain motion information from the motion sensor unit through a sensor interface circuit unit of the low-power processor based on the IS control instruction group; using the low-power processor to process the motion information into IS motion information to be used for IS based on the IS control instruction group; using the low-power processor to generate an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group; and using the low-power processor to provide the IS result to an IS driver included in the camera unit through the sensor interface circuit unit based on the IS control instruction group.

[0208] According to another embodiment, in the OIS method, correcting the shake of the camera unit based on the IS control instruction group may also include: when a data conflict is detected in obtaining motion information based on the IS control instruction group using a low-power processor, controlling to postpone obtaining motion information with a low priority until the transmission of the motion information according to the priority is completed.

[0209] According to another embodiment, an OIS device may include a camera unit, a motion sensor unit, a memory, and a mobile processor, the camera unit including an IS driver configured to correct shake according to an IS result, the motion sensor unit configured to sense motion and output UI motion information and IS motion information, the mobile processor including a general processor and a low-power processor, wherein the general processor may be configured to store an IS control instruction group in the memory when an anti-shake start event occurs, and the low-power processor may be configured to drive a polling timer to perform IS at preset intervals by executing instructions included in the IS control instruction group stored in the memory, obtain IS motion information, generate IS results according to the IS motion information, and provide the IS results to the IS driver.

[0210] According to another embodiment, in an OIS device, the IS control instruction group may include an IS motion sensor control instruction to use a low-power processor to obtain IS motion information from a motion sensor unit through a sensor interface circuit unit of the low-power processor, wherein the low-power processor may be configured to obtain IS motion information from the motion sensor unit based at least on the IS motion sensor control instruction.

[0211] According to another embodiment, in an OIS device, the IS control instruction group may include a polling timer instruction for using a low-power processor to drive a polling timer to perform IS at preset intervals, wherein the low-power processor may be configured to verify whether a preset interval of the polling timer has expired based on at least the polling timer instruction when an anti-shake end event does not occur after correcting the jitter; when the preset interval expires, restart the preset interval of the polling timer; and when the preset interval of the polling timer has not expired, wait for a preset time until the preset interval expires.

[0212] According to another embodiment, in an OIS device, an IS control instruction group may include a communication interface control instruction and an IS drive control instruction, the communication interface control instruction being used to connect a first virtual communication interface between a memory and the general-purpose processor using a general-purpose processor, the IS drive control instruction being used to receive an IS adjustment value stored in a camera unit through the first virtual communication interface using the general-purpose processor, initialize the IS based on the IS adjustment value using a low-power processor, generate an IS result corresponding to the IS motion information by executing an IS algorithm using the low-power processor, and provide the IS result to an IS driver through a sensor interface circuit unit using the low-power processor, wherein the general-purpose processor may be configured to connect a second virtual communication interface between the memory and the general-purpose processor when a video-specific image quality correction mode is executed in a video shooting mode, so as to obtain information for setting the IS intensity according to ambient lighting based on at least the communication interface control instruction, verify the IS intensity based on the information for setting the IS intensity, and provide the IS intensity to the low-power processor that executes the IS drive control instruction, and the low-power processor may be configured to consider the IS intensity when generating the IS result based on at least the IS drive control instruction.

[0213] According to another embodiment, in the OIS device, the general processor may be configured to delete the IS control instruction group from the memory when the anti-shake end event occurs.

[0214] According to another embodiment, in the OIS device, the motion sensor unit may be configured to output the UI motion information and the IS motion information through different communication channels.

[0215] According to another embodiment, an OIS device may include a camera unit, a motion sensor, a memory, and a mobile processor, the camera unit including an IS driver configured to correct shake according to an IS result, the motion sensor configured to sense motion and output motion information, the mobile processor including a general processor and a low-power processor, the general processor may be configured to store an IS control instruction group in the memory when an anti-shake start event occurs, and the low-power processor may be configured to drive a polling timer to perform IS at preset intervals by executing instructions included in the IS control instruction group stored in the memory, obtain motion information, generate an IS result according to the motion information, and provide the IS result to the IS driver.

[0216] According to another embodiment, in an OIS device, the IS control instruction group may include IS sensor data processing instructions for using a low-power processor to receive IS motion attribute setting information required to convert motion information into IS motion information to be used for IS from a general processor, and process the motion information into IS motion information based on the IS motion attribute setting information, wherein the low-power processor may be configured to receive the IS motion attribute setting information from the general processor based on at least the IS sensor data processing instructions, process the motion information into IS motion information based on the IS motion attribute setting information, and generate an IS result based on the IS motion information.

[0217] According to another embodiment, in an OIS device, the IS control instruction group may include a sensor input / output data control instruction for controlling to postpone obtaining motion information with a low priority until the transmission of motion information according to the priority is completed when a data conflict is detected in obtaining motion information based on the IS control instruction group using a low power processor, wherein the low power processor may be configured to postpone obtaining motion information with a low priority based on the sensor input / output data control instruction when a data conflict is detected in obtaining motion information until the transmission of motion information according to the priority is completed.

[0218] According to another embodiment, in an OIS device, an IS control instruction group may include a communication interface control instruction and an IS drive control instruction, the communication interface control instruction being used to connect a first virtual communication interface between a memory and the general processor using a general processor, the IS drive control instruction being used to receive an IS adjustment value stored in a camera unit through the first virtual communication interface using the general processor, initialize the IS based on the IS adjustment value using a low-power processor, generate an IS result corresponding to motion information by executing an IS algorithm using the low-power processor, and provide the IS result to an IS driver through a sensor interface circuit unit using the low-power processor, wherein the general processor may be configured to: when a video-specific image quality correction mode is executed in a video shooting mode, connect a second virtual communication interface between the memory and the general processor to obtain information for setting the IS intensity according to ambient lighting based on at least the communication interface control instruction, verify the IS intensity based on the information for setting the IS intensity, and provide the IS intensity to the low-power processor that executes the IS drive control instruction, and the low-power processor may be configured to consider the IS intensity when generating the IS result based on the IS drive control instruction.

[0219] According to another embodiment, in the OIS device, the motion sensor unit may be configured to output the motion information through a communication channel.

[0220] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. An optical image stabilization (OIS) method, comprising: storing an image stabilization (IS) control instruction set in a memory using a general purpose processor in the event of an anti-shake start event; Using a low power processor, driving a polling timer to perform IS at predetermined intervals by executing a polling timer instruction included in an IS control instruction group; and Using a low-power processor, camera shake is corrected based on the IS control instruction set.

2. The OIS method according to claim 1, further comprising: In the case where the anti-shake end event does not occur after correcting the camera shake, verifying, using the low power processor, whether a preset interval of the polling timer expires; Repeatedly correcting camera shake using a low power processor upon expiration of a preset interval of the polling timer; and In the case that the preset interval of the polling timer has not expired, the low power processor is used to wait for a preset time until the preset interval of the polling timer expires.

3. The OIS method according to claim 1, further comprising: When an anti-shake end event occurs, the IS control instruction group is deleted from the memory using a general purpose processor.

4. The OIS method according to claim 1, further comprising: Before driving the polling timer, Using a general purpose processor, connecting a first virtual communication interface between the memory and the general purpose processor to obtain an IS adjustment value; Using a general purpose processor, receiving an IS adjustment value stored in a camera through a first virtual communication interface and storing the received IS adjustment value in a memory; as well as Using a low power processor, an IS control instruction set is initialized based on the IS adjustment value.

5. The OIS method according to claim 1, wherein: Correcting camera shake based on the IS control command group includes: Using a low-power processor, based on an IS control instruction set, and obtaining IS motion information from a motion sensor through a sensor interface circuit of the low-power processor; Using a low power processor, generating an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group; and Using a low power processor, the IS result is provided to an IS driver included in the camera through the sensor interface circuit based on an IS control instruction group.

6. The OIS method according to claim 5, in, Correcting camera shake based on the control command group also includes: In the case where the video-specific image quality correction mode is executed in the video shooting mode, using the general-purpose processor, connecting a second virtual communication interface between the memory and the general-purpose processor to obtain information for setting the IS intensity according to the ambient lighting; Using the low-power processor, based on the IS control instruction group, obtaining information for setting IS strength from the general processor through the second virtual communication interface; and Using the low power processor, based on the IS control instruction set, verifying the IS strength applied by the information for setting the IS strength, and The generating of the IS result includes generating the IS result by taking into account the IS strength.

7. The OIS method according to claim 1, further comprising: Before driving the polling timer, Using a general purpose processor, connecting a first virtual communication interface between the memory and the general purpose processor to obtain an IS adjustment value; Using a general purpose processor, receiving an IS adjustment value stored in a camera through a first virtual communication interface and storing the received IS adjustment value in a memory; Using a general purpose processor, receiving IS motion attribute setting information required to convert motion information sensed by a motion sensor into motion information to be used for IS; as well as Using a low power processor, an IS control instruction group is initialized based on the IS adjustment value and the IS motion attribute setting information.

8. The OIS method according to claim 1, wherein: Correcting camera shake based on the IS control command group includes: Using a low power processor, based on an IS control instruction set, obtaining motion information from a motion sensor unit through a sensor interface circuit of the low power processor; Using a low power processor, based on an IS control instruction set, processing the motion information into IS motion information to be used for the IS; Using a low power processor, generating an IS result corresponding to the IS motion information by executing an IS algorithm based on the IS control instruction group; and Using a low power processor, the IS result is provided to an IS driver included in the camera through the sensor interface circuit based on an IS control instruction group.

9. The OIS method according to claim 8, wherein: Correcting camera shake based on the IS control instruction group also includes: when a data conflict is detected in obtaining motion information based on the IS control instruction group using a low-power processor, controlling to postpone obtaining motion information with a low priority until transmission of motion information according to the priority is completed.

10. An optical image stabilization (OIS) device, comprising: A camera including an image stabilization (IS) driver configured to correct shake based on an IS result; a motion sensor configured to sense motion and output user interface (UI) motion information and IS motion information; Memory; as well as Mobile processors, including general-purpose processors and low-power processors, The general processor is configured to store the IS control instruction group in the memory when an anti-shake start event occurs, and Wherein, the low power processor is configured as: The polling timer is driven to perform IS at preset intervals by executing instructions included in the IS control instruction group stored in the memory, Get IS movement information, Generate IS results based on IS movement information, and Provide the IS results to the IS driver.

11. The OIS device according to claim 10, in, The IS control instruction group includes an IS motion sensor control instruction to use a low power processor to obtain IS motion information from a motion sensor through a sensor interface circuit of the low power processor, and The low-power processor is further configured to obtain IS motion information from the motion sensor based at least on the IS motion sensor control instruction.

12. The OIS device according to claim 10, in, The IS control instruction group includes a polling timer instruction, the polling timer instruction is used to drive a polling timer to execute IS at a preset interval using a low power processor, and Wherein, the low-power processor is further configured as: In the case that the anti-shake end event does not occur after the jitter correction, verifying whether a preset interval of the polling timer has expired based on at least the polling timer instruction, In case a preset interval of the polling timer expires, restarting the preset interval, and When the preset interval of the polling timer has not expired, wait for a preset time until the preset interval expires.

13. The OIS device according to claim 10, in, The IS control instruction group includes: The communication interface control instruction is used to connect a first virtual communication interface between the memory and the general-purpose processor using the general-purpose processor, The IS drive control instruction is used to use a general processor to receive the IS adjustment value stored in the camera through the first virtual communication interface, Initialize the IS based on the IS adjustment value using a low power processor, generating an IS result corresponding to the IS motion information by executing the IS algorithm using a low power processor, and Using a low-power processor, the IS results are provided to the IS driver via the sensor interface circuit. Wherein, the general processor is further configured as: In the case where the video-specific image quality correction mode is executed in the video shooting mode, a second virtual communication interface is connected between the memory and the general processor to obtain information for setting the IS intensity according to the ambient lighting based on at least the communication interface control instruction, verifying the IS strength based on the information used to set the IS strength, and The IS strength is provided to a low power processor that executes the IS drive control instruction, and wherein the low power processor is further configured to consider the IS strength when generating an IS result based at least on the IS drive control instruction.

14. The OIS device according to claim 10, wherein: The general processor is further configured to delete the IS control instruction group from the memory when an anti-shake end event occurs.

15. The OIS device according to claim 10, wherein: The motion sensor is further configured to output the UI motion information and the IS motion information through different communication channels.

Images