Imaging Device, Control Method, and Storage Medium

Abstract

The present disclosure provides an electronic device, a control method, and a storage medium. The electronic device includes: an acquisition unit configured to acquire first information regarding jitter; a calculation unit configured to input the first information into a machine learning model and output second information regarding a jitter type; and a first control unit configured to control image stabilization using the second information. The first control unit controls image stabilization after giving the shooting instruction by using the second information output from the calculation unit based on the first information before giving the shooting instruction.

Description

Technical Field

[0001] Aspects of the embodiments relate to an electronic device used in an imaging system having an image stabilization function, and a control method for controlling the electronic device. Background Art

[0002] There are cases where an electronic device having a photographing function such as a camera or a smartphone includes an image stabilization function for reducing image blurring caused by camera shake. As control of the image stabilization function, the following techniques are known: determining a photographing state based on an angular velocity signal or an acceleration signal, and calculating an image stabilization signal for performing image stabilization according to the determination result.

[0003] Japanese Patent No. 5743838 discusses an image stabilization device that distinguishes a stationary photographing state and a walking photographing state based on the time elapsed since an angular velocity signal exceeded a threshold value.

[0004] However, in the image stabilization device discussed in Japanese Patent No. 5743838, there is room for further improvement in the determination accuracy of the photographing state. Summary of the Invention

[0005] According to an aspect of an embodiment, there is provided an electronic device including: an acquisition unit configured to acquire first information regarding shake; a calculation unit configured to input the first information into a machine learning model and output second information regarding a shake type; and a first control unit configured to control image stabilization using the second information. The first control unit controls image stabilization after a photographing instruction is given by using the second information output from the calculation unit based on the first information before the photographing instruction is given.

[0006] According to another aspect of an embodiment, there is provided an electronic device used in a system including a first device and a second device having a control unit configured to communicate with the first device and configured to control image stabilization after a photographing instruction is given by using information regarding a shake type before the photographing instruction is given. The electronic device includes: an acquisition unit configured to acquire first information regarding shake occurring in at least one of the first device and the second device; a calculation unit configured to input the first information into a machine learning model and output information regarding a shake type; and a communication unit configured to send, to the second device, information regarding a shake type output from the calculation unit using the first information before the photographing instruction is given.

[0007] According to another aspect of the embodiment, a method is provided, which includes the following steps: obtaining first information about jitter; inputting the first information into a machine learning model and outputting second information about the type of jitter; and using the second information to control image stabilization. In the control step, the image stabilization after giving the shooting instruction is controlled by using the second information output based on the first information before giving the shooting instruction.

[0008] Other features of the present disclosure will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram showing an imaging system.

[0010] Figure 2 is a block diagram showing the functions of a microprocessing unit (MPU) included in an imaging device.

[0011] Figure 3 is a flowchart showing a shooting operation according to a first exemplary embodiment.

[0012] Figure 4 is a conceptual diagram showing a machine learning model.

[0013] Figure 5 is a graph showing the determination accuracy of the type of jitter.

[0014] Figure 6 is a flowchart showing a determination process for determining the type of jitter.

[0015] Figure 7 is a flowchart showing an image stabilization process.

[0016] Figure 8 is a flowchart showing a shooting operation according to a second exemplary embodiment.

[0017] Figure 9 is a flowchart showing a shooting operation according to a third exemplary embodiment.

[0018] Figure 10 is a flowchart showing a shooting operation according to a fourth exemplary embodiment.

[0019] Figure 11 is a block diagram showing the functions of an MPU included in a lens device.

[0020] Figure 12 is a flowchart showing a shooting operation according to a fifth exemplary embodiment.

[0021] Figure 13A and Figure 13B is a diagram showing a smart phone.

[0022] Figure 14 is a block diagram showing a camera unit and an MPU included in a smartphone.

[0023] Figure 15 is a block diagram showing the functions of the MPU included in a smartphone.

[0024] Figure 16 is a diagram showing a display unit when a camera application for a smartphone is started.

[0025] Figure 17 is a flowchart showing a camera operation according to a sixth exemplary embodiment. Detailed Description of the Invention

[0026] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 is a block diagram showing an imaging system including an imaging device 200 and a lens device 100 capable of communicating with the imaging device 200. In the first exemplary embodiment, the imaging device 200 has the functions of an electronic device according to the present disclosure.

[0028] The imaging device 200 includes an image sensor 201, a signal processing circuit 202, a recording processing unit 203, a display unit 204, an operation unit 205, a camera microprocessing unit (MPU) 206, and a camera communication unit 207. The imaging device 200 further includes a camera angular velocity sensor 208, a camera acceleration sensor 209, an image sensor driving motor 210, a camera image stabilization control unit 211, and an image sensor encoder 212.

[0029] The image sensor 201 photoelectrically converts a subject image (optical image) formed by an imaging optical system in the lens device 100 into an electrical signal (analog signal) and outputs the analog signal. The image sensor 201 is composed of, for example, a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. The analog signal output from the image sensor 201 is converted into a digital signal by an analog-to-digital (A / D) conversion circuit (not shown).

[0030] The signal processing circuit 202 performs various types of image processing on the digital signal from an A / D conversion circuit (not shown), thereby generating a video signal. The signal processing circuit 202 also generates focus information indicating the focus state of the imaging optical system and luminance information indicating the exposure state of the imaging optical system. The signal processing circuit 202 also outputs the video signal to the display unit 204, and the display unit 204 displays the video signal as a live view image for confirming the composition or focus state. The signal processing circuit 202 also outputs the video signal to the recording processing unit 203. The recording processing unit 203 stores the video signal as image data in an external memory.

[0031] The operation unit 205 is composed of, for example, a mode dial for setting various shooting modes and a shutter release button for performing shooting preparation operations and giving an instruction to start shooting. If a first shutter release signal (Sw1) generated by the photographer half-pressing the shutter release button is input to the camera MPU 206, the lens device 100 performs shooting preparation operations such as autofocus. If the photographer fully presses the shutter release button, a second shutter release signal (Sw2) is generated. Generation and recording of a still image (image signal) are performed by inputting the second shutter release signal Sw2 to the camera MPU 206. That is, the full press of the shutter release button corresponds to giving a shooting instruction.

[0032] The camera MPU 206 controls the imaging device 200. The camera MPU 206 also communicates with the lens device 100 via the camera communication unit 207 and exchanges data with the lens device 100.

[0033] Although the camera MPU 206 is an MPU in the present exemplary embodiment, a circuit or various processors such as a central processing unit (CPU) may be used.

[0034] The camera angular velocity sensor 208 outputs an angular velocity signal related to the angular velocity of the imaging device 200. The camera acceleration sensor 209 outputs an acceleration signal related to the acceleration of the imaging device 200. The angular velocity signal output from the camera angular velocity sensor 208 and the acceleration signal output from the camera acceleration sensor 209 are input to the camera MPU 206 as camera shake detection signals.

[0035] The camera MPU 206 uses a camera shake detection signal to calculate a drive target signal for driving the image sensor 201. An image stabilization signal based on the difference between the drive target signal and the position signal of the image sensor 201 output from the image sensor encoder 212 is output to the camera image stabilization control unit 211. The camera image stabilization control unit 211 drives the image sensor drive motor 210 based on the image stabilization signal, thereby moving the image sensor 201 in a direction orthogonal to the optical axis of the imaging optical system in the lens device 100. This movement of the image sensor 201 reduces image blurring caused by shake of the imaging system.

[0036] The imaging device 200 may include only either the camera angular velocity sensor 208 or the camera acceleration sensor 209. However, when the imaging device 200 includes both the camera angular velocity sensor 208 and the camera acceleration sensor 209, the imaging device 200 can obtain more information about the camera shake detection signal compared to the case where the imaging device 200 includes only either one. As a result, image stabilization can be performed with high precision and the shake type can be determined. Therefore, in one embodiment, the imaging device 200 should include both the camera angular velocity sensor 208 and the camera acceleration sensor 209.

[0037] The lens device 100 includes an imaging optical system, a drive motor for driving the imaging optical system, and a control circuit for controlling the drive motor. The lens device 100 further includes a lens MPU 101, a lens communication unit 102, a lens angular velocity sensor 103, a lens acceleration sensor 104, and an image stabilization lens encoder 105.

[0038] The imaging optical system includes an image stabilization lens 106 and a focusing lens 107. The drive motor for driving the imaging optical system includes an image stabilization lens drive motor 108 and a focusing lens drive motor 109. The control circuit for controlling the drive motor includes a lens image stabilization control unit 110 and a focusing lens control unit 111.

[0039] The lens MPU 101 controls the operation of components in the lens device 100. The lens MPU 101 also communicates with the imaging device 200 via the lens communication unit 102 and exchanges data with the imaging device 200. Although the lens MPU 101 is an MPU in the present exemplary embodiment, various circuits or processors such as a CPU can be used.

[0040] The lens angular velocity sensor 103 outputs an angular velocity signal related to the angular velocity of the lens device 100. The lens acceleration sensor 104 outputs an acceleration signal related to the acceleration of the lens device 100. The angular velocity signal output from the lens angular velocity sensor 103 and the acceleration signal output from the lens acceleration sensor 104 are input to the lens MPU 101 as lens shake detection signals. The lens MPU 101 uses the lens shake detection signals to calculate a drive target signal for driving the image stabilization lens 106. An image stabilization signal based on the difference between the position signal of the image stabilization lens 106 output from the image stabilization lens encoder 105 and the drive target signal is output to the lens image stabilization control unit 110. The lens image stabilization control unit 110 drives the image stabilization lens drive motor 108 based on the image stabilization signal, so that the image stabilization lens 106, which is part of the imaging optical system, moves in a direction orthogonal to the optical axis of the imaging optical system. This movement of the image stabilization lens 106 reduces image blurring caused by the shake of the imaging system.

[0041] The focus lens control unit 111 uses the control signal received from the lens MPU 101 and the focus lens drive motor 109 to move the focus lens 107 in the optical axis direction, thereby adjusting the focus. The focus lens control unit 111 includes: a focus encoder and a circuit for driving the focus lens 107, and the focus encoder outputs an area pattern signal or a pulse signal according to the movement of the focus lens 107. The subject distance can be detected based on the output of the focus encoder.

[0042] The lens shake detection signal output from at least one of the lens angular velocity sensor 103 and the lens acceleration sensor 104 can be input to the camera MPU 206 instead of the camera shake detection signal. The camera MPU 206 obtains the lens shake detection signal from the lens MPU 101 via the camera communication unit 207. In this case, the camera angular velocity sensor 208 and the camera acceleration sensor 209 may not be provided in the imaging device 200.

[0043] Now, reference will be made to Figure 2 Describe the functions of the camera MPU 206. In the present exemplary embodiment, the functions of the camera MPU 206 use information about the camera shake detection signal to determine the shake type.

[0044] A machine learning model is also used to determine the shake type to improve the determination accuracy of the shake type.

[0045] The camera MPU 206 includes a camera acquisition unit 206a and a camera calculation unit 206b.

[0046] The camera acquisition unit 206a acquires first information regarding shake. In the present exemplary embodiment, the camera acquisition unit 206a acquires a feature amount of a preprocessed camera shake detection signal as the first information.

[0047] The camera calculation unit 206b inputs the first information acquired by the camera acquisition unit 206a into a machine learning model, and outputs information regarding the shake type as second information.

[0048] However, generally, processing using a machine learning model requires a high computational load. Therefore, in the case of using a machine learning model to determine the shake type, the determination process of the camera calculation unit 206b for determining the shake type may take longer than conventional methods. In this case, a delay may occur during the period from when the user gives a shooting instruction through the operation unit 205 during the shooting of a still image to when the actual shooting of an image starts. This is because the camera image stabilization control unit 211 performs image stabilization after waiting for the determination process of the camera calculation unit 206b for determining the shake type to complete.

[0049] Therefore, in the present exemplary embodiment, information regarding the shake type determined by the camera calculation unit 206b before the user gives a shooting instruction through the operation unit 205 is used for image stabilization performed after the user gives a shooting instruction. This eliminates the need for the camera image stabilization control unit 211 to wait for the determination process of the camera calculation unit 206b for determining the shake type to complete after the shooting instruction is given. As a result, even during the shooting of a still image, the shake type can be determined with high accuracy using a machine learning model, and the delay during the shooting of a still image can also be reduced.

[0050] Now, the imaging operation according to the first exemplary embodiment will be described with reference to Figure 3 the flowchart shown. If the imaging device 200 is powered on, then Figure 3 the flowchart in

[0051] In step S101, the imaging device 200 first communicates with the lens device 100 and differentiates the model of the lens device 100. At this time, the lens device 100 differentiates the model of the imaging device 200.

[0052] In step S102, the camera MPU 206 communicates the status with the lens MPU 101 via the camera communication unit 207. The camera MPU 206 also sends the status of the camera (e.g., the aperture value and shutter speed set by the user) to the lens MPU 101. The camera MPU 206 also receives the status of the lens (e.g., the current focal length, the status of the aperture, and the driving status of the focusing lens 107) from the lens MPU 101.

[0053] In step S103, the camera MPU 206 determines whether a first shutter release signal Sw1 is input from the operation unit 205 to the camera MPU 206. If it is determined that the first shutter release signal Sw1 is input (Yes in step S103), the process proceeds to step S104. If it is determined that the first shutter release signal Sw1 is not input (No in step S103), the process returns to step S102.

[0054] In step S104, the camera calculation unit 206b starts to determine the type of shake using a machine learning model. The camera calculation unit 206b records the information about the determined shake type in the camera MPU 206. Until step S109, the camera calculation unit 206b continuously determines the shake type using machine learning. The details of step S104 will be described below.

[0055] In step S105, the camera MPU 206 measures the distance for focusing on the subject and calculates the driving amount of the focusing lens 107. The camera MPU 206 sends the calculated driving amount of the focusing lens 107 to the focusing lens control unit 111.

[0056] In step S106, the camera MPU 206 measures the distance again. If it is determined that the distance is within the depth of focus (Yes in step S106), the process proceeds to step S107. If the distance is outside the depth of focus (No in step S106), the process returns to step S105.

[0057] In step S107, the camera MPU 206 determines whether a second shutter release signal Sw2 is input from the operation unit 205 to the camera MPU 206. If it is determined that the second shutter release signal Sw2 is input (Yes in step S107), the process proceeds to step S108. On the contrary, if it is determined that the second shutter release signal Sw2 is not input (No in step S107), the process returns to step S105.

[0058] In step S108, the camera image stabilization control unit 211 performs image stabilization. The image stabilization is performed based on the determination result of the shake type made by the camera calculation unit 206b before the second shutter release signal Sw2 is input to the camera MPU 206. Therefore, the time from when the second shutter release signal Sw2 is input to the camera MPU 206 to when the camera image stabilization control unit 211 starts image stabilization can be shortened. In step S108, an aperture (not shown) included in the lens device 100 is also driven, and a shutter (not shown) included in the imaging device 200 is also driven. Details of the processing related to image stabilization based on the determination result of the shake type performed in step S108 will be described below.

[0059] In step S109, the signal processing circuit 202 reads a signal from the image sensor 201. The read signal is converted into digital data, and the digital data is stored in an external memory (not shown) by the recording processing unit 203. The camera calculation unit 206b also stops the determination processing for determining the shake type.

[0060] Now, the machine learning model will be described. Figure 4 is a conceptual diagram showing the relationship between the input and output of the machine learning model according to the present exemplary embodiment. The machine learning model used in the present exemplary embodiment is an algorithm for outputting the class to which the input data belongs based on the input data.

[0061] The machine learning model has a learning phase and an estimation phase. In the learning phase, the parameters in the machine learning model are updated to improve the determination accuracy of the class of the input data. The "class" is associated with the shake type and is a predefined number of classes. The data obtained by combining the input data and the correct answer label indicating the class of the input data is called "teacher data". The learning is performed as follows. The teacher data is input to the machine learning model, and the machine learning model determines the class. Then, the machine learning model compares the obtained determined class with the class indicated by the correct answer label and updates the parameters in the machine learning model to minimize the difference (loss) between the classes. In the learning phase, the learning is performed a predetermined number of times. Alternatively, the learning is repeated until, for example, an end condition is satisfied by achieving a predetermined determination accuracy. Using the trained machine learning model thus obtained in the estimation phase, the camera calculation unit 206b can determine the shake type with high accuracy based on the input data.

[0062] In the present exemplary embodiment, a random forest is used as a machine learning model. A random forest is an algorithm that uses feature quantities as input data. Accordingly, the input data as the first information according to the present exemplary embodiment is the feature quantity of the preprocessed camera shake detection signal. As the algorithm of the machine learning model, logistic regression, support vector machine, naive Bayes, neural network, or state space model may be used.

[0063] The input data as the first information may be appropriately changed according to the algorithm. In this case, possible examples of the input data as the first information include only the preprocessed camera shake detection signal, the unprocessed camera shake detection signal, and the feature quantity of the unprocessed camera shake detection signal. Alternatively, information on the motion vector obtained from the image sensor 201 may be used as the first information instead of the camera shake detection signal.

[0064] For example, in the case where an algorithm that does not require a feature quantity (such as a neural network) is used as the machine learning model, the camera shake detection signal is used as the first information without generating a feature quantity. In this case, preprocessing of the camera shake detection signal is not necessary.

[0065] Now, the preprocessing and feature quantity of the camera shake detection signal according to the present exemplary embodiment will be described. The content of the preprocessing and the feature quantity may be appropriately changed according to the learning environment.

[0066] In the preprocessing, the camera shake detection signal is normalized, unit conversion is performed on the camera shake detection signal, and missing values and outliers in the camera shake detection signal are removed.

[0067] Normalization is a process of converting the camera shake detection signal so that the average value of the data is 0 and the variance of the data is 1. Unit conversion is a process of matching the resolution of the angular velocity signal and the acceleration signal as the camera shake detection signal. Normalization and unit conversion are processes performed to improve the learning accuracy by matching the scale of the camera shake detection signal.

[0068] Missing values are null data that appear due to detection errors in the camera angular velocity sensor 208 or the camera acceleration sensor 209. Outliers are data that are extremely different from the values of other data in the camera shake detection. Outliers and missing values cause a decrease in learning accuracy. Accordingly, the camera MPU 206 detects and excludes outliers and missing values from the shake detection signal, thereby preventing outliers and missing values from being input into the machine learning model.

[0069] Examples of the feature amount include the average value, maximum value, minimum value, root sum, variance, standard deviation, kurtosis, and skewness of the camera shake detection signal.

[0070] The output data as the second information according to the present exemplary embodiment is any shake type, such as a walking state, a tripod state, a panning state, a holding state, and other states. The categories to be output from the machine learning model can be a walking state, a tripod state, or a panning state, and the output result can be set as the shake type as it is. Alternatively, the categories can be defined as large shake, vertical shake, horizontal shake, and minute shake. Then, the large shake can be associated with the walking state. The vertical shake and the horizontal shake can be associated with the panning state. The minute shake can be associated with the tripod state. The shake types to be determined are not limited to the above five types, and the shake types can be added or changed.

[0071] Details of the learning phase of the machine learning model according to the present exemplary embodiment will now be described. As input data when performing learning, video recordings regarding a walking state, a tripod state, a panning state, a holding state, and other states are used.

[0072] In the walking state, images are taken under conditions of four moving directions (i.e., forward, backward, left, and right directions). The shooting time varies depending on the photographer.

[0073] In the tripod state, while changing the combination of three shooting angles (i.e., upward angle, horizontal angle, and downward angle) and two attitudes of the imaging device 200 (i.e., vertical attitude and horizontal attitude) in a state where the imaging device 200 is placed on a tripod, images are taken for 2 minutes and 30 seconds.

[0074] In the panning state, while changing the combination of two directions (i.e., left direction and right direction), two attitudes of the imaging device 200 (i.e., vertical attitude and horizontal attitude), and two methods for holding the imaging device 200 (i.e., hand - holding method and tripod method), images are taken for 1 minute.

[0075] In the holding state, while changing the combination of two attitudes (i.e., the state where the photographer stands and the state where the photographer leans on the elbow) and two shooting methods (i.e., live - view shooting and shooting when the photographer views the viewfinder), images are taken for 2 minutes. The holding state is a state where the photographer firmly holds up the camera, so even when performing hand - held shooting, the shake is relatively small.

[0076] In other states, with the imaging device 200 placed on a shake table (not shown) and generating vibrations, an image is captured for 2 minutes and 30 seconds. "Other states" refer to states that do not correspond to any of the walking state, tripod state, panning state, and holding state.

[0077] The input data is divided into learning data and test data. The test data is data used to confirm the determination accuracy of a trained machine learning model trained using the learning data. At this time, data of the same subject is classified into one of the learning data and the test data. This is to prevent the determination accuracy from being too high due to data of the same subject leading to correct answers.

[0078] Now, reference will be made to Figure 5 Describe the determination accuracy of the shake type using a machine learning model according to this exemplary embodiment. The determination accuracy of the shake type using the machine learning model has an accuracy rate of 100% in the walking state, 49% in the tripod state, 88% in the panning state, 99% in the holding state, and 100% in other states. The determination accuracy of the shake type, especially in the walking state, panning state, holding state, and other states, indicates a very high accuracy rate.

[0079] Now, reference will be made to Figure 6 the flowchart in Figure 3 to describe the method of determining the shake type using a machine learning model in step S104 in

[0080] In step S201, the camera MPU 206 obtains a camera shake detection signal from the camera angular velocity sensor 208 and the camera acceleration sensor 209.

[0081] In step S202, the camera MPU 206 reads the offset of the sensor from a non-volatile memory (not shown) and subtracts the value corresponding to the offset from the camera shake detection signal.

[0082] In step S203, the camera MPU 206 sets information about the camera shake detection signal from which the offset has been removed so that the information is input into the machine learning model as time-series data having a predetermined length. The appropriate length of the time-series data can be greater than or equal to 0.1 second and less than or equal to 5.0 seconds. If the time-series data exceeds 5.0 seconds, it takes too much time to obtain data for determining the shake type, which is not desirable. If the time-series data is less than 0.1 second, sufficient accuracy cannot be obtained, which is not desirable. In order to determine the shake type during the imaging time, in one embodiment, the length of the time-series data is greater than or equal to 0.1 second and less than or equal to 0.4 second.

[0083] In step S204, the camera MPU 206 preprocesses the camera shake detection signal.

[0084] In step S205, the camera MPU 206 generates a feature amount of the preprocessed camera shake detection signal. The camera acquisition unit 206a acquires the feature amount of the preprocessed camera shake detection signal.

[0085] In step S206, the camera calculation unit 206b inputs the feature amount of the preprocessed camera shake detection signal as first information into the machine learning model.

[0086] In step S207, the camera calculation unit 206b determines the shake type as second information based on the category output from the machine learning model.

[0087] Now, the processing of image stabilization based on the determination result of the shake type using the machine learning model in step S108 in Figure 7 will be described with reference to the flowchart in Figure 3 .

[0088] In step S401, the camera MPU 206 acquires the camera shake detection signal from the camera angular velocity sensor 208 and the camera acceleration sensor 209.

[0089] In step S402, the camera MPU 206 switches the processing of the camera shake detection signal based on the determination result of the shake type. If the shake type is the walking state, the processing proceeds to step S403. If the shake type is the tripod state, the processing proceeds to step S404. If the shake type is the panning state, the processing proceeds to step S405. If the shake type is the holding state, the processing proceeds to step S406. If the shake type is other states, the processing proceeds to step S407.

[0090] In step S403, the camera MPU 206 sets the cut-off frequency in the low-pass filter calculation of the camera shake detection signal such that the cut-off frequency is high when the value of the camera shake detection signal is greater than that in other states. This is because greater image blurring occurs in the walking state than in other states.

[0091] In step S404, the camera MPU 206 sets the cut-off frequency in the high-pass filter calculation of the camera shake detection signal such that the cut-off frequency is higher than that in other states. This is to prevent incorrect image stabilization due to sensor offset because low-frequency shakes are less likely to occur in the tripod state.

[0092] In step S405, the camera MPU 206 sends a driving signal to the camera image stabilization control unit 211. This driving signal is used to drive the image sensor 201 to gradually return to the central position of the optical axis over a predetermined time. As a result, the movement of the image sensor 201 can be stopped for image stabilization. This is to prevent the intentional movement of the photographer from being corrected as shake. The camera image stabilization control unit 211 can stop image stabilization in the panning direction and drive the image sensor 201 to perform image stabilization in a direction other than the panning direction.

[0093] In step S406, the camera MPU 206 detects a signal regarding low-frequency shake in the vector information obtained from the image sensor 201 and adds this signal to the position signal of the image sensor 201 output from the image sensor encoder 212. If it is determined that the shake type is the holding state, the image stabilization accuracy at low frequencies is improved more than in other states.

[0094] In step S407, the camera MPU 206 sets the cut-off frequency in the calculation of the low-pass filter for the camera shake detection signal such that the cut-off frequency is high when the value of the camera shake detection signal is less than the value in the walking state.

[0095] In step S408, the camera MPU 206 generates an image stabilization signal. The camera MPU 206 uses the camera shake detection signal that has undergone any of the processes in steps S403 to S407 to calculate a drive target signal for driving the image sensor 201. An image stabilization signal is generated based on the difference between the drive target signal and the position signal of the image sensor 201 output from the image sensor encoder 212.

[0096] In step S409, the camera image stabilization control unit 211 acquires the image stabilization signal from the camera MPU 206 and performs image stabilization.

[0097] As described above, according to the imaging operation described in the first exemplary embodiment of the present disclosure, a machine learning model can be used to accurately determine the shake type and reduce the delay when taking a still image.

[0098] In the present exemplary embodiment, the imaging device 200 performs image stabilization and determines the type of shake. However, in the present exemplary embodiment, the lens device 100 may perform image stabilization and determine the type of shake. In this case, the function for determining the type of shake is provided in the lens MPU 101 instead of the camera MPU 206. The lens device 100 uses the lens image stabilization control unit 110 instead of the camera image stabilization control unit 211 that controls image stabilization. The lens device 100 uses the image stabilization lens 106 instead of the image sensor 201 as the drive target member that corrects shake by moving in the optical axis direction of the imaging optical system. The lens device 100 uses the image stabilization lens drive motor 108 that drives the image stabilization lens 106 instead of the image sensor drive motor 210 that drives the image sensor 201. The lens device 100 uses the image stabilization lens encoder 105 that outputs the position signal of the image stabilization lens 106 instead of the image sensor encoder 212 that outputs the position signal of the image sensor 201.

[0099] In the present exemplary embodiment, time-series data having a first length and time-series data having a second length longer than the first length may be used as the input to the machine learning model to determine the type of shake. In this case, the camera calculation unit 206b determines the type of shake based on the time-series data having the first length and the type of shake based on the time-series data having the second length. Then, the camera image stabilization control unit 211 performs image stabilization using at least one of the type of shake based on the time-series data having the first length and the type of shake based on the time-series data having the second length.

[0100] In this case, the advantage of the determination of the type of shake based on the time-series data having the first length is that it takes a shorter time to acquire the data for the determination. However, the determination accuracy of the type of shake is relatively low. On the other hand, in the determination of the type of shake based on the time-series data having the second length, the estimation accuracy at the time of determining the type of shake is relatively high, but it takes a longer time to acquire the data for the determination. As described above, there is a trade-off relationship between the benefit of the determination of the type of shake based on the time-series data having the first length and the benefit of the determination of the type of shake based on the time-series data having the second length.

[0101] Therefore, it is preferable to determine the jitter type based on the timing data having a first length and determine the jitter type based on the timing data having a second length. Thereby, the jitter type can be determined with high precision, and the jitter type can also be determined based on the timing data having a first length during the acquisition period of the timing data having a second length. However, even after the jitter type based on the timing data having a second length is output, the jitter type based on the timing data having a first length can be used for image stabilization. This is to immediately switch the jitter type used for image stabilization when the jitter type changes. In one embodiment, the first length is greater than or equal to 0.1 second and less than or equal to 0.4 second, and the second length is greater than 0.4 second and less than or equal to 5.0 seconds. Alternatively, the timing data having a first length and the timing data having a second length can be input into different machine learning models, and the jitter type can be determined. In this case, a machine learning model that has been learned to be suitable for inputting the timing data having a first length and a machine learning model that has been learned to be suitable for inputting the timing data having a second length can be used.

[0102] In the present exemplary embodiment, optical image stabilization is performed by driving the image sensor 201 however, in the present exemplary embodiment, electronic image stabilization can be performed instead of optical image stabilization. In this case, the imaging device 200 does not need to include the image sensor driving motor 210, and thus, the space in the imaging device 200 is effectively saved.

[0103] Similar to optical image stabilization, electronic image stabilization is performed using a camera jitter detection signal. In this case, the camera MPU 206 can obtain a lens jitter detection signal from the lens angular velocity sensor 103 and the lens acceleration sensor 104 instead of the camera jitter detection signal. The camera MPU 206 calculates an image stabilization amount based on the camera jitter detection signal. Then, the image stabilization control unit 211 crops the frame image based on the image stabilization amount according to the image displayed on the display unit 204 or the image recorded in an external memory (not shown), thereby performing image stabilization.

[0104] In the present exemplary embodiment, vector information indicating the moving direction and moving speed of the subject can be used to perform image stabilization. The vector information is calculated from the image sensor 201. Then, the camera image stabilization control unit 211 moves the image sensor 201 in a direction orthogonal to the optical axis of the imaging optical system in the lens device 100 based on the vector information, thereby performing image stabilization.

[0105] Now, a second exemplary embodiment will be described.

[0106] In a first exemplary embodiment, the imaging device 200 determines a shake type and performs image stabilization. In contrast, in a second exemplary embodiment, the imaging device 200 determines a shake type, and the lens device 100 performs image stabilization. That is, the difference between the first exemplary embodiment and the second exemplary embodiment is that in the first exemplary embodiment, the same electronic device determines the shake type and performs image stabilization, while in the second exemplary embodiment, different electronic devices determine the shake type and perform image stabilization. Accordingly, in the imaging device 200 according to the present exemplary embodiment, the image sensor encoder 212, the image sensor drive motor 210, and the camera image stabilization control unit 211 may not be provided.

[0107] Now, the imaging operation according to the second exemplary embodiment will be described with reference to Figure 8 the flowchart in Figure 8 The processes of steps S501 to S504 in Figure 3 are similar to the processes of steps S101 to S104 in Figure 8 and thus will not be described. Figure 3 The processes of steps S506 to S508 in Figure 8 are similar to the processes of steps S105 to S107 in Figure 3 and thus will not be described.

[0108] In step S505, the camera MPU 206 sends information about the shake type determined by the camera calculation unit 206b to the lens MPU 101. At this time, the lens MPU 101 records the determination result of the shake type. Until step S510, the camera MPU 206 continuously sends information about the shake type to the lens MPU 101.

[0109] In step S509, the camera MPU 206 drives, for example, a shutter (not shown) included in the imaging device 200. At this time, an aperture (not shown) included in the lens device 100 is driven. In step S509, the lens image stabilization control unit 110 also performs image stabilization. The image stabilization is performed based on the determination result of the shake type made by the camera calculation unit 206b before the second shutter release signal Sw2 is input to the camera MPU 206. [[ID=—]]

[0110] As described above, according to the imaging operation in the case where image stabilization and determination of the shake type are performed by different electronic devices as described in the second exemplary embodiment of the present disclosure, a machine learning model can be used to determine the shake type with high accuracy and reduce the delay when shooting a still image.

[0111] In this exemplary embodiment, the imaging device 200 determines the shake type, and the lens device 100 performs image stabilization. However, in this exemplary embodiment, the lens device 100 may determine the shake type, and the imaging device 200 may perform image stabilization. In this case, the function for determining the shake type is provided in the lens MPU 101 rather than in the camera MPU 206. The imaging device 200 uses a camera image stabilization control unit 211 instead of the lens image stabilization control unit 110 that controls image stabilization. The imaging device 200 uses an image sensor 201 instead of the image stabilization lens 106, which corrects shake by moving in the optical axis direction of the imaging optical system. The imaging device 200 uses an image sensor drive motor 210 that drives the image sensor 201 instead of the image stabilization lens drive motor 108 that drives the image stabilization lens 106. The imaging device 200 uses an image sensor encoder 212 that outputs a position signal of the image sensor 201 instead of the image stabilization lens encoder 105 that outputs a position signal of the image stabilization lens 106.

[0112] A third exemplary embodiment will now be described.

[0113] In the first exemplary embodiment, even after the user gives the shooting instruction, the camera computing unit 206b puts a large computational load on determining the shake type. In contrast, in the third exemplary embodiment, after the user gives the still image shooting instruction, the camera computing unit 206b stops determining the shake type.

[0114] In this exemplary embodiment, the computational load after the user gives an instruction to shoot an image can be reduced. Therefore, in this exemplary embodiment, the delay from when a still image shooting instruction is given due to determination of the shake type to when an image is shot can be reduced accordingly compared to the first exemplary embodiment.

[0115] In this exemplary embodiment, the imaging device 200 performs image stabilization and determines the shake type. Therefore, the lens device 100 does not need to be provided with the image stabilization lens encoder 105, the image stabilization lens 106, the image stabilization lens drive motor 108, and the lens image stabilization control unit 110.

[0116] Now refer to Figure 9 The flowchart in describes an imaging operation according to the third exemplary embodiment of the present disclosure. Figure 9 The processing of steps S601 to S603 in Figure 3 The processing of steps S101 to S103 in is omitted and therefore will not be described again. Figure 9 The processing of steps S605 to S607 in Figure 3The processing of steps S105 to S107 in [reference] is not described again. Figure 9 The processing of steps S609 and S610 in [reference] is similar to Figure 3 the processing of steps S108 and S109 in [reference], and thus will not be described again.

[0117] In step S604, the camera calculation unit 206b starts to determine the type of shake using a machine learning model. The camera calculation unit 206b records information about the determined type of shake in the camera MPU 206. The camera calculation unit 206b continuously determines the type of shake using machine learning until the determination stops in step S608.

[0118] In step S608, the camera calculation unit 206b stops determining the type of shake. Thus, the computational load in the camera calculation unit 206b can be reduced.

[0119] As described above, according to the imaging operation described in the third exemplary embodiment of the present disclosure, the determination of the type of shake is stopped after a shooting instruction is given, whereby the delay when shooting a still image can be reduced to a greater extent compared with the first exemplary embodiment.

[0120] In the present exemplary embodiment, the imaging device 200 performs image stabilization and determines the type of shake. However, in the present exemplary embodiment, the lens device 100 can perform image stabilization and determine the type of shake. In this case, a function for determining the type of shake is provided in the lens MPU 101 instead of the camera MPU 206. In step S607, the lens MPU 101 obtains information about the given shooting instruction from the camera MPU 206 and stops determining the type of shake.

[0121] The lens device 100 uses the lens image stabilization control unit 110 instead of the camera image stabilization control unit 211 that controls image stabilization. The lens device 100 uses the image stabilization lens 106 instead of the image sensor 201 as a driving target member that corrects shake by moving in the optical axis direction of the imaging optical system. The lens device 100 uses the image stabilization lens driving motor 108 that drives the image stabilization lens 106 instead of the image sensor driving motor 210 that drives the image sensor 201. The lens device 100 uses the image stabilization lens encoder 105 that outputs the position signal of the image stabilization lens 106 instead of the image sensor encoder 212 that outputs the position signal of the image sensor 201.

[0122] The present exemplary embodiment can be applied to the configuration in the case where the determination of the type of shake and image stabilization are performed by different electronic devices as described in the second exemplary embodiment.

[0123] Now, a fourth exemplary embodiment will be described.

[0124] In the fourth exemplary embodiment, the camera MPU 206 determines the type of shake, and the imaging device 200 and the lens device 100 perform image stabilization based on the determination result. In the present exemplary embodiment, the correction ratio between the image stabilization to be performed by the lens device 100 and the image stabilization to be performed by the imaging device 200 can perform image stabilization with higher accuracy compared to the first exemplary embodiment.

[0125] Now, the imaging operation according to the fourth exemplary embodiment will be described. Figure 10 The processing of steps S701 to S704 in Figure 3 is similar to the processing of steps S101 to S104 in Figure 10 and thus will not be described again. Figure 3 The processing of steps S706 to S708 in Figure 10 is similar to the processing of steps S105 to S107 in Figure 3 and thus will not be described again.

[0126] In step S705, the camera MPU 206 sends information about the determined shake type to the lens MPU 101.

[0127] In step S709, the camera MPU 206 obtains the correction ratio of the image stabilization to be performed by the camera image stabilization control unit 211 from the lens MPU 101. The correction ratio is the ratio between the amount of shake correction to be performed by the camera image stabilization control unit 211 and the amount of shake correction to be performed by the lens image stabilization control unit 110. The correction ratio is calculated by the lens MPU 101.

[0128] In step S710, the camera image stabilization control unit 211 performs image stabilization. At this time, the lens image stabilization control unit 110 also performs image stabilization. Based on the result of determining the shake type performed before the correction ratio and the second shutter release signal Sw2 are input to the camera MPU 206, the camera image stabilization control unit 211 and the lens image stabilization control unit 110 perform image stabilization. In step S710, an aperture (not shown) included in the lens device 100 is also driven, and a shutter (not shown) included in the imaging device 200 is also driven.

[0129] As described above, according to the imaging operation in the case of performing image stabilization using multiple image stabilization control units described in the fourth exemplary embodiment of the present disclosure, the type of shake can be determined with high accuracy using a machine learning model and the delay in shooting a still image can be reduced.

[0130] Although different electronic devices determine the type of jitter and calculate the correction ratio in the present exemplary embodiment, the same electronic device may determine the type of jitter and calculate the correction ratio. If different electronic devices determine the type of jitter and calculate the correction ratio, the imaging device 200 and the lens device 100 may share the determination of the type of jitter and the calculation of the correction ratio. This achieves efficient processing. Therefore, in one embodiment, the electronic device is configured to determine the type of jitter and calculate the correction ratio.

[0131] In the present exemplary embodiment, the imaging device 200 determines the type of jitter. However, in the present exemplary embodiment, the lens device 100 may determine the type of jitter. In this case, the function for determining the type of jitter is provided in the lens MPU 101 instead of the camera MPU 206.

[0132] In the present exemplary embodiment, the lens device 100 calculates the correction ratio. However, in the present exemplary embodiment, the imaging device 200 may calculate the correction ratio. In this case, the camera MPU 206 sends the correction ratio for image stabilization to be performed by the lens image stabilization control unit 110 to the lens MPU 101.

[0133] In the present exemplary embodiment, similar to the third exemplary embodiment, the camera calculation unit 206b may stop determining the type of jitter after giving a shooting instruction.

[0134] Now, the fifth exemplary embodiment will be described.

[0135] In the fifth exemplary embodiment, the imaging device 200 and the lens device 100 determine the type of jitter, and the imaging device 200 performs image stabilization. If the determination results made by the imaging device 200 and the lens device 100 match each other, it can be determined that the results have high reliability. If the determination results do not match each other, it can be determined that the results have low reliability. In the present exemplary embodiment, if the determination result (second information) of the type of jitter made by the imaging device 200 and the determination result (third information) of the type of jitter made by the lens device 100 match each other, image stabilization is performed according to the determination result. If the determination results do not match each other, image stabilization is performed according to other states. As a result, image stabilization can be performed according to the type of jitter only when the determination result of the type of jitter has high reliability. It is also possible to reduce the deterioration of the correction accuracy caused by image stabilization based on an incorrect determination result of the type of jitter.

[0136] In the present exemplary embodiment, the imaging device 200 performs image stabilization. Therefore, the lens device 100 may not include an image stabilization lens encoder 105, an image stabilization lens 106, an image stabilization lens drive motor 108, and a lens image stabilization control unit 110.

[0137] Figure 11 is a block diagram showing the functions of the lens MPU 101. The determination of the shake type by the lens device 100 according to the present exemplary embodiment is performed using Figure 11 the functions shown.

[0138] The lens MPU 101 includes a lens acquisition unit 101a and a lens calculation unit 101b. The lens acquisition unit 101a has a similar function to the camera acquisition unit 206a. The lens calculation unit 101b has a similar function to the camera calculation unit 206b.

[0139] Now, the imaging operation according to the fifth exemplary embodiment will be described with reference to Figure 12 the flowchart in. Figure 12 The processing of steps S801 to S803 in is similar to Figure 3 the processing of steps S101 to S103 in, and thus will not be described again. Figure 12 The processing of steps S805 to S807 in is similar to Figure 3 the processing of steps S105 to S107 in, and thus will not be described again. Figure 12 The processing of steps S811 and S812 in is similar to Figure 3 the processing of steps S108 and S109 in, and thus will not be described again.

[0140] In step S804, the camera calculation unit 206b starts to determine the shake type using a machine learning model. The camera calculation unit 206b records information related to the determined shake type in the camera MPU 206. At this time, the lens calculation unit 101b also starts to determine the shake type by a method similar to the camera calculation unit 206b. The lens MPU 101 records information about the shake type determined by the lens calculation unit 101b. Until step S811, the camera calculation unit 206b and the lens calculation unit 101b continuously determine the shake type using the machine learning model.

[0141] In step S808, the camera MPU 206 obtains the determination result of the shake type from the lens MPU 101.

[0142] In step S809, the camera MPU 206 compares the determination result of the shake type made by the camera calculation unit 206b with the determination result of the shake type made by the lens calculation unit 101b.

[0143] If the determination results of the shake types do not match each other (No in step S809), the process proceeds to step S810. On the contrary, if the determination results of the shake types match each other (Yes in step S809), the process proceeds to step S811.

[0144] In step S810, the camera image stabilization control unit 211 performs image stabilization based on other states. Therefore, it is possible to prevent image stabilization from being performed based on a determination result of the shake type with low reliability. It is also possible to reduce the deterioration of the image stabilization accuracy based on an erroneous determination result of the shake type.

[0145] As described above, according to the imaging operation described in the fifth exemplary embodiment of the present disclosure, image stabilization can be performed according to the shake type based on the reliability of the determination result of the shake type.

[0146] In the present exemplary embodiment, the imaging device 200 performs image stabilization. However, in the present exemplary embodiment, the lens device 100 can perform image stabilization. In this case, the lens device 100 uses the lens image stabilization control unit 110 instead of the camera image stabilization control unit 211 that controls image stabilization.

[0147] The lens device 100 uses the image stabilization lens 106 instead of the image sensor 20 as the drive target member, and the drive target member corrects shake by moving in the optical axis direction of the imaging optical system. The lens device 100 uses the image stabilization lens drive motor 108 that drives the image stabilization lens 106 instead of the image sensor drive motor 210 that drives the image sensor 201. The lens device 100 uses the image stabilization lens encoder 105 that outputs the position signal of the image stabilization lens 106 instead of the image sensor encoder 212 that outputs the position signal of the image sensor 201.

[0148] In the present exemplary embodiment, similar to the third exemplary embodiment, the camera calculation unit 206b may stop the determination of the shake type after giving a shooting instruction. At this time, the lens calculation unit 101b may also stop the determination of the shake type.

[0149] In the present exemplary embodiment, similar to the fourth exemplary embodiment, both the camera image stabilization control unit 211 and the lens image stabilization control unit 110 can be used to perform image stabilization.

[0150] In the present exemplary embodiment, the camera MPU 206 compares the determination results of the shake type made by the camera calculation unit 206b and the lens calculation unit 101b. However, instead of the camera MPU 206, the lens MPU 101 may compare the determination results made by the camera calculation unit 206b and the lens calculation unit 101b. In this case, the camera MPU 206 sends the determination result of the shake type to the lens MPU 101 in step S808.

[0151] Now, a smartphone according to the sixth exemplary embodiment will be described.

[0152] The smartphone according to the sixth exemplary embodiment is different from the imaging device 200 according to the first exemplary embodiment in that the smartphone according to the sixth exemplary embodiment has a photographing function but does not include a shutter release button. Figure 13A and Figure 13B is a diagram showing the smartphone 400. Figure 13A shows the front of the smartphone 400. Figure 13B shows the back of the smartphone 400. The smartphone 400 does not include a shutter release button like the operation unit 205 shown in Figure 1 Therefore, the sixth exemplary embodiment is different from the first exemplary embodiment in the processing of the shutter release signals Sw1 and Sw2 generated by the operation unit 205.

[0153] The smartphone 400 includes a display unit 401, an operation unit 402, and a photographing unit 403.

[0154] The display unit 401 is composed of a liquid crystal panel and includes a touch sensor. That is, in addition to the operation unit 402, the user can also operate the smartphone 400 by touching the display unit 401.

[0155] The photographing unit 403 is used to photograph a subject.

[0156] Figure 14 is a block diagram showing the photographing unit 403 and the MPU 404.

[0157] The photographing unit 403 includes an image sensor 501, a focusing lens 502, an image sensor driving motor 503, a focusing lens driving motor 504, an image stabilization control unit 505, and a focusing lens control unit 506. The photographing unit 403 further includes an angular velocity sensor 507, an image sensor encoder 508, a signal processing circuit 509, and a recording processing unit 510.

[0158] The MPU 404 controls the smartphone 400 based on the inputs provided through the display unit 401 and the operation unit 402.

[0159] Figure 15 is a block diagram showing the functions of the MPU 404. The functions shown in Figure 15 are used to determine the type of jitter in the smart phone 400 according to the present exemplary embodiment.

[0160] The MPU 404 includes an acquisition unit 404a and a calculation unit 404b. The acquisition unit 404a has a function similar to that of the camera acquisition unit 206a. The calculation unit 404b has a function similar to that of the camera calculation unit 206b.

[0161] Figure 16 shows the state of the display unit 401 when the camera application is started in the smart phone 400.

[0162] When the camera application is started, a first area 601 and a second area 602 are displayed on the display unit 401 of the smart phone 400. The first area 601 displays an image of the subject, and the second area 602 is a virtual button for receiving a camera instruction.

[0163] In the present exemplary embodiment, before a camera instruction is given by operating the second area 602 in the shooting of a still image, the determination result of the type of jitter made by the calculation unit 404b is used for image stabilization after the camera instruction is given. Therefore, similar to the first exemplary embodiment, even when shooting a still image, the type of jitter can be determined with high accuracy using a machine learning model, and the delay in the process from when the camera instruction is given to when the image is shot can also be reduced.

[0164] Now, the sixth exemplary embodiment of the present disclosure will be described with reference to the Figure 17 flowchart shown. The flowchart in Figure 17 starts when the camera application of the smart phone 400 is started.

[0165] In step S901, the calculation unit 404b starts to determine the type of jitter using a machine learning model. As input data for the machine learning model, the feature amount of the preprocessed jitter detection signal acquired by the acquisition unit 404a is used. The calculation unit 404b records the information about the determined type of jitter in the MPU 404. Until step S906, the calculation unit 404b continuously determines the type of jitter using machine learning.

[0166] In step S902, the MPU 404 measures the distance for focusing on the subject and calculates the driving amount of the focusing lens 502.

[0167] In step S903, the MPU 404 measures the distance again. If it is determined that the distance is within the depth of focus (Yes in step S903), the process proceeds to step S904. If the distance is outside the depth of focus (No in step S903), the process returns to step S902 within a predetermined time or a predetermined number of times.

[0168] In step S904, the MPU 404 determines whether a shooting instruction is given by a user's operation of touching the second area 602. If a shooting instruction is given (Yes in step S904), the process proceeds to step S905. If no shooting instruction is given (No in step S904), the process returns to step S902.

[0169] In step S905, the image stabilization control unit 505 performs image stabilization. The image stabilization is performed based on the determination result of the shake type made by the calculation unit 404b before a shooting instruction is given by a user's operation of touching the second area 602. Therefore, the time from when the shooting instruction is given to when the image stabilization control unit 505 starts image stabilization can be shortened.

[0170] In step S906, the signal processing circuit 509 reads a signal from the image sensor 501 and stores the captured image in the recording processing unit 510.

[0171] As described above, according to the shooting operation in the smart phone described in the sixth exemplary embodiment of the present disclosure, a machine learning model can be used to accurately determine the shake type and reduce the delay when shooting a still image.

[0172] In the present exemplary embodiment, the image stabilization of the smart phone 400 is performed by driving the image sensor 501. However, if the imaging unit 403 includes an image stabilization lens, the image stabilization of the smart phone 400 can be performed by driving the image stabilization lens. Electronic image stabilization can be performed as the image stabilization of the smart phone 400.

[0173] In the present exemplary embodiment, similar to the third exemplary embodiment, the calculation unit 404b can stop determining the shake type after a shooting instruction is given to the MPU 404.

[0174] The present disclosure can also be implemented by a process of providing a program for realizing one or more more functions of the above exemplary embodiments to a system or device via a network or a storage medium and causing one or more processors of a computer of the system or device to read and execute the program. The present disclosure can also be implemented by a circuit (e.g., an application specific integrated circuit (ASIC)) for realizing one or more functions.

[0175] Although the exemplary embodiments of the present disclosure have been described above, the present disclosure is not limited to these exemplary embodiments and can be modified and changed in various ways within the scope of the present disclosure.

[0176] According to an aspect of an embodiment, an electronic device that performs determination processing for accurately determining a shooting state can be provided.

[0177] Other embodiments

[0178] Embodiments of the present disclosure can also be implemented by a computer of a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above embodiments, and / or includes one or more circuits (e.g., an application specific integrated circuit (ASIC)) for performing the functions of one or more of the above embodiments. Moreover, embodiments of the present disclosure can be implemented by a method of, for example, the computer of the system or apparatus reading and executing the computer-executable instructions from the storage medium to perform the functions of one or more of the above embodiments, and / or controlling the one or more circuits to perform the functions of one or more of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read only memory (ROM), a memory of a distributed computing system, an optical disc (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), a flash device, and a memory card, etc.

[0179] Embodiments of the present invention can also be implemented by a method of providing software (a program) that performs the functions of the above embodiments to a system or apparatus via a network or various storage media, and the computer or the central processing unit (CPU), the microprocessing unit (MPU) of the system or apparatus reads and executes the program.

[0180] Although the present disclosure has been described with reference to exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and equivalent structures and functions.

Claims

1. An imaging device comprising: an acquiring unit configured to acquire first information about jitter; a computing unit configured to input the first information into a machine learning model and output second information about a jitter type; as well as a first control unit configured to control image stabilization using the second information, wherein the first control unit controls image stabilization after the shooting instruction is given by using the second information based on the first information before the shooting instruction is given, and The calculation unit outputs the second information based on the first information as time series data having a first length, and the second information based on the first information as time series data having a second length, the second length being longer than the first length.

2. The imaging device according to claim 1, wherein The computing unit stops processing using the machine learning model according to the shooting instruction.

3. The imaging device according to claim 1, wherein The first information is time series data having a length greater than or equal to 0.1 seconds and less than or equal to 5.0 seconds.

4. The imaging device according to claim 1, wherein The first information is time series data having a length greater than or equal to 0.1 seconds and less than or equal to 0.4 seconds.

5. The imaging device according to claim 1, wherein The first control unit controls image stabilization using at least one of the second information based on the first information having the first length and the second information based on the first information having the second length.

6. The imaging device according to claim 5, wherein The first control unit controls image stabilization using the second information based on the first information having the first length before the second information based on the first information having the second length is output from the calculation unit.

7. The imaging device according to claim 1, further comprising at least one of a velocity sensor configured to output a velocity signal and an acceleration sensor configured to output an acceleration signal, in, The first information is data based on a detection signal associated with at least one of the velocity signal and the acceleration signal.

8. The imaging device according to claim 7, wherein The first information is data based on the detection signal that has been subjected to at least one of normalization processing, missing value removal processing, outlier removal processing, and unit conversion processing.

9. The imaging device according to claim 7, wherein The first information includes at least one of an average value, a maximum value, a minimum value, a root sum, a variance, a standard deviation, a kurtosis, and a skewness of the detection signal.

10. The imaging device according to claim 7, wherein The first control unit performs image stabilization using the detection signal and an image stabilization signal based on the second information.

11. The imaging device according to claim 1, wherein The second information is information on a shake type selected from a plurality of shake types including at least one of a walking state, a tripod state, and a panning state.

12. The imaging device according to claim 1, further comprising an image sensor, in, The first control unit controls image stabilization by driving the image sensor.

13. The imaging device according to claim 1, further comprising an image stabilization lens, in, The first control unit controls image stabilization by driving the image stabilization lens.

14. The imaging device according to claim 1, in, The imaging device communicates with another device including a second control unit configured to control image stabilization, and wherein the first control unit acquires a correction ratio between image stabilization to be performed by the first control unit and image stabilization to be performed by the second control unit, and controls image stabilization using the correction ratio and the second information.

15. The imaging device according to any one of claims 1 to 14, in, The imaging device acquires third information on the type of jitter from another device configured to communicate with the imaging device, and The first control unit controls image stabilization using information based on the second information and the third information.

16. The imaging device according to claim 15, wherein The first control unit controls image stabilization using the second information in a case where the second information and the third information match each other.

17. An imaging device for use in an imaging system, the imaging system comprising a first device and a second device having a control unit, the control unit being configured to communicate with the first device and configured to control image stabilization after a shooting instruction is given, using information about a shake type before the shooting instruction is given, the imaging device comprising: an acquiring unit configured to acquire first information about jitter occurring in at least one of the first device and the second device; a computing unit configured to input the first information into a machine learning model and output second information about a jitter type; as well as a communication unit configured to transmit, to the second device, second information on the shake type output from the calculation unit using the first information before the shooting instruction is given, The calculation unit outputs the second information based on the first information as time series data having a first length, and the second information based on the first information as time series data having a second length, the second length being longer than the first length.

18. A method for controlling an imaging device, comprising the steps of: An acquisition step of acquiring first information about jitter; an input-output step of inputting the first information into a machine learning model and outputting second information about the jitter type; and a control step of controlling image stabilization using the second information, in, In the control step, image stabilization after the shooting instruction is given is controlled by using the second information based on the first information before the shooting instruction is given, and In the input and output step, the second information based on the first information as time series data having a first length and the second information based on the first information as time series data having a second length are output, and the second length is longer than the first length.

19. A non-volatile computer-readable storage medium storing a program for causing a computer to execute a method for controlling an imaging device, the method comprising the steps of: An acquisition step of acquiring first information about jitter; An input-output step of inputting the first information into a machine learning model and outputting second information about the jitter type; as well as a control step of controlling image stabilization using the second information, wherein, in the control step, image stabilization after the shooting instruction is given is controlled by using the second information based on the first information before the shooting instruction is given, and In the input and output step, the second information based on the first information as time series data having a first length and the second information based on the first information as time series data having a second length are output, and the second length is longer than the first length.

Images