Lens device, imaging device, control method, and program

Abstract

To provide a lens device that is advantageous in terms of continuity in change in the details of control or change in an image to be obtained.SOLUTION: A lens device (100) has: an optical member (106); a driving unit (111) that drives the optical member; a detection unit (121) that detects a state related to the drive; and a processing unit (113) that generates a control signal for the driving unit based on first information on the state detected by the detection unit. The processing unit includes a machine learning model that generates an output related to the control signal based on the first information and second information on the lens device different from the first information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a lens apparatus, an imaging apparatus, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a zoom lens that controls a combination of multiple actuators by conditional branching based on limit values ​​set for each actuator. In order to switch the control method under specific conditions, it is common to change the calculation formula (calculation method) for generating a command signal to the actuator. The lens operation device of Patent Document 2 changes the calculation method (adds a calculation) in order to switch the control method under specific conditions, such as when the zoom lens group is at the telephoto end or wide-angle end, or when the focus lens group is at the infinity end or close-up end. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 05-127061 [Patent Document 2] Japanese Patent Application Publication No. 06-175000 Summary of the Invention [Problem to be solved by the invention]

[0004] When control is performed under multiple factors that affect the control, it is difficult to control based on the synergistic effect of the multiple factors. In the controls of Patent Documents 1 and 2, the change in the control content is discontinuous due to branching in which the processing content (control content) is divided based on whether a condition is satisfied or not, and as a result, the change in the obtained image is also discontinuous, which may give a sense of discomfort to a viewer of the image. The present invention aims to provide a lens device that is advantageous in terms of continuity in the change in the control content or the change in the obtained image, for example. [Means for solving the problem]

[0005] One aspect of the present invention is No. 1 Optical Components and a lens device having a second optical member. , the above No. 1 Driving optical components A first method for changing an optical characteristic of the lens device by Drive unit and a second driving unit that changes the optical characteristic of the second optical member by driving the second optical member. And the above The first optical member Regarding drive No. 1 Detecting the state No. 1 Detection Unit and a second detection unit for detecting a second state related to the driving of the second optical member. And the above No. 1 First information about the condition and second information regarding the second state. Based on 、 The above No. 1 Drive unit and the first control signal to the second driving unit for Second A processing unit that generates a control signal. death The processing unit and the above second emotion To Based on the above No. 1 Control Signals and the second control signal Output regarding in parallel Generate Single Including machine learning models, The first information, and When the second information is input Regarding the first information Based on reward , the machine learning parameters of the machine learning model are adjustment The learning unit sets the reward so that the reward becomes higher as an error between the value related to the first state and a target value becomes smaller. The lens device is characterized in that Effect of the Invention

[0006] According to the present invention, for example, it is possible to provide a lens device that is advantageous in terms of continuity in changes in the control content or changes in the obtained image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration according to each embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of the configuration of a lens device according to a first embodiment; [Diagram 3] A diagram showing an example of a system configuration for learning [Figure 4] A diagram illustrating the input and output of a neural network [Diagram 5] FIG. 1 illustrates a flowchart for learning [Figure 6] FIG. 13 is a diagram showing an example of a system configuration for learning according to the second embodiment. [Figure 7] A diagram illustrating the input and output of a neural network regarding zooming. [Figure 8] A diagram illustrating the input and output of a neural network regarding focus. [Figure 9] FIG. 13 is a diagram showing an example of a system configuration for learning according to a third embodiment. [Figure 10] A diagram illustrating the input and output of a neural network [Figure 11] FIG. 13 is a diagram showing an example of a system configuration for learning according to a fourth embodiment. [Figure 12] FIG. 13 is a diagram illustrating an example of an allowable range of position error of a focus lens group; [Figure 13] FIG. 13 is a diagram showing an example of the configuration of a control system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In addition, in principle (unless otherwise specified) throughout all the drawings for explaining the embodiment, the same members are given the same reference numerals and the repeated description thereof will be omitted.

[0009] [Embodiment 1] Fig. 1 is a diagram showing an example of a system configuration according to each embodiment including embodiment 1. Fig. 3 is a diagram showing an example of a system configuration for learning, and also a diagram showing an example of the hardware resources and software configuration of each device in the example of the system configuration in Fig. 1.

[0010] <Example of lens device configuration> FIG. 2 is a diagram showing a configuration example of a lens device according to the first embodiment, and is also a diagram showing a configuration example of a system (imaging device) including a configuration example of a camera body. The system is composed of a camera body 200 and a lens device 100 (also called an interchangeable lens). The camera body 200 and the lens device 100 are mechanically and electrically connected via a mount 300 as a coupling mechanism. The mount 300 may be composed of a mount unit belonging to the camera body 200 and a mount unit belonging to the lens device 100, or may be composed including both mount units. The camera body 200 can supply power to the lens device 100 via a power supply terminal included in the mount 300. The camera body 200 and the lens device 100 can communicate with each other via a communication terminal included in the mount 300.

[0011] The lens device 100 may include a field lens group 101, a zoom lens group 102 movable for varying magnification, an aperture unit 103 for adjusting the amount of light, an image shake correction lens group 104, and a focus lens group 105 movable for adjusting focus. Each of the above lens groups may be composed of a single lens element, and may be simply referred to as a lens for simplicity. The zoom lens group 102 and the focus lens group 105 are each held by a lens holding frame (not shown). The lens holding frame is movable along a guide axis (not shown) (along the optical axis indicated by the dashed line in the figure).

[0012] The zoom lens group 102 moves along the optical axis via a zoom lens driving unit 106, and its position is detected by a detection unit 107. The focus lens group 105 moves along the optical axis via a driving unit 111, and its position is detected by a detection unit 112. The aperture unit 103 includes aperture blades driven via a driving unit 108, and adjusts the amount of light. The F-number of the aperture is detected by a detection unit 109. The image stabilization lens group 104 moves along a direction perpendicular to the optical axis via a driving unit 110, and reduces image shake caused by hand shake or the like. The position of the image stabilization lens group 104 is detected by a detection unit 121.

[0013] The driving units 106, 111, 108, and 110 are each configured to include an actuator such as an ultrasonic motor. The actuator is not limited to this, and may be a voice coil motor, a DC motor, a stepping motor, etc. The shake sensor 120 (also called a shaking sensor) is a sensor that detects shake (vibration) of the lens device 100, and is, for example, an angular velocity sensor configured to include a gyro.

[0014] The microcomputer 113 (also referred to as a lens microcomputer or a processing unit) includes a storage unit 114, an estimation unit 115, a control unit 116, a communication unit 117, and a determination unit 118 (also referred to as a lens device information determination unit). Here, the storage unit 114 stores machine learning parameters. The estimation unit 115 (generation unit) estimates (generates) a control signal for the drive unit 111 for the focus lens group. The control unit 116 controls the positions of the zoom lens group 102, the aperture unit 103, and the image blur correction lens group 104. The control unit generates a control signal for the corresponding drive unit by PID control based on the deviation between the target position or speed of the control object and the position or speed of the control object, for example. The communication unit 117 communicates with the camera body 200. The determination unit 118 determines input information related to the lens device 100 to be used for estimation (generation) in the estimation unit 115. The machine learning parameters and the input information will be described later.

[0015] The camera body 200 may include an image sensor 201, an A / D conversion circuit 202, a signal processing circuit 203, a recording unit 204, a microcomputer 205 (also referred to as a camera microcomputer or a processing unit), and a display unit 206. The image sensor 201 is an image sensor that captures an image formed by the lens device 100, and may include a CCD sensor or a CMOS sensor. The A / D conversion circuit 202 converts an analog signal (electrical signal) output from the image sensor 201 into a digital signal. The signal processing circuit 203 converts the digital signal output from the A / D conversion circuit 202 into video (image) data. The recording unit 204 records the video data output from the signal processing circuit 203. The display unit 206 displays the video data output from the signal processing circuit 203. The operation unit 207 is used by the user to operate the camera. The processing unit 205 of the camera body controls the camera body 200. The processing unit 205 has a communication unit 208 and a control unit 209. The communication unit 208 communicates with the lens device 100 via the communication unit 117. The control unit 209 issues commands to the lens device 100 based on video data from the signal processing circuit 203 and operation information from the operation unit 207 by the user. The communication unit 208 transmits the command from the control unit 209 to the lens device 100.

[0016] <Video recording and display> Here, the recording and display of images in the exemplary configuration in Fig. 2 will be described. Light incident on the lens device 100 forms an image on the image sensor 201 via the field lens group 101, the zoom lens group 102, the aperture unit 103, the image shake correction lens group 104, and the focus lens group 105. The image is converted into an electrical signal by the image sensor 201, the electrical signal is converted into a digital signal by the A / D conversion circuit 202, and the digital signal is converted into image data by the signal processing circuit 203. The image data is recorded in the recording unit 204. Moreover, the display unit 206 displays an image based on the image data.

[0017] <Focus control> Here, the focus control of the lens device 100 by the camera body 200 will be described. The control unit 209 performs autofocus control (also referred to as AF control) based on the video data output from the signal processing circuit 203. For example, the control unit 209 performs control to move the focus lens group 105 based on the contrast of the video data so that the subject is in focus. The control unit 209 outputs a command related to the movement of the focus lens group 105 to the communication unit 208. When the communication unit 208 receives a command from the control unit 209, it converts the command into a control command and transmits the control command to the lens device 100 via the communication contact of the mount 300. The communication unit 117 converts the control command received from the communication unit 208 into the above command and outputs the command to the estimation unit 115. When the command is input, the estimation unit 115 generates a control signal based on a plurality of inputs including the input using the learned machine learning parameters stored in the storage unit 114, and outputs the control signal to the drive unit 111 for the focus lens group. The multiple inputs are information about the position of the focus lens group detected by the focus lens group detection unit 112, and input information about the lens device 100 determined by the determination unit 118. A method in which the estimation unit 115 generates a control signal and the input information will be described later. As described above, appropriate AF control can be performed by moving the focus lens group 105 based on a command from the control unit 209 so as to maximize the contrast of the video data.

[0018] <Aperture control> Here, the control of the aperture of the lens device 100 by the camera body 200 will be described. The control unit 209 controls the aperture (exposure control) based on the image data output from the signal processing circuit 203. Specifically, the control unit 209 determines a target F-number so that the luminance value of the image data is constant. The control unit 209 outputs the determined F-number as a command to the communication unit 208. The communication unit 208 converts the command received from the control unit 209 into a control command, and transmits the control command to the lens device 100 via the communication contact of the mount 300. The communication unit 117 converts the control command received from the communication unit 208 into a command related to the F-number, and outputs the command to the control unit 116. The control unit 116 determines a control signal based on the command and the F-number of the aperture detected by the detection unit 109, and outputs the control signal to the drive unit 108. As a result, appropriate exposure control can be performed such that the F-number is controlled so that the luminance value of the image data is constant.

[0019] <Zoom control> Here, the zoom control of the lens device 100 by the camera body 200 will be described. The user operates the zoom of the lens device 100 via the operation unit 207. The control unit 209 outputs a command (e.g., a speed command) for moving the zoom lens group 102 to the communication unit 208 based on the amount of zoom operation by the user output from the operation unit 207. The communication unit 208 converts the command received from the control unit 209 into a control command and transmits the control command to the lens device 100 via the communication contact of the mount 300. The communication unit 117 converts the control command received from the communication unit 208 into a command for zooming and outputs the command to the control unit 116. The control unit 116 determines a control signal based on the command and the position of the zoom lens group detected by the detection unit 107, and outputs the control signal to the drive unit 111 for the zoom lens group. As described above, the zoom lens group 102 can be moved according to the operation of the operation unit 207 by the user.

[0020] <Anti-vibration control> Here, the control of vibration reduction (image stabilization) in the lens device 100 will be described. Based on a signal related to the vibration of the lens device 100 output from the vibration sensor 120, the control unit 116 determines a target position of the image blur correction lens 104 so as to reduce image blur caused by the vibration of the lens 100. The control unit 116 determines a control signal based on the target position and the position of the image blur correction lens 104 detected by the detection unit 121, and outputs the control signal to the drive unit 110 for the image blur correction lens 104. In this way, it is possible to reduce the vibration of the image obtained by imaging with the image sensor 201.

[0021] <Requirements for focus control> Here, the autofocus control in this embodiment will be described. It is assumed here that the requirements for the autofocus control include those related to position error, speed, and power consumption.

[0022] <Position error related to requirements> The position error is an index that indicates how accurately the focus lens group can be moved relative to the target position when the focus lens is moved to the target position. The focus lens group moves to a specific target position to bring a specific subject into focus. In autofocus control, the greater the error between the target position and the actual position, the more blurred the image becomes. It is well known that even if the error is the same, if the depth of focus determined by the F-number and the circle of least confusion is deep, the image will appear in focus, but if the depth of focus is shallow, the image may appear out of focus.

[0023] FIG. 12 is a diagram illustrating an example of an allowable range of a position error of a focus lens group. More specifically, it is a diagram illustrating an allowable range corresponding to a focal depth when the focus lens group is moved from an initial position to a target position. (a) of FIG. 12 shows a case where the focal depth is deep, and (b) of FIG. 12 shows a case where the focal depth is shallow. A1 and B1 show the initial positions of the focus lens group, and A2 and B2 show the positions of the focus lens group after it is moved. The dotted line in the diagram shows the optical axis, and the focus lens group moves along the optical axis. The focal depth is deep in the case of (a) and shallow in the case of (b). Both (a) and (b) show a case where the focus lens group stops short of the target position by a control error. It should be noted that the initial position, the target position, and the control error are the same in (a) and (b).

[0024] In the case of (a), the position A2 of the focus lens after movement falls within the allowable range of the control error (position error) corresponding to the focal depth, and an image in focus is obtained for the specific subject. On the other hand, in the case of (b), the position B2 of the focus lens after movement falls outside the allowable range of the control error corresponding to the focal depth, and an image out of focus is obtained for the specific subject. In this way, even if the control error (position error) is the same, the allowable range of the control error changes depending on the focal depth, and the ratio of the control error to the allowable range also changes. The focal depth is a parameter for evaluating the image plane position, but by using the focus sensitivity of the following equation (1), it can be converted into a parameter for evaluating the position or control error (position error) of the focus lens group. [Image plane movement amount] = [Focus sensitivity] x [Focus lens group movement amount] (1)

[0025] Here, the focus sensitivity is a value that changes depending on the positions of the focus lens group and the zoom lens group. Also, the focal depth changes depending on the F-number. Therefore, the control error (position error) of the focus lens group required to adjust the focus changes depending on the positions of the focus lens group, the zoom lens group, and the aperture position (the state of a specific optical member in the lens device).

[0026] <Required speed> Here, the speed as a requirement refers to the movement speed when the focus lens group is moved. The speed may be a predetermined movement amount per unit time. In addition, if the movement amount of the plane (image plane) on which the subject is focused is the image plane movement amount, and the movement speed of the plane is the image plane movement speed, the movement amount of the focus lens group can be considered to be in a proportional relationship with the image plane movement amount. However, the proportional constant can change depending on the state of the optical system constituting the lens device (the positional relationship of multiple optical members). This proportional constant corresponds to the focus sensitivity in the above formula (1).

[0027] Even if the movement amount of the focus lens group is the same, when the focus sensitivity is high depending on the state of the lens device, the image plane movement amount is relatively large, and when the focus sensitivity is low, the image plane movement amount is relatively small. Therefore, in order to move the focus lens group so that the image plane movement amount is constant, it is necessary to move the focus lens group by the movement amount of the focus lens group obtained by formula (1). In addition, when controlling the image plane movement speed in a state where the moving subject is always in focus, the allowable error between the target image plane movement speed and the actual image plane movement speed can be determined by the focal depth.

[0028] As described above, the focus sensitivity changes depending on the positions of the focus lens group and the zoom lens group. Also, the focal depth changes depending on the F-number. Therefore, the moving speed of the focus lens group required for a constant image plane moving speed or for subject tracking or the like changes depending on the positions of the focus lens group, the zoom lens group, and the state of the aperture.

[0029] In addition, there are cases where it is required to keep the noise that may be recorded during the driving of the focus lens group below a specific level (threshold) during video shooting, in which case the moving speed of the focus lens group can be set to a specific threshold or lower.

[0030] <Power consumption related to requirements> Power consumption refers to the power consumed to drive the focus lens group, and varies depending on the time and speed at which the focus lens group is moved.

[0031] Reducing power consumption allows battery capacity to be used more effectively, which, for example, makes it possible to increase the number of images or the duration that can be captured on a single charge, and to reduce the size of the battery.

[0032] By satisfying the specific requirements regarding the position error, speed, and power consumption described above, for example, it is possible to perform focusing that is preferable for the user. Note that a plurality of requirements may have a trade-off relationship with each other. For example, the position error and the power consumption may have a trade-off relationship with each other. When the focal depth is deep, the influence of the position deviation of the focus lens group on the focus is small compared to when the focal depth is shallow. Therefore, if the control error is allowed as long as the position error of the focus lens group falls within the allowable range corresponding to the focal depth, the power consumption can be reduced. Conversely, when the focal depth is shallow, the influence of the position deviation of the focus lens group on the focus is large. Therefore, if the control error is reduced so that the position error of the focus lens group falls within the allowable range corresponding to the focal depth, the power consumption will increase.

[0033] Furthermore, when performing focus control to keep the subject in focus, it is necessary to keep the error (also called control error or control deviation) between the target position and the actual position of the focus lens group within an allowable range corresponding to the focal depth. For this purpose, for example, the moving speed of the focus lens group may be relatively slowed down when the control error is within a specific range inside the allowable range, and the moving speed of the focus lens group may be relatively fast when the control error exceeds the specific range. In this case, the slower the moving speed of the focus lens group, the more power consumption can be reduced. Here, the allowable range may be determined based on the focal depth. Furthermore, the relationship between the moving speed of the focus lens group and the image plane moving speed may be determined based on the focus sensitivity. Therefore, the moving speed of the focus lens group for keeping the focus may be determined based on the position of the focus lens group, the position of the zoom lens group, and the state of the aperture.

[0034] For the above reasons, the control signal for the drive unit for the focus lens group can be changed based on requirements regarding the position error, speed, and power consumption associated with the movement of the focus lens group.

[0035] <Determination of Control Signal by Estimation Unit 115> The estimation unit 115 is implemented with a neural network (also referred to as NN) algorithm, refers to the machine learning parameters stored in the storage unit 114, and determines a control signal by the NN algorithm using the machine learning parameters. FIG. 4 is a diagram illustrating input and output of a neural network. In the NN as a learning model (trained model) according to the first embodiment, the input X1 is a target position for the focus lens group output from the communication unit 117. The input X2 is the actual position of the focus lens group 105 obtained from the detection unit 112. The input X3 is the focal depth as information related to the lens device, and the input X4 is the focus sensitivity as information related to the lens device. The output Y1 is a control signal for the drive unit 111 for the focus lens group 105. Here, the information related to the optical characteristics of the lens device such as the focal depth and the focus sensitivity can be determined by the determination unit 118 based on specific information related to the lens device (such as the minimum circle of confusion, the F value, the position of the focus lens group, the position of the zoom lens group, etc.). A control signal for the drive unit for the focus lens group is generated by the trained model having the above-mentioned inputs and outputs.

[0036] <Learning Section Structure> FIG. 3 is a diagram showing an example of a system configuration for learning. In FIG. 3, a communication unit 122 (also referred to as a PC communication unit) is a communication unit for communicating with a PC 400 (also referred to as an information processing device). The PC 400 is configured to include a processor (CPU, GPU) and a storage device (ROM, RAM, HDD). The storage device stores programs and data for the processor to operate. A communication unit 404 is a communication unit for communicating with the communication unit 122. A learning unit 403 (also referred to as a machine learning unit) acquires input and output in the estimation unit 115 and performs learning. The learning will be described in detail later.

[0037] The learning unit 403 and the communication unit 404 in the PC 400 can be realized by developing an execution program stored in the HDD on the memory (ROM, RAM) and executing it by hardware resources such as a CPU. The program also describes rules for setting rewards for reinforcement learning. The rules will be described later.

[0038] The communication unit 404 receives information on the operation log of the lens device 100 from the communication unit 122 of the lens device 100. The operation log includes input and output in the estimation unit 115. The information on the operation log is input to the learning unit 403. The learning unit 403 performs machine learning based on the information on the operation log and outputs parameters of the machine learning. A learning method in the learning unit 403 will be described later. The machine learning parameters output from the learning unit 403 are transmitted to the lens device 100 via the communication unit 404. The machine learning parameters received by the communication unit 122 are stored in the storage unit 114. The machine learning parameters stored in the storage unit 114 are referred to by the estimation unit 115 and reflected in the NN algorithm of the estimation unit 115. The estimation unit 115 outputs a control signal by the NN algorithm, and the driving unit 111 drives the focus lens group 105 based on the control signal.

[0039] Here, since the GPU can efficiently perform parallel processing of data, it is effective when performing repeated learning using a learning model such as a learning model in deep learning. Therefore, the GPU can be used instead of or in addition to the CPU for the processing by the learning unit 403. For example, it is preferable that the CPU and the GPU cooperate to execute a learning program including a learning model. Note that the processing of the learning unit 403 may be performed only by the GPU.

[0040] <Method of manufacturing trained machine learning parameters by the training unit 403> Next, a method for learning machine learning parameters (a method for producing learned machine learning parameters) executed by the learning unit 403 according to this embodiment will be described. The parameters are learned in the system of Fig. 3. As described above, the system includes the lens device 100 and the PC 400.

[0041] The estimation unit 115 incorporates a program that generates a control signal for moving the focus lens group 105 from a start position to a stop position in a specific pattern for learning that is determined in advance. The focus lens group 105 is controlled according to this program. By repeating the control and the associated learning, the machine learning parameters in the control unit 116 are rewritten by the PC 400. FIG. 5 is a diagram illustrating an example of a flowchart for learning. Each step in FIG. 5 can be executed by the learning unit 403 of the PC 400 or the estimation unit 115 of the lens device 100.

[0042] In the figure, first, in step S101 (initialization of machine learning parameters), the learning unit 403 transmits an initial value (for example, 0) of the machine learning parameters to the lens device 100 via the communication unit 404. In step S102 (control of lens group, acquisition of log), the estimation unit 115 moves the focus lens group 105 from a predetermined start position to a stop position. The communication unit 122 acquires a control signal, a control amount, a focal depth, a focus sensitivity, and a power consumption as a log related to the control of the focus lens group 105. The control signal may be acquired from the estimation unit 115. The control amount (here, position) may be acquired from the detection unit 112. The focal depth and the focus sensitivity may be acquired from the determination unit 118. The power consumption of the drive unit 111 may be acquired from the drive unit 111. The power consumption of the drive unit 111 is obtained from a current detected by a current detection unit provided in the power supply of the motor. The above log is transmitted from the communication unit 112 to the PC 400.

[0043] Next, in step S103 (giving reward), the learning unit 403 gives reward to the control related to the log according to a rule previously stored in the storage unit 410 (also called a data storage unit) or the like. The rule will be described later. Next, in step S104 (updating machine learning parameters), the learning unit 403 updates the machine learning parameters so as to maximize the reward. The machine learning parameters can be updated by, but are not limited to, the backpropagation method. The updated machine learning parameters are stored in the storage unit 410, and then stored in the storage unit 114 via the communication unit 404 and the communication unit 122. In step S105 (determining whether machine learning is finished), the update unit 412 (also called a data update unit) determines whether learning of the machine learning parameters is finished. The end of machine learning can be determined by whether the number of iterations of machine learning (updating machine learning parameters) has reached a threshold value, or whether the amount of change in the reward in machine learning has become smaller than a threshold value. If the determination is no (machine learning is not finished), the process returns to step S101. If the determination is yes (machine learning is completed), the learning unit 403 stores information on the machine learning parameters obtained by machine learning in the storage unit 410, and ends the process.

[0044] Examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. In addition, deep learning that uses neural networks is also included. The algorithms are not limited to these, and any available algorithm can be appropriately selected and applied.

[0045] <How to set rewards> Here, a method for setting the reward will be described. Here, the reward is determined by individually determining rules for the three items of position error, speed error, and power consumption, and the reward is calculated by summing the three items. Since the position error and speed error have different allowable amounts depending on the focal depth and focus sensitivity, the rule is based on the ratio to the allowable amount. For example, the lower the ratio is below 100%, the higher the reward (score) is given, and the higher the ratio is above 100%, the lower the reward (score) is given. The position error is the difference between the target position and the actual position. The target position is the target control amount determined by the estimation unit 115, and is obtained from the log. The actual position is obtained from the log via the detection unit. The speed error is the difference between the target image plane movement speed and the actual image plane movement speed, and can be obtained or calculated from the log.

[0046] For example, the target image plane movement speed can be calculated from the log (history) of the target position generated by the estimation unit 115. The actual image plane movement speed can be obtained from the focus sensitivity and the movement speed of the focus lens group obtained from the log using the following formula (2) in which the movement amount in formula (1), which can be calculated from the log (history) of the actual image plane position, is replaced with a speed. [Image plane movement speed] = [Focus sensitivity] x [Focus lens group movement speed] …(2)

[0047] The lower the power consumption is below a predefined upper limit, the higher the reward will be. To obtain a machine learning model, we can use the reward determined based on the above three items.

[0048] <Correlation between input data and output data in machine learning models> By setting such rewards, it is possible to adjust the parameters of the machine learning model that are advantageous in terms of focusing performance and power consumption. The input of the machine learning model (here, a neural network) contains information that is correlated with the control signal for the drive unit as the output of the machine learning model. The amount of movement of the focus lens group required to adjust the focus is obtained from the target position and the actual position. The focal depth determines the allowable range for the error of the actual position with respect to the target position. Therefore, by setting a reward that increases as the error becomes smaller within the allowable range, a preferable control signal can be expected. In addition, since the amount of movement of the focus lens group can be replaced with the amount of image plane movement from the focus sensitivity, a preferable control signal can be expected by setting a reward that increases as the error of the image plane movement amount or the image plane movement speed with respect to the target becomes smaller.

[0049] <Advantages of this embodiment> In conventional lens control, an algorithm was developed in which the control content was switched (branched) according to conditions individually set based on requirements regarding position error, speed error, and power consumption.

[0050] Specifically, the allowable range of the position error was set for each range of the focal depth or focus sensitivity. Therefore, the control signal changed discontinuously as the ranges changed, and seamless control was not possible. In this embodiment, since the control signal is based on the above-mentioned machine learning model, rather than on such conditional branching, it is possible to output a control signal that makes the changes in position, speed, and power consumption seamless. Therefore, according to this embodiment, it is possible to provide a lens device that is advantageous in terms of continuity in changes in control content or changes in images.

[0051] <Other Control Targets> The controlled object in this embodiment is not limited to the focus lens group, and may be, for example, a zoom lens group, an image stabilization lens group, an aperture stop, etc. In these cases, information (information group) that has a correlation with the output control signal may be selected as information (information group) related to the lens device to be input to a machine learning model (for example, a neural network). For example, information (information group) including the rotation angle of the zoom operation ring, focal length, brightness, etc. may be selected. By doing so, the effect according to the embodiment described above can be enjoyed even in such control.

[0052] For example, when a machine learning model is applied to the drive control of a zoom lens, a control signal for the drive unit of the zoom lens group that keeps the angle of view change rate (angle of view change per unit time) constant can be obtained from information on the target position and angle of view for the zoom lens group. In this case, a reward that increases as the angle of view change rate approaches a constant can be set. In addition, when a machine learning model is applied to the drive control of an aperture diaphragm, a control signal for the drive unit of the aperture diaphragm based on (suitable for) the luminance change of the image can be obtained using the luminance and F-number of the image as inputs. For example, even if the drive amount of the aperture diaphragm is the same, when the luminance is high and the F-number is bright (small), the luminance changes more abruptly than when the luminance is low and the F-number is dark (large), which may give a sense of incongruity to the viewer of the image. In this case, a reward that increases as the luminance change decreases can be set. In addition, when a machine learning model is applied to the drive control of an image stabilization lens group, a control signal for the drive unit of the image stabilization lens group based on the focal length can be obtained using the focal length and the shift amount of the image as inputs. Even if the movement amount of the image stabilization lens group is the same, if the focal length is long, the shift amount of the image becomes large, so it is necessary to drive the image stabilization lens group faster. In this case, the smaller the position error of the image stabilization lens group, the higher the reward can be set. In addition, since the shake sensor including the gyro has a time during which the output is not stable after power-on, by inputting the time after the shake sensor is started, a control signal for the drive unit of the image stabilization lens group based on the time can be obtained. In this case, the larger the drift of the output of the shake sensor, the lower the reward can be set. In addition, by inputting information regarding the shift vector (motion vector) of the image obtained from the imaging device, a control signal for the drive unit of the image stabilization lens group based on the shift vector can be obtained. In this case, the smaller the absolute value of the shift vector, the higher the reward can be set.

[0053] In addition, the attitude of the lens device, temperature, and ambient sound volume are also useful as inputs to the machine learning model. The attitude changes the effect of gravity when driving the lens or aperture, and as a result, the drive torque required for the drive unit also changes. The temperature changes the characteristics of the lubricating oil in the drive unit, and as a result, the drive torque required for the drive unit also changes. The ambient sound volume restricts the drive speed through the constraint of the drive sound of the drive unit. Therefore, by using the above state quantities as inputs, a control signal for the drive unit based on (compatible with) the state quantity can be obtained.

[0054] [Embodiment 2] <Embodiments Related to Multi-Group Control (Multi-Model)> The second embodiment relates to a system that controls a plurality of lens groups in parallel, and will be described by taking as an example a case of cooperative control of a zoom lens group and a focus lens group. In this embodiment, when the shooting conditions are determined by two optical members, a control signal for a drive unit of one optical member is obtained by inputting control information of the other optical member into a machine learning model (here, a neural network). Note that, like the first embodiment, this embodiment is also applicable to cooperative control in a combination of optical members other than the combination of a focus lens group and a zoom lens group. FIG. 6 is a diagram showing an example of a system configuration for learning according to the second embodiment. The difference from the first embodiment is that at least one of the inputs to the neural network that outputs a control signal for a drive unit of one lens group is the position of the other lens group. The differences from the first embodiment in the configuration example of FIG. 6 will be described.

[0055] The lens microcomputer 113 (also referred to as a processing unit) includes an estimation unit A715a and an estimation unit B715b (depending on the number of lens groups to be controlled; here, two). The estimation unit A715a estimates (outputs) a control signal for the driving unit of the focus lens group 105, and the estimation unit B715b estimates (outputs) a control signal for the driving unit of the zoom lens group 102. The estimation unit A715a determines a target position of the focus lens group 105 from the position of the zoom lens group 102 based on a driving command from the camera body 200, and outputs (generates) a control signal for the driving unit of the focus lens group based on the target position. The estimation unit A715a holds table information indicating the relationship between the subject distance and the positions of the zoom lens group 102 and the focus lens group 105, and determines the target position of the focus lens group 105 from the table information. The determined target position of the focus lens group 105 becomes an input of the NN algorithm that has learned the machine learning parameters. The driver 111 receives the control signal as the output of the estimation unit A715a and moves the focus lens group 105 along the optical axis. On the other hand, the estimation unit B715b obtains a control signal for the driver of the zoom lens 102, and the driver 106 receives the control signal as the output of the estimation unit B715b and moves the zoom lens 102 along the optical axis. The storage unit 114 stores the learned machine learning parameters used by the estimation units A715a and B715b.

[0056] Next, the learning method of the machine learning parameters (the manufacturing method of the trained machine learning model) executed by the learning unit 703 in this embodiment will be described with respect to the points different from those in the first embodiment. The estimation unit A715a, the control unit A716a, and the learning unit A703a function in controlling the zoom lens group 102, and the estimation unit B715b, the control unit B716b, and the learning unit B703b function in controlling the focus lens group 105.

[0057] Here, FIG. 7 is a diagram illustrating input and output of a neural network related to zoom. The neural network (machine learning model) is in the estimation unit A715a. First, the input information (X1 to X6) is as follows. X1 is the target position of the focus lens group 105 determined by the control unit 116. X2 is the actual position (current position) of the focus lens group 105 obtained from the detection unit 112. X3 is the focal depth of the lens device. X4 is the focus sensitivity of the lens device. X5 is the target position of the zoom lens group 102 determined by the control unit 116, and X6 is the actual position of the zoom lens group 102 obtained from the detection unit 107. It should be noted that X4 (focus sensitivity) is information (value) for obtaining the image plane movement amount from the movement amount of the focus lens group 105. The output information Y2 is a control signal for the drive unit 106 of the zoom lens group 102. As can be seen from the above, when outputting a control signal related to the zoom lens group 102, information about the focus lens group 105, which is a lens group separate from the zoom lens group 102, is input to the neural network (machine learning model).

[0058] FIG. 8 is a diagram illustrating input and output of a neural network related to focus. The neural network (machine learning model) is in the estimation unit B715b. First, the input information (X1 to X6) is as follows. X1 is the target position of the focus lens group 105 determined by the control unit 116. X2 is the actual position (current position) of the focus lens group 105 obtained from the detection unit 112. X3 is the focal depth of the lens device. X4 is the focus sensitivity of the lens device. X5 is the target position of the zoom lens group 102 determined by the control unit 116. X6 is the actual position of the zoom lens group 102 obtained from the detection unit 107. The output information Y1 is a control signal for the drive unit 111 of the focus lens group 105. X4 (focus sensitivity) is information (value) for obtaining the image plane movement amount from the movement amount of the zoom lens group 102. As can be seen from the above, when outputting a control signal related to the focus lens group 105, information about the zoom lens group 102, which is a lens group separate from the focus lens group 105, is input to the neural network (machine learning model).

[0059] The method for producing the machine learning model (machine learning parameters) may be executed by the learning unit A703a and the learning unit B703b in the PC 400 and the estimation unit A715a and the estimation unit B715b in the processing unit 113. The procedure may be similar to that in the first embodiment.

[0060] <Advantages of this embodiment> By incorporating the two different machine-learned models (neural networks) according to this embodiment described above into the control of the zoom lens group 102 and the control of the focus lens group 105, respectively, it becomes possible to control the two lens groups in a coordinated manner. This makes it possible to control the zoom lens group and the focus lens group in a coordinated manner, unlike the prior art in which the focus lens group is simply made to follow the zoom lens group. As a result, it becomes possible to compensate for insufficient positional accuracy or insufficient speed of the focus lens group by controlling the zoom lens group, which was not possible with the prior art.

[0061] <Other Control Targets> In this embodiment, the focus control is described, but the cooperative control is not limited to this. For example, the cooperative control can be applied to the cooperative control between the focus lens group and the zoom lens group for compensating the change in the angle of view (field change) caused by the movement of the focus lens group by the movement of the zoom lens group. The cooperative control can also be applied to the cooperative control between the zoom lens group and the aperture diaphragm for compensating the change in the F-number caused by the movement of the zoom lens group by adjusting the aperture diameter (aperture) of the aperture diaphragm. The cooperative control can also be applied to the cooperative control between multiple control systems that perform image blur correction, such as the cooperative control between multiple image blur correction lenses for image blur correction, the cooperative control between the image blur correction lens and the image blur correction imaging element, and the cooperative control between the image blur correction lens and the image blur correction image processing.

[0062] [Embodiment 3] <Embodiment relating to multi-group control (one model)> Like the second embodiment, the third embodiment also relates to a system that controls multiple lens groups in parallel. It differs from the second embodiment in that the parallel control of multiple lens groups is performed by a single machine learning model (here, a neural network). An example of focus control will be described below.

[0063] This embodiment relates to a case where the shooting conditions (here, the focus state or the object distance) are determined by two lens groups. In that case, a single machine learning model (here, a neural network) is used that receives information about both lens groups as input and outputs control signals for the drive units of both lens groups. FIG. 9 is a diagram showing an example of a system configuration for learning according to the third embodiment. Differences from the first and second embodiments are described below. An estimation unit 815 included in the processing unit 113 is configured to include a single machine learning model (here, a neural network). Information about the lens device input to the neural network includes position information about the zoom lens group 102 and the focus lens group 105, and information output from the neural network is a control signal for the drive units of both lens groups. Based on these control signals, the zoom lens group 102 and the focus lens group 105 are controlled in parallel.

[0064] As described above, the estimation unit 815 outputs control signals in parallel for both the zoom lens group 102 and the focus lens group 105. In accordance with the control signals, the focus lens driving unit 111 moves the focus lens 105 along the optical axis, and the zoom lens driving unit 106 moves the zoom lens 102 along the optical axis. The storage unit 114 stores learned machine learning parameters used by the estimation unit 815.

[0065] Here, the difference between the learning method of the machine learning parameters executed by the learning unit 803 in this embodiment (the manufacturing method of the trained machine learning model) and the first embodiment will be described. The estimation unit 815 incorporates a learning program that generates commands for the driving unit 106 and the driving unit 111 in a predetermined specific change pattern. When the learning program is operating, information input to the estimation unit 815 and information output by the estimation unit 815 are acquired as a log. Here, FIG. 10 is a diagram illustrating input and output of a neural network. The following illustrates examples of input information to the neural network. X1 is a command for the focus lens group determined by the control unit 116. X2 is the actual position (current position) of the focus lens group 105 obtained from the detection unit 112. X3 is the focal depth of the lens device. X4 is the above-mentioned focus sensitivity of the lens device. X5 is a command for the zoom lens group determined by the control unit 116. X6 is the actual position of the zoom lens group 102 obtained from the detection unit 107. X7 is the focus sensitivity related to the movement of the zoom lens group 102 (the "focus sensitivity" obtained by replacing [the movement amount of the focus lens group] with [the movement amount of the zoom lens group] in the above-mentioned formula (1)). Y1 is a control signal for the driver 111, and Y2 is a control signal for the driver 106. The manufacturing (method) of the trained machine learning model described above can be executed by the learning unit 803 of the PC 400 and the estimation unit 815 of the processing unit 113. The manufacturing (learning) procedure can be similar to that in the first embodiment (FIG. 5).

[0066] <Advantages of this embodiment> According to this embodiment, cooperative control of two lens groups can be performed by a single machine learning model (neural network). When there is a common process between the control of the zoom lens group 102 and the control of the focus lens group 105, the process can be integrated into one process inside the neural network, so that the load on the processing unit 113 can be reduced. This can be advantageous in shortening the processing time of the estimation unit 815 compared to the second embodiment. It can also be advantageous in terms of synchronism of the cooperative control.

[0067] <Other Control Targets> Although the present embodiment deals with focus control, the cooperative control is not limited to this. This point is similar to the second embodiment.

[0068] [Embodiment 4] <Embodiments Related to Auxiliary Control> 11 is a diagram showing an example of a system configuration for learning according to embodiment 4. This embodiment differs from the above-described embodiments in that it has a control unit 1516 that performs PID control on the focus lens group 105, and an estimation unit 1515 that performs control to assist the control by the control unit 1516. An example of focus control will be described below.

[0069] For example, it is difficult to perform control adapted to various conditions or usage environments of the lens device, such as the attitude and temperature of the lens device, gravity acting on the lens device, and lubricating oil in the driving unit of the lens device, using only PID control. In addition, when the driven object starts to move and stops, dynamic friction and static friction switch back and forth, making it difficult to achieve accurate and stable control. In an imaging device, a viewer may feel uncomfortable with an image obtained under such difficult control. In view of the difficulty, this embodiment performs auxiliary control using a machine learning model (here, a neural network) in cases where the accuracy or stability of control is disadvantageous, such as when the object starts to move or stops.

[0070] This embodiment differs from the first embodiment in that it includes a focus control unit 1516, an estimation unit 1515, a PC communication unit 1522, and a learning unit 1503. Below, the configuration example in FIG. 11 will be described with respect to the differences from the first embodiment.

[0071] <Focus Control According to the Present Embodiment> The focus control according to this embodiment is different from that of the first embodiment in that it has a control unit 1516 and an estimation unit 1515. The control unit 1516 in this embodiment will be described with reference to FIG. 13. Here, FIG. 13 is a diagram showing an example of the configuration of a control system according to the fourth embodiment. In the same figure, G1 is a target position for the drive unit 111 output from the communication unit 117. P1 is an actual position (current position) of the focus lens group 105 obtained from the detection unit 112. E1 is a difference (deviation) between G1 and P1. 1516a is a P controller that performs P control processing, 1516b is an I controller that performs I control processing, and 1516c is a D controller that performs D control processing. O1 is a control signal for the drive unit 111. X1, X2, X3, and X4 are the same as those in FIG. 5. Y1 is an output of a machine learning model for making the output O1 of the control unit 1516 advantageous in terms of control accuracy and stability. Σ1 is a calculation unit (subtractor) for obtaining deviation E1, and Σ2 is a calculation unit (adder) for obtaining control signal O1 by adding the outputs from each of controllers 1516a to 1516c.

[0072] The control unit 1516 generates a deviation E1 between the target position G1 and the actual position P1. The P controller 1516a outputs a value obtained by multiplying the deviation E1 by a specific gain. The I controller 1516b outputs a value obtained by multiplying the integral value of the deviation E1 by a specific gain. The D controller 1516c outputs a value obtained by multiplying the differential value of the deviation E1 by a specific gain. The method in which the estimation unit 1515 outputs Y1 based on X1, X2, X3, and X4 is the same as that in the first embodiment. The control unit 1516 adds the output of the P controller 1516a, the output of the I controller 1516b, the output of the D controller 1516c, and the output Y1 of the estimation unit 1515 by Σ2, and outputs the obtained control signal O1. In this manner, the focus lens group 105 is controlled.

[0073] Here, whether or not to add the output Y1 of the estimation unit 1515 in Σ2 may be determined based on a specific condition. For example, based on information from the processing unit 205 of the camera body 200 on whether or not a video is being shot, Y1 may be added if the video is being shot, and Y1 may not be added otherwise. In the case of video shooting in which shooting may be performed when the focus lens group 105 starts moving or stops, the output Y1 of the estimation unit 1515 may be added to realize smooth movement of the focus lens group 105 when the focus lens group 105 starts moving or stops. By doing so, the processing load of the processing unit 113 can be reduced. In addition, when the focus lens group 105, in which speed unevenness is easily noticeable, is moved at a low speed equal to or less than a threshold, the output Y1 of the estimation unit 1515 may be added. In addition, when position accuracy is required because the focal depth is equal to or less than a threshold, the output Y1 of the estimation unit 1515 may be added. In addition, when a video is being recorded, the output Y1 of the estimation unit 1515 may be added.

[0074] <Method for producing trained machine learning parameters (machine learning model)> The manufacturing method differs from that of the first embodiment in the configurations of the estimation unit 1515 and the communication unit 1522 in the processing unit 113, and the learning unit 1503 in the PC 400. The estimation unit 1515 is different in that the learning and operation of the machine learning model are limited to specific cases in which the focus lens group 105 is moved as described above. For example, the learning program may locally repeat the start or stop of the movement of the focus lens group 105. Furthermore, the program may be configured to perform learning for each different state or environment, such as the attitude or temperature of the lens device 100.

[0075] Another difference is in the log information transmitted from the communication unit 1522 to the communication unit 404. The log includes not only the input and output related to the estimation unit 115 but also the input and output related to the control unit 1516. Note that the learning method other than the above points may be similar to that in the first embodiment.

[0076] <Advantages of this embodiment> According to this embodiment, it is possible to combine control using a machine learning model with other control for lens control. This makes it possible to limit the learning range of the machine learning model to shorten the time required for learning, or to reduce the scale of the machine learning model to shorten the processing time of the estimation unit 115. In addition, for example, it is possible to apply control using a machine learning model in situations where high-precision control is required, and use other control in other cases, thereby reducing the processing load and power consumption of the processing unit 113.

[0077] <Other Control Targets> Although this embodiment deals with focus control, the auxiliary control is not limited to this. This point is similar to the case of embodiment 2. For example, in zoom control, when zooming (moving the zoom lens group 102) at a low speed equal to or lower than a threshold at which speed unevenness is easily noticeable, auxiliary control (addition of the output of an estimation unit) may be performed.

[0078] In addition, although the present embodiment has been described by way of example with a combination of PID control and a machine learning model, a combination of different machine learning models may be used. Also, a plurality of different machine learning models, such as two learning models corresponding to a start of movement and a stop, may be combined with other control such as PID control.

[0079] <Embodiments Related to Programs, Storage Media, and Data Structures> The present invention can also be realized by providing a program or data structure implementing one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having the computer of the system or device read and execute the program. The computer has one or more processors or circuits, and may include separate computers or a network of separate processors or circuits to read and execute computer-executable instructions.

[0080] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0081] Although the preferred embodiments of the present invention have been described above, it goes without saying that the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0082] 100 Lens device 106 Optical Components 111 Drive unit 113 Processing section 121 Detection unit

Claims

1. A lens device having a first optical member and a second optical member, a first driving unit that changes an optical characteristic of the lens device by driving the first optical member, and a second driving unit that changes the optical characteristic of the lens device by driving the second optical member; a first detector that detects a first state related to the driving of the first optical member, and a second detector that detects a second state related to the driving of the second optical member; a processing unit that generates a first control signal for the first driving unit and a second control signal for the second driving unit based on first information about the first state and second information about the second state, The processing unit includes a single machine learning model that generates outputs related to the first control signal and the second control signal in parallel based on the first information and the second information; Further comprising a learning unit that adjusts a machine learning parameter of the machine learning model based on a reward related to the first information when the first information and the second information are input, The lens device according to claim 1, wherein the learning section sets the reward so that the reward becomes higher as an error between the value relating to the first state and a target value becomes smaller.

2. 2. The lens device according to claim 1, wherein the first information includes information regarding a position of the first optical member in the first state and a target position corresponding to the position.

3. The first drive unit moves the first optical member during zooming, The lens device described in claim 1 or 2, characterized in that the first information includes at least one of information regarding the relationship between the position of the first optical member and the angle of view of the lens device, information regarding the relationship between the amount of movement of the first optical member and the amount of movement of an image plane of the lens device, and information regarding the focal depth of the lens device.

4. The second drive unit moves the second optical member during focusing, The lens device according to claim 3 , wherein the second information includes at least one of information regarding a relationship between a movement amount of the second optical member and a movement amount of an image plane of the lens device, information regarding a focal depth of the lens device, and information regarding an object distance of the lens device.

5. The second drive unit moves the second optical member during image shake correction, 4. The lens device according to claim 3, wherein the second information includes information indicating a relationship between an amount of movement of the second optical member and an amount of movement of an image of the lens device.

6. The second drive unit changes an aperture diameter of an aperture stop serving as the second optical member, The lens device according to claim 3, characterized in that the second information includes at least one of information regarding the F-number of the lens device, information regarding the brightness of an image obtained through the lens device, and information regarding the relationship between the position of the first optical member, the aperture of the aperture stop, and the F-number of the lens device.

7. A lens device described in any one of claims 1 to 6, characterized in that the machine learning model includes a neural network.

8. The processing unit includes a control unit, separate from the machine learning model, that generates an output related to the control signal; The lens device according to claim 1 , wherein the machine learning model generates an output for obtaining the control signal by modifying an output generated by the control unit.

9. The lens apparatus according to claim 8 , wherein the processing unit performs the change when a predetermined condition is satisfied.

10. The lens device according to claim 9 , wherein the conditions include a condition related to imaging performed through the lens device.

11. The lens device according to claim 10, wherein the conditions related to the imaging include at least one of the following: the imaging is a moving image imaging, the imaging results in recording of the moving image, zooming is performed at a low speed equal to or less than a threshold, and the imaging of the moving image is performed with a focal depth equal to or less than a threshold.

12. A lens device according to any one of claims 1 to 11, and an image sensor for capturing an image formed by the lens device.

13. A method for controlling a lens device whose optical characteristics change by driving a first optical member and a second optical member, comprising: a first step of detecting a first state related to driving of the first optical member and a second state related to driving of the second optical member; a second step of generating a first control signal for controlling driving of the first optical member and a second control signal for controlling driving of the second optical member based on first information on the first state and second information on the second state, In the second step, the first information and the second information are input to a single machine learning model to generate outputs related to the first control signal and the second control signal in parallel; a machine learning parameter of the machine learning model is adjusted based on a reward related to the first information when the first information and the second information are input; A control method, characterized in that the reward is set to be higher as an error between a value relating to the first state and a target value becomes smaller.

14. A program causing a computer to execute the control method according to claim 13.

Images