Lens device, imaging device, processing device, processing method, and program

Abstract

To provide a lens device which is advantageous in terms of the adaptation of drive performance.SOLUTION: A lens device (100) includes an optical member (101), a driving part (105) for driving the optical member, a detection part (106) for detecting a state related to driving, and a processing part (120) for generating a control signal with respect to the driving part on the basis of first information related to the state detected by the detection part. The processing part includes a machine learning model for generating output related to the control signal on the basis of the first information and second information related to the lens device which is different from the first information and outputs the first information and the second information to a generation part for performing the generation of the machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a lens apparatus, an imaging apparatus, a processing apparatus, a processing method, and a program. [Background technology]

[0002] In recent years, digital cameras can capture not only still images but also videos. To capture still images quickly, high-speed operations such as autofocus, zoom, and aperture are required. However, when capturing videos, if the operating noise of the drive system for the high-speed operations is loud, the quality of the sound recorded together with the image can be impaired. Therefore, Patent Document 1 discloses an imaging device that sets the operating mode of the actuator to a silent mode when capturing videos. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-006305 A Summary of the Invention [Problem to be solved by the invention]

[0004] In an imaging device, actuators that drive optical members are required to have a wide range of performance. For example, they include drive speed related to control tracking, positioning accuracy related to setting accurate imaging conditions, power consumption related to continuous imaging time, and quietness related to audio quality during video imaging. These performances are mutually dependent. For example, the imaging device described in Patent Document 1 improves quietness by limiting the drive speed and acceleration.

[0005] However, the required quietness may vary depending on the shooting conditions. The required driving speed and acceleration may also vary depending on the shooting conditions. The same can be said about other performance such as positioning accuracy and power consumption. Furthermore, the priority of each performance may vary depending on the shooting conditions and the operator. Therefore, it is desirable for the actuator to operate with driving performance suitable for various shooting conditions and operators. An object of the present invention is to provide a lens device that is advantageous in terms of adaptability of driving performance, for example. [Means for solving the problem]

[0006] One aspect of the present invention is a lens device having an optical member, a drive unit that drives the optical member, a detection unit that detects a state related to the drive, and a processing unit that generates a control signal for the drive unit based on first information related to the state detected by the detection unit, an operation unit for inputting information regarding requirements for the drive performance of the drive unit; and a learning unit for causing a machine learning model to learn based on the information regarding the requirements; The processing unit receives the first information and the Regarding the driving performance of the drive unit Based on the second information tree , Using machine learning models, generating an output related to said control signal death , Previous learning The first information and the second information are output to the The learning unit acquires reward information to be used for the learning based on the information related to the request and the second information. The lens device is characterized by the above. Effect of the Invention

[0007] According to the present invention, for example, it is possible to provide a lens device that is advantageous in terms of adaptability of driving performance. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a lens device according to a first embodiment; [Diagram 2] FIG. 13 is a diagram illustrating the positioning accuracy as an example of drive performance. [Diagram 3] FIG. 13 is a diagram illustrating a driving speed as an example of driving performance. [Figure 4] FIG. 1 is a diagram illustrating the relationship between positioning accuracy, drive speed, power consumption, and noise reduction. [Diagram 5] A diagram illustrating the relationship between drive speed, positioning accuracy, power consumption, and noise reduction. [Figure 6] A diagram illustrating the input and output of a neural network [Figure 7] A diagram illustrating the flow of machine learning processing [Figure 8] A diagram illustrating remuneration information [Figure 9] FIG. 13 is a diagram illustrating a data structure of remuneration information. [Figure 10] FIG. 13 is a diagram illustrating a data structure of information regarding options for the second reward portion; [Figure 11] FIG. 13 is a diagram showing an example of the configuration of a lens device according to a second embodiment. [Figure 12] A diagram illustrating the input and output of a neural network [Figure 13] A diagram illustrating remuneration information [Figure 14] FIG. 13 is a diagram illustrating a data structure of remuneration information. [Figure 15] FIG. 13 is a diagram illustrating a data structure of information regarding options for the second reward portion; [Figure 16] FIG. 13 is a diagram showing an example of the configuration of a lens device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In addition, in principle (unless otherwise specified) throughout all the drawings for explaining the embodiment, the same members are given the same reference numerals and the repeated description thereof will be omitted.

[0010] [Embodiment 1] <<Configuration example in which the camera body (processing device) includes a learning unit (generation unit)>> FIG. 1 is a diagram showing a configuration example of a lens device according to the first embodiment, and is also a diagram showing a configuration example of a system (imaging device) including a configuration example of a camera body (also called a camera device body, an imaging device body, or a processing device). The system is composed of a camera body 200 and a lens device 100 (also called an interchangeable lens) attached to the camera body 200. The camera body 200 and the lens device 100 are mechanically and electrically connected via a mount 300 as a coupling mechanism. The mount 300 may be composed of a mount unit belonging to the camera body 200 and a mount unit belonging to the lens device 100, or may be configured to include both mount units. The camera body 200 can supply power to the lens device 100 via a power terminal included in the mount 300. In addition, the camera body 200 and the lens device 100 can communicate with each other via a communication terminal included in the mount 300. In this embodiment, a configuration in which the lens device and the camera body are connected via a mount is illustrated, but the lens device and the camera body may be integrated without the mount.

[0011] The lens device 100 may include a focus lens group 101 for changing an object distance, a zoom lens group 102 for changing a focal length, an aperture stop 103 for adjusting the amount of light, and an image blur correction lens group 104 for image blur correction. The focus lens group 101 and the zoom lens group 102 are each held by a holding frame. The holding frame is configured to be movable along the optical axis direction (the direction of the dashed line in the figure) via, for example, a guide shaft. The focus lens group 101 is driven along the optical axis direction by a driving unit 105, and its position is detected by a detection unit 106. The zoom lens group 102 is driven along the optical axis direction by a driving unit 107, and its position is detected by a detection unit 108. The aperture stop 103 includes aperture blades, and the aperture blades are driven by a driving unit 109 to adjust the amount of light. The aperture amount (also called the aperture degree or F value) of the aperture stop is detected by a detection unit 110. The image stabilization lens group 104 is driven by a driving unit 112 in a direction having a component perpendicular to the optical axis, and reduces image stabilization caused by camera shake or the like. The position of the image stabilization lens group 104 is detected by a detection unit 113. The driving units 105, 107, 109, and 112 may each be configured to include an ultrasonic motor, for example. Note that each driving unit is not limited to an ultrasonic motor, and may be configured to include other motors such as a voice coil motor, a DC motor, and a stepping motor.

[0012] The detection units 106, 108, 110, and 113 may each be configured to include, for example, a potentiometer or an encoder. When the driving unit is configured to include a motor such as a stepping motor that can drive an arbitrary driving amount without feedback of the driving amount (control amount), a detection unit that detects a specific position (reference position or origin) may be provided. In this case, the detection unit may include, for example, a photointerrupter. The detection unit 111 detects the shaking of the lens device 100, and may include, for example, a gyro.

[0013] The processing unit 120 may be a microcomputer, and may include an AI control unit 121, a determination unit 122, a storage unit 123, a log storage unit 124, a drive control unit 125, and a communication unit 126. The AI ​​control unit 121 is a control unit that controls the drive of the focus lens group 101. The AI ​​control unit 121 may perform an operation based on an algorithm of a neural network (also referred to as NN) here, and more generally, generates a drive command for the drive unit of the focus lens group 101 using a machine learning model. The determination unit 122 is a determination unit that determines information (second information) about the lens device used in the AI ​​control unit 121. The storage unit 123 is a storage unit that stores information that specifies the type of input (feature amount) in the neural network and information about weights for the input to each layer. The log storage unit 124 stores information about the operation log of the lens device related to the drive control of the focus lens group 101. The drive control unit 125 controls the drive of the zoom lens group 102, the aperture stop 103, and the image shake correction lens group 104. The drive control unit 125 may generate drive commands for each drive unit, for example, by PID control, based on the deviation between a target position or a target speed of the control object and an actual position or an actual speed of the control object. The communication unit 126 is a communication unit for communicating with the camera body 200. The NN algorithm, weights, second information, and operation log will be described later.

[0014] The camera body 200 (processing device) may be configured to include an image sensor 201, an A / D conversion unit 202, a signal processing circuit 203, a recording unit 204, a display unit 205, an operation unit 206, a processing unit 210 (also called a camera microcomputer or camera microcomputer), and a learning unit 220. The image sensor 201 captures an image formed by the lens device 100, and may be configured to include, for example, a CCD or CMOS device. The A / D conversion unit 202 converts an analog signal (image signal) captured and output by the image sensor 201 into a digital signal. The signal processing circuit 203 converts the digital signal output from the A / D conversion unit 202 into image data. The recording unit 204 records the image data output from the signal processing circuit 203. The display unit 205 displays the image data output from the signal processing circuit 203. The operation unit 206 is for an operator (user) to operate the image capturing device.

[0015] The processing unit 210 is for controlling the camera body 200, and may be configured to include a control unit 211 and a communication unit 212. The control unit 211 generates a drive command to the lens device 100 based on image data from the signal processing circuit 203 and input information from the operation unit 206 by an operator. The control unit 211 also transmits commands and information to a learning unit 220 (described later). The communication unit 212 communicates with the lens device 100. The communication unit 212 transmits the drive command from the control unit 211 to the lens device 100 as a control command. The communication unit 212 also receives information from the lens device 100.

[0016] The learning unit 220 (generation unit) may be configured to include a processor (CPU, GPU, etc.) and a storage device (ROM, RAM, HDD, etc.). The learning unit 220 may include a machine learning unit 221, a reward holding unit 223, a first reward portion holding unit 224, a second reward portion holding unit 225, and a log holding unit 222. The learning unit 220 also stores a program for controlling the operation of each of these units. The reward information held by the reward holding unit 223, the first reward portion information held by the first reward portion holding unit 224, the second reward portion information held by the second reward portion holding unit 225, and information for obtaining the second reward portion information from information input by the operator will be described later.

[0017] <Recording and displaying image data> Here, the recording and display of image data in the imaging device shown in Fig. 1 will be described. Light incident on the lens device 100 passes through a focus lens group 101, a zoom lens group 102, an aperture stop 103, and an image shake correction lens group 104 to form an image on an imaging element 201. The image is converted into an electrical analog signal by the imaging element 201, the analog signal is converted into a digital signal by an A / D conversion unit 202, and the digital signal is converted into image data by a signal processing circuit 203. The image data output from the signal processing circuit 203 is recorded in a recording unit 204. The image data is also displayed by a display unit 205.

[0018] <Focus control> Next, the focus control in the lens device 100 by the camera body 200 will be described. The control unit 211 performs autofocus control (also referred to as AF control) based on image data output from the signal processing circuit 203. For example, the control unit 211 performs AF control to drive the focus lens group 101 so that the contrast of the image data is maximized. The control unit 211 outputs the driving amount of the focus lens group 101 as a driving command to the communication unit 212. When the communication unit 212 receives the driving command from the control unit 211, it converts the driving command into a control command and transmits the control command to the lens device 100 via the communication contact unit of the mount 300. When the communication unit 126 receives the control command from the communication unit 212, it converts the control command into a driving command, and the driving command is output to the AI ​​control unit 121 via the driving control unit 125. When a drive command is input, the AI ​​control unit 121 generates a drive signal based on a machine learning model (learned weights) stored in the storage unit 123, and outputs the drive signal to the drive unit 105. Details of the generation of the drive signal by the AI ​​control unit 121 will be described later. In this manner, the focus lens group 101 is driven based on a drive command from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform AF control that drives the focus lens group 101 so as to maximize the contrast of image data.

[0019] <Aperture control> Next, the aperture diaphragm control in the lens device 100 by the camera body 200 will be described. The control unit 211 performs aperture diaphragm control (exposure control) based on image data output from the signal processing circuit 203. Specifically, the control unit 211 determines a target F-number so that the luminance value of the image data is constant. The control unit 211 outputs the determined F-number to the communication unit 212 as a drive command. When the communication unit 212 receives a drive command from the control unit 211, it converts the drive command into a control command and transmits the control command to the lens device 100 via the communication contact unit of the mount 300. When the communication unit 126 receives a control command from the communication unit 212, it converts the control command into a drive command and outputs the drive command to the drive control unit 125. When the drive command is input, the drive control unit 125 determines a drive signal based on the drive command and the F-number of the aperture diaphragm detected by the detection unit 110, and outputs the drive signal to the drive unit 109. In this manner, the aperture diaphragm 103 is driven so that the luminance value of the image data is constant, based on a drive command from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform exposure control to drive the aperture diaphragm 103 so that the amount of exposure of the image sensor becomes appropriate.

[0020] <Zoom control explanation> Next, zoom control of the lens device 100 by the camera body 200 will be described. The operator operates the zoom of the lens device 100 via the operation unit 206. The control unit 211 outputs the drive amount of the zoom lens group 102 to the communication unit 212 as a drive command based on the zoom operation amount output from the operation unit 206. When the communication unit 212 receives the drive command, it converts the drive command into a control command and transmits the control command to the lens device 100 via the communication contact unit of the mount 300. When the communication unit 126 receives a control command from the communication unit 212, it converts the control command into a drive command and outputs the drive command to the drive control unit 125. When the drive command is input, the drive control unit 125 generates a drive signal based on the drive command and the position of the zoom lens group detected by the detection unit 108, and outputs the drive signal to the drive unit 107. In this manner, the zoom lens group 102 is driven based on the drive command from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform zoom control for driving the zoom lens group 102 based on the amount of zoom operation output from the operation unit 206 .

[0021] <Image stabilization control> Next, image blur correction control in the lens device 100 will be described. Based on a signal indicating the shake of the lens device 100 output from the detection unit 111, the drive control unit 125 determines a target position of the image blur correction lens group so as to reduce image blur caused by the shake of the lens device 100. Based on the target position and the position of the image blur correction lens group detected by the detection unit 113, the drive control unit 125 generates a drive signal and outputs the drive signal to the drive unit 112. In this manner, the image blur correction lens group is driven based on a drive command from the drive control unit 125. Therefore, the drive control unit 125 can perform image blur correction control that reduces image blur caused by the shake of the lens device 100.

[0022] <Driving performance related to focus control> Here, we will discuss four driving performances related to focus control: positioning accuracy, driving speed, power consumption, and quietness. It is preferable that these driving performances are adapted to various situations in which focus control is performed. The four driving performances will be explained below.

[0023] (1) Positioning accuracy The positioning accuracy will be described with reference to FIG. 2. Here, FIG. 2 is a diagram illustrating the positioning accuracy as a driving performance. (a) and (b) in the figure show cases where the focal depth is shallow and deep, respectively. (a) and (b) in the figure have different F-numbers. G shows the target position of the focus lens group, that is, the position of the focus lens group where a point-like object (subject) S on the optical axis is focused on the image sensor 201. C shows the actual position of the focus lens group after driving the focus lens group with G as the target position. Position C is on the object S side with respect to the target position G by a control error (control deviation) E. Bp shows the imaging position (focus position) of the object S when the focus lens group is at position C. δ shows the allowable circle of confusion (diameter) corresponding to the image sensor 201.

[0024] The F value (Fa) in (a) of the figure is brighter (smaller) than the F value (Fb) in (b) of the figure. Therefore, the focal depth (2Faδ) in (a) is shallower than the focal depth (2Fbδ) in (b). The light rays Ca and Ga in (a) indicate the outermost light rays from the object S at the position C of the focus lens group and the target position G, respectively. The light rays Cb and Gb in (b) indicate the outermost light rays from the object S at the position C of the focus lens group and the target position G, respectively. In (a), Ia indicates the diameter of a point image of the object S on the image sensor 201 when the focus lens group is at the position C. In (b), Ib indicates the diameter of a point image of the object S on the image sensor 201 when the focus lens group is at the position C.

[0025] In (a), the focal position Bp is outside the range of the focal depth (2Faδ). Also, the diameter Ia of the point image is larger than the permissible circle of confusion δ, and does not fall within the pixel at the center of the image sensor 201, but extends to the adjacent pixel. Therefore, in (a), the object S is out of focus at the position C of the focus lens group. In contrast, in (b), the focal position Bp is within the range of the focal depth (2Fbδ). Also, the diameter Ib of the point image is smaller than the permissible circle of confusion δ, and falls within the pixel at the center of the image sensor 201. Therefore, in (b), the object S is in focus at the position C of the focus lens group. From the above, even if the positioning accuracy of the focus lens group is the same, a focused state may or may not be obtained depending on the imaging conditions. In other words, the required positioning accuracy changes depending on the imaging conditions.

[0026] (2) Drive speed The drive speed is the amount of movement per unit time. Also, the focus movement speed is the amount of focus movement per unit time. The amount of movement of the focus lens group is proportional to the amount of focus movement. The proportional constant in this proportional relationship is called focus sensitivity. In other words, focus sensitivity is the amount of focus movement of the lens device per unit amount of movement of the focus lens group. Focus sensitivity changes depending on the state of the optical system that constitutes the lens device. The amount of focus movement ΔBp can be expressed by the following equation (1), where Se is the focus sensitivity and ΔP is the amount of movement of the focus lens group. ΔBp = Se × ΔP (1)

[0027] Here, the drive speed required for focus control will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating the drive speed as the drive performance. (a) and (b) in the figure show the case where the focus sensitivity S is large and the case where the focus sensitivity S is small, respectively. (a) and (b) in the figure have different object distances. In (a), when the focus position is moved from Bp1 to Bp2, it is necessary to move the position of the focus lens group from Pa1 to Pa2. Here, the relationship between the movement amount ΔPa of the focus lens group and the focus movement amount ΔBp is shown in formula (1). In (b), when the focus position is moved from Bp1 to Bp2, it is necessary to move the position of the focus lens group from Pb1 to Pb2. Here, the relationship between the movement amount ΔPa of the focus lens group and the focus movement amount ΔBp is shown in formula (1).

[0028] As shown in (a) and (b), the focus sensitivity in (a) is smaller than that in (b), so the amount of movement of the focus lens group required to move the same focus movement amount ΔBp is larger in the case of (a) than in the case of (b). Therefore, compared to the case of (a), the amount of movement of the focus lens group per unit time can be made smaller in the case of (b), and the focus movement speed can be made the same even if the drive speed of the focus lens group is slowed down. From the above, the drive speed of the focus lens group required to obtain a specific focus movement speed differs depending on the imaging conditions. In other words, the required drive speed of the focus lens group changes depending on the imaging conditions.

[0029] (3) Power consumption Power consumption varies depending on the drive time, drive speed, and drive acceleration of the focus lens group. In other words, when the drive time is long, the drive speed is high, or the drive acceleration is high, power consumption increases more than when the drive time is not long. Therefore, if power consumption can be reduced by adapting the drive performance, for example, battery capacity can be effectively utilized, which can increase the image capture time on a single charge and reduce the size of the battery.

[0030] (4) Quietness Drive noise (noise) is generated due to vibrations and friction caused by the drive of the focus lens group. The drive noise changes depending on the drive speed and drive acceleration of the focus lens group. In other words, when the drive speed and drive acceleration are high, the drive noise is louder than when they are not. Also, the longer the time that the focus lens group is stopped, the more advantageous it is in terms of quietness. When imaging in a quiet place, harsh drive noise may be recorded. Therefore, it may be required to be able to change the volume of the drive noise depending on the imaging environment (ambient volume).

[0031] <Relationship between positioning accuracy, drive speed, power consumption, and noise reduction> The relationship between positioning accuracy and drive speed, power consumption, and quietness will be described with reference to FIG. 4. FIG. 4 is a diagram illustrating the relationship between positioning accuracy and drive speed, power consumption, and quietness. (a) and (b) in the figure illustrate the movement of the focus lens group to keep the focus on a moving subject when the focal depth is deep and when the focal depth is shallow, respectively. In (a) and (b), the horizontal axis indicates time, and the vertical axis indicates the position of the focus lens group. The upper side of the vertical axis is the direction toward infinity, and the lower side of the vertical axis is the direction toward the closest point.

[0032] G indicates the target position of the focus lens group, and indicates the position of the focus lens group when the subject is imaged on the image sensor 201. The focal depth in (a) and the focal depth in (b) are 2Faδ and 2Fbδ, respectively. (a) indicates, with respect to the target position G, the position GalimI of the focus lens group where the focal position is the boundary on the infinity side of the focal depth, and the position GalimM of the focus lens where the focal position is the boundary on the close side of the focal depth. (b) indicates, with respect to the target position G, the position GblimI of the focus lens where the focal position is the boundary on the infinity side of the focal depth, and the position GblimM of the focus lens group where the focal position is the boundary on the close side of the focal depth. Ca in (a) and Cb in (b) indicate the positions (trajectories) of the focus lens group controlled so that the subject falls within the focal depth.

[0033] In (a), the focal depth is deep, so the subject is less likely to go out of focus by controlling the focus lens group. In contrast, in (b), the focal depth is shallow, so compared to the case of (a), it is necessary to control the trajectory Cb of the focus lens group so that the deviation from the target position G is smaller. In other words, although the subject will not go out of focus in either (a) or (b), the drive amount and drive speed can be reduced when driving along the trajectory Ca in (a) compared to when driving along the trajectory Cb in (b). In other words, under imaging conditions where low positioning accuracy is required, the focus lens group can be controlled at a low speed, with low power consumption, and quietly.

[0034] <Relationship between drive speed, positioning accuracy, power consumption, and noise reduction> The relationship between the driving speed, the positioning accuracy, the power consumption, and the quietness will be described with reference to FIG. 5. Here, FIG. 5 is a diagram illustrating the relationship between the driving speed, the positioning accuracy, the power consumption, and the quietness. In (a) and (b) of the same figure, the horizontal axis indicates time (time), and the vertical axis indicates the position of the focus lens group. (a) shows the position Ca of the focus lens group when it is driven from the position Pa1 shown in (a) of FIG. 3 to the position Pa2 in the time period T0 to T1. Also, (b) shows the position Cb of the focus lens group when it is driven from the position Pb1 shown in (b) of FIG. 3 to the position Pb2 in the time period T0 to T1. Here, as shown in FIG. 3, the focus movement amount when the focus lens group moves from the position Pa1 to the position Pa2 is the same as the focus movement amount when the focus lens group moves from the position Pb1 to the position Pb2. Here, the slopes of Ca and Cb in (a) and (b) of FIG. 5 correspond to the driving speed of the focus lens group.

[0035] As shown in (a) and (b) of FIG. 5, the driving speed of the focus lens group in Ca needs to be higher than that in Cb to obtain the same focus movement amount ΔBp at the time T0 to T1. In addition, since the driving speed of Ca is higher than that of Cb, it takes a long time for the position of the focus lens group to stabilize after it reaches the target position Pa2. In contrast, since the driving speed of Cb is lower than that of Ca, it takes only a short time for the position of the focus lens group to stabilize after it reaches the target position Pb2. That is, the driving speed affects the positioning accuracy. In addition, since the driving acceleration of the focus lens group in Ca is higher than that of Cb, the power consumption is also higher and the driving sound is also louder. That is, under imaging conditions where the required driving speed is low, the focus lens group can be controlled with high positioning accuracy, low power consumption, and quiet operation.

[0036] <Second information regarding lens device> Next, the second information regarding the lens device will be described. The second information is information that affects the driving performance of the focus lens group. As described above, in order to adapt the driving performance in the driving control of the focus lens group, it is necessary to generate a control signal based on the second information that affects the driving performance. The second information is determined by the determination unit 122. The second information includes, for example, information on the depth of focus and the focus sensitivity. The determination unit 122 obtains information on the depth of focus from information on the F value and information on the permissible circle of confusion. In addition, the determination unit 122 stores information (table) indicating the relationship between the focus sensitivity and the position of the focus lens group and the position of the zoom lens group, and obtains information on the focus sensitivity from each of the information on the relationship, the position of the focus lens group, and the position of the zoom lens group. By generating a control signal based on such second information, it is possible to provide a lens device that is advantageous in terms of adapting (customizing) driving performance such as positioning accuracy, driving speed, power consumption, and quietness. Note that a machine learning algorithm that generates a control signal based on the second information will be described later.

[0037] <Machine learning model> Here, a method in which the AI ​​control unit 121 generates a control signal using a machine learning model will be described. The AI ​​control unit 121 has a machine learning model and operates according to a machine learning algorithm. The machine learning algorithm is not limited to this, but is assumed here to be an algorithm (also referred to as an NN algorithm) using a neural network (also referred to as an NN). The AI ​​control unit 121 refers to the feature amount that is the input to the NN stored in the storage unit 123 and the weights for the input to each layer, and generates an output related to the control signal by the NN algorithm using the feature amount and weights obtained by the reference. A method for creating a machine learning model (weights) will be described later.

[0038] A concept showing the input / output structure in the learning model according to the first embodiment will be described with reference to FIG. 6. FIG. 6 is a diagram illustrating an example of input / output of a neural network. In the figure, X1 is information on a drive command output from the drive control unit 125. X2 is information on the position of the focus lens group 101 obtained from the detection unit 106. X3 is information on the focal depth as the second information, and X4 is information on the focus sensitivity as the second information. Y1 is information on an output related to a control signal for the drive unit 105. In this way, based on X1 to X4 as inputs, Y1 is generated as an output of the trained machine learning model. The AI ​​control unit 121 uses Y1 as a control signal or generates a control signal based on Y1, and controls the drive unit 105 using the control signal.

[0039] <How to create a learning model (weights)> Next, a method of creating a learning model (weights) (learning in the machine learning unit) will be described. Here, the control unit 211 transmits a command to the machine learning unit 221 to implement machine learning in response to an operation by the operator on the operation unit 206. Upon receiving the command, the machine learning unit 221 starts machine learning. The flow of machine learning by the machine learning unit 221 will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating an example of the flow of the machine learning process.

[0040] In the figure, in step S101, a learning model (weights) is initialized. Specifically, the machine learning unit 221 outputs an initial value of the weights to the control unit 211. When the control unit 211 receives the initial value of the weights from the machine learning unit 221, the control unit 211 transmits the initial value of the weights to the lens device 100 via the communication unit 212. When the drive control unit 125 of the lens device 100 receives the initial value of the weights via the communication unit 126, the drive control unit 125 stores the initial value in the storage unit 123. In the following step S102, log information is acquired. Specifically, the machine learning unit 221 requests the control unit 211 to acquire log information in the lens device 100. When the control unit 211 receives the request, the control unit 211 requests the log information to the lens device 100 via the communication unit 212. When the drive control unit 125 of the lens device 100 receives the request for the log information via the communication unit 126, it instructs the AI ​​control unit 121 to drive the focus lens group 101. When the AI ​​control unit 121 receives the drive command, it generates a control signal for the drive unit 105 based on a machine learning model using the weights stored in the storage unit 123. The machine learning unit 221 stores a predetermined learning drive pattern for driving the focus lens group 101 from the start position to the stop position, and generates a control signal corresponding to the pattern. Instead of the predetermined learning drive pattern, a learning drive pattern determined according to an autofocus algorithm may be used. When the drive control unit 125 receives a request for the log information via the communication unit 126, it requests the log storage unit 124 to output the log information. When the log storage unit 124 receives the request for output, it transmits the log information of the lens device during the drive of the focus lens group 101 to the camera body 200 via the drive control unit 125 and the communication unit 126.

[0041] In step S103, the driving performance of the focus lens group 101 is evaluated. Specifically, the machine learning unit 221 evaluates the driving performance of the focus lens group 101 by the control signal generated by the AI ​​learning unit 121 based on the reward information held by the reward holding unit 223 and the log information held by the log holding unit 222. The details of the evaluation will be described later. In step S104, the machine learning model (weight) is updated. Specifically, the machine learning unit 221 updates the machine learning model (weight) based on the evaluation value in the evaluation (for example, so that the evaluation value is maximized). The weight can be updated using backpropagation, but is not limited to this. The generated weight (machine learning model) is stored in the storage unit 123 by the same process as the process in step S101.

[0042] In step S105, a determination (judgment) is made as to whether or not to end the machine learning. Specifically, for example, the machine learning unit 221 makes this determination based on whether the number of times of learning (weight updates) has reached a predetermined value, or whether the amount of change in the evaluation value of the driving performance is smaller than a predetermined value. If the machine learning unit 221 has determined not to end the machine learning, the process returns to step S101 and the machine learning continues. If the machine learning unit 221 has determined to end the machine learning, the machine learning unit 221 ends the machine learning process. Note that the machine learning unit 221 adopts a machine learning model whose evaluation satisfies an allowable condition (for example, the amount of change in the evaluation value of the driving performance has become smaller than a predetermined value). In addition, the machine learning unit 221 rejects a machine learning model whose evaluation satisfies an end condition (for example, the number of times of learning has reached a predetermined value) and whose evaluation does not satisfy an allowable condition.

[0043] The machine learning algorithm may be deep learning that uses a neural network and generates weights for inputs to each layer by itself. Deep learning may also generate features by itself. The algorithm is not limited to deep learning and may be other algorithms, for example, may include at least one of the nearest neighbor method, the naive Bayes method, the decision tree, and the support vector machine. Any of these algorithms that can be used may be appropriately applied to this embodiment.

[0044] Here, since the GPU can efficiently perform parallel processing of data, it is effective when performing repeated learning using a machine learning model such as a machine learning model in deep learning. Therefore, the GPU can be used instead of or in addition to the CPU for processing by the machine learning unit 221. For example, it is preferable that the CPU and the GPU cooperate to execute a machine learning program including a machine learning model.

[0045] <Log information> Next, the log information will be described. The log information includes information to be used for evaluating the driving performance of the focus lens group 101. The log storage unit 124 collects and stores input / output information of the machine learning model such as X1 to X4 and Y1 shown in FIG. 6 for each operation cycle of the machine learning model. The log storage unit 124 stores information on the power consumption of the drive unit 105 obtained by the processing unit 120 as log information. The log storage unit 124 stores information on the drive command input to the AI ​​control unit 121 and information on the position of the focus lens group detected by the detection unit 106 as log information. The log storage unit 124 stores information on the target position and positioning accuracy of the focus lens group obtained by the processing unit 120 as log information. The log storage unit 124 stores information on the drive speed and drive acceleration of the focus lens group obtained from the information on the position of the focus lens group as log information. The log storage unit 124 transmits the stored log information to the camera body 200 via the drive control unit 125 and the communication unit 126. When the control unit 211 of the camera body 200 receives the log information via the communication unit 212, it causes the log information to be stored in the log storage unit 222.

[0046] <Reward information and driving performance evaluation> The reward information is information necessary for evaluating the driving performance. The reward information includes information on boundary values ​​that determine a range for each driving performance, and information on rewards that are predetermined for each range. The reward information will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating reward information. In the figure, (a1), (b1), (c1), and (d1) show the relationship between time (time) and rewards when a machine learning model is being learned for positioning accuracy, driving speed, driving acceleration, and power consumption as driving performance. The horizontal axis of the graph shows time, and the vertical axis of the graph shows driving performance and boundary values. In the figure, (a2), (b2), (c2), and (d2) show the data structure of reward information for positioning accuracy, driving speed, driving acceleration, and power consumption, respectively. The data structure includes data on the boundary values ​​and data on rewards for each of the ranges.

[0047] Here, the machine learning model is trained to give a high evaluation to the driving performance. Therefore, for example, when the driving performance is the positioning accuracy, the highest reward is set in a range including zero position deviation. In addition, a specific driving performance is given a higher reward than other driving performances by being set with a relatively high reward. For example, power consumption is given a higher reward than positioning accuracy by being set with a relatively high reward. In this embodiment, an example is shown in which the reward information includes information on two boundary values ​​and information on three rewards.

[0048] The vertical axis of (a1) indicates the value of the position deviation E, which is the difference between the target position and the actual position of the focus lens group. The positive direction of the position deviation E indicates that the actual position of the focus lens group is on the infinity side with respect to the target position, and the negative direction of the position deviation E indicates that the actual position is on the close side with respect to the target position. The frequency of the position deviation E being close to 0 (E 2The smaller the sum of the position deviations E and E2, the higher the positioning accuracy of the focus lens group. (a2) shows reward information RE regarding the positioning accuracy. The reward information RE includes boundary values ​​E1 and E2 of the position deviation, and rewards SE1, SE2, and SE3 that can be obtained in each range. Here, the range of the position deviation from E1×-1 to E1 is set as range AE1. Also, the range obtained by excluding range AE1 from the range from E2×-1 to E2 is set as range AE2. Also, the range obtained by excluding range AE1 and range AE2 from the entire range is set as range AE3. As shown in (a2), rewards SE1, SE2, and SE3 are set for range AE1, range AE2, and range AE3, respectively. Here, the magnitude relationship of the rewards is reward SE1>reward SE2>reward SE3, and the higher the reward is set as the position deviation E is closer to 0. As shown in (a1), the position deviation E at time Tp1, time Tp2, and time Tp3 belong to the range AE2, range AE3, and range AE1, respectively. Therefore, the rewards that can be obtained at time Tp1, time Tp2, and time Tp3 are reward SE2, reward SE3, and reward SE1, respectively. Here, for example, the boundary value E1 can be a value of Fδ / 2, and the boundary value E2 can be a value of Fδ. In other words, if the actual position of the focus lens group has a deviation of less than half the focal depth from the target position (|E|≦Fδ / 2), the highest reward is obtained. On the other hand, if the actual position of the focus lens group has a deviation of more than half the focal depth from the target position to the focal depth (Fδ / 2<|E|≦Fδ), an intermediate reward is obtained. Also, if the actual position of the focus lens group has a deviation of more than the focal depth from the target position (|E|>Fδ), the lowest reward is obtained.

[0049] The vertical axis of (b1) indicates the value of the drive speed V of the focus lens group. The positive direction of the drive speed V indicates the direction toward infinity, and the negative direction of the drive speed V indicates the direction toward the closest point. The drive sound becomes smaller as the drive speed V approaches 0. (b2) indicates reward information RV regarding the drive speed. The reward information RV includes boundary values ​​V1 and V2 of the drive speed, and rewards SV1, SV2, and SV3 that can be obtained in each range. Here, the range of drive speeds V1×-1 to V1 is set as range AV1. Also, the range obtained by excluding range AV1 from the range of drive speeds V2×-1 to V2 is set as range AV2. Also, the range obtained by excluding range AV1 and range AV2 from the entire range is set as range AV3. As shown in (b2), rewards SV1, SV2, and SV3 are set for ranges AV1, AV2, and AV3, respectively. Here, the magnitude relationship of the reward is reward SV1>reward SV2>reward SV3, and the closer the drive speed V is to 0, the higher the reward is set. As shown in (b1), the drive speeds V at time Tp1, time Tp2, and time Tp3 belong to ranges AV2, AV3, and AV1, respectively. Therefore, the rewards that can be acquired at time Tp1, time Tp2, and time Tp3 are reward SV2, reward SV3, and reward SV1, respectively. Here, the boundary values ​​V1 and V2 are set, for example, based on the relationship between the drive speed and the drive sound. If the reward is set so that the lower the drive speed, the higher the reward that can be acquired, and the lower the drive speed, the quieter the drive sound, so that a machine learning model that takes quietness into consideration can be obtained.

[0050] The vertical axis of (c1) indicates the value of the driving acceleration A of the focus lens group. The positive direction of the driving acceleration A indicates the direction toward infinity, and the negative direction of the driving acceleration A indicates the direction toward the closest point. The driving sound becomes smaller as the driving acceleration A approaches 0. (c2) indicates the reward information RA regarding the driving acceleration. The reward information RA includes boundary values ​​A1 and A2 of the driving acceleration, and rewards SA1, SA2, and SA3 that can be obtained in each range. Here, the range of the driving acceleration A1×-1 to A1 is set as range AA1. Moreover, the range obtained by excluding the range AA1 from the range of A2×-1 to A2 is set as range AV2. Moreover, the range obtained by excluding the range AA1 and the range AA2 from the entire range is set as range AA3. As shown in (c2), rewards SA1, SA2, and SA3 are set for the range AA1, range AA2, and range AA3. Here, the magnitude relationship of the reward is reward SA1>reward SA2>reward SA3, and the closer the driving acceleration A is to 0, the higher the reward is set. As shown in (c1), the driving acceleration A at time Tp1, time Tp2, and time Tp3 belong to ranges AA1, AA3, and AA2, respectively. Therefore, the rewards that can be acquired at time Tp1, time Tp2, and time Tp3 are reward SA1, reward SA3, and reward SA2, respectively. Here, the boundary values ​​A1 and A2 are set, for example, based on the relationship between the driving acceleration and the driving sound. If the reward is set so that the lower the driving acceleration, the higher the reward that can be acquired, and the lower the driving acceleration, the quieter the driving sound, so that a machine learning model that takes quietness into consideration can be obtained.

[0051] The vertical axis of (d1) indicates the value of the power consumption P of the drive unit 105. (d2) indicates the reward information RP regarding the power consumption. The reward information RP includes boundary values ​​P1 and P2 of the power consumption, and rewards SP1, SP2, and SP3 that can be obtained in each range. Here, the range of power consumption from 0 to P1 is set as range AP1. The range exceeding P1 and not exceeding P2 is set as range AP2. The range obtained by removing range AP1 and range AP2 from the entire range is set as range AP3. As shown in (d2), rewards SP1, SP2, and SP3 are set for range AP1, range AP2, and range AP3. Here, the magnitude relationship of the rewards is reward SP1>reward SP2>reward SP3, and the closer the power consumption P is to 0, the higher the reward is set. As shown in (d1), the power consumption P at time Tp1, time Tp2, and time Tp3 belongs to range AP1, range AP3, and range AP2, respectively. Therefore, the rewards that can be acquired at time Tp1, time Tp2, and time Tp3 are reward SP1, reward SP3, and reward SP2, respectively. If the rewards are set so that the smaller the power consumption, the more reward can be acquired, it is possible to obtain a machine learning model that takes low power consumption into consideration.

[0052] As described above, reward information for evaluating driving performance such as positioning accuracy (position deviation), driving speed, driving acceleration, and power consumption can be set. By using such reward information, a reward is generated for each unit time for each driving performance based on log information when the focus lens group is driven, and the rewards are accumulated to evaluate the machine learning model. Note that, being based on rewards for a plurality of driving performances is advantageous for customizing the machine learning model. Furthermore, the power consumption may be measured based on the current in the driving unit, or may be estimated based on the driving speed or driving acceleration. Furthermore, the boundary value is not limited to being constant, and may be variable as necessary. Furthermore, the reward is not limited to being specified based on the boundary value, and may be specified based on a function related to each driving performance. In this case, the reward information may be configured to include information related to the function.

[0053] <First and second remuneration parts> Next, the first reward part and the second reward part of the reward information will be described. FIG. 9 is a diagram illustrating an example of the data structure of the reward information. The information of the first reward part (information of the first reward prepared in advance) is configured to include information of the reward REb related to the positioning accuracy, the reward RVb related to the drive speed, the reward RAb related to the drive acceleration, and the reward RPb related to the power consumption. The information of the second reward part (information of the second reward) is configured to include information of the reward REu related to the positioning accuracy, the reward RVu related to the drive speed, the reward RAu related to the drive acceleration, and the reward RPu related to the power consumption. REb and REu have the same data structure as the reward RE related to the positioning accuracy shown in (a2) of FIG. 8. RVb and RVu have the same data structure as the reward RV related to the drive speed shown in (b2) of FIG. 8. RAb and RAu have the same data structure as the reward RA related to the drive acceleration shown in (c2) of FIG. 8. RPb and RPu have the same data structure as the reward RP related to the power consumption shown in (d2) of FIG. 8.

[0054] Here, the information of the first remuneration portion is information on remuneration specific to the lens apparatus 100. The information of the first remuneration portion is pre-stored in the first remuneration portion holding unit 224 as remuneration information specific to the lens apparatus 100. The information of the second remuneration portion is information on remuneration that is variable based on a request of the operator of the lens apparatus 100. The information of the second remuneration portion is held in the second remuneration portion holding unit 225 based on the request of the operator. The remuneration holding unit 223 holds the information of the first remuneration portion and the information of the second remuneration portion.

[0055] The information of the first reward portion is reward information for obtaining an acceptable driving performance in the lens device 100, and therefore a wider range of rewards, including negative values, is set than the information of the second reward portion. The information of the second reward portion is variable according to the request of the operator, and can be acquired based on the request information and the option information of the second reward portion. The reward information is obtained from the information of the first reward portion and the information of the second reward portion. Then, based on the reward information, an evaluation value of the machine learning model is obtained, as illustrated with reference to FIG. 8, to learn (generate) the machine learning model.

[0056] Here, a method of obtaining information on the second reward part at the request of the user will be described. FIG. 10 is a diagram illustrating an example of a data structure of information on options of the second reward part. (a) of the figure shows a data structure of information on an option UREu of the second reward part related to positioning accuracy. The information on the option UREu includes, for each level, a boundary value of the position deviation and reward information for each range defined by the boundary value. (b) of the figure shows a data structure of information on an option URSu of the second reward part related to quietness. The information on the option URSu includes information on an option URVu of the second reward part related to the drive speed and information on an option URAu of the second reward part related to the drive acceleration. The information on URVu includes, for each level, a boundary value of the drive speed and reward information for each range defined by the boundary value. The information on URAu includes, for each level, a boundary value of the drive acceleration and reward information for each range defined by the boundary value. (c) of the figure shows a data structure of information on an option URPu of the second reward part related to power consumption. The information on the option URPu includes, for each level, boundary values ​​of power consumption and reward information for each range defined by the boundary values.

[0057] The information of the second reward part options UREu regarding positioning accuracy, the second reward part options URSu regarding quietness, and the second reward part options URPu regarding power consumption is set as follows. That is, the boundary values ​​and reward values ​​of each information are set in the order (ascending order) of level 1, level 2, and level 3 so that the operator's required level decreases. More specifically, for example, level 1 has a boundary value closer to the target value of the drive performance and a higher reward value than the other levels.

[0058] The operator's request may be input via the operation unit 206 shown in FIG. 1. Based on the request, a level may be selected from level 1 to level 3 for each driving performance. Information regarding the level is transmitted to the second reward portion holding unit 225 via the control unit 211. The second reward portion holding unit 225 identifies (selects) information on the second reward portion for each driving performance based on information regarding the level of each driving performance. By doing so, learning of the machine learning model (weighting) is performed based on the customized reward information, and a customized machine learning model (weighting) can be generated. Information on the generated machine learning model (weighting) is transmitted from the camera body 200 to the lens device 100, stored in the storage unit 123, and used to control the driving of the focus lens group (driving unit 105).

[0059] <Other examples of control targets> Although the focus lens group has been exemplified as an object of drive control, this embodiment is not limited thereto. In this embodiment, other optical members such as a zoom lens group, an image stabilization lens group, a flange back adjustment lens group, and an aperture diaphragm may be objects of drive control. Positioning accuracy, quietness, and power consumption are driving performances that are also emphasized when driving these optical members. The required positioning accuracy may vary depending on the relationship between the drive amount and the change amount of the angle of view or the size of the subject for the zoom lens group. The required positioning accuracy may vary depending on the focal length for the image stabilization lens group. The required positioning accuracy may vary depending on the relationship between the drive amount and the change amount of the brightness of the image for the aperture diaphragm.

[0060] <Other examples of secondary information> Although the focus sensitivity and focal depth are exemplified as the second information related to the lens device, the information is not limited thereto and may include at least one of the following information: the attitude of the lens device, the temperature, and the ambient sound volume. The attitude of the lens device can change the load (torque) of the drive unit by changing the effect of gravity on the optical member. The temperature of the lens device can change the load (torque) of the drive unit by changing the performance of the lubricating oil in the drive system. The ambient sound volume of the lens device can change the effect on the constraints on the drive sound by the drive unit, thereby changing the limitations on the speed and acceleration of the drive unit.

[0061] As described above, according to the present embodiment, it is possible to provide a lens device or an imaging device that is advantageous in terms of adapting (customizing) driving performance, for example.

[0062] [Embodiment 2] Example of configuration in which the lens device includes a learning unit (generation unit) The second embodiment will be described with reference to Figs. 11 to 15. Fig. 11 is a diagram showing an example of the configuration of a lens device according to the second embodiment, and is also a diagram showing an example of the configuration of a system (imaging device) including an example of the configuration of a camera body. This system differs from the first embodiment in that the lens device 100 includes a learning unit. Also, this system differs from the first embodiment in that the second information about the lens device 100 includes information about recording by the camera body.

[0063] The learning unit 1220 may be configured to include a processor (such as a CPU or GPU) and a storage device (such as a ROM, RAM, or HDD). The learning unit 1220 may include a machine learning unit 1221, a log storage unit 1222, a reward storage unit 1223, a first reward portion storage unit 1224, and a second reward portion storage unit 1225. The learning unit 1220 also stores a program for controlling the operation of each of these units.

[0064] The drive control unit 1125 has a function of exchanging information with the learning unit 1220, in addition to the functions of the drive control unit 125 in the first embodiment. The AI ​​control unit 1121 controls the drive of the focus lens group 101 (drive unit 105) based on the machine learning model generated by the learning unit 1220. The determination unit 1122 is a determination unit that determines information (second information) about the lens apparatus used in the AI ​​control unit 1121. The second information here will be described later. The operation unit 1206 is an operation unit that allows an operator to operate the lens apparatus 100 (imaging apparatus).

[0065] <Second information> Here, the second information includes information regarding the influence of the control of the drive of the focus lens group on the recording in the camera body. In this embodiment, by using such second information in addition to or instead of the second information in the first embodiment, the control of the drive of the focus lens group can be performed while taking into consideration the influence of the control on the recording. The second information may include information obtained by analyzing the image data obtained by the signal processing circuit 203 in the control unit 211, and may be determined based on information sent from the control unit 211 to the determination unit 1122 via the communication unit 212, the communication unit 126, and the drive control unit 1125. The second information may be, for example, information regarding at least one of the allowable circle of confusion diameter, the defocus amount of the subject obtained by imaging with the camera body, and the volume (volume of the recorded ambient sound) obtained by a microphone or the like included in the camera body. The determination unit 1122 can obtain information on the focal depth from the information on the F-number and the allowable circle of confusion diameter.

[0066] <Machine learning model> Here, the machine learning model in the AI ​​control unit 1121 will be described. Here, FIG. 12 is a diagram illustrating an example of input and output of a neural network. In the neural network according to the second embodiment shown in the figure, X21 is information of a drive command output from the drive control unit 1125. X22 is information of the position of the focus lens group 101 obtained from the detection unit 106. X23 is information of the focal depth obtained as described above as the second information. X24 is information of the focus sensitivity as the second information. X25 is information on the defocus amount of the subject obtained as described above as the second information. X26 is information on the sound volume obtained as the second information as described above. Y21 is information of the output related to the control signal for the drive unit 105. In this way, Y21 is generated as an output of the learned machine learning model based on X21 to X26 as inputs. The AI ​​control unit 121 generates a control signal based on Y21 or the Y21 as the control signal, and controls the drive unit 105 by the control signal.

[0067] <Log information> Here, the log information in the second embodiment will be described. The log storage unit 1124 collects and stores input / output information of the machine learning model such as X21 to X26 and Y21 shown in FIG. 12 for each operation cycle of the machine learning model. The log storage unit 1124 stores information on the power consumption of the drive unit 105 obtained by the processing unit 120 as log information. The log storage unit 1124 stores information on the drive command input to the AI ​​control unit 1121 and information on the position of the focus lens group detected by the detection unit 106 as log information. The log storage unit 1124 stores information on the target position and positioning accuracy of the focus lens group obtained by the processing unit 120 as log information. The log storage unit 1124 stores information on the drive speed and drive acceleration of the focus lens group obtained from information on the position of the focus lens group as log information. The log storage unit 1124 stores information indicating the relationship between at least one of the drive speed and drive acceleration and the drive sound volume, and stores information on the drive sound volume generated based on the at least one information and the information on the relationship. Furthermore, the log storage unit 1124 obtains the ratio between the recording volume and the drive volume (S / N ratio with the drive sound as noise) and stores information on this ratio. Note that this S / N ratio indicates the influence that the drive sound, which becomes noise, has on the recording, and the larger the S / N ratio, the smaller the influence of the drive sound on the recording. The log storage unit 1124 causes the log information it stores to be stored in the log storage unit 1222 via the drive control unit 1125.

[0068] <Reward information and driving performance evaluation> The reward information according to the second embodiment will be described with reference to FIG. 13. FIG. 13 is a diagram illustrating reward information. (a1) and (b1) in the figure show the relationship between time (hours) and reward when a machine learning model is being learned for the defocus amount as a driving performance and the above-mentioned S / N ratio, respectively. The horizontal axis of the graphs in (a1) and (b1) shows time (hours). In the figure, (a2) and (b2) show the data structure of reward information for the defocus amount and the S / N ratio, respectively. The data structure, like that in the first embodiment, is configured to include boundary value data and reward data for each range defined by the boundary value for each driving performance.

[0069] The vertical axis of (a1) indicates the value of the defocus amount D. The defocus amount D is a positive value when the focus is shifted to the infinity side, and a negative value when the focus is shifted to the close distance side. (a2) indicates the reward RD for the defocus amount. The reward information RD includes the boundary value D1 and boundary value D2 of the defocus amount, and the reward SD1, reward SD2, and reward SD3 that can be obtained in each range. Here, the range of the defocus amount from D1×-1 to D1 is set as range AD1. The range obtained by excluding the range AD1 from the range from D2×-1 to D2 is set as range AD2. The range obtained by excluding the range AD1 and the range AD2 from the entire range is set as range AD3. As shown in (a2), rewards SD1, SD2, and SD3 are set for the range AD1, range AD2, and range AD3, respectively. Here, the magnitude relationship of the reward is reward SD1>reward SD2>reward SD3, and the higher the reward is set as the defocus amount D is closer to 0. As shown in (a1), the defocus amount D at time Tp1, time Tp2, and time Tp3 belong to the range AD2, range AD3, and range AD1, respectively. Therefore, the rewards that can be obtained at time Tp1, time Tp2, and time Tp3 are reward SD2, reward SD3, and reward SD1, respectively. Here, for example, the boundary value D1 can be a value of Fδ / 2, and the boundary value E2 can be a value of Fδ. In other words, if the defocus amount has a value equal to or less than half the focal depth (|D|≦Fδ / 2), the highest reward is obtained. On the other hand, if the defocus amount has a value exceeding half the focal depth and up to the focal depth (Fδ / 2<|D|≦Fδ), an intermediate reward is obtained. Also, if the defocus amount has a value exceeding the focal depth (|D|>Fδ), the lowest reward is obtained.

[0070] The vertical axis of (b1) indicates the value of the S / N ratio N. The larger the S / N ratio N, the smaller the impact of the driving sound on the quality of the recording. (b2) indicates the reward RN for the S / N ratio. The reward information RN includes the boundary value N1 and boundary value N2 of the S / N ratio, and the reward SN1, reward SN2, and reward SN3 that can be obtained in each range. Here, the range of the S / N ratio from 0 to N1 is set as the range AN1. The range from N1 to N2 is set as the range AN2. The range obtained by excluding the range AN1 and the range AN2 from the entire range is set as the range AN3. As shown in (a2), the reward SN1, reward SN2, and reward SN3 are set for the range AN1, range AN2, and range AN3, respectively. Here, the magnitude relationship of the reward is reward SN1<reward SN2<reward SN3, and the closer the S / N ratio N is to 0, the lower the reward is set. As shown in (b1), the S / N ratios N at times Tp1, Tp2, and Tp3 belong to the ranges AN1, AN3, and AN2, respectively. Therefore, the rewards that can be earned at times Tp1, Tp2, and Tp3 are SN1, SN3, and SN2, respectively. Here, the rewards are set so that the larger the S / N ratio, the more rewards can be earned, so that a machine learning model that is advantageous in terms of recording quality can be generated.

[0071] As described above, reward information for evaluating the defocus amount as a driving performance and the S / N ratio related to the driving sound can be set. By using such reward information, a reward is generated for each unit time for each driving performance based on log information when the focus lens group is driven, and the rewards are accumulated to evaluate the machine learning model. Note that, being based on rewards for a plurality of driving performances is advantageous for customizing the machine learning model. In addition, the boundary value is not limited to being invariant, and may be variable as necessary. In addition, the reward is not limited to being specified based on the boundary value, and may be specified based on a function related to each driving performance. In this case, the reward information may be configured to include information related to the function.

[0072] <First and second remuneration parts> Next, the information of the first reward portion and the information of the second reward portion in this embodiment will be described. Here, FIG. 14 is a diagram illustrating an example of the data structure of the reward information. The information of the first reward portion includes information of the reward RDb related to the defocus amount and the reward RNb related to the S / N ratio. Moreover, the information of the second reward portion includes information of the reward RDu related to the defocus amount and the reward RNu related to the S / N ratio. RDb and RDu have the same data structure as the reward RD related to the defocus amount shown in (a2) of FIG. 13. RNb and RNu have the same data structure as the reward RN related to the S / N ratio shown in (b2) of FIG. 13.

[0073] Here, the information of the first remuneration portion is information regarding remuneration unique to the lens apparatus 100. The information of the first remuneration portion is pre-stored in the first remuneration portion holding unit 1224 as remuneration information unique to the lens apparatus 100. The information of the second remuneration portion is information regarding a variable remuneration based on a request of the operator of the lens apparatus 100. The information of the second remuneration portion is held in the second remuneration portion holding unit 1225 based on the request of the operator. The remuneration holding unit 1223 holds the information of the first remuneration portion and the information of the second remuneration portion.

[0074] The information of the first reward portion is reward information for obtaining an acceptable driving performance in the lens device 100, and therefore a wider range of rewards, including negative values, is set than the information of the second reward portion. The information of the second reward portion is variable at the request of the operator, and can be acquired based on the request information and the option information of the second reward portion. The reward information is obtained from the information of the first reward portion and the information of the second reward portion. Then, based on the reward information, an evaluation value of the machine learning model is obtained, as illustrated with reference to FIG. 13, to learn (generate) the machine learning model.

[0075] Here, a method of obtaining information on the second reward part at the request of a user will be described. FIG. 15 is a diagram illustrating an example of a data structure of information on options of the second reward part. (a) of the figure shows a data structure of information on an option URDu of the second reward part related to the amount of defocus. The information on the option URDu includes, for each level, a boundary value of the amount of defocus, and reward information for each range defined by the boundary value. (b) of the figure shows a data structure of information on an option URNu of the second reward part related to quietness (S / N ratio). The information on URNu includes, for each level, a boundary value of the S / N ratio, and reward information for each range defined by the boundary value.

[0076] The information of the second reward part option URDu related to the defocus amount and the second reward part option URNu related to the quietness (S / N ratio) has boundary values ​​and reward values ​​set so that the operator's requested level decreases in the order (ascending order) of level 1, level 2, and level 3. More specifically, for example, level 1 has a boundary value closer to the target value of the drive performance than the other levels, and the reward is a high value.

[0077] The operator's request may be input via the operation unit 1206 shown in FIG. 11. Based on the request, a level may be selected from level 1 to level 3 for each driving performance. Information on the level is transmitted to the second reward portion holding unit 1225 via the driving control unit 1125. The second reward portion holding unit 1225 identifies (selects) information on the second reward portion for each driving performance based on information on the level of each driving performance. By doing so, learning of the machine learning model (weighting) is performed based on the customized reward information, and a customized machine learning model (weighting) can be generated. Information on the generated machine learning model (weighting) is transmitted from the machine learning unit 1221, stored in the storage unit 123, and used to control the driving of the focus lens group (driving unit 105).

[0078] <Other examples of control targets> Although the focus lens group has been exemplified as an object of drive control, this embodiment is not limited thereto. In this embodiment, other optical members such as a zoom lens group, an image stabilization lens group, a flange back adjustment lens group, and an aperture stop can be objects of drive control. The defocus amount and quietness (S / N ratio) are drive performances that can be taken into consideration when driving these optical members. When these other optical members are objects of drive control, information on other drive performances may be taken into consideration as second information in addition to or instead of the defocus amount.

[0079] As described above, according to the present embodiment, it is possible to provide a lens device or an imaging device that is advantageous in terms of adapting (customizing) driving performance, for example.

[0080] [Embodiment 3] <<Configuration example in which a remote device (processing device) includes a learning unit (generation unit)>> The third embodiment will be described with reference to FIG. 16. FIG. 16 is a diagram showing a configuration example of a lens device according to the third embodiment, and is also a diagram showing a configuration example of a system (imaging device) including a configuration example of a camera body. The system includes a remote device 400, and is different from the first embodiment in that the remote device 400 includes a learning unit. Here, the camera body 200 has a communication unit 230 for communicating with the remote device 400. The remote device 400 can be, for example, a processing device such as a mobile terminal or a computer terminal. The remote device 400 has a display unit 401, an operation unit 402, a processing unit 410, and a learning unit 420. The processing unit 410 includes a control unit 411 and a communication unit 412. The communication unit 412 is for communicating with the camera body 200. Note that the communication unit 412 and the communication unit 230 of the camera body 200 communicate wirelessly, although this is not limited thereto. The wireless communication can be a known wireless communication such as a wireless LAN.

[0081] The learning unit 420 may be configured to include a processor (CPU, GPU, etc.) and a storage device (ROM, RAM, HDD, etc.). The learning unit 420 may include a machine learning unit 421, a log retention unit 422, a reward retention unit 423, a first reward portion retention unit 424, and a second reward portion retention unit 425. The learning unit 420 also stores a program for controlling the operation of each of these units. The operation of the learning unit 420 may be similar to the operation of the learning unit 220 in the first embodiment.

[0082] In this embodiment, unlike the first embodiment, the learning unit is included in the remote device 400, not in the camera body 200, and therefore information is transmitted between the processing unit 210 of the camera body and the learning unit 420 via the communication unit 230, the communication unit 412, and the control unit 411. In addition, image data output from the signal processing circuit 203 is transmitted to the control unit 411 via the control unit 211, the communication unit 230, and the communication unit 412. The image data transmitted to the control unit 411 is displayed by the display unit 401.

[0083] Here, the control unit 411 may transmit a command for implementing machine learning to the machine learning unit 421 in response to an operation of the operator on the operation unit 402. Also, the control unit 211 may transmit a command for implementing machine learning to the machine learning unit 421 via the control unit 411 in response to an operation of the operator on the operation unit 206. Upon receiving the command, the machine learning unit 421 starts machine learning. Similarly, information on the level of the second reward portion for each driving performance input by the user via the operation unit 402 or the operation unit 206 is transmitted to the second reward portion holding unit 425 via the control unit 411. The second reward portion holding unit 425 identifies (selects) information on the second reward portion for each driving performance based on information on the level of each driving performance. By doing so, learning of the machine learning model (weighting) is performed based on the customized reward information, and a customized machine learning model (weighting) can be generated. Information on the generated machine learning model (weighting) is transmitted from the remote device 400 to the lens device 100, stored in the storage unit 123, and used to control the driving of the focus lens group (driving unit 105).

[0084] In this way, a customized machine learning model can be generated in a remote location away from the lens device, where the image obtained by the camera body can be confirmed (viewed). Note that the camera body may issue commands for executing machine learning and commands for setting the second reward portion via the operation unit 206, and the remote device may perform only the machine learning process that requires high-speed arithmetic processing.

[0085] As described above, according to the present embodiment, it is possible to provide a lens device, an imaging device, or a processing device that is advantageous in terms of adapting (customizing) driving performance, for example.

[0086] Here, the second information related to the lens device used for training the machine learning model is only information specific to the lens device in the first and third embodiments, and is both information specific to the lens device and information specific to the camera body in the second embodiment, but is not limited thereto. The second information may be only information specific to the camera body.

[0087] [Embodiments relating to programs, storage media, and data structures] The present invention may also be realized by supplying a program or data (structure) that realizes one or more functions or methods of the above-described embodiments to a system or device via a network or storage medium. In this case, the computer of the system or device may read the program or data (structure) and execute processing based on it. The computer may have one or more processors or circuits, and may include separate computers or a network of separate processors or circuits to read and execute computer-executable instructions.

[0088] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), and the processor or circuitry may include a digital signal processor (DSP), a data flow processor (DFP) or a neural processing unit (NPU).

[0089] Although the preferred embodiments of the present invention have been described above, it goes without saying that the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0090] 100 Lens device 101 Optical components 105 Drive unit 106 Detection unit 120 Processing section

Claims

1. A lens device comprising: an optical member; a drive unit that drives the optical member; a detection unit that detects a state related to the drive; and a processing unit that generates a control signal for the drive unit based on first information related to the state detected by the detection unit, an operation unit for inputting information regarding requirements for the drive performance of the drive unit; A learning unit for learning a machine learning model, The processing unit includes: generating an output related to the control signal using the machine learning model based on the first information and second information related to the driving performance of the driving unit; outputting the first information and the second information to the learning unit; The lens device according to claim 1, wherein the learning section acquires reward information to be used for the learning based on the information related to the request and the second information.

2. 2. The lens device according to claim 1, wherein the information regarding the request input from the operation unit relates to each of a plurality of performances as the driving performance.

3. The lens device described in claim 1 or 2, characterized in that the learning unit has first reward information prepared in advance, and trains the machine learning model based on the first reward information and second reward information different from the first reward information obtained based on information regarding the request.

4. the optical member includes a focus lens group for changing a focus position; The lens device according to claim 1 , wherein the second information includes information regarding at least one of a focal depth of the lens device, a focus movement amount of the lens device per unit movement amount of the focus lens group, and a sound around the lens device.

5. The lens device according to claim 1 , wherein the learning unit acquires the reward information based on information from an image capture device body to which the lens device is attached.

6. 6. The lens device according to claim 5, wherein the information from the imaging device body includes information on at least one of a diameter of an allowable circle of confusion, a defocus amount, and a sound volume.

7. 7. The lens device according to claim 5, wherein the processing unit generates the control signal based on information from the main body of the image pickup device.

8. The lens device according to claim 1 , wherein the learning unit evaluates the machine learning model based on the reward information, and determines the end of the learning based on the evaluation.

9. The lens apparatus according to claim 8 , wherein the learning unit adopts the machine learning model for which the evaluation satisfies an allowable condition.

10. The lens apparatus according to claim 8 , wherein the learning unit rejects the machine learning model that satisfies a termination condition and the evaluation does not satisfy an allowance condition.

11. A lens device according to any one of claims 1 to 10; an image pickup element for capturing an image formed by the lens device; An imaging device comprising:

12. A processing device that performs processing related to a machine learning model in a lens device, a learning unit that causes the machine learning model to learn; and an operation unit that inputs information regarding requirements for a drive performance of a drive unit of the lens device, the lens device includes an optical member, a drive unit that drives the optical member, a detection unit that detects a state related to the drive, and a processing unit that generates a control signal for the drive unit based on first information related to the state detected by the detection unit; The processing unit includes: generating an output related to the control signal using the machine learning model based on the first information and second information related to the driving performance of the driving unit; outputting the first information and the second information to the learning unit; The processing device, characterized in that the learning unit acquires reward information to be used in the learning based on information regarding the request and the second information.

13. The processing device described in claim 12, characterized in that the learning unit has information on a first reward prepared in advance, and trains the machine learning model based on the first reward information and information on a second reward different from the first reward information obtained based on information regarding the request.

14. The processing device according to claim 12 or 13, wherein the learning unit acquires the reward information based on the second information.

15. A processing method for performing processing related to a machine learning model in a lens device, comprising: The lens device includes an optical member, a driving unit that drives the optical member, a detection unit that detects a state related to the driving, a processing unit that generates a control signal for the driving unit based on first information related to the state detected by the detection unit, an operation unit for inputting information related to requirements for driving performance of the driving unit, and a learning unit that causes the machine learning model to learn, generating an output related to the control signal by using the machine learning model based on the first information and second information related to the driving performance of the driving unit, by the processing unit; outputting the first information and the second information to the learning unit by the processing unit; The processing method comprises a step of acquiring, by the learning unit, reward information to be used in the learning based on the information regarding the request and the second information.

16. The processing method described in Claim 15, characterized in that the learning unit trains the machine learning model based on the reward information.

17. A program for causing a computer to execute the processing method according to claim 15 or 16.

Images