Lens device, camera device, processing device, processing method and storage medium

Abstract

The present invention provides a lens device, an imaging device, a processing device, a processing method, and a storage medium. The lens device includes: an optical component; a drive device configured to drive the optical component; a detector configured to detect a state related to the drive; and a processor configured to generate a control signal for the drive device based on first information about the detected state. The processor includes a machine learning model configured to generate an output related to the control signal based on first and second information about the lens device, and the processor outputs the first and second information to a generator configured to generate the machine learning model.

Description

Technical Field

[0001] Aspects of the embodiments relate to a lens device, an imaging device, a processing device, a processing method, and a computer-readable storage medium. Background Art

[0002] Some recent digital cameras can capture not only still images but also moving images. To achieve fast still image capture, high-speed automatic focusing, zooming, and aperture operations are required. Conversely, when capturing moving images, the high operating noise from the drive system responsible for high-speed operation can impair the quality of the sound recorded along with the images. In light of this, Japanese Patent Application Laid-Open No. 2007-006305 discusses an imaging device that switches the operating mode of its actuator to a silent mode during moving image capture.

[0003] Actuators used to drive optical components in imaging devices require various performance characteristics. Examples include drive speed for control followability, positioning accuracy for accurate imaging condition setting, power consumption for continuous imaging duration, and quietness for sound quality during motion picture capture. These performance characteristics are interdependent. For example, the imaging device discussed in Japanese Patent Application Laid-Open No. 2007-006305 improves quietness by limiting drive speed and acceleration.

[0004] The desired quietness can vary depending on the photographic situation. The desired driving speed and acceleration can also vary depending on the photographic situation. The same applies to other performance characteristics, such as positioning accuracy and power consumption. Furthermore, the priority of each performance characteristic can vary depending on the photographic situation and the operator. Therefore, it is desirable to operate the actuator with driving performance that is appropriate for each photographic situation and operator. Summary of the Invention

[0005] The present disclosure provides, in its first aspect, a lens device, comprising: an optical component; a driving device configured to drive the optical component; a detector configured to detect a state related to the driving; and a processor configured to generate a control signal for the driving device based on first information about the detected state, wherein the processor includes a machine learning model configured to generate an output related to the control signal based on first information and second information about the lens device, and the processor outputs the first information and the second information to a generator configured to generate the machine learning model.

[0006] A second aspect of the present disclosure provides an image pickup apparatus including the lens apparatus according to the first aspect; and an image pickup element configured to pick up an image formed by the lens apparatus.

[0007] A third aspect of the present disclosure provides a processing device configured to perform processing related to a machine learning model in a lens device, the lens device comprising: an optical component; a drive device configured to drive the optical component; a detector configured to detect a state related to the drive; and a processor configured to generate a control signal for the drive device based on first information about the detected state, the processor comprising a machine learning model configured to generate an output related to the control signal based on first information and second information about the lens device, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing device comprising: an operating device for an operator to input information related to requirements for driving performance of the drive device, wherein the processing device obtains information about a reward for generating the machine learning model based on the information about the requirements.

[0008] The present disclosure provides, in its fourth aspect, a processing method for performing processing related to a machine learning model in a lens device, the lens device comprising: an optical component; a drive device configured to drive the optical component; a detector configured to detect a state related to the drive; and a processor configured to generate a control signal for the drive device based on first information about the detected state, the processor comprising a machine learning model configured to generate an output related to the control signal based on first information and second information about the lens device, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing method comprising: obtaining information about a reward for generating the machine learning model based on information related to requirements for driving performance of the drive device, the information about the requirements being input from an operating device operated by an operator.

[0009] The present disclosure, in its fifth aspect, provides a computer-readable storage medium storing a program for causing a computer to execute a processing method for performing processing related to a machine learning model in a lens device, the lens device comprising: an optical component; a drive device configured to drive the optical component; a detector configured to detect a state related to the drive; and a processor configured to generate a control signal for the drive device based on first information about the detected state, the processor comprising a machine learning model configured to generate an output related to the control signal based on first information and second information about the lens device, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing method comprising: obtaining information about a reward for generating the machine learning model based on information related to requirements for driving performance of the drive device, the information about the requirements being input from an operating device operated by an operator.

[0010] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating a configuration example of a lens apparatus according to a first exemplary embodiment.

[0012] Figure 2A and Figure 2B is a graph showing positioning accuracy used as driving performance.

[0013] Figure 3A and Figure 3B is a graph showing driving speed used as driving performance.

[0014] Figure 4A and Figure 4B Graph showing the relationship between positioning accuracy, driving speed, power consumption, and quietness.

[0015] Figure 5A and 5B Graph showing the relationship between driving speed, positioning accuracy, power consumption, and quietness.

[0016] Figure 6 is a diagram showing the input and output of a neural network.

[0017] Figure 7 is a flowchart showing the processing procedure of machine learning.

[0018] Figure 8A1 、 Figure 8A2 、 Figure 8B1 、 Figure 8B2 、 Figure 8C1 、 Figure 8C2 、 Figure 8D1 and Figure 8D2 It is a diagram showing reward information.

[0019] Figure 9 This is a diagram showing the data structure of reward information.

[0020] Figures 10A to 10C 2 is a diagram showing the data structure of information related to options of the second bonus part.

[0021] Figure 11 is a diagram illustrating a configuration example of a lens apparatus according to a second exemplary embodiment.

[0022] Figure 12 is a diagram showing the input and output of a neural network.

[0023] Figure 13A1 、 Figure 13A2 、 Figure 13B1 and Figure 13B2 It is a diagram showing reward information.

[0024] Figure 14 This is a diagram showing the data structure of reward information.

[0025] Figure 15A and Figure 15B 2 is a diagram showing the data structure of information related to options of the second bonus part.

[0026] Figure 16 is a diagram illustrating a configuration example of a lens apparatus according to a third exemplary embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. In all drawings used to describe exemplary embodiments, similar components are denoted by the same reference numerals in principle (unless otherwise specified), and redundant descriptions thereof will be omitted.

[0028] [First exemplary embodiment]

[0029] Example of a camera body (processing device) including a training unit (generator)

[0030] Figure 1This figure illustrates an example configuration of a lens device according to the first exemplary embodiment and, by extension, also illustrates an example configuration of a system (imaging device) including an example configuration of a camera body (also referred to as a camera device body, imaging device body, or processing device). The system includes a camera body 200 and a lens device 100 (also referred to as an interchangeable lens) mounted on the camera body 200. The camera body 200 and the lens device 100 are mechanically and electrically connected via a bracket 300 serving as a coupling mechanism. The bracket 300 may be composed of a bracket unit belonging to the camera body 200 or a bracket unit belonging to the lens device 100, or may be configured to include both bracket units. The camera body 200 can supply power to the lens device 100 via a power supply terminal included in the bracket 300. The camera body 200 and the lens device 100 can communicate with each other via a communication terminal included in the bracket 300. In this exemplary embodiment, the lens device 100 and the camera body 200 are connected via the bracket 300. However, the lens device 100 and the camera body 200 may be integrally configured without a bracket.

[0031] The lens device 100 may include a focus lens unit 101 for changing the object distance, a zoom lens unit 102 for changing the focal length, an aperture stop 103 for adjusting the amount of light, and an image stabilization lens unit 104 intended for image stabilization. The focus lens unit 101 and the zoom lens unit 102 are held by respective retaining frames. The retaining frames are configured to be movable in the optical axis direction (the direction of the dashed line in the figure) via, for example, a guide shaft. The focus lens unit 101 is driven along the optical axis by a drive device 105. A detector 106 detects the position of the focus lens unit 101. The zoom lens unit 102 is driven along the optical axis by a drive device 107. A detector 108 detects the position of the zoom lens unit 102. The aperture stop 103 includes aperture blades. The aperture blades are driven by a drive device 109 to adjust the amount of light. A detector 110 detects the amount of opening (also known as the aperture or f-number) of the aperture stop 103. The image stabilization lens unit 104 is driven by a drive device 112 in a direction including a component orthogonal to the optical axis, thereby reducing image shake caused by camera shake. A detector 113 detects the position of the image stabilization lens unit 104. For example, the drive devices 105, 107, 109, and 112 can be configured to include ultrasonic motors. The drive devices 105, 107, 109, and 112 are not limited to ultrasonic motors and can be configured to include other motors such as voice coil motors, direct current (DC) motors, and stepping motors.

[0032] For example, detectors 106, 108, 110, and 113 may be configured to include potentiometers or encoders. If the drive device includes a motor, such as a stepper motor, capable of driving at a given drive amount without requiring drive amount (control amount) feedback, a detector for detecting a specific position (reference position or origin) may be provided. In this case, for example, the detector may include a photointerrupter. Detector 111 detects shake of lens device 100. For example, detector 111 may include a gyroscope.

[0033] The processor 120 may be a microcomputer and may include an artificial intelligence (AI) control unit 121, a determination unit 122, a storage unit 123, a log storage unit 124, a drive control unit 125, and a communication unit 126. The AI control unit 121 controls the driving of the focus lens unit 101. The AI control unit 121 may operate based on a neural network (NN) algorithm. More generally, the AI control unit 121 generates drive instructions for the drive device 105 of the focus lens unit 101 using a machine learning model. The determination unit 122 determines information (second information) about the lens apparatus 100 for use by the AI control unit 121. The storage unit 123 stores information used to identify the type of input (feature quantity) to the NN and information regarding the weights assigned to the inputs of each layer. The log storage unit 124 stores information related to the operation log of the lens apparatus 100 regarding the drive control of the focus lens unit 101. The drive control unit 125 controls the driving of the zoom lens unit 102, the aperture stop 103, and the image stabilization lens unit 104. For example, the drive control unit 125 can generate drive instructions for the drive devices 107, 109, and 112 through proportional-integral-derivative (PID) control based on the deviation between the target position or target speed of the object to be controlled and the actual position or actual speed of the object to be controlled. The communication unit 126 is a communication unit for communicating with the camera body 200. The NN algorithm, weights, second information, and operation logs will be described below.

[0034] The camera body 200 (processing device) may include an imaging element 201, an analog-to-digital (A / D) conversion unit 202, a signal processing circuit 203, a recording unit 204, a display unit 205, an operating device 206, a processor 210 (also referred to as a camera microcomputer), and a training unit 220. The imaging element 201 captures images formed by the lens apparatus 100. For example, the imaging element 201 may include a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) device. The A / D conversion unit 202 converts the analog signal (image signal) captured and output by the imaging element 201 into a digital signal. The signal processing circuit 203 converts the digital signal output from the A / D conversion unit 202 into image data. The recording unit 204 records the image data output from the signal processing circuit 203. The display unit 205 displays the image data output from the signal processing circuit 203. The operating device 206 is provided for an operator (user) to operate the imaging device.

[0035] The processor 210 is designed to control the camera body 200 and may include a control unit 211 and a communication unit 212. The control unit 211 generates drive instructions for the lens device 100 based on image data from the signal processing circuit 203 and input information from the operator of the operating device 206. The control unit 211 also issues instructions and transmits information to the training unit 220 (described below). The communication unit 212 communicates with the lens device 100. The communication unit 212 transmits the drive instructions from the control unit 211 to the lens device 100 as control commands. The communication unit 212 also receives information from the lens device 100.

[0036] The training unit 220 (generator) may include a processor (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)) and a storage device (e.g., a read-only memory (ROM), a random access memory (RAM), and a hard disk drive (HDD)). The training unit 220 may include a machine learning unit 221, a reward storage unit 223, a first reward portion storage unit 224, a second reward portion storage unit 225, and a log storage unit 222. The training unit 220 also stores a program for controlling the operations of the units 221 to 225. The reward information stored in the reward storage unit 223, the information about the first reward portion stored in the first reward portion storage unit 224, the information about the second reward portion stored in the second reward portion storage unit 225, and the information for obtaining the information about the second reward portion from the information input by the operator will be described below.

[0037] <Recording and Display of Image Data>

[0038] We will now describe Figure 1 The camera device shown records and displays image data.

[0039] Light entering the lens apparatus 100 forms an image on the imaging element 201 via the focus lens unit 101, zoom lens unit 102, aperture stop 103, and image stabilization lens unit 104. The imaging element 201 converts the image into an electrical analog signal. The A / D conversion unit 202 converts the analog signal into a digital signal. The signal processing circuit 203 converts the digital signal into image data. The image data output from the signal processing circuit 203 is recorded in the recording unit 204. The image data is also displayed on the display unit 205.

[0040] <Focus Control>

[0041] Next, the focus control of the lens apparatus 100 by the camera body 200 will be described. The control unit 211 performs autofocus (AF) control based on the image data output from the signal processing circuit 203. For example, the control unit 211 performs AF control to drive the focus lens unit 101 so as to maximize the contrast of the image data. The control unit 211 outputs the amount of drive for the focus lens unit 101 as a drive command to the communication unit 212. The communication unit 212 receives the drive command from the control unit 211, converts the drive command into a control command, and transmits the control command to the lens apparatus 100 via the communication contact member of the bracket 300. The communication unit 126 receives the control command from the communication unit 212, converts the control command into a drive command, and outputs the drive command to the AI control unit 121 via the drive control unit 125. Upon receiving the drive command, the AI control unit 121 generates a drive signal based on the machine learning model (trained weights) stored in the storage unit 123, and outputs the drive signal to the drive device 105. The details of how the AI control unit 121 generates the drive signal will be described below. In this manner, the focus lens unit 101 is driven based on a drive instruction from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform AF control to drive the focus lens unit 101 so that the contrast of image data is maximized.

[0042] <Aperture diaphragm control>

[0043] Next, the aperture stop control of the lens apparatus 100 by the camera body 200 will be described. The control unit 211 performs aperture stop control (exposure control) based on the image data output from the signal processing circuit 203. Specifically, the control unit 211 determines a target f-number so that the image data has a constant luminance value. The control unit 211 outputs the determined f-number as a drive command to the communication unit 212. The communication unit 212 receives the drive command from the control unit 211, converts the drive command into a control command, and transmits the control command to the lens apparatus 100 via the communication contact member of the bracket 300. The communication unit 126 receives the control command from the communication unit 212, converts the control command into a drive command, and outputs the drive command to the drive control unit 125. Upon receiving the drive command, the drive control unit 125 determines a drive signal based on the drive command and the f-number of the aperture stop 103 detected by the detector 110, and outputs the drive signal to the drive device 109. In this manner, the aperture stop 103 is driven to maintain the brightness of the image data constant based on a drive instruction from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform exposure control to drive the aperture stop 103 so that the exposure amount of the imaging element 201 is appropriate.

[0044] <Explanation of zoom control>

[0045] Next, the zoom control of the lens apparatus 100 by the camera body 200 will be described. The operator performs a zoom operation on the lens apparatus 100 via the operation device 206. The control unit 211 outputs the drive amount of the zoom lens unit 102 as a drive command to the communication unit 212 based on the zoom operation amount output from the operation device 206. The communication unit 212 receives the drive command, converts the drive command into a control command, and transmits the control command to the lens apparatus 100 via the communication contact member of the bracket 300. The communication unit 126 receives the control command from the communication unit 212, converts the control command into a drive command, and outputs the drive command to the drive control unit 125. Upon receiving the drive command, the drive control unit 125 generates a drive signal based on the drive command and the position of the zoom lens unit 102 detected by the detector 108, and outputs the drive signal to the drive device 107. In this manner, the zoom lens unit 102 is driven based on the drive command from the control unit 211 of the camera body 200. Therefore, the control unit 211 can perform zoom control to drive the zoom lens unit 102 based on the zoom operation amount output from the operation device 206 .

[0046] <Image Stabilization Control>

[0047] Next, the image stabilization control of the lens apparatus 100 will be described. The drive control unit 125 determines a target position for the image stabilization lens unit 104 based on a signal indicating vibration of the lens apparatus 100, output from the detector 111, to reduce image shake caused by the vibration of the lens apparatus 100. The drive control unit 125 generates a drive signal based on the target position and the position of the image stabilization lens unit 104 detected by the detector 113, and outputs the drive signal to the drive device 112. In this manner, the image stabilization lens unit 104 is driven based on the drive signal from the drive control unit 125. Thus, the drive control unit 125 can perform image stabilization control to reduce image shake caused by the vibration of the lens apparatus 100.

[0048] <Drive Performance Related to Focus Control>

[0049] Four types of driving performance related to focus control, namely, positioning accuracy, driving speed, power consumption, and quietness, will be described. These types of driving performance are applicable to various situations in which focus control is performed.

[0050] (1) Positioning accuracy

[0051] Will refer to Figure 2A and Figure 2B Describes the positioning accuracy. Figure 2A and Figure 2B is a graph showing positioning accuracy used as driving performance. Figure 2A and Figure 2B The case where the depth of focus is small and the case where the depth of focus is large are shown respectively. Figure 2A and 2B The f-number is different in the image sensor 201. The target position G of the focus lens unit 101 indicates the position of the focus lens unit 101 at which the point-shaped subject S on the optical axis is focused on the image sensor 201. Position C indicates the actual position of the focus lens unit 101 after the focus lens unit 101 is driven to the target position G. Position C is on the subject S side of the target position G due to a control error (control deviation) E. The image formation position (focal position) Bp indicates the position at which the image of the subject S is formed when the focus lens unit 101 is at position C. The image sensor 201 has an allowable circle of confusion (diameter) δ.

[0052] Figure 2A The f-number (Fa) in Figure 2B The f-number (Fb) in is smaller (brighter). Therefore, Figure 2A The focal depth (2Faδ) is less than Figure 2B Depth of focus (2Fbδ) in. Figure 2A The ray Ca and the ray Ga in φ represent the outermost rays from the subject S when the focus lens unit 101 is located at the position C and the target position G, respectively. Figure 2B The ray Cb and the ray Gb in represent the outermost rays from the subject S when the focus lens unit 101 is located at position C and target position G, respectively. Figure 2A In FIG. 1 , when the focus lens unit 101 is located at position C, the point image of the subject S on the image pickup element 201 has a diameter Ia. Figure 2B In FIG. 2 , when the focus lens unit 101 is located at position C, the point image of the subject S on the imaging element 201 has a diameter Ib.

[0053] exist Figure 2A In the example, the focal position Bp falls outside the focal depth (2Faδ). The diameter Ia of the point image is larger than the permissible circle of confusion δ, and the point image extends beyond the central pixel of the imaging element 201 and to the adjacent pixels. Figure 2A In the case where the focus lens unit 101 is at position C, the subject S is in a defocused state. Figure 2B In the image, the focal position Bp falls within the focal depth (2Fbδ). The diameter Ib of the point image is smaller than the permissible circle of confusion δ, and the point image is located within the center pixel of the imaging element 201. Therefore, in Figure 2B In the example, when the focus lens unit 101 is at point C, the subject S is in focus. Therefore, given the same positioning accuracy of the focus lens unit 101, the focus state may or may not be achieved depending on the photographic conditions. In other words, the desired positioning accuracy varies with the photographic conditions.

[0054] (2) Drive speed

[0055] The driving speed refers to the amount of movement per unit time. The focus movement speed refers to the amount of focus movement per unit time. The amount of movement of the focus lens unit 101 is proportional to the amount of focus movement. The proportional constant in this proportional relationship will be referred to as focus sensitivity. In other words, the focus sensitivity is the amount of focus movement of the lens device 100 per unit movement of the focus lens unit 101. The focus sensitivity varies depending on the state of the optical system constituting the lens device 100. The focus movement amount ΔBp can be expressed by the following formula (1):

[0056] ΔBp=Se×ΔP, (1)

[0057] Here, Se is the focus sensitivity, and ΔP is the movement amount of the focus lens unit 101 .

[0058] Now refer to Figure 3A and Figure 3B Describes the drive speed required for focus control. Figure 3A and Figure 3B is a graph showing driving speed used as driving performance. Figure 3A and 3BThe cases where the focus sensitivity Se is high and the focus sensitivity Se is low are shown respectively. Figure 3A and 3B In , the object distance is different. Figure 3A In the example, when the focus position is moved from position Bp1 to position Bp2, the position of the focus lens unit 101 is moved from position Pa1 to position Pa2. The relationship between the movement amount ΔPa (ΔP) of the focus lens unit 101 and the focus movement amount ΔBp is given by equation (1). Figure 3B In FIG. 1 , when the focus position is moved from position Bp1 to position Bp2, the focus lens unit 101 is moved from position Pb1 to position Pb2. The relationship between the movement amount ΔPb (ΔP) of the focus lens unit 101 and the focus movement amount ΔBp is given by equation (1).

[0059] like Figure 3A and Figure 3B As shown, due to Figure 3A The focus sensitivity is lower than Figure 3B The focus sensitivity in Figure 3A The amount of movement of the focus lens unit 101 required to move the same focus movement amount ΔBp is greater than Figure 3B The amount of movement of the focus lens unit 101 required in Figure 3A Compared with the situation in Figure 3B In this case, the amount of movement of the focus lens unit 101 per unit time can be reduced. In other words, the same focus movement speed can be achieved by reducing the drive speed of the focus lens unit 101. Therefore, the drive speed of the focus lens unit 101 required to achieve a specific focus movement speed depends on the photographic conditions. In other words, the desired movement speed of the focus lens unit 101 varies depending on the photographic conditions.

[0060] (3) Power consumption

[0061] Power consumption varies depending on the drive duration, drive speed, and drive acceleration of the focus lens unit 101. Specifically, when the drive duration is long, the drive speed is high, or the drive acceleration is high, power consumption increases compared to when this is not the case. In other words, if power consumption can be reduced by adapting the drive performance, for example, since the battery capacity can be effectively utilized, the shooting duration per battery charge can be increased or the battery can be miniaturized.

[0062] (4) Quietness

[0063] The driving of the focus lens unit 101 generates drive noise due to vibration and friction. The drive noise varies with the drive speed and drive acceleration of the focus lens unit 101. Specifically, when the drive speed is high or the drive acceleration is high, the drive noise increases compared to when it is not. The longer the focus lens unit 101 remains stationary, the more beneficial the focus control is in terms of quietness. When shooting in a quiet place, unpleasant drive noise may be recorded. Therefore, it may be necessary to adjust the drive noise capability according to the shooting environment (ambient sound level).

[0064] <Relationship between Positioning Accuracy, Drive Speed, Power Consumption, and Quietness>

[0065] Will refer to Figure 4A and Figure 4B Describe the relationship between positioning accuracy and drive speed, power consumption, and quietness. Figure 4A and Figure 4B Graph showing the relationship between positioning accuracy, driving speed, power consumption, and quietness. Figure 4A and Figure 4B The movement of the focus lens unit 101 for continuously focusing on a moving subject is shown in the case where the depth of focus is large and in the case where the depth of focus is small. Figure 4A and Figure 4B , the horizontal axis represents time, and the vertical axis represents the position of the focus lens unit 101. The vertical axis indicates a direction upward toward infinity and indicates a direction downward toward the closest distance.

[0066] The target position G of the focus lens unit 101 indicates the position of the focus lens unit 101 when an image of a subject is focused on the image pickup element 201 . Figure 4A and 4B The focal depths are 2Faδ and 2Fbδ respectively. Figure 4A In FIG. 1 , referring to the target position G, the position GalimI indicates the position of the focus lens unit 101 where the focus is located at the boundary of the focal depth 2Faδ on the infinite distance side, and the position GalimM indicates the position of the focus lens unit 101 where the focus is located at the boundary of the focal depth 2Faδ on the closest distance side. Figure 4B , referring to the target position G, the position GblimI indicates the position of the focus lens unit 101 where the focus is located at the boundary of the focal depth 2Fbδ on the infinity side, and the position GblimM indicates the position of the focus lens unit 101 where the focus is located at the boundary of the focal depth 2Fbδ on the closest distance side. Figure 4A The position (trajectory) Ca and Figure 4B The positions (trajectories) Cb in φ indicate the positions of the focus lens unit 101 controlled so that the subject falls within the depths of focus 2Faδ and 2Fbδ, respectively.

[0067] exist Figure 4AIn , the depth of focus 2Faδ is large, and the subject is less likely to be out of focus due to the control of the focus lens unit 101. In contrast, in Figure 4B In the embodiment, the focal depth 2Fbδ is small, and the trajectory Cb of the focus lens unit 101 is controlled so that the deviation from the target position G is less than Figure 4A More specifically, in Figure 4A and Figure 4B Of the two, when the subject remains in focus, Figure 4B Compared with the driving of trajectory Cb along Figure 4A In other words, under photographic conditions where positioning accuracy is low, the focus lens unit 101 can be controlled at low speed, low power consumption, and low noise.

[0068] <Relationship between drive speed, positioning accuracy, power consumption, and quietness>

[0069] Will refer to Figure 5A and Figure 5B Describe the relationship between drive speed and positioning accuracy, power consumption, and quietness. Figure 5A and 5B Graph showing the relationship between drive speed, positioning accuracy, power consumption, and quietness. Figure 5A and Figure 5B , the horizontal axis represents time, and the vertical axis represents the position of the focus lens unit 101. Figure 5A The focus lens unit 101 is shown in FIG. 1 from time T0 to T1. Figure 3A When the position Pa1 shown is driven to the position Pa2, the position Ca of the lens unit 101 is focused. Figure 5B The focus lens unit 101 is shown in FIG. 1 from time T0 to T1. Figure 3B When the position Pb1 shown in FIG. 1 is driven to the position Pb2, the position Cb of the focusing lens unit 101 is adjusted. Figure 3A and 3B As shown, the focus movement amount in the case where the focus lens unit 101 moves from the position Pa1 to the position Pa2 is the same as the focus movement amount in the case where the focus lens unit 101 moves from the position Pb1 to the position Pb2. Figure 5A and Figure 5B The slopes of the positions Ca and Cb in φ correspond to the driving speed of the focus lens unit 101.

[0070] like Figure 5A and 5BAs shown, in the case of position Ca, the drive speed of the focus lens unit 101 required to achieve the same focus shift amount ΔBp from time T0 to T1 is higher than that required in position Cb. Furthermore, because the drive speed at position Ca is higher than that at position Cb, position Ca takes a longer time to stabilize after the focus lens unit 101 reaches target position Pa2. Conversely, because the drive speed at position Cb is lower than that at position Ca, position Cb takes only a shorter time to stabilize after the focus lens unit 101 reaches target position Pb2. In other words, drive speed affects positioning accuracy. Compared to the focus lens unit 101 at position Cb, the drive acceleration of the focus lens unit 101 at position Ca is higher, resulting in higher power consumption and drive noise. In other words, under photographic conditions requiring a lower drive speed, the focus lens unit 101 can be controlled with high positioning accuracy, low power consumption, and low noise.

[0071] <Second Information Regarding Lens Device>

[0072] Next, the second information regarding the lens apparatus 100 will be described. This second information affects the driving performance of the focus lens unit 101. As described above, in order to adapt the driving performance in the drive control of the focus lens unit 101, a control signal (drive signal) is generated based on the second information affecting the driving performance. The second information is determined by the determination unit 122. For example, the second information includes information regarding the depth of focus and focus sensitivity. The determination unit 122 obtains information regarding the depth of focus from information regarding the f-number and information regarding the permissible circle of confusion. The determination unit 122 stores information (a table) indicating the relationship between focus sensitivity and the position of the focus lens unit 101 and the position of the zoom lens unit 102, and obtains information regarding focus sensitivity from this relationship, information regarding the position of the focus lens unit 101, and information regarding the position of the zoom lens unit 102. Generating a control signal based on this second information can benefit the lens apparatus in terms of adapting (customizing) driving performance such as positioning accuracy, drive speed, power consumption, and quietness. The machine learning algorithm used to generate the control signal based on the second information will be described below.

[0073] <Machine Learning Model>

[0074] The method by which the AI control unit 121 generates a control signal by using a machine learning model will be described. The AI control unit 121 includes a machine learning model and operates based on a machine learning algorithm. The machine learning algorithm here is, but is not limited to, an NN-based algorithm (also referred to as an NN algorithm). The AI control unit 121 refers to the feature quantity to be input to the NN stored in the storage unit 123 and the weights assigned to the inputs of each layer, and uses the feature quantity and weights obtained by reference to generate an output related to the control signal through the NN algorithm. The method for generating a machine learning model (weight) will be described below.

[0075] Will refer to Figure 6 Concepts representing the input and output structures of the machine learning model according to the first exemplary embodiment are described. Figure 6 is a diagram showing the input and output of NN. Figure 6 , input X1 is information about a driving instruction output from the driving control unit 125. Input X2 is information about the position of the focus lens unit 101 obtained from the detector 106. Input X3 is information about a focal depth serving as the second information. Input X4 is information about a focus sensitivity serving as the second information. Output Y1 is information about an output related to a control signal for the driving device 105. Thus, output Y1 of the trained machine learning model is generated based on inputs X1 to X4. The AI control unit 121 generates output Y1 as a control signal or generates a control signal based on output Y1, and controls the driving device 105 by using the control signal.

[0076] <Method for generating machine learning models (weights)>

[0077] Next, a method for generating a machine learning model (weight) (trained by the machine learning unit 221) will be described. The control unit 211 sends an instruction related to executing machine learning to the machine learning unit 221 based on the operator's operation on the operation device 206. Upon receiving the instruction, the machine learning unit 221 starts machine learning. Figure 7 Describe the process of machine learning performed by the machine learning unit 221. Figure 7 is a flowchart showing the processing procedure of machine learning.

[0078] exist Figure 7In step S101, the machine learning unit 221 initializes the machine learning model (weight). Specifically, the machine learning unit 221 outputs the initial value of the weight to the control unit 211. The control unit 211 receives the initial value of the weight from the machine learning unit 221 and sends the initial value of the weight to the lens device 100 via the communication unit 212. The drive control unit 125 of the lens device 100 receives the initial value of the weight via the communication unit 126 and stores the initial value in the storage unit 123. Subsequently, in step S102, the machine learning unit 221 obtains log information. Specifically, the machine learning unit 221 requests the control unit 211 to obtain log information about the lens device 100. Upon receiving the request, the control unit 211 requests the lens device 100 for log information via the communication unit 212. The drive control unit 125 of the lens device 100 receives the request for log information via the communication unit 126 and instructs the AI control unit 121 to drive the focus lens unit 101. The AI control unit 121 receives a driving instruction and generates a control signal for driving the device 105 based on a machine learning model using weights stored in the storage unit 123. The machine learning unit 221 stores a predetermined training driving pattern for driving the focus lens unit 101 from a start position to a stop position and generates a control signal corresponding to the training driving pattern. A training driving pattern determined based on an autofocus algorithm can be used instead of the predetermined training driving pattern. The drive control unit 125 receives a request for log information via the communication unit 126 and requests the log storage unit 124 to output the log information. The log storage unit 124 receives the output request and transmits log information about the lens apparatus 100 during the driving of the focus lens unit 101 to the camera body 200 via the drive control unit 125 and the communication unit 126. The log information is stored in the log storage unit 222.

[0079] In step S103, the machine learning unit 221 evaluates the driving performance of the focusing lens unit 101. Specifically, the machine learning unit 221 evaluates the driving performance of the focusing lens unit 101 driven by using the control signal generated by the AI control unit 121 based on the reward information stored in the reward storage unit 223 and the log information stored in the log storage unit 222. The details of the evaluation will be described below. In step S104, the machine learning unit 221 updates the machine learning model (weight). Specifically, the machine learning unit 221 updates the machine learning model (weight) based on the evaluation value obtained from the evaluation (for example, so that the evaluation value is maximized). The weight can be updated by, but not limited to, back propagation. Through processing similar to that of step S101, the generated weight (machine learning model) is stored in the storage unit 123.

[0080] In step S105, the machine learning unit 221 determines whether to end machine learning. Specifically, for example, the machine learning unit 221 makes a determination based on whether the number of trainings (weight updates) reaches a predetermined value or whether the amount of change in the evaluation value of the driving performance is less than a predetermined value. If the machine learning unit 221 determines not to end machine learning ("No" in step S105), the process returns to step S101, and the machine learning unit 221 continues machine learning. If the machine learning unit 221 determines to end machine learning ("Yes" in step S105), the process ends. The machine learning unit 221 adopts a machine learning model that is evaluated to meet the acceptance condition (for example, the amount of change in the evaluation value of the driving performance is less than a predetermined value). The machine learning unit 221 does not adopt a machine learning model that meets the end condition (for example, the number of trainings reaches a predetermined value) but does not meet the acceptance condition.

[0081] The machine learning algorithm may be deep learning, which uses a neural network and generates its own weights for layer inputs. Deep learning can even generate its own feature quantities. Machine learning algorithms are not limited to deep learning, and other algorithms may be used. Examples may include at least one of the following: a nearest neighbor algorithm, a naive Bayes algorithm, a decision tree, and a support vector machine. Any such available algorithm may be appropriately applied to this exemplary embodiment.

[0082] The GPU can efficiently perform parallel data processing and is therefore effective when performing repeated training using machine learning models such as models used in deep learning. Therefore, the GPU can be used for processing in the machine learning unit 221 instead of or in addition to the CPU. For example, a machine learning program including a machine learning model can be executed through the collaboration of the CPU and the GPU.

[0083] <Log Information>

[0084] Next, the log information will be described. The log information includes information for evaluating the driving performance of the focus lens unit 101. The log storage unit 124 collects and stores input / output information about the machine learning model in each operation period of the machine learning model, such as Figure 6Inputs X1 to X4 and output Y1 are shown. The log storage unit 124 stores information about the power consumption of the drive device 105 obtained by the processor 120 as log information. The log storage unit 124 also stores information about drive instructions input to the AI control unit 121 and the position of the focus lens unit 101 detected by the detector 106 as log information. The log storage unit 124 also stores information about the target position and positioning accuracy of the focus lens unit 101 obtained by the processor 120 as log information. The log storage unit 124 also stores information about the drive speed and drive acceleration of the focus lens unit 101, derived from the information about the position of the focus lens unit 101, as log information. The log storage unit 124 transmits the stored log information to the camera body 200 via the drive control unit 125 and the communication unit 126. The control unit 211 of the camera body 200 receives the log information via the communication unit 212 and stores it in the log storage unit 222.

[0085] <Reward Information and Evaluation of Driver Performance>

[0086] The reward information is information for evaluating driving performance. The reward information includes information on boundary values for determining ranges for each type of driving performance and information on rewards determined in advance for each range. Figures 8A1 to 8D2 Describes the reward information. Figures 8A1 to 8D2 is a diagram showing an example of reward information. Figure 8A1 、 Figure 8B1 、 Figure 8C1 and Figure 8D1 The graph shows the relationship between time and reward when training a machine learning model for positioning accuracy, driving speed, driving acceleration, and power consumption, respectively, as driving performance indicators. The horizontal axis of the graph represents time, and the vertical axis represents driving performance and boundary values. Figure 8A2 、 Figure 8B2 、 Figure 8C2 and Figure 8D2 The data structures of reward information for positioning accuracy, driving speed, driving acceleration, and power consumption are shown separately. The data structures include data on boundary values and data on rewards within each range.

[0087] The machine learning model is trained to improve the evaluation of driving performance. For example, if the expected driving performance is positioning accuracy, the highest reward is assigned to the range that includes a position deviation of 0. Certain types of driving performance are assigned relatively high rewards, thereby giving them priority over other types of driving performance. For example, a relatively high reward is assigned to power consumption, thereby giving it priority over positioning accuracy. In this exemplary embodiment, the reward information will be described as including information with two boundary values and information with three rewards.

[0088] Figure 8A1 The vertical axis indicates the value of positional deviation E, which is the difference between the target position and the actual position of the focus lens unit 101. The positive direction of positional deviation E corresponds to a case where the actual position of the focus lens unit 101 is on the infinitely far side of the target position. The negative direction of positional deviation E corresponds to a case where the actual position is on the closest distance side of the target position. The more frequently the positional deviation E approaches 0 (the smaller the sum of the positional deviations E), the higher the positioning accuracy of the focus lens unit 101. Figure 8A2 The reward information RE regarding positioning accuracy is shown. The reward information RE includes the boundary value E1 and the boundary value E2 of the position deviation E, and the rewards SE1, SE2, and SE3 that can be obtained within each range. The range in which the position deviation E is E1×-1 to E1 will be referred to as the range AE1. The range obtained by excluding the range AE1 from the range in which the position deviation E is E2×-1 to E2 will be referred to as the range AE2. The range obtained by excluding the ranges AE1 and AE2 from the entire range will be referred to as the range AE3. Figure 8A2 As shown, ranges AE1, AE2, and AE3 are assigned rewards SE1, SE2, and SE3, respectively. The relationship between the rewards is reward SE1 > reward SE2 > reward SE3. The closer the position deviation E is to 0, the higher the reward assigned. Figure 8A1 As shown, the position deviation E at time Tp1, Tp2 and Tp3 belongs to the ranges AE2, AE3 and AE1, respectively. Therefore, the rewards obtainable at time Tp1, Tp2 and Tp3 are rewards SE2, SE3 and SE1, respectively. Here, for example, the boundary value E1 can have a value Fδ / 2, and the boundary value E2 can have a value Fδ. In other words, if the deviation between the actual position of the focusing lens unit 101 and the target position is less than or equal to half the depth of focus (|E|≤Fδ / 2), the highest reward SE1 is obtained. If the deviation between the actual position of the focusing lens unit 101 and the target position is greater than half the depth of focus and up to the depth of focus (Fδ / 2<|E|≤Fδ), the intermediate reward SE2 is obtained. If the deviation between the actual position of the focusing lens unit 101 and the target position exceeds the depth of focus (|E|>Fδ), the lowest reward SE3 is obtained.

[0089] Figure 8B1 The vertical axis indicates the value of the driving speed V of the focus lens unit 101. The positive direction of the driving speed V indicates the direction toward infinity. The negative direction of the driving speed V indicates the direction toward the minimum distance. The closer the driving speed V is to 0, the lower the driving noise. Figure 8B2The reward information RV about the driving speed V is shown. The reward information RV includes the boundary values V1 and V2 of the driving speed V, and the rewards SV1, SV2, and SV3 obtainable within each range. The range in which the driving speed V is V1×-1 to V1 will be referred to as the range AV1. The range obtained by excluding the range AV1 from the range in which the driving speed V is V2×-1 to V2 will be referred to as the range AV2. The range obtained by excluding the ranges AV1 and AV2 from the entire range will be referred to as the range AV3. Figure 8B2 As shown, ranges AV1, AV2, and AV3 are assigned rewards SV1, SV2, and SV3, respectively. The relationship between the rewards is reward SV1 > reward SV2 > reward SV3. The closer the driving speed V is to 0, the higher the reward assigned. Figure 8B1 As shown, the driving speed V at times Tp1, Tp2, and Tp3 falls within the ranges AV2, AV3, and AV1, respectively. Therefore, the rewards available at times Tp1, Tp2, and Tp3 are rewards SV2, SV3, and SV1, respectively. Here, for example, the boundary values V1 and V2 are set based on the relationship between the driving speed V and the driving noise. By setting the rewards so that the reward increases as the driving speed V decreases, since the driving noise decreases as the driving speed V decreases, a machine learning model that takes quietness into account can be obtained.

[0090] Figure 8C1 The vertical axis indicates the value of the driving acceleration A of the focus lens unit 101. The positive direction of the driving acceleration A indicates the direction toward infinity. The negative direction of the driving acceleration A indicates the direction toward the closest distance. The closer the driving acceleration A is to 0, the lower the driving noise. Figure 8C2 The reward information RA about the driving acceleration A is shown. The reward information RA includes the boundary values A1 and A2 of the driving acceleration A, and the rewards SA1, SA2, and SA3 obtainable within each range. The range in which the driving acceleration A is A1×-1 to A1 will be referred to as the range AA1. The range obtained by excluding the range AA1 from the range A2×-1 to A2 will be referred to as the range AA2. The range obtained by excluding the ranges AA1 and AA2 from the entire range will be referred to as the range AA3. Figure 8C2 As shown, ranges AA1, AA2, and AA3 are assigned rewards SA1, SA2, and SA3, respectively. The magnitude relationship between the rewards is reward SA1 > reward SA2 > reward SA3. The closer the driving acceleration A is to 0, the higher the reward assigned. Figure 8C1As shown, the driving acceleration A at times Tp1, Tp2, and Tp3 falls within ranges AA1, AA3, and AA2, respectively. Therefore, the rewards available at times Tp1, Tp2, and Tp3 are rewards SA1, SA3, and SA2, respectively. Here, for example, boundary values A1 and A2 are set based on the relationship between driving acceleration A and driving noise. By setting the rewards so that the available rewards increase as driving acceleration A decreases, and since driving noise decreases as driving acceleration A decreases, a machine learning model that takes quietness into account can be obtained.

[0091] Figure 8D1 The vertical axis indicates the value of the power consumption P of the driving device 105 . Figure 8D2 The reward information RP about the power consumption P is shown. The reward information RP includes the boundary values P1 and P2 of the power consumption, and the rewards SP1, SP2, and SP3 that can be obtained within the respective ranges. The range in which the power consumption P is 0 to P1 will be referred to as the range AP1. The range in which the power consumption P is higher than P1 and not higher than P2 will be referred to as the range AP2. The range obtained by excluding the ranges AP1 and AP2 from the entire range is referred to as the range AP3. Figure 8D2 As shown, ranges AP1, AP2, and AP3 are assigned bonuses SP1, SP2, and SP3, respectively. The magnitude relationship between the bonuses is bonus SP1 > bonus SP2 > bonus SP3. The closer the power consumption P is to 0, the higher the bonus assigned. Figure 8D1 As shown, the power consumption P at times Tp1, Tp2, and Tp3 falls within ranges AP1, AP3, and AP2, respectively. Therefore, the rewards available at times Tp1, Tp2, and Tp3 are rewards SP1, SP3, and SP2, respectively. By setting the rewards so that the available rewards increase as power consumption decreases, a machine learning model that takes low power consumption into account can be obtained.

[0092] In this way, reward information for evaluating driving performance such as positioning accuracy (position deviation), driving speed, driving acceleration and power consumption can be set. Using the reward information, the machine learning unit 221 can generate rewards for each type of driving performance per unit time based on the log information when driving the focusing lens unit 101, and accumulate the rewards to evaluate the machine learning model. It is beneficial to customize the machine learning model based on rewards related to multiple types of driving performance. The power consumption can be measured based on the current flowing through the driving device 105, or the power consumption can be estimated based on the driving speed and / or driving acceleration. The boundary value is not limited to a constant value and can be changed appropriately. The reward is not limited to a reward determined based on the boundary value, but can be determined based on a function related to each type of driving performance. In this case, the reward information may include information about the function.

[0093] <First Reward Part and Second Reward Part>

[0094] Next, the first bonus part and the second bonus part of the bonus information will be described. Figure 9 : is a diagram showing the data structure of reward information. Information about the first reward part (pre-prepared first reward information) includes information about the reward REb related to positioning accuracy, the reward RVb related to driving speed, the reward RAb related to driving acceleration, and the reward RPb related to power consumption. Information about the second reward part (second reward information) includes information about the reward REu related to positioning accuracy, the reward RVu related to driving speed, the reward RAu related to driving acceleration, and the reward RPu related to power consumption. Rewards REb and REu have Figure 8A2 The data structure of the reward information RE about positioning accuracy is similar to that shown in FIG. The rewards RVb and RVu have the same Figure 8B2 The data structure of the reward information RV about the driving speed is similar to that shown in FIG. The rewards RAb and RAu have the same Figure 8C2 The data structure of the reward information RA about driving acceleration is similar to that shown in FIG. The rewards RPb and RPU have the same Figure 8D2 The data structure of the reward information RP on power consumption shown is similar to the data structure.

[0095] The information regarding the first reward portion is information related to a reward unique to the lens apparatus 100. The information regarding the first reward portion is pre-stored in the first reward portion storage unit 224 as reward information unique to the lens apparatus 100. The information regarding the second reward portion is information related to a reward that is variable based on a request from the operator of the lens apparatus 100. The information regarding the second reward portion is stored in the second reward portion storage unit 225 based on the operator's request. The reward storage unit 223 stores the information regarding the first reward portion and the information regarding the second reward portion.

[0096] The information about the first reward part is reward information for obtaining the permissible driving performance of the lens device 100, and therefore includes a wider range of reward settings including negative values than the information about the second reward part. The information about the second reward part is variable based on the operator's request and can be obtained based on the information about the request and the information about the options of the second reward part. The reward information is obtained from the information about the first reward part and the information about the second reward part. As shown in FIG. Figures 8A1 to 8D2 As described, the machine learning model is trained (generated) by obtaining an evaluation value of the machine learning model based on reward information.

[0097] A method for obtaining information about the second bonus part based on the operator's request will now be described. Figures 10A to 10C 2 is a diagram showing the data structure of information related to options of the second bonus part. Figure 10A The data structure of information related to option UREu of the second bonus part related to positioning accuracy is shown. The information about option UREu includes the boundary value of the position deviation and the bonus information related to each range defined by the boundary value of each level. Figure 10B The data structure of information related to option URSu of the second bonus component related to quietness is shown. Information related to option URSu includes information related to option URVu of the second bonus component related to driving speed and information related to option URAu of the second bonus component related to driving acceleration. Information related to option URVu includes boundary values for driving speed and bonus information related to the respective ranges defined by the boundary values for each level. Information related to option URAu includes boundary values for driving acceleration and bonus information related to the respective ranges defined by the boundary values for each level. Figure 10C The data structure of information related to option URPu of the second bonus part related to power consumption is shown. The information about option URPu includes the boundary value of power consumption and bonus information related to each range defined by the boundary value of each level.

[0098] Information related to the second bonus option UREu for positioning accuracy, information related to the second bonus option URSu for quietness, and information related to the second bonus option URPu for power consumption are set as follows. Within each of these information types, the boundary value and bonus value are set so that the operator's request level decreases in the order of Levels 1, 2, and 3 (in ascending order). More specifically, for example, Level 1 has a boundary value closer to the target value for driving performance and a higher bonus value than the other levels.

[0099] Can be passed Figure 1 The operation device 206 shown inputs the operator's request. Based on the request, the level of each type of driving performance can be selected from levels 1 to 3. Information about the level is sent to the second reward part storage unit 225 via the control unit 211. The second reward part storage unit 225 identifies (selects) information about the second reward part related to each type of driving performance based on the information about the level of each type of driving performance. Therefore, a customized machine learning model (weight) can be generated by training the machine learning model (weight) based on the customized information about the reward. Information about the generated machine learning model (weight) is sent from the camera body 200 to the lens device 100, stored in the storage unit 123, and used to control the drive of the focus lens unit 101 (drive device 105).

[0100] <Other examples of objects to be controlled>

[0101] Although the drive control is described as being directed to the focus lens unit 101, the present exemplary embodiment is not limited thereto. In the present exemplary embodiment, the drive control may be directed to other optical components, such as the zoom lens unit 102, the image stabilization lens unit 104, the flange rear adjustment lens unit, and the aperture stop 103. Positioning accuracy, quietness, and power consumption are also considered when driving such optical components. The required positioning accuracy of the zoom lens unit 102 may vary depending on the relationship between the drive amount and the amount of change in the angle of view or the amount of change in the size of the subject. The required positioning accuracy of the image stabilization lens unit 104 may vary depending on the focal length. The required positioning accuracy of the aperture stop 103 may vary depending on the relationship between the drive amount and the amount of change in the brightness of the video image.

[0102] <Other Examples of Second Information>

[0103] Information regarding focus sensitivity and depth of focus has been described as the second information regarding lens apparatus 100. However, this is not restrictive, and the second information may include information regarding at least one of the posture, temperature, and ambient sound level of lens apparatus 100. Depending on the posture of lens apparatus 100, the influence of gravity on the optical components changes, thereby changing the load (torque) of drive device 105. Depending on the temperature of lens apparatus 100, the properties of the lubricant in the drive system change, thereby changing the load (torque) of drive device 105. The sound level surrounding lens apparatus 100 affects the limit on the drive noise of drive device 105, thereby changing the limit on the speed and acceleration of drive device 105.

[0104] As described above, in the present exemplary embodiment, for example, a lens apparatus or an imaging apparatus that is advantageous in terms of adaptation (customization) of driving performance can be provided.

[0105] [Second exemplary embodiment]

[0106] Example of a lens device including a training unit (generator)

[0107] Will refer to Figures 11 to 15B A second exemplary embodiment is described. Figure 11 This figure illustrates an example configuration of a lens apparatus according to the second exemplary embodiment, and by extension, also illustrates an example configuration of a system (imaging apparatus) including an example configuration of a camera body. This system differs from the system of the first exemplary embodiment in that the lens apparatus 100 includes a training unit. Another difference from the first exemplary embodiment is that the second information regarding the lens apparatus 100 includes information regarding recordings made by the camera body.

[0108] The training unit 1220 may include a processor (such as a CPU or GPU) and a storage device (such as a ROM, RAM, or HDD). The training unit 1220 may include a machine learning unit 1221, a log storage unit 1222, a reward storage unit 1223, a first reward portion storage unit 1224, and a second reward portion storage unit 1225. The training unit 1220 also stores a program for controlling the operations of these units.

[0109] In addition to the functions of the drive control unit 125 according to the first exemplary embodiment, the drive control unit 1125 also has a function of exchanging information with the training unit 1220. The AI control unit 1121 controls the driving of the focus lens unit 101 (driving device 105) based on the machine learning model generated by the training unit 1220. The determination unit 1122 is a determination unit that determines information (second information) about the lens apparatus 100 for use by the AI control unit 1121. The second information will be described below. The operation device 1206 is an operation device for the operator to operate the lens apparatus 100 (imaging device).

[0110] <Second Information>

[0111] The second information here includes information regarding the impact of the drive control of the focus lens unit 101 on recording by the camera body 200. In this exemplary embodiment, the drive of the focus lens unit 101 can be controlled by taking into account the impact of the control on recording based on this second information in addition to or instead of the second information according to the first exemplary embodiment. The second information may include information obtained by the control unit 211 analyzing image data obtained by the signal processing circuit 203. The second information can be determined based on information transmitted from the control unit 211 to the determination unit 1122 via the communication unit 212, the communication unit 126, and the drive control unit 1125. For example, the second information may include information regarding at least one of the following: the permissible circle of confusion, the amount of defocus of the subject obtained by imaging with the camera body 200, and the sound level (the level of recorded ambient sound) obtained by the microphone 200 included in the camera body 200. The determination unit 1122 can obtain information regarding the depth of focus from the information regarding the f-number and the permissible circle of confusion.

[0112] <Machine Learning Model>

[0113] The machine learning model in the AI control unit 1121 will now be described. Figure 12 is a diagram showing the input and output of NN. Figure 12In the illustrated NN according to the second exemplary embodiment, input X21 is information regarding a drive instruction output from drive control unit 1125. Input X22 is information regarding the position of focus lens unit 101 obtained from detector 106. Input X23 is information regarding the depth of focus obtained as the second information as described above. Input X24 is information regarding focus sensitivity used as the second information. Input X25 is information regarding the amount of defocus of the subject obtained as the second information as described above. Input X26 is information regarding the sound level obtained as the second information as described above. Output Y21 is information regarding an output related to a control signal for drive device 105. In this manner, output Y21 of the trained machine learning model is generated based on inputs X21 to X26. AI control unit 1121 generates output Y21 as a control signal or generates a control signal based on output Y21, and controls drive device 105 using the control signal.

[0114] <Log Information>

[0115] The log information according to the second exemplary embodiment will be described. The log storage unit 1124 collects and stores input / output information about the machine learning model in each operation period of the machine learning model, such as Figure 12 , and the inputs X21 to X26 and output Y21 shown in FIG. The log storage unit 1124 stores information about the power consumption of the drive device 105 obtained by the processor 120 as log information. The log storage unit 1124 also stores information about the drive instructions input to the AI control unit 1121 and the position of the focus lens unit 101 detected by the detector 106 as log information. The log storage unit 1124 also stores information about the target position and positioning accuracy of the focus lens unit 101 obtained by the processor 120 as log information. The log storage unit 1124 also stores information about the drive speed and drive acceleration of the focus lens unit 101 obtained from the information about the position of the focus lens unit 101 as log information. The log storage unit 1124 also stores information indicating the relationship between at least one of the drive speed and drive acceleration and the drive noise level, and stores information about the drive noise level generated based on the information about at least one of the drive speed and drive acceleration and the information indicating the relationship. The log storage unit 1124 also obtains the ratio of the recorded sound level to the driving noise level (the signal-to-noise (S / N) ratio, with the driving noise as noise), and stores information about this ratio. The S / N ratio indicates the impact of the driving noise on the recording. The higher the S / N ratio, the smaller the impact of the driving noise on the recording. The log storage unit 1124 stores the stored log information in the log storage unit 1222 via the driving control unit 1125.

[0116] <Reward Information and Driver Performance Evaluation>

[0117] Will refer to Figures 13A1 to 13B2 The reward information according to the second exemplary embodiment is described. Figures 13A1 to 13B2 It is a diagram showing reward information. Figure 13A1 and Figure 13B1 The relationship between time and reward when training a machine learning model is shown, respectively, with respect to the defocus amount and S / N ratio used as driving performance. Figure 13A1 and Figure 13B1 The horizontal axis of the graph represents time. Figure 13A2 and Figure 13B2 The data structure of reward information for defocus amount and S / N ratio is shown separately. Similar to the data structure in the first exemplary embodiment, this data structure includes data on boundary values for each type of driving performance and data on rewards within each range defined by the boundary values.

[0118] Figure 13A1 The vertical axis indicates the value of the defocus amount D. If the focus moves to the infinity side, the defocus amount D has a positive value, and if the focus moves to the closest distance side, the defocus amount D has a negative value. Figure 13A2 Reward information RD about the defocus amount D is shown. The reward information RD includes a boundary value D1 and a boundary value D2 of the defocus amount D, and rewards SD1, SD2, and SD3 that can be obtained within each range. The range in which the defocus amount D is D1×-1 to D1 will be referred to as range AD1. The range obtained by excluding range AD1 from the range of D2×-1 to D2 will be referred to as range AD2. The range obtained by excluding ranges AD1 and AD2 from the entire range will be referred to as range AD3. Figure 13A2 As shown, ranges AD1, AD2, and AD3 are assigned rewards SD1, SD2, and SD3, respectively. The relationship between the rewards is reward SD1 > reward SD2 > reward SD3. The closer the defocus amount D is to 0, the higher the reward assigned. Figure 13A1 As shown, the defocus amount D at times Tp1, Tp2, and Tp3 belongs to the ranges AD2, AD3, and AD1, respectively. Therefore, the rewards obtainable at times Tp1, Tp2, and Tp3 are rewards SD2, SD3, and SD1, respectively. Here, for example, the boundary value D1 may have a value Fδ / 2, and the boundary value D2 may have a value Fδ. In other words, if the defocus amount D has a value less than or equal to half the depth of focus (|D|≤Fδ / 2), the highest reward SD1 is obtained. If the value of the defocus amount D is greater than half the depth of focus and up to the depth of focus (Fδ / 2<|D|≤Fδ), the intermediate reward SD2 is obtained. If the defocus amount D has a value exceeding the depth of focus (|D|>Fδ), the lowest reward SD3 is obtained.

[0119] Figure 13B1The vertical axis indicates the value of the S / N ratio N. The higher the S / N ratio N, the smaller the influence of drive noise on recording quality. Figure 13B2 The reward information RN about the S / N ratio is shown. The reward information RN includes the boundary value N1 and the boundary value N2 of the S / N ratio, and the rewards SN1, SN2, and SN3 that can be obtained within each range. The range of the S / N ratio from 0 to N1 will be referred to as the range AN1. The range from N1 to N2 will be referred to as the range AN2. The range obtained by excluding the ranges AN1 and AN2 from the entire range will be referred to as the range AN3. Figure 13B2 As shown, ranges AN1, AN2, and AN3 are assigned rewards SN1, SN2, and SN3, respectively. The relationship between the rewards is reward SN1 < reward SN2 < reward SN3. The closer the signal-to-noise ratio N is to 0, the lower the assigned reward. Figure 13B1 As shown, the S / N ratios N at times Tp1, Tp2, and Tp3 fall within ranges AN1, AN3, and AN2, respectively. Therefore, the rewards available at times Tp1, Tp2, and Tp3 are rewards SN1, SN3, and SN2, respectively. Since the rewards are set so that the available rewards increase as the signal-to-noise ratio increases, a machine learning model that is beneficial in terms of recording quality can be generated.

[0120] As described above, reward information for evaluating the defocus amount and the S / N ratio related to the driving noise used as driving performance can be set. Using such reward information, the machine learning unit 1221 can generate rewards for each type of driving performance per unit time based on the log information when the focusing lens unit 101 is driven, and accumulate the rewards to evaluate the machine learning model. It is beneficial to customize the machine learning model based on rewards related to multiple types of driving performance. The boundary value is not limited to a constant value and can be changed appropriately. The reward is not limited to a reward determined based on the boundary value, but can be determined based on a function related to each type of driving performance. In this case, the reward information may include information about the function.

[0121] <First Reward Part and Second Reward Part>

[0122] Next, information about the first bonus part and information about the second bonus part according to the present exemplary embodiment will be described. Figure 14 : is a diagram showing the data structure of reward information. The information about the first reward part includes information about the reward RDb related to the defocus amount and the reward RNb related to the S / N ratio. The information about the second reward part includes information about the reward RDu related to the defocus amount and the reward RNi related to the S / N ratio. The rewards RDb and RDu have Figure 13A2 The reward RNb and RNi have the same data structure as the reward information RD about the defocus amount. Figure 13B2The data structure of the reward information RN regarding the S / N ratio shown is similar to the data structure.

[0123] The information regarding the first reward portion is information regarding a reward unique to the lens apparatus 100. The information regarding the first reward portion is pre-stored in the first reward portion storage unit 1224 as reward information unique to the lens apparatus 100. The information regarding the second reward portion is information regarding a reward that is variable based on a request from the operator of the lens apparatus 100. The information regarding the second reward portion is stored in the second reward portion storage unit 1225 based on the operator's request. The reward storage unit 1223 stores the information regarding the first reward portion and the information regarding the second reward portion.

[0124] The information about the first reward part is reward information for obtaining the permissible driving performance of the lens device 100, and therefore includes a wider range of reward settings including negative values than the information about the second reward part. The information about the second reward part is variable based on the operator's request and can be obtained based on the information about the request and the information about the options of the second reward part. The reward information is obtained from the information about the first reward part and the information about the second reward part. By referring to Figures 13A1 to 13B2 The reward information obtains the evaluation value of the machine learning model to train (generate) the machine learning model.

[0125] A method for obtaining information about the second bonus part based on the operator's request will now be described. Figure 15A and Figure 15B 2 is a diagram showing the data structure of information related to options of the second bonus part. Figure 15A The data structure of information related to the option URDu of the second bonus part related to the defocus amount is shown. The information on the option URDu includes the boundary value of the defocus amount and bonus information on each range defined by the boundary value of each level. Figure 15B The data structure of information related to option URNu of the second bonus part related to quietness (S / N ratio) is shown. The information about option URNu includes the boundary value of the S / N ratio and the bonus information about each range defined by the boundary value of each level.

[0126] In the information regarding the option URDu of the second bonus section related to the defocus amount and the information regarding the option URDu of the second bonus section related to quietness (S / N ratio), the boundary value and the bonus value are both set so that the operator's request level decreases in the order (ascending order) of levels 1, 2, and 3. More specifically, for example, the boundary value of level 1 is close to the target value of driving performance and the bonus value is high compared to the other levels.

[0127] Can be passed Figure 11The operating device 1206 shown inputs the operator's request. Based on the request, the level of each type of driving performance can be selected from levels 1 to 3. Information about the level is sent to the second reward part storage unit 1225 via the drive control unit 1125. The second reward part storage unit 1225 identifies (selects) information about the second reward part related to each type of driving performance based on the information about the level of each type of driving performance. Therefore, a customized machine learning model (weight) can be generated by training the machine learning model (weight) based on the customized information about the reward. Information about the generated machine learning model (weight) is sent from the machine learning unit 1221, stored in the storage unit 123, and used to control the drive of the focus lens unit 101 (drive device 105).

[0128] <Other examples of objects to be controlled>

[0129] While drive control has been described for the focus lens unit 101, this exemplary embodiment is not limited thereto. In this exemplary embodiment, drive control may be applied to other optical components, such as the zoom lens unit, image stabilization lens unit, flange rear adjustment lens unit, and aperture stop. Defocus amount and quietness (S / N ratio) are drive performance characteristics that should also be considered when driving such optical components. If such other optical components are subjected to drive control, information regarding other types of drive performance may be considered as the second information in addition to or in place of the defocus amount.

[0130] As described above, in the present exemplary embodiment, for example, a lens apparatus or an imaging apparatus that is advantageous in terms of adaptation (customization) of driving performance can be provided.

[0131] [Third exemplary embodiment]

[0132] Example of a remote device (processing device) including a training unit (generator)

[0133] Will refer to Figure 16 A third exemplary embodiment is described. Figure 16This figure illustrates an example configuration of a lens device according to the third exemplary embodiment, and by extension, also illustrates an example configuration of a system (imaging device) including an example configuration of a camera body. This system differs from the system of the first exemplary embodiment in that a remote device 400 is included, and that remote device 400 includes a training unit. The camera body 200 includes a communication unit 230 for communicating with the remote device 400. For example, the remote device 400 may be a processing device such as a mobile terminal or a computer terminal. The remote device 400 includes a display unit 401, an operating device 402, a processor 410, and a training unit 420. The processor 410 includes a control unit 411 and a communication unit 412. The communication unit 412 is used to communicate with the camera body 200. The communication unit 412 wirelessly communicates with the communication unit 230 of the camera body 200, but the communication method is not limited to wireless communication. The wireless communication may be known wireless communication via a wireless local area network (LAN).

[0134] The training unit 420 may include a processor (e.g., a CPU or GPU) and a storage device (e.g., a ROM, RAM, or HDD). The training unit 420 may include a machine learning unit 421, a log storage unit 422, a reward storage unit 423, a first reward portion storage unit 424, and a second reward portion storage unit 425. The training unit 420 also stores a program for controlling the operations of these units. The training unit 420 may perform operations similar to those of the training unit 220 according to the first exemplary embodiment.

[0135] In this exemplary embodiment, unlike the first exemplary embodiment, the training unit is not included in the camera body 200 but is included in the remote device 400. Therefore, information transmission between the processor 210 of the camera body 200 and the training unit 420 is performed via the communication unit 230, the communication unit 412, and the control unit 411. Image data output from the signal processing circuit 203 is transmitted to the control unit 411 via the control unit 211, the communication unit 230, and the communication unit 412. The image data transmitted to the control unit 411 is displayed on the display unit 401.

[0136] The control unit 411 can transmit an instruction related to executing machine learning to the machine learning unit 421 based on the operator's operation of the operating device 402. The control unit 211 can transmit an instruction related to executing machine learning to the machine learning unit 421 via the control unit 411 based on the operator's operation of the operating device 206. Upon receiving this instruction, the machine learning unit 421 begins machine learning. Similarly, information related to the level of the second reward portion associated with each type of driving performance, input by the operator from the operating device 402 or the operating device 206, is transmitted via the control unit 411 to the second reward portion storage unit 425. The second reward portion storage unit 425 identifies (selects) information related to the second reward portion associated with each type of driving performance based on the information related to the level of the second reward portion associated with each type of driving performance. Thus, a customized machine learning model (weights) can be generated by training the machine learning model (weights) based on the customized information regarding the reward. Information regarding the generated machine learning model (weights) is transmitted from the remote device 400 to the lens device 100, stored in the storage unit 123, and used to control the driving of the focus lens unit 101 (driving device 105).

[0137] In this manner, a customized machine learning model can be generated at a remote location away from the lens device 100 while being able to observe (view) an image obtained by the camera body 200. While the remote device 400 performs only machine learning processing requiring high-speed computing, the camera body 200 can issue an instruction for executing machine learning and an instruction for setting the second bonus portion via the operation device 206.

[0138] As described above, in the present exemplary embodiment, for example, a lens device, an imaging device, or a processing device that is advantageous in terms of adaptation (customization) of driving performance can be provided.

[0139] In the first and third exemplary embodiments, the second information about the lens apparatus 100 to be used for training the machine learning model is described as being information specific to the lens apparatus 100 only. In the second exemplary embodiment, the second information is described as including information specific to the lens apparatus 100 and information specific to the camera body 200. However, this is not restrictive. The second information may include only information specific to the camera body 200.

[0140] [Exemplary Embodiments Related to Program, Storage Medium, and Data Structure]

[0141] The exemplary embodiments of the present disclosure may be implemented by providing a program or data (structure) for implementing one or more functions or methods of the aforementioned exemplary embodiments to a system or device via a network or storage medium. In this case, a computer in the system or device may read the program or data (structure) and process based on the program or data (structure). The computer may include one or more processors or circuits, and may include a network including multiple separate computers or multiple separate processors or circuits to read and execute computer-executable instructions.

[0142] The processor or circuit may include a CPU, a microprocessing unit (MPU), a GPU, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0143] Although the exemplary embodiments of the present disclosure have been described above, it should be understood that the present disclosure is not limited to the exemplary embodiments, and various modifications and changes may be made without departing from the gist thereof.

[0144] Other embodiments

[0145] The embodiments of the present disclosure may also be implemented by reading out and executing computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the above-described embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions of the above-described embodiments. Furthermore, the embodiments of the present disclosure may be implemented by a method executed by a computer of a system or device, for example, reading out and executing computer-executable instructions from a storage medium to perform one or more functions of the above-described embodiments, and / or controlling one or more circuits to perform one or more functions of the above-described embodiments. A computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)) and may include a network composed of separate computers or separate processors to read and execute computer-executable instructions. Computer-executable instructions may be supplied to a computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD). TM ), one or more of a flash memory device, a memory card, etc.

[0146] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0147] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A lens device comprising: optical components; a driving device configured to drive the optical component; a detector configured to detect a state related to driving; as well as a processor configured to generate a control signal for the drive device based on first information about the detected state, wherein the processor generates an output related to the control signal based on the first information and the second information by using a machine learning model, and the processor outputs the first information and the second information to a generator configured to generate the machine learning model, The second information includes information on a driving speed of the optical member by the driving device, power consumption of the driving device, and a sound level caused by driving the optical member.

2. The lens device according to claim 1, further comprising: the generator; as well as an operating device for an operator to input information related to a requirement for driving performance of the driving device, wherein the generator obtains information about a reward for generating the machine learning model based on information about the requirements.

3. The lens device according to claim 2, wherein: The information on the request input from the operating device is related to each of a plurality of types of performance serving as driving performance.

4. The lens device according to claim 2, wherein: The generator includes previously prepared information about the first reward, and generates the machine learning model based on the information about the first reward and information about the second reward, and the information about the second reward is obtained based on the information about the requirement.

5. The lens device according to claim 2, in, The generator obtains information about the reward based on the second information.

6. The lens device according to claim 5, in, The optical member is a lens unit configured to change an object distance, and The second information includes information related to at least one of a focal depth of the lens device, a focus movement amount of the lens device per unit movement amount of a lens unit, and sounds around the lens device.

7. The lens device according to claim 2, wherein: The generator obtains information about a bonus based on information from an imaging device body on which the lens device is mounted.

8. The lens device according to claim 7, wherein: The information from the imaging device main body includes information on at least one of a diameter of an allowable circle of confusion, a defocus amount, and a sound level.

9. The lens device according to claim 7, wherein: The processor generates a control signal based on information from the imaging device body.

10. The lens device according to claim 2, wherein: The generator evaluates the machine learning model based on information about the reward and determines to end generation based on the evaluation.

11. The lens device according to claim 10, wherein: The generator adopts the machine learning model whose evaluation satisfies the acceptance condition.

12. The lens device according to claim 10, wherein: The generator does not adopt the machine learning model that meets the end condition and the evaluation does not meet the acceptance condition.

13. A camera device comprising: The lens device according to claim 1; as well as An image pickup element is configured to pick up an image formed by the lens device.

14. A processing device configured to perform processing related to a machine learning model in a lens device, the lens device comprising: optical components; a driving device configured to drive the optical component; a detector configured to detect a state related to driving; and a processor configured to generate a control signal for the drive device based on first information regarding the detected state, the processor generating an output related to the control signal based on the first information and second information using a machine learning model, wherein the second information includes information regarding a driving speed of the optical member by the drive device, power consumption of the drive device, and a sound level caused by driving the optical member, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing device comprising: an operating device for an operator to input information related to a requirement for driving performance of the driving device, Wherein, the processing device obtains information about the reward for generating the machine learning model based on the information about the requirements. The processing device according to claim 14 , further comprising the generator.

16. The processing device according to claim 15, wherein The generator includes previously prepared information about the first reward, and generates a machine learning model based on the information about the first reward and information about the second reward, where the information about the second reward is obtained based on the information about the requirement.

17. The processing device according to claim 15, in, The second information includes information on a state of the lens device related to driving performance of the driving apparatus, and The generator obtains information about the reward based on the second information.

18. A processing method for performing processing related to a machine learning model in a lens device, the lens device comprising: optical components; a driving device configured to drive the optical component; a detector configured to detect a state related to driving; and a processor configured to generate a control signal for the driving device based on first information regarding the detected state, the processor generating an output related to the control signal based on the first information and second information using a machine learning model, wherein the second information includes information regarding a driving speed of the driving device for the optical member, power consumption of the driving device, and a sound level caused by driving the optical member, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing method comprising: Based on information about requirements for driving performance of the driving device, information about a reward for generating the machine learning model is obtained, the information about the requirements being input from an operating device operated by an operator.

19. The processing method according to claim 18, further comprising: The machine learning model is generated by the generator based on information about the reward.

20. A computer-readable storage medium storing a program for causing a computer to execute a processing method for performing processing related to a machine learning model in a lens device, the lens device comprising: optical components; a driving device configured to drive the optical component; a detector configured to detect a state related to driving; and a processor configured to generate a control signal for the driving device based on first information regarding the detected state, the processor generating an output related to the control signal based on the first information and second information using a machine learning model, wherein the second information includes information regarding a driving speed of the driving device for the optical member, power consumption of the driving device, and a sound level caused by driving the optical member, the processor outputting the first information and the second information to a generator configured to generate the machine learning model, the processing method comprising: Based on information about requirements for driving performance of the driving device, information about a reward for generating the machine learning model is obtained, the information about the requirements being input from an operating device operated by an operator.

Images