Lens device, imaging device, camera device, control method, and program

Abstract

To provide a lens device advantageous for obtaining a learned model.SOLUTION: A lens device includes an optical member, a drive section that drives the optical member, and a control section that controls the drive section on the basis of a first learned model. The first learned model is a learned model obtained by learning about the lens device using a second learned model obtained by learning about a device different from the lens device as an initial learned model.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a lens apparatus, an imaging apparatus, a camera apparatus, a control method, and a program. [Background technology]

[0002] In the past, digital cameras were used to take still images, while video cameras were used to take moving images. However, in recent years, digital cameras that can take and record moving images have also been commercialized.

[0003] Furthermore, still image shooting places importance on speed, so high-speed operations such as autofocus, aperture, and electric zoom are necessary. On the other hand, when shooting video, if the operating noise of the driving parts such as focus, aperture, and zoom is loud, the operating noise will be recorded as noise along with the audio that should be recorded. Patent Document 1 discloses a lens device that solves these two problems.

[0004] Furthermore, in an imaging device, a wide range of performance is required of the driving unit of the optical member. For example, driving speed for tracking, optical member positioning accuracy for setting accurate imaging conditions, low power consumption for securing imaging time, and quiet performance for video shooting are some of the requirements. These performance requirements are mutually dependent. The lens device described in Patent Document 1 realizes quiet performance by limiting the speed and acceleration at which the optical member is driven during video shooting.

[0005] However, the required noise reduction performance may differ depending on the user, the required speed and acceleration may differ depending on the user and the state of the lens apparatus, and the required positioning accuracy of the optical member may differ depending on the user and the state of the lens apparatus. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2007-6305 A Summary of the Invention [Problem to be solved by the invention]

[0007] Here, drive control of the lens device can be performed based on a machine learning model (trained model) obtained by machine learning performed using the lens device, but the machine learning may require a lot of driving of the lens device.

[0008] The present invention aims to provide, for example, a lens device that is advantageous for obtaining a trained model. [Means for solving the problem]

[0009] A lens device according to one embodiment of the present invention includes an optical member, a driving unit that drives the optical member, and a control unit that controls the driving unit, wherein the control unit controls the driving unit so that the optical member is driven at a speed or acceleration based on a first trained model, and the first trained model is a second trained model obtained by learning about a device different from the lens device as an initial trained model. Based on compensation information This is a trained model obtained by training. The reward information includes information regarding power consumption by the drive unit. It is characterized by: Effect of the Invention

[0010] According to the present invention, for example, a lens device that is advantageous for obtaining a trained model can be provided. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a system configuration according to a first embodiment. [Diagram 2] 1A and 1B are diagrams for explaining the positional accuracy required for focus lens control. [Diagram 3] 11 is a diagram for explaining a speed required for focus lens control. FIG. [Figure 4]1A and 1B are diagrams for explaining the relationship between position accuracy, speed, power consumption, and noise reduction. [Diagram 5] 11 is a diagram for explaining the relationship between speed, position accuracy, power consumption, and noise reduction. FIG. [Figure 6] 1 shows an example of input and output of a neural network. [Figure 7] 4 is a flowchart showing a flow of a series of imaging processes according to the first embodiment. [Figure 8] 1 illustrates an example of remuneration information according to the first embodiment. [Figure 9] 13 shows an example of a data structure of device constraint remuneration information and user requested remuneration information. [Figure 10] FIG. 11 is a flowchart showing the flow of a remuneration information determination process. [Figure 11] 13 shows an example of a data structure of a device constraint remuneration information database. [Figure 12] 1 shows an example of the structure of a user request information database. [Figure 13] 1 shows an example of a data structure of a user desired reward conversion information database. [Figure 14] 1 shows an example of a data structure of a remuneration information database. [Figure 15] FIG. 11 is a flowchart showing the flow of a machine learning process. [Figure 16] 1 shows an example of a data structure of a machine learning model database. [Figure 17] FIG. 11 is a flowchart showing the flow of a weight initial value determination process. [Figure 18] 11 shows an example of a value of user requested remuneration information in the remuneration information. [Figure 19] 11 shows an example of a user request value in the user request information. [Figure 20] FIG. 11 is a block diagram showing a system configuration according to a second embodiment. [Figure 21] FIG. 11 is a block diagram showing a system configuration according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

[0013] In the following, a trained model refers to a model that has undergone prior training (learning) on ​​a machine learning model that follows any machine learning algorithm, such as deep learning. However, a trained model is one that has undergone prior training, but this does not mean that it does not undergo further learning, and that it is also possible for the model to undergo additional learning.

[0014] First Embodiment An imaging system according to a first embodiment of the present invention will be described below with reference to Fig. 1 to Fig. 19. In this embodiment, a camera system will be described as an example of an imaging system.

[0015] <Camera system configuration> Hereinafter, the system configuration of the camera system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the system configuration of the camera system according to this embodiment.

[0016] The camera system according to this embodiment is provided with a camera body 200, which is an example of an imaging device, and a lens 100, which is an example of a lens device. The camera body 200 and the lens 100 are mechanically and electrically connected via a mount 300, which is a coupling mechanism. The camera body 200 supplies power to the lens 100 via a power terminal unit (not shown) provided on the mount 300. In addition, the camera body 200 and the lens 100 communicate with each other via a communication terminal unit (not shown) provided on the mount 300.

[0017] The lens 100 has an imaging optical system. The imaging optical system includes a focus lens 101 that adjusts the focus, a zoom lens 102 that changes the magnification, an aperture unit 103 that adjusts the amount of light, and an image stabilization lens 104. The focus lens 101 and the zoom lens 102 are held by a lens holding frame (not shown). The lens holding frame is guided by a guide shaft (not shown) so as to be movable in the optical axis direction (indicated by a dashed line in the figure).

[0018] The focus lens 101 is moved in the optical axis direction via a focus lens driving unit 105. The position of the focus lens 101 is detected by a focus lens detection unit .

[0019] The zoom lens 102 is moved in the optical axis direction via a zoom lens driving unit 107. The position of the zoom lens 102 is detected by a zoom lens detection unit 108.

[0020] The aperture unit 103 is provided with aperture blades, and the aperture blades are driven via an aperture drive unit 109 to adjust the amount of light. The F-number of the aperture is detected by an aperture detection unit 110.

[0021] The image stabilization lens 104 is moved in a direction perpendicular to the optical axis via an image stabilization lens drive unit 112, and reduces image shake caused by camera shake, etc. The position of the image stabilization lens 104 is detected by an image stabilization lens detection unit 113.

[0022] For example, ultrasonic motors are used for the focus lens driving unit 105, the zoom lens driving unit 107, the aperture driving unit 109, and the image blur correction lens driving unit 112. Although ultrasonic motors are used in this embodiment, other motors (such as a voice coil motor, a DC motor, or a stepping motor) may also be used.

[0023] The focus lens detection unit 106, the zoom lens detection unit 108, the aperture detection unit 110, and the image shake correction lens detection unit 113 each include, for example, a potentiometer or an encoder. In addition, when the drive unit is configured using a motor such as a stepping motor that can drive a predetermined drive amount without feedback, a means for detecting the position of the lens or the like based on the drive amount of the drive unit may be provided. In this case, the optical member is initially driven to a predetermined position where a detection sensor such as a photointerrupter is provided, and after the initial drive, the position of the optical member can be identified based on the motor drive amount.

[0024] The vibration sensor 111 is a sensor that detects the vibration of the lens 100. The vibration sensor 111 may be configured using, for example, a gyro sensor.

[0025] Next, a description will be given of a lens microcomputer (hereinafter referred to as a lens microcomputer 120) that controls the drive and communication of the lens 100. The lens microcomputer 120 includes an NN control unit 121, a lens device information management unit 122, an NN data storage unit 123, an operation log management unit 124, a control unit 125, a communication unit 126, and a lens individual information management unit 127.

[0026] The NN control unit 121 is a control unit that controls the position of the focus lens 101. A neural network (hereinafter, NN) algorithm is implemented in the NN control unit 121, and the NN control unit 121 determines a drive command for the focus lens drive unit 105 by the NN algorithm based on the target position of the focus lens 101, etc. The details of the NN algorithm will be described later.

[0027] The lens device information management unit 122 is a management unit that manages lens device information used in the NN control unit 121. Here, the lens device information is information related to shooting conditions of the lens device that affect the shot image, and includes, for example, focal depth and focus sensitivity.

[0028] The NN data storage unit 123 is a storage unit that holds weights, which are the coupling weighting coefficients of the NN. The operation log management unit 124 is a management unit that manages operation log information related to the drive control of the focus lens 101. Here, the operation log information is control result information that is the subject of determining a score when converting the control result of the NN algorithm into a score. The operation log information can include, for example, the target position and position information of the focus lens, the drive speed and acceleration of the focus lens, and the power consumption of the focus lens drive unit 105.

[0029] The control unit 125 controls the positions of the zoom lens 102, the aperture unit 103, and the image stabilization lens 104, and also controls the transmission of information between the lens 100 and the camera body 200. For example, the control unit 125 generates a drive command by PID (Proportional-Integral-Differential) control in response to the deviation between a target position or speed of the control object and the current position or speed of the control object, and controls the control object.

[0030] The communication unit 126 is a communication unit for communicating with the camera body 200. Furthermore, the lens individual information management unit 127 is a management unit for managing lens individual information (lens information) which is information indicating at least one of the lens model and the individual identification number of the lens 100.

[0031] Next, a description will be given of the camera body 200. The camera body 200 is provided with an image sensor 201, an A / D conversion circuit 202, a signal processing circuit 203, a recording unit 204, a display unit 205, an operation unit 206, a camera microcomputer (hereinafter referred to as a camera microcomputer 210), and a learning unit 220.

[0032] The imaging element 201 is an imaging element that converts light incident from the lens 100 into an image electrical signal. The imaging element 201 is, for example, a CCD sensor or a CMOS sensor. The A / D conversion circuit 202 is a conversion circuit for converting the image electrical signal output from the imaging element 201 into a digital signal. The signal processing circuit 203 is a signal processing circuit that converts the digital signal output from the A / D conversion circuit 202 into image data.

[0033] The recording unit 204 is a recording unit that records the video data output from the signal processing circuit 203. The display unit 205 is a display unit that displays various information such as the video data output from the signal processing circuit 203 and user request information. The display unit 205 may be configured using any known monitor.

[0034] The operation unit 206 is an operation unit for the user to operate the camera. The operation unit 206 may be configured using any known input member such as a button. In addition, when the display unit 205 is configured using a touch panel, the operation unit 206 may be configured integrally with the display unit 205.

[0035] The camera microcomputer (hereinafter, camera microcomputer 210) is a control microcomputer that controls the camera body 200. The camera microcomputer 210 includes a control unit 211 and a communication unit 212.

[0036] The control unit 211 is a control unit that issues drive commands to the lens 100 based on video data from the signal processing circuit 203 and user operation information from the operation unit 206. The control unit 211 also controls commands and information transmission to the learning unit 220.

[0037] The communication unit 212 is a communication unit for communicating with the lens 100. More specifically, the communication unit 212 transmits a drive command from the control unit 211 to the lens 100 as a control command. The communication unit 212 also receives information from the lens 100.

[0038] The learning unit 220 can be configured using a processor and a storage device. Here, the processor may be any processor such as a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit). The storage device may be configured using any storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a HDD (Hard Disc Drive).

[0039] The learning unit 220 includes a learning processing unit 221, an operation log storage unit 222, a reward management unit 223, a device constraint reward management unit 224, a user request reward management unit 225, a user request management unit 226, a user request storage unit 227, and a device constraint reward information storage unit 228. The learning unit 220 also includes a user request reward conversion information storage unit 229, a reward information storage unit 230, a machine learning model storage unit 231, and a machine learning initial value management unit 232. Each component of the learning unit 220 may be realized by a processor such as a CPU or MPU executing a software module stored in a storage device. The processor may be, for example, a GPU or a Field-Programmable Gate Array (FPGA). Each component of the learning unit 220 may be configured by a circuit or the like that performs a specific function, such as an Application Specific Integrated Circuit (ASIC).

[0040] The learning processing unit 221 performs learning processing of the NN provided in the NN control unit 121. In this embodiment, the learning processing unit 221 performs reinforcement learning on the NN so as to maximize the reward with respect to the control of the focus lens 101 using the NN, using as a reward an evaluation value of an operation according to reward information based on equipment constraint information for each lens device and request information by a user.

[0041] The device constraint compensation management unit 224 manages device constraint compensation information, which is compensation information related to device constraint information for each lens device. Specifically, the device constraint compensation management unit 224 determines device constraint compensation information corresponding to the lens 100 from a device constraint compensation information database held in the device constraint compensation information holding unit 228, according to the lens model information included in the lens individual information. Note that the device constraint compensation information database stores lens model information and device constraint compensation information in association with each other.

[0042] The user-requested reward management unit 225 manages user-requested reward information, which is reward information related to the request information by the user. The user-requested reward conversion information storage unit 229 stores a user-requested reward conversion information database in which the lens model information of the lens individual information and the user-requested reward conversion information are stored in association with each other. In addition, the user-requested reward conversion information storage unit 229 determines user-requested reward conversion information corresponding to the lens 100 from the user-requested reward conversion information database according to the lens model information of the lens individual information. The user-requested reward conversion information storage unit 229 converts the user-requested information set by the user into user-requested reward information by referring to the determined user-requested reward conversion information. The user-requested reward management unit 225 manages the converted user-requested reward information in association with the user-requested information set by the user.

[0043] The user request management unit 226 manages user request information set by the user and to be applied to the current control. The user request storage unit 227 stores a user request information database that stores the user request information set by the user.

[0044] The reward management unit 223 combines the device constraint reward information with the user requested reward information to determine and manage reward information. The reward information storage unit 230 stores a reward information database that stores reward information.

[0045] The machine learning model holding unit 231 holds a machine learning database that stores machine learning models. The machine learning model holding unit 231 also selects a trained model used to determine an initial value of machine learning of the NN from the machine learning model database according to the lens individual information and a user's request. The machine learning initial value management unit 232 manages the weight of the trained model selected by the machine learning model holding unit 231 as an initial value of the weight (machine learning initial value) when performing machine learning of the NN of the NN control unit 121. Note that the machine learning model database stores lens individual information, user request information, remuneration information, and machine learning models in association with each other. Note that the remuneration information stored in the machine learning model database may be an identification number of the remuneration information stored in the remuneration information database, or the like.

[0046] The operation log storage unit 222 stores operation log information related to drive control of the focus lens 101 in response to a drive command. The storage device of the learning unit 220 stores programs for implementing various components of the learning unit 220, and various information and various databases held by each component of the learning unit 220, such as the operation log information held by the operation log storage unit 222.

[0047] <Recording and displaying footage> The following describes the recording and display of captured images in the camera system shown in Fig. 1. Light incident on the lens 100 passes through the focus lens 101, the zoom lens 102, the aperture unit 103, and the image stabilization lens 104, and forms an image on the image sensor 201. The light formed on the image sensor 201 is converted into an electrical signal by the image sensor 201, converted into a digital signal by the A / D conversion circuit 202, and converted into image data by the signal processing circuit 203. The image data output from the signal processing circuit 203 is recorded in the recording unit 204. Furthermore, the display unit 205 displays an image based on the image data output from the signal processing circuit 203.

[0048] <Focus control> Next, a description will be given of a method in which the camera body 200 controls the focus of the lens 100. The control unit 211 performs AF (autofocus) control based on the video data output from the signal processing circuit 203.

[0049] Specifically, the control unit 211 controls the focus lens 101 to move so that the contrast of the video data is maximized, thereby focusing on the subject to be photographed. First, the control unit 211 outputs a focus drive amount for moving the focus lens 101 as a drive command to the communication unit 212. Upon receiving the drive command from the control unit 211, the communication unit 212 converts the drive command for focus drive into a control command and transmits it to the lens 100 via the communication contact unit of the mount 300.

[0050] When the communication unit 126 of the lens 100 receives a control command from the communication unit 212, it converts the control command into a drive command for focus drive and outputs it to the NN control unit 121 via the control unit 125. The NN control unit 121 determines a drive signal using the drive command for focus drive, the focus lens position detected by the focus lens detection unit 106, and the shooting conditions described later as inputs to the NN, and outputs the drive signal to the focus lens drive unit 105. Here, the NN used to determine the drive signal has the weight of the learned model stored in the NN data storage unit 123, and can perform processing similar to that of the learned model. Therefore, the NN can output the drive amount of the focus lens according to the configuration of the lens 100 and the user's request according to the learning tendency.

[0051] As a result, the focus lens 101 is driven according to a drive command from the control unit 211. With this operation, the control unit 211 can perform appropriate AF control by moving the focus lens 101 so that the contrast of the video data is maximized. In addition, by using an NN having weights of a trained model to determine a drive signal, AF control can be performed according to the configuration of the lens 100 and the user's requests.

[0052] <Aperture control> Next, a method in which the camera body 200 controls the aperture of the lens 100 will be described. The control unit 211 performs exposure control based on the video data output from the signal processing circuit 203. Specifically, the control unit 211 determines a target F-number so that the luminance value of the video data is constant. The control unit 211 outputs the determined F-number to the communication unit 212 as a drive command. Upon receiving a drive command from the control unit 211, the communication unit 212 converts the drive command for the F-number into a control command and transmits it to the lens 100 via the communication contact unit of the mount 300.

[0053] When the communication unit 126 of the lens 100 receives a control command from the communication unit 212, it converts it into an F-number drive command and outputs it to the control unit 125. The control unit 125 determines a drive signal based on the F-number drive command and the F-number of the aperture detected by the aperture detection unit 110, and outputs the drive signal to the aperture drive unit 109. In this way, the F-number is controlled so that the luminance value of the video data is constant, and appropriate exposure control can be performed.

[0054] <Zoom control> Next, a method in which the camera body 200 controls the zoom of the lens 100 will be described. First, a user performs a zoom operation via the operation unit 206. The control unit 211 determines a zoom drive amount for moving the zoom lens 102 based on the zoom operation amount output from the operation unit 206, and outputs the zoom drive amount as a drive command to the communication unit 212. When the communication unit 212 receives a drive command from the control unit 211, it converts the drive command for zoom drive into a control command and transmits it to the lens 100 via the communication contact unit of the mount 300.

[0055] When the communication unit 126 of the lens 100 receives a control command from the communication unit 212, it converts it into a drive command for zoom drive and outputs it to the control unit 125. The control unit 125 determines a drive signal based on the drive command for zoom drive and the zoom lens position detected by the zoom lens detection unit 108, and outputs the drive signal to the zoom lens drive unit 107. As a result, the zoom lens 102 is driven according to the zoom operation input to the operation unit 206, allowing the user to operate the zoom.

[0056] <Anti-vibration control> Next, a method for the lens 100 to perform vibration reduction control will be described. Based on the vibration signal of the lens 100 output from the vibration sensor 111, the control unit 125 determines an image blur correction lens target position so as to cancel out the vibration of the lens 100. The control unit 125 determines a drive signal based on the image blur correction lens position detected by the image blur correction lens detection unit 113 and the determined image blur correction lens target position, and outputs the drive signal to the image blur correction lens drive unit 112. As a result of the above, vibration reduction is correctly controlled, and vibration of the image captured by the image sensor 201 can be prevented.

[0057] <Four indices required for focus lens control> Next, we will explain the requirements for focus lens control. There are four requirements for focus lens control: position accuracy, speed, power consumption, and quietness. Focus lens control requires that each of these requirements be controlled in a well-balanced manner. Each requirement will be explained below.

[0058] <Position accuracy required for focus lens control> The positional accuracy in focus lens control is an index that indicates how accurately the focus lens can be driven to a target position when the focus lens is driven to the target position. The positional accuracy in focus lens control will be described below with reference to Figs. 2(a) and 2(b). Fig. 2(a) shows the relationship between the focus lens and the focus position when the focal depth is shallow, and Fig. 2(b) shows the relationship between the focus lens and the focus position when the focal depth is deep. Figs. 2(a) and 2(b) show the relationship between the focus lens and the focus position when the lens configuration is the same and only the F-number is different. In Figs. 2(a) and 2(b), the same reference symbols are used for common parts.

[0059] 2(a) and 2(b), focus lens target position G indicates the focus lens position where the image of point object S, which is the main subject on the optical axis, is focused on the image sensor 201. In contrast, focus lens position C indicates the focus position after driving the focus lens to target position G. Focus lens position C is a position on the side of point object S relative to focus lens target position G by an amount of control error Er. Focus position Bp indicates the imaging position of point object S when the focus lens position is focus lens position C. Circle of confusion δ is the circle of confusion on the image sensor 201.

[0060] The F value Fa in FIG. 2(a) is brighter (smaller) than the F value Fb in FIG. 2(b). Therefore, the focal depth width 2Faδ in FIG. 2(a) is narrower than the focal depth width 2Fbδ in FIG. 2(b). The light rays Cla and Gla in FIG. 2(a) respectively indicate the outermost light rays of the light rays of the point object S at the focus lens position C and the focus lens target position G. The light rays Clb and Glb in FIG. 2(b) respectively indicate the outermost light rays of the light rays of the point object S at the focus lens position C and the focus lens target position G.

[0061] 2(a), point image diameter Ia indicates the diameter of a point image of point object S on image sensor 201 when the focus lens is at focus lens position C. Also, in Fig. 2(b), point image diameter Ib indicates the diameter of a point image of point object S on image sensor 201 when the focus lens is at focus lens position C.

[0062] Here, in FIG. 2(a), the focus position Bp is outside the range of the focal depth width 2Faδ, the point image diameter Ia is larger than the circle of confusion δ, and the light does not fit into the central pixel and enters the neighboring pixel. As a result, in FIG. 2(a), the point object S is out of focus at the focus lens position C. On the other hand, in FIG. 2(b), the focus position Bp is within the range of the focal depth width 2Fbδ, the point image diameter Ib is smaller than the circle of confusion δ, and all light rays are focused on the central pixel. As a result, in FIG. 2(b), the point object S is in focus at the focus lens position C.

[0063] As described above, even if the same positional accuracy is achieved, the focus state changes to out of focus or in focus depending on the shooting conditions such as the F-number. This shows that the positional accuracy required for focus lens control changes depending on the shooting conditions.

[0064] <Speed ​​required for focus lens control> Next, the speed related to the focus lens control will be described with reference to FIG. 3(a) and FIG. 3(b). Here, the speed related to the focus lens control refers to the movement speed when driving the focus lens (the driving speed of the focus lens). The movement speed can be replaced with the movement amount by considering it as the movement amount per unit time. The movement amount in the optical axis direction of the focused position is called the focus movement amount, and the movement speed is called the focus movement speed. The focus lens movement amount is in a proportional relationship with the focus movement amount. The proportional constant between the focus lens movement amount and the focus movement amount is called the focus sensitivity. The focus sensitivity changes depending on the positional relationship of the optical system that constitutes the lens (the configuration of the lens device). Here, the relationship between the focus movement amount ΔBp, the focus sensitivity Se, and the focus lens movement amount ΔP can be expressed by the following formula 1: focus movement amount ΔBp=focus sensitivity Se×focus lens movement amount ΔP.

[0065] Fig. 3(a) shows the relationship between the focus lens and the focus position when the focus sensitivity Se is small, and Fig. 3(b) shows the relationship when the focus sensitivity Se is large. Fig. 3(a) and Fig. 3(b) show cases where the lens configuration is the same but the distance between the lens and the point object S is different. In Fig. 3(a) and Fig. 3(b), the same reference symbols are used for common parts.

[0066] In FIG. 3(a), when moving the focus position from focus position Bp1 to focus position Bp2, it is necessary to move the focus lens from focus lens position LPa1 to focus lens position LPa2. At this time, the relationship between the focus lens movement amount ΔPa and the focus movement amount ΔBp is as shown in Equation 1. Similarly, in FIG. 3(b), when moving the focus position from focus position Bp1 to focus position Bp2, it is necessary to move the focus lens from focus lens position LPb1 to focus lens position LPb2. The relationship between the focus lens movement amount ΔPa and the focus movement amount ΔBp at this time is also as shown in Equation 1.

[0067] As shown in Figures 3(a) and 3(b), when the focus sensitivity Se is smaller, the focus lens movement amount ΔP required to move the same focus movement amount ΔBp becomes larger. In other words, compared to the case shown in Figure 3(a), the focus movement amount ΔBp per unit time can be made smaller in the case shown in Figure 3(b), so that the focus movement speed can be the same even if the focus lens drive speed is slower.

[0068] As described above, the focus lens driving speed required to achieve a specific focus movement speed differs depending on the shooting conditions such as focus sensitivity, etc. This shows that the required focus lens driving speed changes depending on the shooting conditions.

[0069] <Power consumption required for focus lens control> The power consumption related to focus lens control is the power consumed to drive the focus lens. The power consumption changes depending on the drive time, drive speed, or drive acceleration change of the focus lens. For example, if the drive time is long, the drive speed is fast, or there is a lot of change in drive acceleration, the power consumption increases. On the other hand, by suppressing the power consumption, the battery capacity can be effectively utilized, and the advantages are that the number of shots that can be taken on a single charge can be increased and the battery can be made even smaller.

[0070] <Quiet operation required for focus lens control> Quietness in focus lens control is an index of the drive noise generated when driving the focus lens. When driving the focus lens, drive noise is generated due to vibration, friction, and the like. The drive noise changes according to the drive speed or change in drive acceleration. For example, when the drive speed is fast or when there is a large change in drive acceleration, the drive noise becomes louder. Also, the longer the time that the focus lens is stopped, the longer the time that no drive noise is generated.

[0071] <Relationship between position accuracy, speed, power consumption, and noise> Next, the relationship between the positional accuracy of the focus lens control and the speed, power consumption, and quietness of the focus lens control will be described with reference to Fig. 4(a) and Fig. 4(b). Fig. 4(a) shows an example of the movement of the focus lens control for keeping the focus on a moving subject when the focal depth is shallow, and Fig. 4(b) shows an example of the movement of the focus lens control for keeping the focus on a moving subject when the focal depth is deep. The horizontal axis of Fig. 4(a) and Fig. 4(b) indicates the passage of time, and the vertical axis indicates the focus lens position. Here, in Fig. 4(a) and Fig. 4(b), when the focus lens position moves to the upper side of the paper, the focus is set in the infinity direction, and when the focus lens position moves to the lower side of the paper, the focus is set in the close range direction. In Fig. 4(a) and Fig. 4(b), the same reference symbols are used for common parts.

[0072] In FIG. 4(a) and FIG. 4(b), the focus lens target position G indicates the focus lens position when the image of the subject is focused on the image sensor 201. The focal depth widths in FIG. 4(a) and FIG. 4(b) are 2Faδ and 2Fbδ, respectively. In FIG. 4(a), the focus lens position where the focus position is the boundary on the infinity side of the focal depth is indicated by position GalimI, and the focus lens position where the focus position is the boundary on the close side is indicated by position GalimM, based on the focus lens target position G. Similarly, in FIG. 4(b), the focus lens position where the focus position is the boundary on the infinity side of the focal depth is indicated by position GblimI, and the focus lens position where the focus position is the boundary on the close side is indicated by position GblimM, based on the focus lens target position G. In addition, FIG. 4(a) and FIG. 4(b) each show the focus lens positions Ca and Cb controlled so that the subject falls within the focal depth.

[0073] In the case shown in Fig. 4(b), since the focal depth is deep, even if the focus lens is controlled to the trajectory indicated by the focus lens position Cb, the image of the subject will not deviate from the focal depth. On the other hand, in the case shown in Fig. 4(a), since the focal depth is shallow, it is necessary to control the trajectory of the focus lens position Cb to a trajectory with less deviation from the focus lens target position G compared to the case shown in Fig. 4(b).

[0074] In both cases shown in Figure 4(a) and Figure 4(b), the subject does not go out of focus, but the control of focus lens position Cb shown in Figure 4(b) requires less drive amount and drive speed than the control of focus lens position Ca shown in Figure 4(a). In this way, under shooting conditions where low positional accuracy is required, the focus lens can be controlled at low speed, with low power consumption, and in a quiet state.

[0075] <Relationship between speed, position accuracy, power consumption, and noise> Next, the relationship between the speed of the focus lens control and the position accuracy, power consumption, and quietness of the focus lens control will be described with reference to Fig. 5(a) and Fig. 5(b). The horizontal axis of Fig. 5(a) and Fig. 5(b) indicates the passage of time, and the vertical axis indicates the focus lens position. Fig. 5(a) shows the change in the focus lens position Ca when the focus lens is driven from the focus lens position LPa1 shown in Fig. 3(a) to the focus lens position LPa2 during the time period T0 to T1. Similarly, Fig. 5(b) shows the change in the focus lens position Cb when the focus lens is driven from the focus lens position LPb1 shown in Fig. 3(b) to the focus lens position LPb2 during the time period T0 to T1. In addition, in each of Fig. 5(a) and Fig. 5(b), the slope of the change in the focus lens positions Ca and Cb indicates the focus lens speed.

[0076] Here, as shown in Figures 3(a) and 3(b), the focus movement amount ΔBp when moving from focus lens position LPa1 to focus lens position LPa2 is the same as the focus movement amount ΔBp when moving from focus lens position LPb1 to focus lens position LPb2. In contrast, the gradient of the change in focus lens position Ca shown in Figure 5(a) is larger than the gradient of the change in focus lens position Cb shown in Figure 5(b), and it can be seen that the focus lens speed is fast in the example shown in Figure 5(a).

[0077] As shown in FIG. 5(a) and FIG. 5(b), the focus lens needs to be moved faster in the control of the focus lens position Ca to move the same focus movement amount ΔBp between times T0 and T1 than in the control of the focus lens position Cb. In addition, since the speed is fast in the control of the focus lens position Ca, it takes a certain amount of time for the position to stabilize after the focus lens position LPa2, which is the target position, is reached. On the other hand, since the speed is slow in the control of the focus lens position Cb, the position stabilizes immediately after the focus lens position LPb2, which is the target position, is reached. This affects the position accuracy. In addition, since the control of the focus lens position Ca drives the focus lens quickly, the acceleration change is large when the lens is stopped, so the power consumption is high and the driving sound is louder than the control of the focus lens position Cb. Therefore, under shooting conditions where the required speed is low, the focus lens can be controlled with high position accuracy, low power consumption, and quiet operation.

[0078] <Lens device information> Next, lens device information related to shooting conditions will be described. The lens device information is information about the shooting conditions of the lens device that affect the shot image. In this embodiment, in order to control the requirements in the focus lens control in a well-balanced manner, focus lens control is performed based on lens device information for determining the position accuracy and speed required in the focus lens control. The lens device information is determined by the lens device information management unit 122. The lens device information includes, for example, focal depth and focus sensitivity.

[0079] The lens device information management unit 122 can determine the focal depth from the current F-number and information on the circle of confusion as shown in Equation 1. The lens device information management unit 122 also holds a conversion table (not shown) indicating the relationship between the focus sensitivity and the focus lens position and the zoom lens position, and can determine the focus sensitivity from the focus lens position and the zoom lens position. The focal depth and focus sensitivity may be determined using any known method.

[0080] As described above, the requirements for focus lens control are affected by the shooting conditions. Therefore, by controlling the focus lens based on the lens device information related to these shooting conditions, it is possible to control the requirements for position accuracy, speed, power consumption, and quietness in a well-balanced manner while taking into account the effects of the shooting conditions on the captured image.

[0081] <NNアルゴリズムとウエイト> The following describes a method in which the NN control unit 121 determines a drive command using an NN algorithm. As described above, the NN control unit 121 is implemented with an NN. The NN control unit 121 refers to weights, which are coupling weighting coefficients of the learned model recorded in the NN data storage unit 123, and uses the weights referred to as the weights of the NN. This allows the NN control unit 121 to use the NN as a learned NN. By determining a drive command using a learned NN, the NN control unit 121 can determine a drive command according to the configuration of the lens 100 and the user's request according to the learning tendency of the learned model.

[0082] 6 is a conceptual diagram showing the input / output structure of the NN control unit 121 using the NN according to this embodiment. The target position X1 is a drive command target position for focus drive output from the control unit 125. The current position X2 is the current position of the focus lens 101 obtained from the focus lens detection unit 106. The focal depth X3 is the focal depth included in the lens device information, and the focus sensitivity X4 is the focus sensitivity included in the lens device information. In addition, the drive signal Y1 is the drive signal for the focus lens 101.

[0083] In this manner, in this embodiment, a drive command for focus drive, the current position of the focus lens 101, the focal depth, and the focus sensitivity are input, and a drive signal is determined as an output of the trained model. The NN control unit 121 can control the focus lens 101 by taking into account the influence of the shooting conditions on the shot image by using the lens device information related to the shooting conditions as an input to the NN.

[0084] <Shooting process flow> Next, a series of image capturing processes according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of a series of image capturing processes according to this embodiment. In this embodiment, when the image capturing process is started in response to a user operation, the process proceeds to step S701.

[0085] In step S701, shooting conditions that affect the control using the NN are set. For example, in step S701, the control unit 211 according to this embodiment performs exposure control and sets an F-number based on the video data output from the signal processing circuit 203. The depth of focus is set by setting the F-number. The F-number may be set in response to the user's operation of the operation unit 206.

[0086] In step S702, the control unit 211 determines whether or not to perform machine learning in response to a user's operation of the operation unit 206. If the control unit 211 determines to perform machine learning, the process proceeds to step S703.

[0087] In step S703, reward information related to the learning of the NN used for the focus lens control according to this embodiment is determined. As described above, in this embodiment, as the machine learning process, the evaluation value of the operation according to the reward information based on the equipment constraint information for each lens device and the user's request information is used as a reward, and reinforcement learning is performed on the NN so that the reward is maximized. Therefore, in this embodiment, prior to the machine learning process, the reward information is determined according to the configuration of the lens 100 and the user's request. Specifically, the equipment constraint reward management unit 224, the user request reward management unit 225, and the reward management unit 223 determine the reward information based on the lens individual information and the user request information set by the user. Details of the reward information and the reward information determination process will be described later. When the reward information is determined in step S703, the process proceeds to step S704.

[0088] In step S704, the learning unit 220 performs machine learning processing on the NN. Here, as described above, when machine learning is performed using a lens device, many drives are required during machine learning, which may cause the drive unit to wear out and lead to performance degradation of the drive unit. In addition, since machine learning requires a lot of time, a burden may be placed on the user, such as not being able to take a picture when the user actually wants to take a picture. Therefore, in this embodiment, the learning unit 220 selects a trained model that has been trained using the configuration of the lens 100 and reward information corresponding to a configuration and request similar to the user's request from a plurality of trained models that have been trained in advance. The learning unit 220 uses the weight of the selected trained model as the initial value of the weight of the NN.

[0089] Thereafter, the learning unit 220 acquires operation log information of focus control using the lens 100 equipped with the NN in which the initial value of the weight is set, and performs additional learning with the evaluation value of focus control as a reward based on the reward information determined in step S703. In this way, in this embodiment, the number of times the drive unit is driven during machine learning can be reduced by performing so-called transfer learning using a trained model trained using the configuration of the lens 100 and the reward information corresponding to the configuration and request that matches or is similar to the user's request. Therefore, wear on the drive unit during machine learning can be suppressed, and the time required for machine learning can be shortened. Details of the machine learning process will be described later.

[0090] On the other hand, if the control unit 211 determines in step S702 that machine learning is not to be performed, the process proceeds to step S705. In step S705, the control unit 211 selects a trained model to be used for the NN control unit 121 in response to a user operation. For example, the display unit 205 displays trained models stored in the machine learning model database held by the machine learning model holding unit 231 as options of trained models. The user can select a trained model to be used from the options of trained models displayed on the display unit 205, and the NN control unit 121 can perform drive control according to the learning tendency by using the weight of the selected trained model as the weight of the NN. Here, when the display unit 205 displays the options of trained models, user request information at the time of learning, etc. may be displayed as a description of the trained model.

[0091] The selection of the trained model in step S705 may be performed so as to automatically select the trained model used last time. In this case, the user's selection can be omitted, and the user's effort in operation can be reduced. In addition, the control unit 211 may automatically select a trained model trained using, for example, a lens device of a lens model that matches or is similar to the lens 100 and data acquired under shooting conditions similar to the shooting conditions set in step S701. When the processing in step S704 or step S705 ends, the processing proceeds to step S706.

[0092] In step S706, shooting processing is performed in response to the user's operation. In the shooting processing, zooming, shake correction, and AF focus control by the NN control unit 121 are performed in response to the user's operation. Also, in response to the user's operation, acquisition of video data using the imaging element 201, storage of the video data in the recording unit 204, display of the video on the display unit 205, etc. are performed.

[0093] In step S707, the control unit 211 determines whether or not an instruction to end image capture has been input from the user via the operation unit 206. If the control unit 211 determines that an instruction to end image capture has not been input, the process returns to step S706, and the image capture process continues. On the other hand, if the control unit 211 determines that an instruction to end image capture has been input, the control unit 211 ends the series of image capture processes.

[0094] In such a series of shooting processes according to the present embodiment, the weights of a trained model that has been trained in advance are used as the initial values ​​of the weights of the NN, thereby reducing the number of times the drive unit is driven during training of the NN and suppressing wear on the drive unit. In addition, the time required for machine learning can be shortened, and the burden on the user can be reduced. Furthermore, when setting the initial values ​​of the weights of the NN, the weights of a trained model that has been trained using reward information corresponding to a request that matches or is similar to the request of the user can be referred to, thereby realizing focus control that meets the request of the user.

[0095] In this embodiment, the setting of the photographing conditions is performed before the decision to perform machine learning, but the timing of setting the photographing conditions is not limited to this. The setting of the photographing conditions may be performed before the machine learning process in step S704 and the selection of the trained model in step S705. Therefore, for example, the setting of the photographing conditions may be performed after the decision process to perform machine learning in step S702 and the decision process of the reward information in step S703.

[0096] <Reward Information> Next, the reward information will be described in detail with reference to Fig. 8 and Fig. 9. The reward information is information that serves as a standard for scoring (evaluating) the control result of the NN algorithm obtained from the operation log information. The reward information has information on the boundary value of the score and the score assigned to each range delimited by the boundary value for the control result of the NN algorithm. Here, the reward information will be described with reference to Fig. 8.

[0097] (a1), (b1), (c1), and (d1) in Fig. 8 show examples of the relationship between the change over time during learning and the score boundary value for the items indicating the control results of the NN algorithm, namely, position accuracy, speed, acceleration, and power consumption, respectively. Here, the horizontal axis in (a1), (b1), (c1), and (d1) in Fig. 8 indicates the passage of time.

[0098] (a2), (b2), (c2), and (d2) in Fig. 8 show examples of the data structure of reward information for position accuracy, speed, acceleration, and power consumption, respectively. The reward information data is composed of multiple boundary values ​​and points that can be obtained within the ranges separated by the boundary values. In this embodiment, an example is shown in which the reward information data is composed of two boundary values ​​and three points.

[0099] Here, the NN learns so that the control result will have a higher score (evaluation value), so that the closer the boundary value is to the target for each target item, the higher the accuracy of the control will be. For example, the closer the boundary value for position accuracy is to 0, the higher the control will be. In addition, by setting a higher score for a certain item in the reward information compared to other items, it is possible to indicate that the learning has a higher priority than the other items. For example, by setting a higher score for power consumption than for position accuracy, the NN learns so that the control prioritizes power consumption over position accuracy.

[0100] The vertical axis in (a1) of Fig. 8 indicates the value of position accuracy E, which is the difference between the target position and the current position of the focus lens. A positive direction of position accuracy E indicates that the current position is on the infinity side of the target position, and a negative direction indicates that the current position is on the close side of the target position. Note that in (a1) of Fig. 8, the closer the position accuracy E is to 0, the higher the position accuracy in the drive control.

[0101] 8(a2) shows an example of the data structure of the location accuracy reward information RE, which is reward information for the location accuracy. In this embodiment, the location accuracy reward information RE is composed of boundary values ​​E1 and E2 that determine the reward range for the location accuracy, and points SE1, SE2, and SE3 that can be acquired within the reward range.

[0102] The boundary values ​​E1 and E2 indicate boundary values ​​of points given as a reward for the position accuracy E. Here, the range from -E1 to E1 is defined as the range AE1, the range from -E2 to E2 excluding the range AE1 is defined as the range AE2, and the range other than the range AE1 and the range AE2 is defined as the range AE3. When the position accuracy E is within the ranges AE1, AE2, and AE3, the scores SE1, SE2, and SE3 shown in (a2) of FIG. 8 are given as rewards. Here, the relationship between the scores SE1, SE2, and SE3 is score SE1>score SE2>score SE3, and the scores are set so that the closer the position accuracy E is to 0, the higher the score.

[0103] As shown in (a1) of Figure 8, the position accuracy E at any time TP1, TP2, Tp3 is within the ranges AE2, AE3, AE1, respectively. Therefore, the rewards that can be obtained at any time TP1, TP2, Tp3 are points SE2, SE3, SE1, respectively.

[0104] Here, for example, the boundary value E1 can be set to ±Fδ / 2, and the boundary value E2 can be set to ±Fδ. In this case, if the current position is controlled to be within the focal depth with respect to the target position of the focus lens, a high score is added, and if it is outside the focal depth, a low score is added. Also, the closer the focus lens is to the target position, the higher the score that can be obtained.

[0105] Next, the vertical axis of (b1) in Fig. 8 indicates the value of the drive speed V of the focus lens. The positive direction of the drive speed V indicates the drive speed toward infinity, and the negative direction indicates the drive speed toward the closest point. The closer the drive speed V is to 0, the quieter the drive sound becomes.

[0106] (b2) of Fig. 8 shows an example of the data structure of speed reward information RV, which is reward information for speed. In this embodiment, the speed reward information RV is composed of boundary values ​​V1 and V2 that determine the reward range of speed, and points SV1, SV2, and SV3 that can be obtained within the reward range.

[0107] The boundary values ​​V1 and V2 indicate boundary values ​​of points given as a reward for the drive speed V. Here, the range from -V1 to V1 is defined as range AV1, the range from -V2 to V2 excluding range AV1 is defined as range AV2, and the range other than range AV1 and range AV2 is defined as range AV3. When the drive speed V is within the ranges AV1, AV2, and AV3, the points SV1, SV2, and SV3 shown in (b2) of FIG. 8 are given as rewards. Here, the relationship of the points SV1, SV2, and SV3 is point SV1>point SV2>point SV3, and the points are set so that the closer the drive speed V is to 0, the higher the point is.

[0108] As shown in (b1) of Fig. 8, the drive speed V at any time TP1, TP2, Tp3 is within the ranges AV2, AV3, AV1, respectively. Therefore, at any time TP1, TP2, Tp3, the reward that can be acquired for the drive speed V related to the drive sound is the score SV2, SV3, SV1, respectively.

[0109] Here, for example, the boundary values ​​V1 and V2 are determined based on the relationship between the drive speed and the drive noise, and the score is set so that the slower the drive speed is controlled, the higher the score that can be obtained. Generally, the slower the drive speed is, the quieter the drive noise is, so the higher the score obtained indicates that the control is more focused on quietness.

[0110] Next, the vertical axis of (c1) in Fig. 8 indicates the value of the drive acceleration A of the focus lens. The positive direction of the drive acceleration A indicates the drive acceleration toward infinity, and the negative direction indicates the drive acceleration toward the close range. The closer the drive acceleration A is to 0, the quieter the drive sound becomes.

[0111] 8(c2) shows an example of the data structure of acceleration reward information RA, which is reward information for acceleration. In this embodiment, the acceleration reward information RA is composed of boundary values ​​A1 and A2 that determine the reward range of acceleration, and points SA1, SA2, and SA3 that can be acquired within the reward range.

[0112] The boundary values ​​A1 and A2 indicate boundary values ​​of points given as a reward for the driving acceleration A. Here, the range from -A1 to A1 is the range AA1, the range from -A2 to A2 excluding the range AA1 is the range AA2, and the range other than the range AA1 and the range AA2 is the range AA3. When the driving acceleration A is within the ranges AA1, AA2, and AA3, the scores SA1, SA2, and SA3 shown in (c2) of FIG. 8 are given as rewards. Here, the relationship of the scores SA1, SA2, and SA3 is score SA1>score SA2>score SA3, and the closer the driving acceleration A is to 0, the higher the score is set.

[0113] As shown in (c1) of Fig. 8, the driving acceleration A at any time TP1, TP2, Tp3 is within the ranges AA1, AA3, AA2, respectively. Therefore, at any time TP1, TP2, Tp3, the reward that can be acquired for the driving acceleration A related to the driving sound is the points SA1, SA3, SA2, respectively.

[0114] Here, for example, the boundary values ​​A1 and A2 are determined based on the relationship between the drive acceleration and the drive noise, and the score is set so that the smaller the drive acceleration is controlled, the higher the score that can be obtained. Generally, the smaller the drive acceleration is, the quieter the drive noise is, so the higher the obtained score is, the more importance is attached to the control for quietness.

[0115] Next, the vertical axis of (d1) in Fig. 8 indicates the value of the power consumption P of the focus lens. The closer the power consumption P is to 0, the smaller the power consumption is.

[0116] (d2) in Fig. 8 shows an example of the data structure of power consumption reward information RP, which is reward information for power consumption. In this embodiment, the power consumption reward information RP is composed of boundary values ​​P1 and P2 that determine the reward range for power consumption, and points SP1, SP2, and SP3 that can be acquired within the reward range.

[0117] Boundary values ​​P1 and P2 indicate boundary values ​​of points given as a reward for power consumption P. Here, the range from 0 to P1 is range AP1, the range from P1 to P2 is range AP2, and the range other than range AP1 and range AP2 is range AP3. When power consumption P is within ranges AP1, AP2, and AP3, respectively, points SP1, SP2, and SP3 shown in (d2) of FIG. 8 are given as rewards. Here, the relationship of points SP1, SP2, and SP3 is point SP1>point SP2>point SP3, and the points are set so that the closer the power consumption P is to 0, the higher the point.

[0118] As shown in (d1) of Fig. 8, the power consumption P at any time TP1, TP2, TP3 is within the ranges AP1, AP3, AP2, respectively. Therefore, the rewards that can be acquired for the power consumption P at any time TP1, TP2, TN3 are points SP1, SP3, SP2, respectively.

[0119] Here, for example, the boundary values ​​P1 and P2 are arbitrarily determined, and the points are set so that the more the power consumption is controlled to be small, the higher the points that can be acquired. Therefore, the higher the acquired points, the more the control that emphasizes low power consumption is being performed.

[0120] As described above, reward information for scoring control results such as position control error, speed, acceleration, and power consumption is set. Using such reward information, the learning processing unit 221 scores the control results of the NN algorithm for each unit time based on operation log information related to focus lens driving during learning, and can determine the cumulative score (accumulated evaluation value) of the control results of the NN algorithm by accumulating the scores for each unit time. In addition, with regard to the control results of the NN algorithm, the respective scores of the position control error, speed, acceleration, and power consumption are added together and the total score is set as the evaluation value, so that the total control result of the NN algorithm can be scored.

[0121] Here, an example is shown in which power consumption is used as one of the control results, but reward information for power consumption may also be set using the results of speed and acceleration based on the relationship between speed and acceleration and power consumption.

[0122] In this embodiment, the number of boundary values ​​is two, but the number of boundary values ​​is not limited to this, and may be three or more depending on the desired configuration, or the number of boundary values ​​may be variable as necessary. In addition, in this embodiment, the score is determined by the boundary value, but a method of scoring using a conversion function that converts the position accuracy E, the driving speed V, the driving acceleration A, and the power consumption P into a score may also be used. In this case, the conversion function and its coefficient are set as the reward information, rather than the boundary value and the score that can be obtained within the range divided by the boundary value.

[0123] In this embodiment, in order to perform learning according to the configuration of the lens device and the user's request, the reward information includes device constraint reward information based on device constraint information for each lens device and user request reward information based on request information by the user. Fig. 9 shows an example of the data structure of the device constraint reward information and user request reward information included in the reward information according to this embodiment.

[0124] First, the user-requested reward information is reward information that can be changed according to the user-requested information set by the user. As described above, the user-requested information is converted into the user-requested reward information using the user-requested reward conversion information. Here, the user-requested reward conversion information is set in advance for each lens model, and is stored in association with the lens model information in the user-requested reward conversion information database held by the user-requested reward conversion information holding unit 229. The user-requested reward conversion information may be, for example, a conversion table for converting the user-requested information into the user-requested reward information.

[0125] On the other hand, the device constraint compensation information is compensation information for prescribing the minimum control that should be observed as a device. Therefore, the device constraint compensation information has a wider range determined by boundary values ​​than the user desired compensation information, while a low score including a negative value is set when deviating from an expected target. In this embodiment, the device constraint compensation information is set in advance for each lens model, and is stored in association with the lens model information in the device constraint compensation information database held by the device constraint compensation information holding unit 228. Note that the device constraint compensation information may be set based on the results of a drive control experiment for each lens model, for example.

[0126] 9, in this embodiment, the device constraint reward information is composed of position accuracy reward information REb, speed reward information RVb, acceleration reward information RAb, and power consumption reward information RPb. Also, the user request reward information is composed of position accuracy reward information REu, speed reward information RVu, acceleration reward information RAu, and power consumption reward information RPu.

[0127] Here, the position accuracy reward information REb and the position accuracy reward information REu have the same data structure as the position accuracy reward information RE shown in (a2) of Fig. 8. Specifically, the position accuracy reward information REb has boundary values ​​Eb1 and Eb2, and has points SEb1, SEb2, and SEb3 that can be acquired within the ranges separated by the boundary values. The position accuracy reward information REu has boundary values ​​Eu1 and Eu2, and has points SEu1, SEu2, and SEu3 that can be acquired within the ranges separated by the boundary values.

[0128] The speed reward information RVb and the speed reward information RVu have the same data structure as the speed reward information RV shown in (b2) of Fig. 8. Specifically, the speed reward information RVb has boundary values ​​Vb1, Vb2, and has points SVb1, SVb2, and SVb3 that can be acquired within the range delimited by the boundary values. The speed reward information RVu has boundary values ​​Vu1, Vu2, and has points SVu1, SVu2, and SVu3 that can be acquired within the range delimited by the boundary values.

[0129] The acceleration reward information RAb and the acceleration reward information RAu have the same data structure as the acceleration reward information RA shown in (c2) of Fig. 8. Specifically, the acceleration reward information RAb has boundary values ​​Ab1 and Ab2, and has points SAb1, SAb2, and SAb3 that can be acquired within the ranges delimited by the boundary values. The acceleration reward information RAu has boundary values ​​Au1 and Au2, and has points SAu1, SAu2, and SAu3 that can be acquired within the ranges delimited by the boundary values.

[0130] The power consumption remuneration information RPb and the power consumption remuneration information RPu have the same data structure as the power consumption remuneration information RP shown in (d2) of Fig. 8. Specifically, the power consumption remuneration information RPb has boundary values ​​Pb1 and Pb2, and has points SPb1, SPb2, and SPb3 that can be acquired within a range bounded by the boundary values. The power consumption remuneration information RPu has boundary values ​​Pu1 and Pu2, and has points SPu1, SPu2, and SPu3 that can be acquired within a range bounded by the boundary values.

[0131] <Remuneration information determination process> Next, the remuneration information determination process in step S703 will be described with reference to Figures 10 to 14. Figure 10 is a flowchart showing the flow of the remuneration information determination process.

[0132] First, when the remuneration information determination process is started in step S703, the process proceeds to step S1001. In step S1001, in a state in which the lens 100 is attached to the camera body 200, the learning unit 220 acquires the lens individual information from the lens individual information management unit 127 via the control unit 125, the communication unit 126, the communication unit 212, and the control unit 211.

[0133] In step S1002, the device constraint compensation management unit 224 determines device constraint compensation information corresponding to the lens 100 from the device constraint compensation information database held by the device constraint compensation information holding unit 228 according to the lens model information of the lens individual information. Here, the device constraint information database and a method for determining device constraint information using the device constraint information database will be described with reference to Figures 11(a) to 11(d). Figures 11(a) to 11(d) show an example of the data structure of the device constraint compensation information database held in the device constraint compensation management unit 224.

[0134] FIG. 11(a) shows an example of the data structure of the database of the position accuracy reward information REb. The database of the position accuracy reward information REb is composed of a plurality of pieces of position accuracy reward information REb with different boundary values ​​and scores for each lens model. FIG. 11(b) shows an example of the data structure of the database of the speed reward information RVb. The database of the speed reward information RVb is composed of a plurality of pieces of speed reward information RVb with different boundary values ​​and scores for each lens model. FIG. 11(c) shows an example of the data structure of the database of the acceleration reward information RAb. The database of the acceleration reward information RAb is composed of a plurality of pieces of acceleration reward information RAb with different boundary values ​​and scores for each lens model. FIG. 11(d) shows an example of the data structure of the database of the power consumption reward information RPb. The database of the power consumption reward information RPb is composed of a plurality of pieces of power consumption reward information RPb with different boundary values ​​and scores for each lens model.

[0135] The device constraint compensation management unit 224 determines the position accuracy compensation information REb corresponding to the lens model of the lens 100 from the position accuracy compensation information REb database according to the lens model information of the lens individual information. For example, when the lens 100 is model A, the device constraint compensation management unit 224 sets the boundary values ​​Eb1 and Eb2 of the position accuracy compensation information REb to Eb1TA and Eb2TA, and sets the scores SEb1, SEb2, and SEb3 to SEb1TA, SEb2TA, and SEb3TA.

[0136] Similarly, the device constraint reward management unit 224 determines the speed reward information RVb, the acceleration reward information RAb, and the power consumption reward information RPb from the speed reward information RVb database, the acceleration reward information RAb database, and the power consumption reward information RPb database, respectively. Note that, although an example in which a device constraint reward information database is configured for each lens model has been shown in this embodiment, a device constraint reward information database may be configured for each individual lens. In this case, the device constraint reward management unit 224 can determine the device constraint reward information by the individual identification number included in the lens individual information.

[0137] Next, in step S1003, the user uses the operation unit 206 to set one of the predetermined user request information as the user request information to be currently applied. The user request management unit 226 acquires the user request information selected by the user via the control unit 211, and determines the acquired user request information as the user request information to be currently applied. Here, the user request information includes each level information of the location accuracy, the quietness, and the power consumption. The level information is information indicating the level required to be achieved for each of the location accuracy, the quietness, and the power consumption included in the request information. For example, as the level information for the location accuracy, levels 1, 2, and 3 corresponding to high, medium, and low levels are set in advance, and the user can select any level of the location accuracy that is desired to be achieved.

[0138] Here, a method for setting user request information using a user request information database will be described with reference to Fig. 12. Here, the user request information includes level information indicating a level to be achieved that is required by the user for the drive control of the focus lens. In this embodiment, information indicating the required levels for the user request items, namely, positional accuracy, quietness, and power consumption, is given as an example of the user request information.

[0139] 12 shows an example of the structure of the user request information database stored in the user request storage unit 227. The user request information database is configured with a user request ID, position accuracy user request information Eu, quietness user request information Su, power consumption user request information Pu, user identification information, shooting conditions, and creation date and time information for each user request information data.

[0140] Here, the user request ID is assigned a unique number that uniquely determines the user request information data. The position accuracy user request information Eu indicates the required level of position accuracy for the user request item. The quietness user request information Su indicates the required level of quietness for the user request item. The power consumption user request information Pu indicates the required level of power consumption for the user request item.

[0141] The user identification information is information that uniquely identifies a user who has set the user request information data, and is set by the user when generating the user request information.

[0142] The shooting conditions are information indicating the shooting conditions when the user request information is set. The shooting conditions can include, for example, settings of the lens 100 that affect the captured image, such as the zoom position, focus position, aperture value, and image shake correction state. The shooting conditions can also include settings of the camera body 200 that affect the captured image, such as information indicating video shooting or still image shooting, shooting mode, shutter speed, autofocus control, and exposure control settings. The creation date and time information records information indicating the date and time when the user request information data was created.

[0143] In the example shown in Fig. 12, the user request information database is composed of three pieces of user request information: user request information u1, u2, and u3. However, the number of pieces of user request information is not limited to this, and may be increased or decreased as desired by the user.

[0144] The above user request information database configuration makes it possible to manage the level value of the user request information and the information at the time the user request information was generated for each piece of user request information. In addition, the above database configuration makes it possible to easily check the association between each piece of user request information used in a trained machine learning model. Furthermore, it is possible to easily search for a trained model that has been trained using reward information generated from similar user request information.

[0145] Next, a method for setting currently applied user desired information from a plurality of pieces of user desired information will be described. The user operates the operation unit 206 to select one of the user desired information u1, u2, and u3. The selected user desired information is transmitted to the user desired management unit 226 via the control unit 211. The user desired management unit 226 manages the selected user desired information as currently applied user desired information. In the following, as an example, it is assumed that the user selects the user desired information u1.

[0146] Next, a method for changing the levels of the user request items for the currently applied user request information will be described. The user operates the operation unit 206 to set the levels of the position accuracy user request information Eu, the quietness user request information Su, and the power consumption user request information Pu of the selected user request information u1.

[0147] Information on the level set by the user is transmitted to the user request management unit 226 via the control unit 211. The user request management unit 226 determines the information on the level set by the user as currently applied user request information, and transmits it to the user request reward management unit 225. The example shown in Fig. 12 shows a case where level 1, level 2, and level 3 are set for the position accuracy user request information Eu, the quietness user request information Su, and the power consumption user request information Pu, respectively.

[0148] As described above, the user can select user request information u1 as user request information and set a level for each user request item. Furthermore, the user request management unit 226 can request the user request storage unit 227 to update the user request information u1 in the user request information database in accordance with the level information set by the user. In this case, the user request storage unit 227 can update the user request information database in response to the request to update the user request information u1.

[0149] In this embodiment, a case where user request information that already exists in the user request information database is changed has been described, but it is also possible to create new user request information initialized at a predetermined level and change the level of each request, etc. Also, it is also possible to create new user request information by copying another user request information and change the level of each request, etc.

[0150] Next, in step S1004, the user request reward management unit 225 identifies the lens model of the lens 100 based on the lens individual information acquired in step S1001. The user request reward management unit 225 also receives currently applied user requests from the user request management unit 226. The user request reward management unit 225 determines user request reward information according to the lens individual information and the received user request information.

[0151] More specifically, when the user requested reward management unit 225 receives each level information of position accuracy, quietness, and power consumption included in the user requested information, it transmits it to the user requested reward conversion information storage unit 229 together with the lens model information. The user requested reward conversion information storage unit 229 first determines user requested reward conversion information to be used for converting the user requested information from the user requested reward conversion information database according to the lens model information. Thereafter, the user requested reward conversion information storage unit 229 converts each level information included in the user requested information into user requested reward information by referring to the determined user requested reward conversion information.

[0152] Here, with reference to Figure 13 (a) to Figure 13 (c), the user desired reward conversion information database and the method of determining user desired reward information using the user desired reward conversion information database will be described. Figure 13 (a) to Figure 13 (c) show an example of the data structure of the user desired reward conversion information database.

[0153] FIG. 13(a) shows an example of the data structure of a database of position accuracy user desired reward conversion information UREu. The position accuracy user desired reward conversion information UREu is composed of multiple position accuracy reward information REu with different boundary values ​​and scores for each level. In addition, the database of position accuracy user desired reward conversion information UREu is composed of position accuracy user desired reward conversion information UREu for each lens model. For example, for model A, the position accuracy reward information REu used when the position accuracy user desired information is level 1 is position accuracy reward information REuTAL1.

[0154] FIG. 13(b) shows an example of the data structure of the database of quiet user requested reward conversion information URSu. Quiet user requested reward conversion information URSu is composed of speed user requested reward conversion information URVu and acceleration user requested reward conversion information URAu. Speed ​​user requested reward conversion information URVu and acceleration user requested reward conversion information URAu are composed of a plurality of speed reward information RVu and acceleration reward information RAu with different boundary values ​​and scores for each level. Furthermore, the quiet user requested reward conversion information URSu database is composed of speed user requested reward conversion information URVu and acceleration user requested reward conversion information URAu for each lens model. For example, for model A, the speed reward information RVu and acceleration reward information RAu used when the quiet user requested information is level 1 are speed reward information RVuTAL1 and acceleration reward information RAuTAL1, respectively.

[0155] 13(c) shows an example of the data structure of the power consumption user requested reward conversion information URPu database. The power consumption user requested reward conversion information URPu is composed of a plurality of pieces of power consumption reward information RPu with different boundary values ​​and points for each level. The power consumption user requested reward conversion information URPu database is composed of the power consumption user requested reward conversion information URPu for each lens model. For example, for model A, the power consumption reward information RPu used when the power consumption user requested information is level 1 is the power consumption reward information RPuTAL1.

[0156] Here, the boundary values ​​and score values ​​of the position accuracy user desired reward conversion information UREu, the quietness user desired reward conversion information URSu, and the power consumption user desired reward conversion information URPu are determined so that user desires increase in the order of level 1, level 2, and level 3. Specifically, level 1 has boundary values ​​that are closer to the targets of each item and has a high score compared to the others.

[0157] For example, when the lens 100 is model A, in the example shown in FIG. 13(a) to FIG. 13(b), the user requested reward conversion information holding unit 229 determines the user requested reward conversion information corresponding to model A from the user requested reward conversion information database. Here, when the user requested information selected by the user is user requested information u1, in the example shown in FIG. 12, the levels of the position accuracy user requested information Eu, the quietness user requested information Su, and the power consumption user requested information Pu are level 1, level 2, and level 3, respectively. In this case, in the example shown in FIG. 13(a) to FIG. 13(b), the user requested reward conversion information holding unit 229 refers to the user requested reward conversion information corresponding to model A and converts each level information included in the user requested information into user requested reward information. As a result, the position accuracy reward information REu, the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu become the position accuracy reward information REuTAL1, the speed reward information RVuTAL2, the acceleration reward information RAuTAL2, and the power consumption reward information RPuTAL3, respectively.

[0158] Note that if the user does not select user request information after the lens 100 is attached to the camera body 200, the user request information used when learning the previously attached lens may be used. Since the user request information is information independent of the lens model, it can be used as is without being changed when the lens 100 is replaced. Furthermore, since users often have the same requests before and after lens replacement, not changing the user request information allows the user to omit cumbersome user request settings.

[0159] In addition, in this embodiment, an example of constructing a user-requested remuneration conversion information database for each lens model has been shown, but a user-requested remuneration conversion information database may be constructed for each individual lens, and the user-requested remuneration conversion information may be determined by the individual identification number of the individual lens information. In this case, the user-requested remuneration management unit 225 can determine the user-requested remuneration information by the individual identification number included in the individual lens information.

[0160] When the user requested reward information is determined in step S1004, the process proceeds to step S1005. In step S1005, the reward management unit 223 combines the device constraint reward information determined in step S1002 and the user requested reward information determined in step S1004 to determine reward information as shown in Fig. 9. Note that, before the device constraint reward information determination process in step S1002, a user requested information acquisition process in step S1003 and a user requested reward information determination process in step S1004 may be performed.

[0161] In the above-mentioned learning of the NN, as described with reference to Fig. 8, the score of the focus control result is determined using the respective reward information of the device constraint reward information and the user request reward information determined in this way. After that, the respective determined scores are added, and the accumulated score is determined as the score of the final control result (accumulated evaluation value). Then, using the score of the final control result as the reward, reinforcement learning is performed so as to maximize the reward.

[0162] Here, the reward information database that stores the above-mentioned reward information will be described with reference to Fig. 14. Fig. 14 shows an example of the data structure of the reward information database. The reward information database is composed of reward information ID, creation date and time information, model information, user request information u, device constraint reward information Rb, user request reward information Ru, and user request reward conversion information URu for each reward information data.

[0163] Here, the remuneration information ID is assigned a unique numerical value that uniquely determines the remuneration information data. The creation date and time information records information indicating the date and time when the remuneration information was created. The model information records lens model information of the individual lens information related to the lens device that is the subject of the remuneration information data. In this embodiment, an example is shown in which the lens model is recorded as the model information, but information that identifies the individual lens that is the subject of the remuneration information data, such as an individual identification number, may also be recorded.

[0164] The user requested information u records the user requested information used when generating the reward information data. The device constraint reward information Rb records a value indicating the device constraint reward information included in the reward information. The user requested reward information Ru records a value indicating the user requested reward information included in the reward information. The user requested reward conversion information URu records the user requested reward conversion information used when generating the user requested reward information Ru.

[0165] 14, for example, reward information with reward information ID RID1 is created at creation date and time Date1, and includes device constraint reward information RbA1 and user requested reward information Ru1A1. For the reward information, the user requested information u and user requested reward conversion information URu used when generating the user requested reward information Ru1A1 are user requested information u1 and user requested reward conversion information URu1, respectively.

[0166] Furthermore, the reward information with reward information IDs RID3 and RID4 indicates reward information in a case where the model information and user requested information are the same, but the device constraint reward information database and the user requested reward conversion information database have changed due to a version upgrade, etc. When the device constraint reward information database and the user requested reward conversion information database have changed, the device constraint reward information Rb, the user requested reward information Ru, and the user requested reward conversion information URu change even if the model information and user requested information are the same.

[0167] The above data structure of the reward information database allows management of the value of reward information and various information when generating reward information for one piece of reward information. In this embodiment, an example is shown in which actual information is recorded for model information, user requested information u, device constraint reward information Rb, user requested reward information Ru, and user requested reward conversion information URu. However, the information data of model information, user requested information u, device constraint reward information Rb, user requested reward information Ru, and user requested reward conversion information URu may be held in each database. In this case, the reward information database can manage these pieces of information by including ID information that uniquely identifies various pieces of information from each database.

[0168] For example, the database of the model information and the user requested reward information may be stored in an information storage unit (not shown). Also, the databases of the user requested information u, the device constraint reward information Rb, and the user requested reward conversion information URu may be stored in the user request storage unit 227, the device constraint reward information storage unit 228, and the user requested reward conversion information storage unit 229, respectively.

[0169] The above database configuration makes it easy to check the associations between the reward information used in trained machine learning models. Furthermore, it makes it easy to search for trained models that have been trained using similar reward information.

[0170] <Machine learning processing> Next, the machine learning process according to this embodiment will be described with reference to Fig. 15 to Fig. 19. Fig. 15 is a flowchart showing the flow of the machine learning process according to this embodiment. When the machine learning process is started in step S704, the process proceeds to step S1501.

[0171] In step S1501, the machine learning initial value management unit 232 selects a trained model to be an initial model from trained models trained in the past or recorded in advance, based on the lens individual information acquired in step S1001. The machine learning initial value management unit 232 also determines the weights of the trained model selected as the initial model as the initial values ​​of the weights, and sends them to the learning processing unit 221. Details of the weight initial value determination process will be described later.

[0172] In step S1502, the learning processing unit 221 outputs the initial values ​​of the weights to the control unit 211. When the control unit 211 receives the initial values ​​of the weights from the learning processing unit 221, it sends the initial values ​​of the weights to the lens 100 via the communication unit 212. When the lens 100 receives the initial values ​​of the weights at the communication unit 126, it stores the initial values ​​of the weights in the NN data storage unit 123 via the control unit 125.

[0173] In step S1503, the learning processing unit 221 transmits a drive command for the focus lens 101 to the lens 100 and requests acquisition of operation log information via the control unit 211 and the communication unit 212. Here, the learning processing unit 221 can hold a specific drive pattern from a start position to a stop position determined in advance for learning as a drive command for the focus lens 101, and send a drive command according to the held drive pattern. Note that the drive pattern may be a random pattern. The learning processing unit 221 may also send a drive command for the focus lens 101 to execute AF (autofocus) control.

[0174] In the lens 100, when the communication unit 126 receives a drive command for the focus lens 101, the control unit 125 sends the drive command for the focus lens 101 to the NN control unit 121. Upon receiving the drive command, the NN control unit 121 performs drive control of the focus lens 101 using the weights stored in the NN data storage unit 123 as NN weights.

[0175] In addition, in the lens 100, when the communication unit 126 receives a request to acquire operation log information, the control unit 125 requests the operation log management unit 124 to output the operation log information. Here, the operation log information will be described. The operation log information is control result information that is used to determine a score when converting the control result of the NN algorithm into a score.

[0176] The operation log management unit 124 collects and records input / output information of the NN algorithm corresponding to the target position X1, the current position X2, the focal depth X3, the focus sensitivity X4, and the drive signal Y1 shown in FIG. 6 for each control period of the NN algorithm. For example, the operation log management unit 124 records the drive command input to the NN control unit 121 and the position information of the focus lens detected by the focus lens detection unit 106 as operation log information. Furthermore, for example, the operation log management unit 124 determines the target position, position information, and position accuracy E of the focus lens from the drive command, and records them as operation log information. Also, for example, the operation log management unit 124 calculates the speed and acceleration of the focus lens from the position information of the focus lens, and records them as operation log information. Furthermore, in the case where a power detection unit (not shown) is provided to measure the power consumption of the focus lens drive unit 105, the operation log management unit 124 can also record the information on the power consumption of the focus lens drive unit 105 obtained from the power detection unit as operation log information.

[0177] When operation log management unit 124 receives a request to output operation log information, it transmits the operation log information recorded when focus lens 101 was driven to camera body 200 via control unit 125 and communication unit 126. The operation log information transmitted to camera body 200 is transmitted to operation log storage unit 222 via communication unit 212 and control unit 211, and is stored by operation log storage unit 222.

[0178] Next, in step S1504, the learning processing unit 221 scores the control result of the NN algorithm based on the weight of the initial value, based on the reward information held by the reward management unit 223 and the operation log information held by the operation log holding unit 222. As for scoring the control result, as described above, the reward information is used to score the control result of the NN algorithm for each unit time based on the operation log information, and the scores for each unit time are accumulated, thereby making it possible to determine the accumulated score of the control result of the NN algorithm.

[0179] Next, in step S1505, the learning processing unit 221 updates the weights so that the cumulative score of the NN algorithm control result is maximized. Although the back propagation method is used to update the weights, the weight update method according to this embodiment is not limited to this, and any known method may be used. The generated weights are stored in the NN data storage unit 123 in the same procedure as in S1502.

[0180] In step S1506, the learning processing unit 221 judges whether or not the learning of the weights is completed. The completion of the learning can be judged by whether the number of iterations of the learning (update of the weights) reaches a specified value, or whether the amount of change in the cumulative score in the operation log information at the time of the update is smaller than a specified value. If the learning processing unit 221 judges that the learning is not completed, the process returns to step S1503, and the machine learning process is continued. On the other hand, if the learning processing unit 221 judges that the learning is completed, the machine learning process is terminated. Note that the machine learning model for which the learning has been completed is added to and held in a machine learning model database stored in the storage device of the learning unit 220.

[0181] By such processing, machine learning processing of the NN is performed based on the device constraint reward information based on the configuration of the lens 100 and the user request reward information according to the user request changed by the user setting. As a result, an NN capable of performing appropriate control according to the configuration of the lens 100 and the user request is generated. Note that the weights of the trained model (NN) updated in the machine learning processing are sent from the camera body 200 to the lens 100 and stored in the NN data storage unit 123 as described in step S1505, and can be used for focus drive control.

[0182] In addition, specific algorithms of the machine learning according to the present embodiment include deep learning, which uses NN to generate features and weighting coefficients for learning. In addition, nearest neighbor methods, naive Bayes methods, decision trees, support vector machines, etc. can be used. Any of the above algorithms that can be used can be applied to the present embodiment as appropriate.

[0183] Since the GPU can perform efficient calculations by processing a larger amount of data in parallel, it is effective to use the GPU for processing when performing learning multiple times using a learning model such as deep learning. Therefore, the GPU may be used in addition to the CPU for processing by the learning processing unit 221. Specifically, when executing a learning program including a learning model, the CPU and GPU may cooperate to perform calculations to perform learning. The processing of the learning processing unit 221 may be performed by only the CPU or the GPU.

[0184] Next, the machine learning model database will be described with reference to Fig. 16. Fig. 16 shows an example of a data structure of the machine learning model database. The machine learning model database is composed of a machine learning model ID, creation date and time information, reward information ID, machine learning model NN, and machine learning model initial value NNi for each piece of machine learning model data. Note that the machine learning models recorded in the machine learning model database are trained models that have undergone training.

[0185] The machine learning model ID is assigned a unique numerical value that uniquely determines the machine learning model data. The creation date and time information records information indicating the date and time when the machine learning model was created. The reward information ID is a unique numerical value that uniquely determines the reward information shown in FIG. 14 that was used to generate the machine learning model. The machine learning model NN is information indicating the weight of the machine learning model. The machine learning model initial value NNi is information indicating the weight of the machine learning model that was used as the initial value of the machine learning model when the machine learning model was generated. Here, the lens individual information, user request information, user request reward conversion information, and reward information used to generate the machine learning model can be acquired from the reward information database based on the reward information ID.

[0186] 16, for example, a machine learning model with a machine learning model ID of NNID1 is generated at creation date and time Date1, and the weight information of the machine learning model is machine learning model NN1. In addition, the reward information used during learning is reward information RID1, and the weight information of the machine learning model at the start of learning is machine learning model initial value NNi1.

[0187] The data structure of the machine learning model database described above allows management of information indicating the weight of a trained model and information when the trained model is generated for one trained model. In this embodiment, an example in which information indicating an actual weight is used for the machine learning model initial value NNi has been shown, but a machine learning model ID indicating an initial model may also be used. In addition, instead of the reward information ID, a value recorded in the reward information database indicated by the actual reward information ID may also be used. In addition, the machine learning model NN and the machine learning model initial value NNi may include not only information indicating the weight, but also information on the neural network structure or logic information on the neural network structure.

[0188] The above database configuration makes it easy to check the relationships between trained machine learning models and to search for similar trained machine learning models.

[0189] <Weight initial value determination process> Next, the details of the NN weight initial value determination process in step S1501 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the flow of the weight initial value determination process. When the weight initial value determination process is started in step S1501, the process proceeds to step S1701.

[0190] In step S1701, the machine learning initial value management unit 232 requests the machine learning model holding unit 231 to select a trained model in which the lens individual information of the lens device used to generate the trained model matches the lens individual information acquired in step S1001. The machine learning model holding unit 231 refers to the machine learning model database and the reward information database, and selects a machine learning model in which the lens individual information of the lens device used to generate the trained model matches the lens individual information of the lens 100.

[0191] Next, in step S1702, the machine learning model holding unit 231 determines whether or not multiple trained models have been selected. If the machine learning model holding unit 231 determines that multiple trained models have been selected, the process proceeds to step S1703. On the other hand, if the machine learning model holding unit 231 determines that multiple trained models have not been selected, the process proceeds to step S1704.

[0192] In step S1703, the machine learning model holding unit 231 selects a trained model trained using reward information or user request information that is closest to the currently applied reward information or user request information. Details of the trained model selection method will be described later. When the machine learning model holding unit 231 selects a trained model trained using reward information or user request information that is closest to the currently applied reward information or user request information, the process proceeds to step S1705.

[0193] In response to this, in step S1704, the machine learning model holding unit 231 determines whether or not there is a trained model selected in step S1701. If the machine learning model holding unit 231 determines that there is a trained model selected in step S1701, the process proceeds to step S1705. On the other hand, if the machine learning model holding unit 231 determines that there is no trained model selected in step S1701, the process proceeds to step S1706.

[0194] In step S1705, the machine learning initial value management unit 232 determines the weights of the trained model selected by the machine learning model holding unit 231 as the initial values ​​of the weights of the NN. Meanwhile, in step S1706, the machine learning initial value management unit 232 determines predetermined weights as the initial values ​​of the weights of the NN. Note that in step S1706, the machine learning initial value management unit 232 may set the weights of the NN currently used by the lens 100 as the initial values ​​of the weights of the NN instead of the predetermined weights.

[0195] As described above, the machine learning initial value management unit 232 can determine the initial values ​​of the weights of the NN from the lens individual information and the remuneration information or the user request information. According to such processing, the NN used by the NN control unit 121 can start machine learning from the weights of the trained model that is closest to the control target, and the number of times the drive unit is driven during machine learning can be reduced, thereby shortening the machine learning time.

[0196] In this embodiment, an example is shown in which a trained model is selected in step S1701 using a lens device of a model that matches the lens individual information for training. Alternatively, information on similar products may be included in the lens individual information in advance, allowing a trained model to be selected in which a similar product to the lens 100 is used for training. Also, in this embodiment, the determination process in step S1704 is performed after the determination process in step S1702, but the determination process in step S1704 may be performed before the determination process in step S1702.

[0197] <How to select the trained model with the closest reward information> Next, with reference to Fig. 18(a) to Fig. 18(d), a method in which the machine learning model holding unit 231 selects a trained model trained using reward information closest to the currently applied reward information in step S1703 will be described. Fig. 18(a) to Fig. 18(d) show an example of the value of user requested reward information in reward information RID1 to RID3. Here, the user requested reward information is composed of position accuracy reward information REu, speed reward information RVu, acceleration reward information RAu, and power consumption reward information RPu, and Fig. 18(a) to Fig. 18(d) show an example of the data structure of each piece of reward information. Below, an example will be described in which reward information RID1 is reward information actually used for machine learning.

[0198] In this case, the machine learning model holding unit 231 selects reward information that is closest to the reward information RID1 from among the reward information RID2 to RID3. For example, the machine learning model holding unit 231 calculates the absolute value of the difference between each of the boundary values ​​Eu1R1 and Eu2R1 of the reward information RID1 and the boundary values ​​Eu1 and Eu2 of the reward information to be compared.

[0199] Also, the learning result changes depending on the score value. Therefore, for example, the machine learning model holding unit 231 calculates the ratio of the scores SEu1R1, SEu2R1, and SEu3R1 of the reward information RID1, and calculates the ratio of the scores SEu1, SEu2, and SEu3 of the reward information to be compared. Furthermore, the machine learning model holding unit 231 calculates the difference as the absolute value of these calculated ratios.

[0200] Here, the smaller the difference between the boundary values ​​Eu1 and Eu2 and the difference between the ratios of the scores SEu1, SEu2, and SEu3, the closer the reward information is to the reward information RID1. Therefore, the machine learning model holding unit 231 converts the difference between the boundary values ​​Eu1 and Eu2 and the difference between the ratios of the scores SEu1, SEu2, and SEu3 into scores indicating similarity so that the smaller the difference, the greater the similarity between the reward information. In this case, the reward information with the highest score indicating similarity is the closest to the reward information RID1.

[0201] The machine learning model holding unit 231 also performs the above scoring on the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu. After that, the machine learning model holding unit 231 selects the reward information with the highest sum of the scores indicating the similarity between the position accuracy reward information REu, the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu.

[0202] In the above manner, reward information closest to reward information RID1 can be selected from reward information RID2 to RID3. The machine learning model holding unit 231 selects the trained model trained using the reward information selected in this manner as the trained model trained using reward information closest to the currently applied reward information in step S1703.

[0203] Regarding the scoring of the position accuracy reward information REu, the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu, the difference between the boundary values ​​and the difference between the score ratios may be multiplied by a fixed coefficient specific to each of them according to the influence of the machine learning result. Also, the score may be calculated from a predetermined polynomial function regarding the difference between the boundary values ​​and the difference between the score ratios.

[0204] Furthermore, the position accuracy reward information REu, the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu may be considered in terms of the degree of influence of each other's reward information, and a specific fixed coefficient may be multiplied to the score indicating the similarity calculated for each of them. Also, a score considering the degree of influence of the reward information may be calculated from a polynomial function set in advance for the score indicating the calculated similarity.

[0205] In this embodiment, the reward information closest to the currently applied reward information is selected using only the user requested reward information in the reward information. Alternatively, a score indicating the similarity may be calculated from both the user requested reward information and the device constraint reward information in the reward information, and the reward information closest to the currently applied reward information may be selected according to the score.

[0206] In this embodiment, the score indicating the similarity is calculated, but the closest reward information may be selected simply based on the difference in the boundary value or the difference in the score ratio. In this case, the closest reward information may be selected for each of the position accuracy reward information REu, the speed reward information RVu, the acceleration reward information RAu, and the power consumption reward information RPu, and the reward information with the most items selected as the closest reward information may be selected as the overall closest reward information. In this case, the overall closest reward information may be selected using a fixed coefficient or a polynomial function, taking into account the influence of the reward information.

[0207] <Trained model with the closest user request information Selection How to select> Next, with reference to Fig. 19, a method in which the machine learning model holding unit 231 selects a trained model trained using user request information that is closest to the currently applied user request information in step S1703 will be described. Fig. 19 shows an example of user request values ​​in user request information u1 to u3. Here, the user request information is composed of position accuracy user request information Eu, quietness user request information Su, and power consumption user request information Pu. Below, an example will be described in which user request information u1 is user request information used to generate reward information actually used in machine learning.

[0208] In this case, the machine learning model holding unit 231 selects the user desired information that is closest to the user desired information u1 from among the user desired information u2 to u3. For example, the machine learning model holding unit 231 calculates the absolute value of the difference between the position accuracy user desired information Eu of the user desired information u1 and the position accuracy user desired information Eu of the user desired information to be compared, for level 1 indicated by the position accuracy user desired information Eu of the user desired information u1.

[0209] Here, the smaller the difference in the position accuracy user desired information Eu, the closer the user desired information is to the user desired information u1. Therefore, the machine learning model holding unit 231 converts the difference in the position accuracy user desired information Eu into a score indicating the similarity so that the smaller the difference, the greater the similarity between the user desired information. In this case, the user desired information with the highest score indicating the similarity is the user desired information closest to the user desired information u1.

[0210] The machine learning model holding unit 231 also performs the above scoring on the quiet user request information Su and the power consumption user request information Pu. After that, the machine learning model holding unit 231 selects the user request information with the highest sum of the scores indicating the similarity between the position accuracy user request information Eu, the quiet user request information Su, and the power consumption user request information Pu.

[0211] In the above manner, it is possible to select the user desired information that is closest to the user desired information u1 from among the user desired information u2 to u3. The machine learning model holding unit 231 selects the trained model trained using the user desired information selected in this manner as the trained model trained using the user desired information that is closest to the currently applied user desired information in step S1703.

[0212] Regarding the conversion of the position accuracy user request information Eu, the quietness user request information Su, and the power consumption user request information Pu into scores, the calculated absolute value difference may be multiplied by a specific fixed coefficient. Also, the calculated absolute value difference may be used to calculate a score from a predetermined polynomial function.

[0213] Furthermore, the position accuracy user request information Eu, the quietness user request information Su, and the power consumption user request information Pu may be multiplied by a specific fixed coefficient to the score indicating the calculated similarity, taking into account the degree of influence of the user request information on each other. Also, a score considering the influence of the user request information may be calculated from a preset polynomial function for the calculated score indicating the similarity.

[0214] In this embodiment, the score indicating the similarity is calculated, but the closest user desired information may be selected simply based on the difference in the user desired information. In this case, the closest user desired information may be selected for each of the position accuracy user desired information Eu, the quiet user desired information Su, and the power consumption user desired information Pu, and the user desired information having the most items selected as the closest user desired information may be selected as the user desired information that is the most overall closest. In this case, too, the user desired information that is the most overall closest may be selected using a fixed coefficient or a polynomial function, taking into account the influence of the user desired information.

[0215] In addition, as a method for selecting a trained model trained using reward information or user request information that is closest to the currently applied reward information or user request information, it may be predetermined whether the trained model is selected based on the reward information or the user request information. Also, for example, when multiple trained models are selected for the trained model with the closest user request information, a trained model with the closest reward information may be further selected from the selected multiple trained models. On the other hand, when multiple trained models are selected for the trained model with the closest reward information, a trained model with the closest user request information may be further selected from the selected multiple trained models.

[0216] Furthermore, when multiple trained models are selected for the trained model with the closest reward information or user request information, the most recently generated trained model may be selected from the selected multiple trained models. Also, the trained model may be managed in association with a captured image captured using the trained model, and the user may be prompted to specify a trained model based on the captured image from the multiple trained models selected by the above-mentioned process, and the specified trained model may be selected.

[0217] As described above, the camera system according to the present embodiment functions as an example of an imaging device including the lens 100 and the camera body 200 including the image sensor 201 that captures an image formed by the lens 100. The lens 100 includes an optical member, a driving unit that drives the optical member, and a control unit that controls the driving unit based on the first trained model. Here, the first trained model is a trained model obtained by learning about the lens 100 with a second trained model obtained by learning about a device different from the lens 100 as an initial trained model. In this embodiment, the focus lens 101 functions as an example of an optical member of the lens 100, and the focus lens driving unit 105 functions as an example of a driving unit that drives the optical member of the lens 100. In addition, the NN control unit 121 and the control unit 125 of the lens 100 can constitute a part of a control unit that controls the focus lens driving unit 105 based on the first trained model.

[0218] Furthermore, the camera body 200 functions as an example of a camera device to which the lens 100 is detachably attached. Furthermore, the camera body 200 includes a learning unit 220 that functions as an example of a control unit that acquires a first trained model. The learning unit 220 can acquire a second trained model, and generate a first trained model based on reward information according to a user's request, using the second trained model as an initial trained model. Note that the control unit 211 and the learning unit 220 can constitute a part of the control unit that acquires the first trained model.

[0219] Here, the learning unit 220 selects the second trained model from the group of trained models based on information about the lens 100, which is a lens device. The learning unit 220 can select the second trained model based on at least one of information about the lens device used to train the trained model, information about the initial value of the trained model, information about the user's request used to train the trained model, information about the reward used to train the trained model, and information about the training date and time. In addition, the learning unit 220 can select the second trained model from the group of trained models based on information about at least one of the reward used in training to obtain the first trained model and the user's request. Note that the information about the lens device can include lens individual information, which is an example of information indicating the model and individual of the lens device. Note that the machine learning model holding unit 231 of the learning unit 220 can function as an example of a selection unit that selects the second trained model from the group of trained models.

[0220] In this embodiment, the first learned model is acquired by applying weights updated by learning on the lens 100, which correspond to the weights of the first learned model, to the NN in the NN control unit 121. Therefore, the NN control unit 121, the NN data storage unit 123, and the control unit 125, which acquire the weights of the first learned model from the camera body 200, can configure a part of the control unit of the lens 100 that acquires the first learned model.

[0221] With this configuration, the camera system according to the present embodiment determines, from among trained models generated in the past, a trained model trained using a lens device that matches or is similar to the lens 100 to be trained, as the initial value of the NN. This makes it possible to start training the NN of the lens 100 from a trained model that is close to the trained model after training using the lens 100.

[0222] As described above, the camera system according to this embodiment can obtain a final trained model with fewer learning iterations compared to starting learning from a fixed machine learning model. As a result, it is possible to suppress the driving of the driving unit during machine learning. Therefore, according to this embodiment, it is possible to provide an imaging device including a lens device and a camera device that are advantageous for obtaining a trained model. In addition, the camera system according to this embodiment can control the driving in accordance with the user's request by using reward information according to the user's request for learning.

[0223] In this embodiment, the lens individual information includes at least one of a lens model and an individual lens. The first trained model can be a trained model that has been trained using information acquired using a lens device having lens information that matches the lens information indicating the lens 100. Furthermore, the remuneration information based on the configuration of the lens 100 can be identified using the individual lens information. Furthermore, the remuneration information based on the user's request is user-requested remuneration information obtained by converting the user-requested information based on the conversion information identified using the individual lens information.

[0224] Furthermore, the camera system according to the present embodiment further includes a machine learning model holding unit 231 that functions as an example of a model management unit that manages a plurality of trained models. The machine learning model holding unit 231 can manage a plurality of trained models, lens individual information of the lens device used in training each trained model, and training-related information in association with each other, using a machine learning model database held by the machine learning model holding unit 231. Here, the training-related information can include items included in the machine learning model database described above. For example, the training-related information includes at least one of the initial value of each training model, user request information, user request reward conversion information used to convert the user request information, reward information, and training date and time. In this case, the machine learning model holding unit 231 can select a second trained model from the group of trained models based on the lens individual information and the training-related information.

[0225] In this configuration, a desired trained model can be easily searched for by linking previously generated trained models with information related to the machine learning and managing them. Therefore, it is possible to easily select an appropriate trained model as an initial value for machine learning to be performed from a group of previously generated trained models.

[0226] Furthermore, the machine learning model holding unit 231 according to the present embodiment can select a second trained model from a group of trained models using at least one of the lens individual information and the user-requested reward information indicating the configuration of the lens 100 and the user-requested information. Therefore, it is possible to easily select an appropriate trained model as an initial value for machine learning to be performed from a group of trained models generated in the past.

[0227] Furthermore, the camera system according to the present embodiment further includes a reward management unit 223 that manages multiple pieces of reward information. The reward management unit 223 can manage multiple pieces of reward information, lens information corresponding to each piece of reward information, and reward generation information in association with each other using the reward information database held by the reward information holding unit 230. Here, the reward generation information can include items included in the reward information database described above. For example, the reward generation information includes at least one of the user's request, conversion information used to convert the user's request, and the creation date and time of the reward information. In this case, the machine learning model holding unit 231 can select a second trained model from the group of trained models based on the lens individual information and the reward generation information.

[0228] In this configuration, previously generated reward information is managed in association with information related to the generation of the reward information, making it easy to search for desired reward information. Therefore, from the previously generated reward information, reward information closest to the reward information to be used in machine learning to be performed in the future can be easily selected, and a machine learning model generated based on the closest reward information can be easily selected.

[0229] The camera system according to the present embodiment further includes a user request management unit 226 that functions as an example of a request management unit that manages information on requests from multiple users. The user request management unit 226 can manage the information on requests from multiple users in association with the lens information when the request information of each user was set and the request-related information by using a user request information database held by the user request holding unit 227. Here, the request-related information can include items included in the above-mentioned user request information database. For example, the request-related information includes at least one of user identification information, shooting conditions when the request information was set, and date and time when the request information was set. In this case, the machine learning model holding unit 231 can select a second trained model from the group of trained models based on the lens individual information and the request-related information.

[0230] In this configuration, the previously created user request information is managed in association with information related to the user request information created, so that the desired user request information can be easily searched for. Therefore, it is possible to easily select the user request information that is closest to the user request information that is the basis of the reward information to be used in the machine learning to be performed from the previously created user request information. Furthermore, it is possible to easily select the machine learning model generated based on the closest user request information.

[0231] In this way, by selecting the second trained model based on the reward generation information or the request-related information, the machine learning model that is closest to the reward information or the user request information can be determined as the initial value. Therefore, learning can be started from a trained model that is closer to the trained model after learning using the lens 100, and the driving of the driving unit during machine learning can be further suppressed.

[0232] In this embodiment, the machine learning model holding unit 231 is configured to function as an example of a selection unit that selects a second trained model using lens individual information and a model management unit that manages a plurality of trained models. In contrast, the selection unit and the model management unit may be provided as separate components. In this case, the machine learning model holding unit 231 may include the selection unit and the model management unit.

[0233] In this embodiment, the four indexes required for the focus lens control are, for example, the F value, the focus sensitivity, and the driving time, the moving speed, and the acceleration change of the focus lens. However, the control-related shooting conditions are not limited to these, and may be changed according to a desired configuration.

[0234] In addition, in this embodiment, the current position of the focus lens, the focal depth, and the focus sensitivity that can be acquired according to the internal configuration of the lens 100 are used as inputs to the NN for learning, but information obtained from the captured video may also be used for learning. Specifically, the S / N ratio between the recorded audio and the actuator driving sound, and the defocus amount during shooting may also be used as inputs to the NN for learning.

[0235] Furthermore, the input to the NN is not limited to this, and for example, the attitude difference, temperature, and the sound volume of the surrounding environment may be used. Regarding the attitude difference, the influence of gravity when driving various lenses and apertures changes, so the drive torque required for the motor changes depending on the attitude difference. Regarding the temperature, the characteristics of the lubricating oil used in the drive connection parts of various lenses and apertures change, so the drive torque required for the motor changes depending on the temperature.

[0236] Also, for example, when shooting in a quiet environment, the user may feel uncomfortable with the driving sound. Also, when shooting a video, audio is recorded at the same time, so the problem occurs that unnecessary driving sound is recorded in the shot video. Therefore, depending on the shooting situation, quietness is required to reduce the driving sound as much as possible. On the other hand, for example, when the driving speed of various lenses and apertures is limited to suppress the driving sound of a motor, if the driving sound of the motor is within a range small compared to the volume of the surrounding environment, removing the limit on the driving speed does not affect the shot video. Therefore, control that changes the maximum speed limit of the driving speed according to the volume of the surrounding environment is also useful.

[0237] In this way, when other items are added to the input items to the NN or the above-mentioned items are reduced, the items used to calculate the reward can be changed according to the input items to the NN. This makes it possible to perform control according to the desired shooting conditions and shooting situations based on the changed items.

[0238] In addition, when the lens 100 is replaced with a different lens device, the camera body 200 can change the user's requested reward information based on the lens individual information of the replaced lens device. Specifically, the learning unit 220 of the camera body 200 uses, as reward information based on the user's request, reward information obtained by converting the user's request based on the user's requested reward conversion information identified using the lens individual information indicating the replaced lens device, for learning. This makes it possible to omit setting a user's request each time the lens device is replaced, even when the lens device is replaced. Therefore, when the lens is replaced, the user's request used in the past can be used as is, and the trouble of having the user set a new user request can be prevented.

[0239] In addition, when changing lenses, the reward information used for machine learning can be optimized for the lens model or individual lens by switching to device constraint reward information and user desired reward conversion information that corresponds to the lens model or individual lens.

[0240] In this embodiment, the focus control in which the focus lens is driven is configured to be controlled using the learned model. However, the control using the learned model according to this embodiment may also be applied to other controls (zoom control, anti-shake control, aperture control, etc.).

[0241] Regarding noise reduction and power consumption, when an optical member such as a zoom lens is driven by an actuator, there are issues similar to those in controlling the drive of a focus lens. Furthermore, regarding positional accuracy, each control has its own issues. For example, in zoom control, the required positional accuracy changes depending on the relationship between the amount of change in the magnification of the subject when the angle of view changes. Furthermore, the required positional accuracy changes depending on the relationship between the amount of zoom lens drive and the amount of change in the angle of view. Furthermore, in vibration reduction control, the required positional accuracy changes depending on information on the relationship between the focal length and the amount of shift of the image. Furthermore, in aperture control, the required positional accuracy changes depending on the relationship between the amount of aperture drive and the amount of change in the brightness of the image.

[0242] Therefore, by applying control using the trained model according to this embodiment to controls other than focus control (zoom control, anti-shake control, aperture control, etc.), it is possible to achieve well-balanced control of the required positional accuracy, noise reduction, and power consumption. The above-mentioned various modified examples can also be applied to the following embodiments as appropriate.

[0243] Example 2 Hereinafter, a camera system according to a second embodiment of the present invention will be described with reference to FIG. 20. FIG. 20 is a block diagram showing a system configuration of a camera system according to this embodiment. This embodiment differs from the first embodiment in that the lens device includes a learning unit that performs machine learning processing. Note that the camera system according to this embodiment has the same configuration as the camera system according to the first embodiment except that the lens device includes a learning unit and an operation unit. Therefore, the same configuration as that of the first embodiment is denoted by the same symbols and the description thereof will be omitted. Hereinafter, the camera system according to this embodiment will be described with respect to the differences from the camera system according to the first embodiment.

[0244] In the camera system according to this embodiment, the camera body 200 does not include a learning unit 220. On the other hand, the lens 100 according to this embodiment is provided with a lens microcomputer 1120, an operation unit 1206, and a learning unit 1220.

[0245] The lens microcomputer 1120 is provided with a control unit 1125 that is different from the control unit 125 associated with the lens microcomputer 120 according to Example 1. In addition, the lens microcomputer 1120 differs from the lens microcomputer 120 according to Example 1 in that the lens microcomputer 1120 is connected to an operation unit 1206 and a learning unit 1220 and transmits information to each of them.

[0246] The control unit 1125 is a control unit that controls the respective positions of the zoom lens 102, the aperture unit 103, and the image stabilization lens 104, and also controls the transmission of information between the learning unit 1220 and the camera body 200. The operation unit 1206 is an operation unit that allows the user to operate the lens 100, and the user can operate the operation unit 1206 to perform operations related to machine learning, such as setting user request information.

[0247] The learning unit 1220 can be configured using a processor (CPU, GPU) and a storage device (ROM, RAM, HDD). The learning unit 1220 includes a learning processing unit 1221, an operation log storage unit 1222, a reward management unit 1223, a device constraint reward management unit 1224, a user request reward management unit 1225, a user request management unit 1226, and a user request storage unit 1227. The learning unit 1220 also includes a reward information storage unit 1230, a machine learning model storage unit 1231, and a machine learning initial value management unit 1232.

[0248] Each component of the learning unit 1220 may be realized by a processor such as a CPU or MPU executing a software module stored in a storage device. The processor may be, for example, a GPU or FPGA. Each component of the learning unit 1220 may be configured by a circuit that performs a specific function, such as an ASIC. Furthermore, the storage device of the learning unit 1220 stores programs for implementing the various components of the learning unit 1220, and various information and various databases held by each component of the learning unit 1220, such as operation log information held by the operation log holding unit 1222.

[0249] The learning unit 1220 does not include the device constraint reward information storage unit 228 and the user requested reward conversion information storage unit 229. The operation of the learning unit 1220 is the same as the operation of the learning unit 220 in the first embodiment, except that the device constraint reward information and the user requested reward conversion information do not depend on the lens model.

[0250] In this embodiment, a learning unit 1220 that performs machine learning is configured in the lens 100. Therefore, information transmission similar to the information transmission between the camera microcomputer 210 and the learning unit 220 in the first embodiment is performed between the zoom lens 102 and the learning unit 1220.

[0251] In this embodiment, the learning unit 1220 performs learning using the lens 100, and therefore the model of the lens device and the individual lens used for learning are uniquely determined. Therefore, a device constraint remuneration information database and a user desired remuneration conversion information database for each lens model do not exist, and the device constraint remuneration information and the user desired remuneration conversion information for the lens 100 are uniquely determined.

[0252] In this embodiment, the NN control unit 121, the control unit 1125, and the learning unit 1220 can configure a part of the control unit of the lens 100. Therefore, the control unit of the lens 100 can control the driving unit based on the first learned model. In addition, the control unit of the lens 100 can acquire the first learned model. Specifically, the control unit of the lens 100 can acquire a second learned model, and generate the first learned model based on reward information according to the user's request, using the second learned model as an initial learned model. Furthermore, the control unit of the lens 100 can select the second learned model from the group of learned models based on information about the lens 100. Note that the method of selecting the second learned model may be the same as the method described in the first embodiment.

[0253] Even in the configuration of this embodiment in which the learning unit 1220 is configured in the lens 100, by performing processing similar to the machine learning processing according to the first embodiment, a final trained model can be obtained with fewer learning iterations compared to the case where learning is started from a fixed machine learning model. As a result, the number of times the driving unit is driven during machine learning can be reduced. Therefore, according to this embodiment, a lens device that is advantageous for obtaining a trained model can be provided. Moreover, the lens 100 can control driving in accordance with the user's request by using reward information according to the user's request for learning.

[0254] Example 3 A camera system according to a third embodiment of the present invention will be described below with reference to Fig. 21. Fig. 21 is a block diagram showing a system configuration of a camera system according to the third embodiment. This embodiment differs from the first embodiment in that the remote device includes a learning unit that performs machine learning processing.

[0255] The camera system according to this embodiment has the same configuration as the camera system according to the first embodiment except that the remote device includes a learning unit and an operation unit, and therefore the same configuration as that of the first embodiment is given the same reference numerals and the description thereof is omitted. Hereinafter, the camera system according to this embodiment will be described with respect to the points different from the camera system according to the first embodiment.

[0256] The camera system according to this embodiment is provided with a remote device 400 in addition to the lens 100 and the camera body 200. Here, the remote device 400 is, for example, a mobile terminal or a personal computer terminal. The remote device 400 may also be any server, such as a cloud server, a fog server, or an edge server.

[0257] The camera body 200 according to this embodiment does not include a learning unit 220. On the other hand, the camera body 200 is provided with a communication unit 240. The communication unit 240 is a communication unit for communicating with the remote device 400.

[0258] The remote device 400 is provided with a display unit 401, an operation unit 402, a remote device microcomputer (hereinafter referred to as a remote device microcomputer 410), and a learning unit 420. The display unit 401 is a display unit of the remote device 400, and can display various information to a user of the remote device 400. The operation unit 402 is an operation unit for a user to operate the remote device 400.

[0259] The remote device microcomputer 410 is provided with a control unit 411 and a communication unit 412. The control unit 411 is a control unit that controls the remote device 400. The communication unit 412 is a communication unit for communicating with the camera body 200.

[0260] The learning unit 420 can be configured using a processor (CPU, GPU) and a storage device (ROM, RAM, HDD). The learning unit 420 includes a learning processing unit 421, an operation log storage unit 422, a reward management unit 423, a device constraint reward management unit 424, a user request reward management unit 425, a user request management unit 426, a user request storage unit 427, and a device constraint reward information storage unit 428. The learning unit 420 also includes a user request reward conversion information storage unit 429, a reward information storage unit 430, a machine learning model storage unit 431, and a machine learning initial value management unit 432.

[0261] Each component of the learning unit 420 may be realized by a processor such as a CPU or MPU executing a software module stored in a storage device. The processor may be, for example, a GPU or FPGA. Each component of the learning unit 420 may be configured by a circuit that performs a specific function such as an ASIC. The storage device of the learning unit 420 stores programs for implementing the various components of the learning unit 420, and various information and various databases held by each component of the learning unit 420, such as operation log information held by the operation log holding unit 422.

[0262] Here, the operation of the learning unit 420 is the same as the operation of the learning unit 220 according to the first embodiment. The communication unit 240 and the communication unit 412 are connected by wireless communication. The wireless communication may be short-distance wireless communication such as Bluetooth (registered trademark) or Wi-Fi, or public wireless communication such as public wireless LAN. The communication between the communication unit 240 and the communication unit 412 may be wired communication.

[0263] In this embodiment, a learning unit 420 that performs machine learning is configured in the remote device 400. Therefore, information transmission between the camera microcomputer 210 and the learning unit 220 in the first embodiment is performed between the camera microcomputer 210 and the learning unit 420. In addition, video data output from the signal processing circuit 203 can be transmitted to the control unit 411 via the control unit 211, the communication unit 240, and the communication unit 412. The video data transmitted to the control unit 411 can be displayed on the display unit 401.

[0264] Furthermore, the user's desired information, such as the level information of positional accuracy, noise level, and power consumption set by the user via the operation unit 206 or the operation unit 402, is sent to the user's desired reward management unit 425 via the control unit 411. The user's desired reward management unit 425 updates the user's desired information database based on the received user's desired information.

[0265] Furthermore, when the user performs an operation indicating the execution of machine learning from the operation unit 206 or the operation unit 402, a command to execute machine learning is transmitted to the learning unit 420 via the control unit 411. Upon receiving the command to execute machine learning, the learning unit 420 determines reward information, determines weight initial values, and starts machine learning processing, similar to the processing in the learning unit 220 according to the first embodiment.

[0266] In this embodiment, an appropriate initial value of machine learning is determined based on the user desired reward information set by the user in the same manner as in the first embodiment, and the learning of the NN algorithm is performed. The weight of the trained model updated in the machine learning process by the learning unit 420 is sent from the remote device 400 to the lens 100 via the camera body 200, stored in the NN data storage unit 123, and can be used for focus drive control. Through the above operations, the user can generate an NN that can perform appropriate control according to the user settings while checking the captured image at a remote location. In this case, the control unit 211 of the camera body 200 that acquires the weight of the first trained model from the remote device 400 functions as an example of a control unit of a camera device that acquires the first trained model.

[0267] It should be noted that the user can also use the operation unit 206 to perform an operation indicating the implementation of machine learning and set user desired reward information from the camera body 200. In this case, only high speed calculation processing related to learning can be performed by the remote device 400. In this embodiment, a large capacity database can be configured in the remote device 400 separate from the lens 100 and the camera body 200, and the lens 100 and the camera body 200 can be made smaller.

[0268] Even in the configuration of this embodiment in which the learning unit 420 is configured in the remote device 400, by performing the same process as the machine learning process according to the first embodiment, a final trained model can be obtained with fewer learning times compared to the case where learning is started from a fixed machine learning model. As a result, the number of times the driving unit is driven during machine learning can be reduced. Therefore, according to this embodiment, it is possible to provide an imaging system including a lens device, a camera device, and a remote device that are advantageous for obtaining a trained model. In addition, the camera system according to this embodiment can control driving in accordance with the user's request by using reward information according to the user's request for learning.

[0269] In addition, all or a part of the user request management unit 426, the user request holding unit 427, the device constraint reward management unit 424, the device constraint reward information holding unit 428, the user request reward management unit 425, the user request reward conversion information holding unit 429, the reward management unit 423, and the reward information holding unit 430 may be configured in a remote device other than the lens 100, the camera body 200, and the remote device 400. Similarly, all or a part of the operation log holding unit 422, the machine learning model holding unit 431, and the machine learning initial value management unit 432 may be configured in a remote device other than the lens 100, the camera body 200, and the remote device 400. In addition, in this embodiment, the lens 100 communicates with the remote device 400 via the camera body 200, but may directly communicate with the remote device 400. In this case, the NN control unit 121, the NN data storage unit 123, and the control unit 125, which acquire the weights of the first learned model from the remote device 400, can constitute part of the control unit of the lens 100 that acquires the first learned model.

[0270] In the above-mentioned first to third embodiments, the configuration in which the focus lens is controlled using the NN included in the NN control unit 121 in the lens 100 has been described. However, the NN does not have to be provided in the lens device. For example, the NN may be implemented in the camera body 200, the remote device 400, or other remote devices. In this case, the NN control unit 121 or the control unit 125 of the lens 100 may transmit the above-mentioned input to the NN to the NN via the communication unit 126, receive a drive command for the focus lens output from the NN, and drive the focus lens drive unit 105. Also, the same operation may be applied to other controls (zoom control, anti-shake control, aperture control, etc.) as described above.

[0271] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0272] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. The present invention also includes inventions modified within the scope of the present invention and inventions equivalent to the present invention. In addition, the above-mentioned embodiments and modifications can be appropriately combined within the scope of the present invention. [Explanation of symbols]

[0273] 100: lens (lens device), 105: focus lens (optical member), 105: focus lens driving unit (driving unit)

Claims

1. An optical member; A drive unit that drives the optical member; A control unit that controls the drive unit; A lens device having The control unit controls the drive unit so that the optical member is driven at a speed or acceleration based on a first trained model; The first trained model is a trained model obtained by training based on reward information regarding the lens device using a second trained model obtained by training regarding a device different from the lens device as an initial trained model, The lens device, wherein the compensation information includes information regarding power consumption by the drive unit.

2. The lens device according to claim 1 , wherein the control unit acquires the first trained model.

3. The lens device according to claim 2 , wherein the control unit selects the second trained model from a group of trained models based on information about the lens device.

4. The lens device described in claim 3, characterized in that the control unit selects the second learned model based on at least one of information regarding the lens device used to train the learned model, information regarding the initial values ​​of the learned model, information regarding user requests used to train the learned model, information regarding rewards used to train the learned model, and information regarding learning date and time.

5. The lens device described in claim 3 or 4, characterized in that the control unit selects the second trained model from the group of trained models based on information regarding at least one of a reward used in learning to obtain the first trained model and a user's request.

6. 6. The lens device according to claim 3, wherein the information relating to the lens device includes information indicating a model and an individual unit of the lens device.

7. A lens device according to any one of claims 1 to 6; and an image sensor for capturing an image formed by the lens device.

8. A camera device to which the lens device according to claim 1 is detachably attached, A camera device comprising: a control unit that acquires the first learned model.

9. The camera device according to claim 8, characterized in that a control unit of the camera device acquires the second trained model, and generates the first trained model based on reward information in accordance with a user's request, using the second trained model as an initial trained model.

10. The camera device according to claim 9 , wherein a control unit of the camera device selects the second trained model from a group of trained models based on information about the lens device.

11. The camera device described in claim 10, characterized in that the control unit of the camera device selects the second trained model based on at least one of information regarding a lens device used to train the trained model, information regarding initial values ​​of the trained model, information regarding user requests used to train the trained model, information regarding rewards used to train the trained model, and information regarding training date and time.

12. The camera device according to claim 10 or 11, characterized in that a control unit of the camera device selects the second trained model from the group of trained models based on information regarding at least one of a reward used in training to obtain the first trained model and a user's request.

13. 13. The camera device according to claim 10, wherein the information relating to the lens device includes information indicating a model and an individual unit of the lens device.

14. Controlling the driving of an optical member in the lens device; generating a first trained model for use in the control; Controlling the driving of the optical member includes controlling the optical member to be driven at a speed or acceleration based on the first trained model; The first trained model is a trained model obtained by training based on reward information regarding the lens device using a second trained model obtained by training regarding a device different from the lens device as an initial trained model, A control method, characterized in that the compensation information includes information regarding power consumption caused by driving the optical element.

15. A program causing a computer to execute the control method according to claim 14.

Images