Imaging device, accessory device, control method and program

Abstract

To achieve a technique that can provide an optimal learning model for every accessory device connectable to an imaging apparatus.SOLUTION: An imaging apparatus has: connection means that connects an accessory device to the imaging apparatus; image processing means that performs image processing using a learning model; and control means that acquires the learning model used by the image processing means for the image processing from the accessory device connected with the imaging apparatus.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a system in which an accessory device and an imaging device are connected.

Background Art

[0002] In a system in which an accessory device is connected to an imaging device, data for controlling the accessory device and the like are transmitted and received between the imaging device and the accessory device. When the accessory device is an interchangeable lens and an image is taken using the interchangeable lens, the imaging device needs to acquire optical information regarding the optical characteristics of the lens from the interchangeable lens and perform appropriate image processing on the image based on the optical information.

[0003] In such an imaging device to which an accessory device is detachable, when performing image processing by a neural network such as a generative adversarial network (GAN), it is impossible to perform image processing based on the optical information of a variety of lenses that can be connected to the imaging device using a single neural network because the model will become too large.

[0004] Patent Document 1 describes a system that connects to an external device capable of transmitting a plurality of learning models and receives a model selected from a list.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, Patent Document 1 requires connecting to an external device capable of transmitting multiple learning models and selecting the target model from a list, making it impossible to obtain a learning model in environments without network connectivity. Furthermore, since it is necessary to select a model corresponding to the various types of lenses for each lens, it impairs user convenience and prevents the camera from being used immediately after connecting the lens.

[0007] This invention has been made in view of the above problems, and its objective is to realize a technology that can provide an optimal learning model for each accessory device that can be connected to an imaging device. [Means for solving the problem]

[0008] To solve the above problems and achieve the objective, the imaging device of the present invention includes a connecting means for connecting an accessory device, neural network Image processing means that performs image processing using a learning model, A storage means for storing the first portion of the learning model from the input layer to a predetermined intermediate layer, From the accessory device connected to the imaging device, Writing Xi Model Of these, the second portion from the downstream of the predetermined intermediate layer to the output layer A control means for acquiring, and The image processing means performs image processing using the learning model which includes the first part and the second part, the parameters of the first part being fixed regardless of the accessory device, and the parameters of the second part being different depending on the type of accessory device. . [Effects of the Invention]

[0009] According to the present invention, an optimal learning model can be provided for each accessory device that can be connected to the imaging device. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the system configuration and device configuration of Embodiment 1. [Figure 2] A diagram showing the communication interface between the lens device and the camera device of Embodiment 1. [Figure 3] A flowchart illustrating the operation of the lens device and camera device in Embodiment 1. [Figure 4] A flowchart illustrating the operation of the camera device according to Embodiment 2. [Figure 5]A diagram illustrating the training method of the neural network model in Embodiment 2. [Figure 6] A diagram showing information about the neural network model of Embodiment 2. [Figure 7] A diagram illustrating how to store the neural network model of Embodiment 3 using a lens device and a camera device. [Figure 8] A flowchart illustrating the operation of the camera device according to Embodiment 3. [Figure 9] A diagram showing the UI for managing the neural network model of Embodiment 3. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] [Embodiment 1] First, Embodiment 1 will be described.

[0013] Figure 1 illustrates the configuration of a system including the accessory device and imaging device according to Embodiment 1.

[0014] In this embodiment, the accessory device 100 is a detachable interchangeable lens (hereinafter referred to as the lens device) attached to the imaging device 200, and the imaging device 200 is a single-lens reflex digital camera (hereinafter referred to as the camera device) with interchangeable lenses. Note that the imaging device of the present invention is not limited to a single-lens reflex digital camera, but may also be a smartphone or the like. Furthermore, the accessory device of the present invention is not limited to an interchangeable lens, but may also be an optical filter or the like. The lens device 100 and the camera device 200 are mechanically and electrically connected by mounts 300 provided on each, and communicate using asynchronous communication.

[0015] <Lens Device Configuration>Next, referring to FIG. 2, the configuration and functions of the lens device 100 and the camera device 200 that make up the system of this embodiment will be described.

[0016] The lens device 100 receives power from the camera device 200 via a power supply terminal (not shown) provided on the mount 300 and supplies power to each component of the lens device 100 described below. Also, the lens device 100 and the camera device 200 communicate via a communication terminal (not shown) provided on the mount 300.

[0017] The lens device 100 has an optical system for forming an optical image for imaging purposes. The optical system includes, in order from the side closer to the subject 50, a field lens 101, a zoom lens 102, an aperture unit 114, an image shake correction lens 103, and a focus lens 104.

[0018] The zoom lens 102 is held by a lens holding frame 105. The focus lens 104 is held by a lens holding frame 106.

[0019] The lens holding frames 105 and 106 are guided to be movable in the optical axis direction AX by a guide shaft (not shown) and are driven in the optical axis direction AX by stepping motors 107 and 108.

[0020] Drive [[ID=twenty-two]]The stepping motor 107 drives the lens holding frame 105 in synchronization with the drive pulses of the zoom drive circuit 119 to reciprocate the zoom lens 102 in the optical axis direction AX. The stepping motor 108 drives the focus lens 104 in the optical axis direction AX via the lens holding frame 1'06 in synchronization with the drive pulses of the focus drive circuit 120. Drive

[0021] The image shake correction lens 103 is moved in a direction orthogonal to the optical axis direction AX by the anti-shake actuator 126 to reduce image shake caused by hand movement or the like.

[0022] <00001'11>The lens control unit 111 has a microcomputer that includes a processor (e.g., CPU), memory, and interface circuits, which comprehensively control the entire lens device 100. The lens control unit 111 executes a program stored in memory to realize the flowchart processing described later. Alternatively, instead of the lens control unit 111 controlling the entire lens device 100, multiple hardware components may share the processing to control the entire device.

[0023] The lens control unit 111 receives requests to transmit camera data and lens data from the camera device 200 via the lens communication unit 112. The lens control unit 111 controls the lens device 100 according to the camera data and transmits lens data to the camera device 200.

[0024] The lens control unit 111 performs operations related to communication with the camera device 200 (camera control unit 205) according to a communication control program, which is a computer program. The lens control unit 111 also outputs control signals to the zoom drive circuit 119 and the focus drive circuit 120 in response to camera data related to zoom control and focus control received from the camera device 200. The zoom drive circuit 119 and the focus drive circuit 120 then output drive pulses to the stepping motors 107 and 108 to perform zoom processing to zoom the zoom lens 102 and AF (autofocus) processing to focus the focus lens 104.

[0025] The lens device 100 includes a manual focus ring 130 that can be rotated by the user, and a focus encoder 131 that detects the amount of rotation of the manual focus ring 130.

[0026] The lens control unit 111 drives the stepping motor 108 via the focus drive circuit 120 to move the focus lens 104 according to the amount of rotation of the manual focus ring 130 detected by the focus encoder 131. This enables manual focus (MF).

[0027] The aperture unit 114 includes aperture blades 114a and 114b. The operating state of the aperture blades 114a and 114b is detected by a Hall element 115 and input to the lens control unit 111 via an amplification circuit 122 and an AD converter 123.

[0028] The lens control unit 111 outputs a control signal to the aperture drive circuit 121 based on the input signal from the AD converter 123, and the aperture drive circuit 121 outputs a drive pulse to the aperture actuator 113 to perform light intensity adjustment processing to operate the aperture blades 114a and 114b of the aperture unit 114.

[0029] Furthermore, the lens control unit 111 outputs a control signal to the vibration damping drive circuit 125 in response to camera shake detected by a shake sensor (not shown), such as a vibration gyroscope, provided on the lens device 100. The vibration damping drive circuit 125 then outputs a drive pulse to the vibration damping actuator (such as a voice coil motor) 126, performing vibration damping processing to move the image shake correction lens 103.

[0030] The memory unit 151 is a ROM or the like, and stores various programs for the operation of the lens control unit 111, lens-specific information, and neural network models (learning models) such as CNN (Convolutional Neural Network). Note that the memory unit 151 is not limited to ROM, but may also be a memory card or hard disk, for example. A neural network (NN) includes an input layer, an output layer, and hidden layers. Data is input to the input layer, features contained in the data are extracted by the hidden layer, and the final result is calculated by the output layer. A neural network model (NN model) includes the structure and parameters (weight coefficients and biases) of each layer, which perform machine learning or deep learning to obtain the desired result from the input data.

[0031] The camera device 200 includes an image sensor 201, an AD converter 202, an image processing unit 203, a storage unit 204, a camera control unit 205, and a display unit 206.

[0032] The image sensor 201 includes a photoelectric conversion element such as a CCD or CMOS that converts the subject image formed by the optical system of the lens device 100 into an electrical signal (analog signal). The AD converter 202 converts the analog signal from the image sensor 201 into a digital signal.

[0033] The image processing unit 203 performs image processing on the digital signal from the AD converter 202 to generate an image signal. The image processing unit 203 executes image processing using an NN model. The image processing unit 203 is equipped with a GPU (Graphics Processing Unit). A GPU is a processor capable of performing processing specialized for computer graphics calculations and has the processing power to perform matrix operations and other calculations necessary for neural networks in a short amount of time. Note that the image processing unit 203 is not limited to a GPU; it is sufficient to have a circuit configuration that can perform matrix operations and other calculations necessary for neural networks.

[0034] The image signal generated by the image processing unit 203 is output to the display unit 206 and stored in the storage unit 204 at any desired timing.

[0035] The display unit 206 is equipped with a display device such as a liquid crystal or organic EL that displays the image signal as a live view image. The user can check the composition, focus status, etc., while viewing the live view image.

[0036] The storage unit 204 includes a ROM that stores the program executed by the camera control unit 205, operational constants necessary for executing the program, and RAM that stores image signals, lens data and NN models received from the lens device 100, and loads operational constants, variables, and programs read from the ROM for the camera control unit 205. Note that the storage unit 204 is not limited to ROM and RAM, but may also be a memory card or a hard disk.

[0037] The camera control unit 205 has a microcomputer that includes a processor (e.g., CPU), memory, and interface circuits, which comprehensively control the entire camera device 200. The camera control unit 205 executes a program stored in memory to implement the flowchart processing described later. Alternatively, instead of the camera control unit 205 controlling the entire camera device 200, multiple hardware components may share the processing to control the entire device.

[0038] The camera control unit 205 controls the camera device 200 in response to user input from the operation unit 207, which includes a shutter switch (not shown) and various setting switches. The camera control unit 205 also transmits camera data related to the zoom operation of the zoom lens 102 to the lens control unit 111 via the camera communication unit 208 in response to operation input from a zoom lever (not shown).

[0039] Furthermore, the camera control unit 205 transmits camera data related to the light intensity adjustment operation of the aperture unit 114 and the AF operation of the focus lens 104 to the lens control unit 111 via the camera communication unit 208. The camera control unit 205 performs communication processing with the lens control unit 111 according to the program stored in the memory unit 204.

[0040] Furthermore, the camera control unit 205 transmits a request to acquire an NN model to the lens control unit 111 via the camera communication unit 208, and acquires the NN model from the lens device 100. The camera control unit 205 then performs image processing using the NN model acquired by the image processing unit 203 from the lens device 100.

[0041] Next, with reference to Figure 2, the communication circuit and communication processing formed between the camera control unit 205 of the camera device 200 and the lens control unit 111 of the lens device 100 will be described.

[0042] The camera control unit 205 has the function of managing communication with the lens control unit 111 and the function of making requests to the lens control unit 111 to transmit lens data. The lens control unit 111 has the function of generating lens data and the function of transmitting lens data.

[0043] The camera control unit 205 has a camera communication interface circuit 208a, and the lens control unit 111 has a lens communication interface circuit 112a.

[0044] The camera data transmission / reception unit 208b of the camera control unit 205 and the lens data transmission / reception unit 112b of the lens control unit 111 communicate with each other via the communication terminal 300a provided on the mount 300, the camera communication interface circuit 208a, and the lens communication interface circuit 112a.

[0045] In this embodiment, the camera control unit 205 and the lens control unit 111 perform serial communication using a three-wire asynchronous serial communication method with three channels.

[0046] The camera communication unit 208 is comprised of the camera communication unit 208 and the camera communication interface circuit 208a, and the lens communication unit 112, which functions as an accessory communication unit, is comprised of the lens communication unit 112 and the lens communication interface circuit 112a.

[0047] Of the three channels mentioned above, the first channel is the send request channel, which serves as a notification channel.

[0048] The second channel is the first data communication channel used for transmitting lens data from the lens control unit 111 to the camera control unit 205.

[0049] The third channel is a second data communication channel used for transmitting camera data from the camera control unit 205 to the lens control unit 111.

[0050] The lens data (accessory data) transmitted from the lens control unit 111 to the camera control unit 205 via the first data communication channel is called the lens data signal DLC. The camera data transmitted from the camera control unit 205 to the lens control unit 111 via the second data communication channel is called the camera data signal DCL.

[0051] Next, referring to Figure 3, the communication process between the camera device 200 (camera control unit 205) and the lens device 100 (lens control unit 111) will be described.

[0052] Steps S301 to S306 represent processing by the camera control unit 205 of the camera device 200, steps S321 to S324 represent processing by the lens control unit 111 of the lens device 100, and steps S331 to S334 represent communication processing between the camera device 200 and the lens device 100.

[0053] First, let's explain the processing performed by the camera device 200.

[0054] In step S301, the camera control unit 205 determines whether or not the lens device 100 is connected to the mount 300. The camera control unit 205 can determine, for example, whether the lens device 100 is connected to the mount 300 by a change in the signal voltage of the communication terminal of the mount 300. If the camera control unit 205 determines that the lens device 100 is connected to the mount 300, it proceeds to step S302. The camera control unit 205 repeatedly performs the process in step S301 until it determines that the lens device 100 is connected to the mount 300.

[0055] In step S302, the camera control unit 205 transmits a lens information request 331 to the lens device 100 to request lens type information. Lens type information is information that identifies the type of lens provided by the lens device 100, such as lens model information. Based on the lens type information, the camera device 200 can obtain the zoom range and F-number of the lens device 100 connected to the camera device 200. The camera control unit 205 transmits a camera data signal DCL including the lens information request 331.

[0056] In step S303, the camera control unit 205 determines whether or not it has received lens information 332 from the lens device 100. The camera control unit 205 can determine that it has received lens information 332 by receiving the lens data signal DLC from the lens device 100. If the camera control unit 205 determines that it has received lens information 332 from the lens device 100, it proceeds to step S304. The camera control unit 205 repeatedly performs the process in step S303 until it determines that it has received lens information 332 from the lens device 100.

[0057] In step S304, the camera control unit 205 receives lens information from the lens device 100. 332 Based on this, the camera control unit 205 obtains information about the NN model held by the lens device 100, which is connected to the camera device 200. The information about the NN model includes information about the number of layers and the time required for processing. If the camera control unit 205 determines that calculation processing is possible in the image processing unit 203 based on the information about the NN model, it sends a model information request 333 to the lens device 100 to request the NN model.

[0058] In step S305, the camera control unit 205 determines whether or not it has received model information 334 from the lens device 100. The camera control unit 205 can determine that it has received model information 334 from the lens device 100 by receiving the lens data signal DLC from the lens device 100. If the amount of data for model information 334 is large, the camera control unit 205 can shorten the data transfer time by switching the direction of the camera data signal DCL and transmitting the model information 334 from the lens device 100 to the camera device 200 on two channels. If the camera control unit 205 determines that it has received model information 334 from the lens device 100, it proceeds to step S306. The camera control unit 205 repeats the process in step S305 until it determines that it has received model information 334 from the lens device 100.

[0059] In step S306, the camera control unit 205 performs image processing using the NN model included in the model information 334 received from the lens device 100 in the image processing unit 203, and then terminates the processing.

[0060] Next, the processing of the lens device 100 will be described.

[0061] In step S321, the lens control unit 111 determines whether or not it has received a lens information request 331 from the camera device 200. The lens control unit 111 can determine that it has received a lens information request 331 from the camera device 200 by receiving a camera data signal DCL from the camera device 200. If the lens control unit 111 determines that it has received a lens information request 331 from the camera device 200, it proceeds to step S322. The lens control unit 111 repeatedly performs the process in step S321 until it determines that it has received a lens information request 331 from the camera device 200.

[0062] In step S322, the lens control unit 111 transmits the lens data signal DLC as lens information 332 to the camera device 200.

[0063] In step S323, the lens control unit 111 determines whether or not it has received a model information request 333 from the camera device 200. The lens control unit 111 can determine that it has received a model information request 333 from the camera device 200 by receiving a camera data signal DCL from the camera device 200. If the lens control unit 111 determines that it has received a model information request 333 from the camera device 200, it proceeds to step S324. The lens control unit 111 repeatedly performs the process in step S323 until it determines that it has received a model information request 333 from the camera device 200.

[0064] In step S324, the lens control unit 111 transmits a lens data signal DLC containing model information 334 to the camera device 200 and terminates the process.

[0065] As described above, according to Embodiment 1, the lens device 100 holds the NN model, and the camera device 200 acquires the NN model from the lens device 100 when it is connected to the lens device 100.

[0066] By configuring it in this way, even in environments where external devices cannot be connected to a network, the optimal NN model can be obtained from the lens device 100 connected to the camera device 200.

[0067] Furthermore, each time a lens unit 100 is connected to the camera unit 200, it becomes unnecessary to select a model for each lens that matches the characteristics of various lenses, thus improving user convenience.

[0068] [Embodiment 2] Next, Embodiment 2 will be described.

[0069] In Embodiment 2, we will describe a case in which image processing is performed using an appropriate NN model according to the performance of the lens device 100 and the camera device 200, respectively.

[0070] In this embodiment, we describe an example in which the lens device 100 has multiple NN models. However, the lens device 100 does not always need to have multiple NN models; it only needs to have one NN model. Furthermore, we will also describe how to handle the case where the lens device 100 does not have an NN model.

[0071] Figure 6 illustrates information regarding NN models when the lens device 100 has multiple NN models. As will be described later in Figure 4, the camera device 200 acquires information about the NN models (model information) before acquiring the NN models from the lens device 100. In this embodiment, we will describe the case where there are three NN models: Model 1, Model 2, and Model 3.

[0072] The model information includes information about Model 1 (611), information about Model 2 (612), and information about Model 3 (613). Furthermore, the information about Models 1, 2, and 3 (611, 612, and 613) includes the number of layers in the NN model (601), the number of matrix operations required for the NN model (602), and the image processing performance of the NN model (603), respectively.

[0073] In this embodiment, for the sake of simplicity, the relative image quality, which is the result of image processing by Model 1, Model 2, and Model 3, is referred to as the image processing performance 603. However, in reality, it represents the characteristics when correcting an image captured using the lens device 100.

[0074] In this embodiment, for the sake of clarity, the relative image processing performance of Model 1, Model 2, and Model 3 is shown, but in reality, it represents the image correction processing performance based on the optical characteristics of the lens device 100.

[0075] Model 1 has the fewest layers (601) and the fewest matrix operations (602), resulting in the lowest image processing performance (603). Model 2 has an intermediate number of matrix operations (602) between Model 1 and Model 3, and its image processing performance (603) is also intermediate. Model 3 has the most matrix operations (602), but its image processing performance (603) is also the highest.

[0076] Next, the operation of the camera device 200 in this embodiment will be described with reference to the flowchart in Figure 4.

[0077] In step S401, the camera control unit 205 acquires type information of the lens device 100. From the type information of the lens device 100, the camera control unit 205 acquires information about the NN model stored in the lens device 100 connected to the camera device 200. The information about the NN model is the model information shown in Figure 6.

[0078] In step S402, the camera control unit 205 determines whether the lens device 100 connected to the camera device 200 has an NN model. The camera control unit 205 determines whether an NN model that enables the processing of correcting an image captured using the lens device 100 connected to the camera device 200 based on the optical characteristics of the lens device 100 (hereinafter referred to as optical correction of the lens device 100) is stored in the storage unit 151 of the lens device 100.

[0079] In this embodiment, it is determined whether or not an NN model is stored in the memory unit 151 of the lens device 100, but it is not limited to this, and for example, it may be determined whether or not an NN model exists in an external device. In this case, the external device must be connected to the system via a network so as to be able to communicate.

[0080] If the camera control unit 205 determines that the lens device 100 has an NN model, it proceeds to step S403. If the camera control unit 205 determines that the lens device 100 does not have an NN model, it proceeds to step S409.

[0081] In step S403, the camera control unit 205 determines whether the image processing unit 203 is capable of image processing using the NN model. Here, the camera control unit 205 determines whether the image processing unit 203 can complete the NN model calculation within a predetermined time.

[0082] In this embodiment, the image processing unit 203 of the camera device 200 determines whether or not image processing using an NN model is possible. However, the invention is not limited to this, and for example, the determination of whether or not image processing using an NN model is possible may be made in an external device that is connected to the device via a network for communication.

[0083] If the camera control unit 205 determines that the image processing unit 203 is capable of image processing using the NN model, it proceeds to step S404. If the camera control unit 205 determines that the image processing unit 203 is not capable of image processing using the NN model, it proceeds to step S409.

[0084] In step S409, the camera control unit 205 acquires optical information regarding the optical characteristics of the lens from the lens device 100. The camera control unit 205 sends an optical information request to the lens device 100 using the camera data signal DCL, and acquires the optical information of the lens using the lens data signal DLC received from the lens device 100.

[0085] In step S410, the camera control unit 205 terminates the process after the image processing unit 203 performs a rulebook-based, normal image processing that does not use a neural network.

[0086] In step S404, the camera control unit 205 determines whether the image processing unit 203 has the capability to perform image processing using an NN model. The camera control unit 205 determines the image processing performance of the NN model that the image processing unit 203 can execute. Since the camera control unit 205 obtained information in step S401 regarding the number of layers in the NN model of the lens device 100 and the time required for matrix calculations, it can calculate the processing time when performing image processing using each NN model based on this information and the computational processing capability of the image processing unit 203.

[0087] If the image processing unit 203 determines that image processing A3 shown in Figure 6 can be completed within a predetermined time, the camera control unit 205 proceeds to step S405; if it determines that image processing A3 is not possible but image processing A2 can be completed within a predetermined time, the process proceeds to step S406; and if it determines that image processing A2 cannot be completed within a predetermined time, the process proceeds to step S407. In this embodiment, the camera device capable of image processing using an NN model is assumed to have at least the performance to complete the image processing of Model 1 within a predetermined time. Whether or not the image processing of Model 1 can be completed within a predetermined time is determined in step S403.

[0088] Also, NThe predetermined time for determining whether or not image processing by the N model is feasible can be arbitrarily set by the user, and for example, when recording video, it corresponds to the frame rate.

[0089] In step S405, the camera control unit 205 acquires the NN model 3 from the lens device 100.

[0090] In step S406, the camera control unit 205 acquires the NN model 2 from the lens device 100.

[0091] In step S407, the camera control unit 205 acquires the NN model 1 from the lens device 100.

[0092] In steps S405 to S407, the camera device 200 sends an NN model request to the lens device 100 using the camera data signal DCL, and acquires the NN model using the lens data signal DLC received from the lens device 100.

[0093] In step S408, the camera control unit 205 executes image processing using the NN model acquired by the image processing unit 203 in steps S405 to S407, and then terminates the processing.

[0094] Next, referring to Figure 5, the learning process of the multiple NN models held by the lens device 100 will be described.

[0095] Figure 5(a) shows the learning process corresponding to Model 1 in Figure 6, and Figure 5(b) shows the learning process corresponding to Model 2 in Figure 6. In this embodiment, the explanation of Model 3 is omitted.

[0096] As shown in Figure 5, in the training process of the NN model (the process of generating a trained model), the training process is performed using the same layers up to a predetermined intermediate layer (in the example in Figure 5, the input layer (501), intermediate layer 1 (502), and intermediate layer 2 (503)) for both NN model 1 and NN model 2.

[0097] For intermediate layers other than the common layer, the number of nodes (number of operations per layer) for each intermediate layer may be set arbitrarily.

[0098] In this embodiment, the training process for the NN model is assumed to be performed in advance by a PC (personal computer) or the like with advanced computing power.

[0099] In this embodiment, the common layers are described up to intermediate layer 2 (503), but this is not limited to this, and the number of intermediate layers to which the common layers are set can be arbitrarily set within a range that does not affect the computational processing accuracy of multiple NN models.

[0100] First, the input layer (501), hidden layer 1 (502), hidden layer 2 (503), and output layer (511) of Model 2 in Figure 5(b) are trained to set optimal parameters. These parameters can be changed arbitrarily. After the training process in Figure 5(b) is completed, the training process for Model 1 in Figure 5(a) is executed. In the training process for Model 1 in Figure 5(a), the parameters of the input layer (501), hidden layer 1 (502), and hidden layer 2 (503) are not changed. In Model 2 in Figure 5(b), the parameters of hidden layer 3 (521), hidden layer 4 (522), and output layer (523) are changed to the optimal ones as needed through the training process.

[0101] By performing the learning process for the NN model in this way, some parameters of multiple NN models can be made common, and the capacity required for the memory unit 151 of the lens device 100 can be minimized.

[0102] As described above, according to Embodiment 2, even lens devices that do not have an NN model and camera devices that cannot perform image processing using an NN model can perform optimal image processing in their respective cases.

[0103] Furthermore, since the lens device 100 has multiple NN models, it is possible to perform image processing using the optimal NN model according to the image processing performance of the image processing unit 203 of the camera device 200.

[0104] Furthermore, if the lens device 100 has multiple NN models, the required capacity of the storage unit 151 of the lens device 100 can be minimized by making some of the parameters of the multiple NN models common.

[0105] Furthermore, when transmitting multiple NN models from the lens device 100 to the camera device 200, it is possible to reduce the communication time.

[0106] [Embodiment 3] Next, Embodiment 3 will be described.

[0107] Embodiment 3 describes a process for reducing the required capacity of the storage unit 151 of the lens device 100 compared to Embodiments 1 and 2.

[0108] Figure 7 illustrates the memory state of the NN model using the camera and lens devices of this embodiment.

[0109] The NN model is assumed to consist of an input layer, N+1 hidden layers, and an output layer. Here, we define the input layer as (701), hidden layer 1 as (702), hidden layer 2 as (703), hidden layer 3 as (704), hidden layer N as (705), hidden layer N+1 as (706), and the output layer as (707).

[0110] The memory unit 204 of the camera device 200 stores the first part of the NN model, which consists of the input layer (701), intermediate layer 1 (702), intermediate layer 2 (703), intermediate layer 3 (704), intermediate layers 4 through N-1, and the output layer (707).

[0111] The memory unit 151 of the lens device 100 stores the second part of the NN model, which consists of the intermediate layer N(705) and the intermediate layer N+1(706).

[0112] The first part, from the input layer (701) to the intermediate layer N-1, has fixed parameters that do not depend on the optical information of the lens device 100, and common parameters are set for any lens.

[0113] The parameters of the second part, intermediate layer N(705) and intermediate layer N+1(706), change depending on the optical information of the lens device 100, and different parameters are set according to the type of lens of the lens device 100.

[0114] In this embodiment, intermediate layer N(705) and intermediate layer N+1(706) are described as layers whose parameters change depending on the optical information of the lens device 100. However, the embodiment is not limited to this, and appropriate parameters can be set for any layer that performs optical correction of the lens device 100.

[0115] Alternatively, as in Embodiment 2, multiple NN models for optical correction of the lens device 100 may be prepared, and the NN model may be changed according to the image processing performance of the image processing unit 203 of the camera device 200. In this case as well, since only a part of the NN model (only the layer that performs optical correction) needs to be stored in the memory unit 151 of the lens device 100, it can be implemented with a smaller memory capacity than in Embodiment 2.

[0116] In this embodiment, we describe an NN model in which the output layer (707) follows the intermediate layer N+1 (706). However, depending on the type of image processing performed by the camera device 200, several intermediate layers may be added between the intermediate layer N+1 (706) and the output layer (707).

[0117] For example, the NN model stored in the memory unit 204 of the camera device 200 can be arbitrarily changed depending on the image processing content, such as when you want to perform image output or super-resolution processing. In this embodiment, only the model with an appropriate number of layers and parameters as the layer for performing optical correction of the lens device 100 is stored in the memory unit 151 of the lens device 100, and the other parameters are stored in the memory unit 204 of the camera device 200.

[0118] The method by which the camera device 200 obtains the NN model of intermediate layer N(705) and intermediate layer N+1(706) from the lens device 100 is as described in Embodiments 1 and 2.

[0119] Next, the training process of the NN model in Embodiment 3 will be described. In this embodiment, the training process is assumed to have been performed in advance by a PC or the like with high computing power.

[0120] First, a learning process is performed by connecting any one lens device 100. In this case, the parameters can be freely changed.

[0121] After completing the learning process, the parameters from the input layer (701), hidden layer 1 (702), hidden layer 2 (703), and hidden layer 3 (704) up to hidden layer N-1 are stored in the memory unit 204 of the camera device 200. Additionally, hidden layer N (705) and hidden layer N+1 (706) are stored in the memory unit 151 of any lens device 100 to which they are connected.

[0122] Subsequently, a different lens device 100 is connected and the learning process is performed. In this case, the parameters of the input layer (701), hidden layer 1 (702), hidden layer 2 (703), and hidden layer 3 (704) up to hidden layer N-1 are not changed. In the different lens device 100, the parameters that are changed are limited to hidden layer N (705) and hidden layer N+1 (706). In this way, the learning process is performed by changing only the parameters of hidden layer N (705) and hidden layer N+1 (706) for each different lens device, and the learning results are stored in each lens device.

[0123] As described above, the number of intermediate layers to which the common layer that fixes parameters regardless of the lens device is set can be arbitrarily determined as long as it does not affect the computational processing accuracy of the NN model. Furthermore, although this embodiment describes two layers, intermediate layer N(705) and intermediate layer N+1(706), regardless of the type of lens device, it is not limited to this, and the number of layers can be freely changed for each type of lens device 100.

[0124] In the learning process of this embodiment, parameters that differ for each lens device are set in the layer after the intermediate layer. This is suitable for transfer learning of the NN model, as the parameters in the layer before the intermediate layer are fixed and the learning process is performed, so parameters that differ for each lens device are set in the layer after the intermediate layer. Furthermore, if it is desired to perform multiple types of image processing with the camera device 200, a layer corresponding to the type of image processing may be added further after the intermediate layer N+1 (706). Also, if the image processing is different, the parameters from the input layer (701), intermediate layer 1 (702), intermediate layer 2 (703), intermediate layer 3 (704) to intermediate layer N-1 may be stored in the memory unit 204 of the camera device 200 and switched according to the mode of the camera device 200. Thus, if the NN model itself is changed when the image processing is different, it is necessary to prepare several intermediate layers N (705) and intermediate layers N+1 (706) with different parameters for each lens device 100.

[0125] By storing the NN model acquired from the lens device 100 in the memory unit 204 of the camera device 200, it becomes unnecessary to transmit the model each time the lens device 100 is connected, thus shortening the time from the second connection of the lens device to image processing.

[0126] Next, with reference to Figure 9, the UI for managing the NN model stored in the memory unit 204 of the camera device 200 will be described.

[0127] The menu screen 901 displays a description of the screen's functions, such as the selection of the NN model. The acquired list 902 displays the NN models stored in the memory unit 204 of the camera device 200. The acquired list 902 displays information indicating the type of lens, information indicating the NN model, information indicating the required capacity of the memory unit 204 of the camera device 200, and the priority for erasing information when an unregistered lens device is connected. Note that the numbers displayed in the acquired list 902 may not represent priority but rather numbers that identify the location where the subject is stored. Also, even if the lens types are different, the information indicating the NN model may be the same.

[0128] If an unregistered lens device is connected to the camera device, the priority selection box 903 for information erasure will display the priority selected by the user. If no priority is selected by the user, less frequently used models may be deleted first, or, although not shown in this embodiment, the date and time when the lens device 100 was connected may be managed and the oldest ones deleted first. Also, since lens A has multiple NN models, its deletion priority may be increased.

[0129] Next, referring to Figure 8, the initial operation of the camera device 200 will be described in two cases: one where the NN model of the lens device 100 has been acquired and stored in the memory unit 204, and another where it has not.

[0130] In step S801, the camera control unit 205 acquires type information of the lens device 100. From the type information of the lens device, the camera control unit 205 acquires information about the NN model stored in the lens device 100 connected to the camera device 200. The information about the NN model includes the number of layers and the time required for processing.

[0131] In step S802, the camera control unit 205 determines whether the NN model of the lens device 100 connected to the camera device 200 is already stored in the storage unit 204. If the camera control unit 205 determines that the NN model of the lens device 100 connected to the camera device 200 is already stored in the storage unit 204, it proceeds to step S803. If the camera control unit 205 determines that the NN model of the lens device 100 connected to the camera device 200 is not already stored in the storage unit 204, it proceeds to step S804.

[0132] In step S803, the camera control unit 205 does not acquire the NN model from the lens device 100 connected to the camera device 200, but instead reads the model stored in the memory unit 204 of the camera device 200 and performs image processing using the NN model. This configuration makes it possible to shorten the time from connecting the lens device to image processing for subsequent connections.

[0133] In step S804, the camera control unit 205 sends a request to the lens device 100 connected to the camera device 200 to acquire the NN model. The subsequent processing is as described in Figure 3. In step S804, until the parameters of intermediate layer N(705) and intermediate layer N+1(706) are acquired from the lens device 100, image processing may be performed using the input layer (701), intermediate layer 1 (702), intermediate layer 2 (703), intermediate layer 3 (704), intermediate layer N-1, and output layer (707). With this configuration, it is possible to shorten the time until image processing even when the lens device is connected for the first time, but the image processing performance will decrease because the parameters of intermediate layer N(705) and intermediate layer N+1(706) have not been acquired. After acquiring the parameters of intermediate layer N(705) and intermediate layer N+1(706) from the lens device 100, the acquired NN model is added and image processing is performed, and the processing is terminated.

[0134] As described above, according to Embodiment 3, by sharing the storage of the NN model between the camera device 200 and the lens device 100, the required capacity of the storage unit 151 of the lens device 100 can be minimized.

[0135] Furthermore, it is possible to reduce the time required to transmit the NN model from the lens device 100 to the camera device 200.

[0136] Furthermore, by having the lens device 100 hold a portion of multiple NN models, it is possible to perform optimal neural network image processing according to the computational processing power of the image processing unit 203 of the camera device 200.

[0137] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more functions of each embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0138] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0139] 100...Lens device, 111...Lens control unit, 112...Communication unit, 151...Storage unit, 200...Camera device, 203...Image processing unit, 205...Camera control unit, 208...Communication unit

Claims

1. A connection means for connecting accessory devices, Image processing means that performs image processing using a neural network learning model, A storage means for storing the first portion of the learning model from the input layer to a predetermined intermediate layer, The system includes a control means for acquiring a second portion of the learning model, from the stage after the predetermined intermediate layer to the output layer, from an accessory device connected to the imaging device. The image processing means performs image processing using the learning model which includes the first part and the second part. An imaging device characterized in that the parameters of the first part are fixed regardless of the accessory device, and the parameters of the second part differ depending on the type of accessory device.

2. The control means selects one of the plurality of second parts of the accessory device according to the computational processing capability of the image processing means, The imaging device according to claim 1, characterized in that it requests the accessory device to transmit the selected second portion.

3. The computational processing capability of the image processing means includes the time required for image processing of each of the second parts. The imaging apparatus according to claim 2, characterized in that the time required for the image processing is determined by whether or not it is completed within a predetermined time.

4. The control means includes a setting means for setting the predetermined time in accordance with user operation. The imaging device according to claim 3, characterized in that, according to a predetermined time set by the setting means, one of the plurality of second parts is selected and the accessory device is requested to transmit the selected second part.

5. The aforementioned accessory device is a lens device. The imaging apparatus according to any one of claims 1 to 4, characterized in that the second part is a layer of a neural network for correcting an image captured using the lens device.

6. The aforementioned accessory device is a lens device. The imaging apparatus according to any one of claims 1 to 4, characterized in that the second part is a layer of a neural network for correcting an image based on the optical characteristics of the lens device.

7. The control means selects one of the plurality of second parts of the accessory device according to the computational processing capability of the image processing means, The imaging device according to claim 6, characterized in that it requires the selected second portion to the accessory device.

8. An accessory device connectable to an imaging device having a first portion from the input layer to a predetermined intermediate layer of a learning model of a neural network that performs image processing, A storage means for storing a second portion of the learning model, from the stage after the predetermined intermediate layer to the output layer, The system includes a control means that transmits the second portion stored in the storage means to the imaging device in response to a request from the imaging device, An accessory device characterized in that the parameters of the first part are fixed regardless of the accessory device, and the parameters of the second part differ depending on the type of accessory device.

9. The storage means stores a plurality of the second parts, The accessory device according to claim 8, characterized in that the control means transmits one of the plurality of second parts to the imaging device in response to a request from the imaging device.

10. The aforementioned accessory device is a lens device. The learning model is a layer of a neural network for correcting images based on the optical characteristics of the lens device. The storage means stores optical information relating to the optical characteristics of the lens device. The accessory device according to claim 8 or 9, characterized in that the control means transmits the optical information to the imaging device in response to a request from the imaging device.

11. The accessory device according to claim 10, characterized in that, in the learning process of the neural network model, the first portion is fixed regardless of the lens device.

12. The aforementioned accessory device is a lens device. The accessory device according to claim 8 or 9, characterized in that the learning model is a layer of a neural network for correcting an image based on the optical characteristics of the lens device.

13. A control method for an imaging device comprising: connection means for connecting an accessory device; image processing means for performing image processing using a neural network learning model; and storage means for storing a first portion of the learning model from the input layer to a predetermined intermediate layer, A step of acquiring a second portion of the learning model from the downstream of the predetermined intermediate layer to the output layer from an accessory device connected to the imaging device, The steps include performing image processing using the learning model which includes the first part and the second part, It has, A control method characterized in that the parameters of the first part are fixed regardless of the accessory device, and the parameters of the second part differ depending on the type of accessory device.

14. A method for controlling an accessory device that can be connected to an imaging device having a first portion from the input layer to a predetermined intermediate layer of a learning model of a neural network that performs image processing, The steps include receiving a request from the imaging device for a second portion of the learning model, from the stage after the predetermined intermediate layer to the output layer, The process includes the step of transmitting to the imaging device, in response to a request from the imaging device, a second portion of the learning model stored in the storage means, from the stage after the predetermined intermediate layer to the output layer, to the imaging device. A control method characterized in that the parameters of the first part are fixed regardless of the accessory device, and the parameters of the second part differ depending on the type of accessory device.

15. A program for causing a computer to function as each of the means of an imaging apparatus described in any one of claims 1 to 7, excluding the connection means and the storage means.

16. A program for causing a computer to function as each of the means of an accessory device described in any one of claims 8 to 12, excluding the storage means.

Images