Image capture device, image capture device control method, image processing device, and image capture system

Abstract

To provide an imaging device that improves the recognition speed and accuracy of an object while suppressing the circuit scale of an object detection circuit in the imaging device.SOLUTION: An imaging device includes imaging means, first acquisition means for acquiring a first detection result as a result of detecting a subject from image data captured by the imaging means, second acquisition means for transmitting the image data to an external image processing device and acquiring a second detection result as a result of detection of the subject by the image processing device from the image data, and control means for executing imaging processing using imaging parameter settings determined based on at least one of the first detection result and the second detection result. The control means executes imaging processing using the imaging parameter settings determined based on the first detection result until the second detection result is obtained from the image processing device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, an image processing device, and an imaging system. [Background technology]

[0002] Image processing methods that automatically detect specific subjects from an image are useful technologies used in imaging devices. For example, imaging devices detect specific areas of a subject, such as a person's face, from a captured image and optimize focus and exposure for the detected area. In recent years, imaging devices equipped with a function for detecting subjects using machine learning, such as neural networks, have become known.

[0003] Patent Document 1 discloses an imaging device that detects subjects such as people and animals by preparing a trained model (dictionary) for each preset category. When attempting to detect a variety of subjects, a trained model must be prepared for each subject to be detected, which increases the amount of storage space used. In contrast, Patent Document 2 proposes an imaging device that can detect a variety of subjects by appropriately switching trained models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 080037 [Patent Document 2] Patent Publication No. 2021-093568 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the storage capacity used is reduced and various subjects can be detected, the performance of subject detection is determined by the circuit size of the subject detection circuit in the imaging device. Therefore, increasing the recognition speed and accuracy can be problematic in terms of cost.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an imaging device that improves the speed and accuracy of object recognition while suppressing the circuit scale of the object detection circuit within the imaging device. [Means for solving the problem]

[0007] The present invention provides An imaging means; a first acquisition means for acquiring a first detection result that is a result of detecting a subject from image data captured by the imaging means; a second acquisition means for transmitting the image data to an external image processing device and acquiring a second detection result, which is a result of the image processing device detecting a subject, from the image data; a control means for executing an image capturing process using image capturing settings determined based on at least one of the first detection result and the second detection result; and the second detection result includes information about the subject detected by the image processing device, The control means performing an imaging process using the imaging parameter settings determined based on the first detection result until the second detection result is obtained from the image processing device; death, When the second detection result is acquired from the image processing device, an image capturing process is performed using the setting of the image capturing parameters determined based on the information of the subject included in the second detection result. do The imaging device is characterized by the above. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the circuit scale of the object detection circuit in the imaging device while increasing the object recognition speed. It is possible to provide an imaging device that improves the image quality and recognition accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an imaging system. [Figure 2] This is a diagram explaining the processing flow using a neural network. [Figure 3]FIG. 10 is a diagram illustrating features of subject detection in a camera and a cloud server. [Figure 4] 10 is a flowchart illustrating a live view shooting process. [Figure 5] FIG. 10 is a diagram illustrating an example in which the same type of subject is detected on the cloud server. [Figure 6] FIG. 10 is a diagram illustrating an example in which a different subject is detected by the cloud server. [Figure 7] 10 is a flowchart illustrating a still image capturing process. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment> Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] 1 is a diagram illustrating the configuration of an imaging system 10 according to an embodiment. The imaging system 10 includes a camera 100 as an imaging device and a cloud server 200 as an image processing device capable of communicating with the imaging device. In this embodiment, the imaging device will be described as a camera 100 (digital camera), but it may be any device capable of capturing images, such as a personal computer, smartphone, tablet terminal, game console, or robot. In addition, the image processing device external to the imaging device will be described as a cloud server 200, but it may be any device capable of image processing, such as detecting a subject from image data, such as a mobile terminal such as a smartphone, or a personal computer.

[0012] First, we will explain the configuration of camera 100. Camera 100 has a CPU 113 that controls the processing of each component. Lens 101 is detachable from camera 100 and adjusts the focal distance (focus) and the amount of light incident on image sensor 103.

[0013] The shutter 102 adjusts the exposure time of the image sensor 103. The shutter 102 has a release switch SW1 and a release switch SW2. The release switch SW1 is turned on when the shutter 102 is pressed halfway (a shooting preparation command) during operation of the shutter 102. The release switch SW2 is turned on when the shutter 102 is pressed fully (a shooting command) after operation of the shutter 102 is completed.

[0014] Pixels each equipped with a photoelectric conversion unit are arranged two-dimensionally in the image sensor 103. The image sensor 103 transmits image signals obtained by photoelectrically converting an optical image of a subject formed by the lens 101 at each pixel to an image processing unit 105. The image signals photoelectrically converted by the image sensor 103 are called RAW data.

[0015] The image processing unit 105 generates image data by developing the received RAW data. The development processing includes white balance processing, matrix conversion processing, gamma processing, color correction processing using a LUT (Look-Up-Table), gradation correction processing, etc. The image processing unit 105 also uses the RAW data to perform AF processing (autofocus processing) and AE processing (autoexposure processing).

[0016] The subject detection unit 106 detects a subject from the image data generated by the image processing unit 105. The subject detection unit 106 detects subjects such as people and animals (dogs, cats, etc.), and transmits the subject detection results to the CPU 113. The subject detection results include the type of subject, such as a person or an animal, and the position of the subject within the image data. If the subject detection unit 106 does not detect a specific subject that has been determined in advance, the CPU 113 The subject is determined based on information from the processing unit 105, for example, color information and histogram information of the image data.

[0017] The operation unit 108 is a group of input devices that can be operated by the user. The operation unit 108 includes, for example, input devices such as a release button, a power switch, direction keys, a decision button, a menu button, an operation mode selection dial, a touch panel, etc. When the CPU 113 detects an operation by the user using the operation unit 108, it executes various processes according to the detected operation.

[0018] The communication unit 109 transmits the image data generated by the image processing unit 105 to the cloud server 200 via the communication unit 203. The communication unit 109 can communicate with the communication unit 203 by, for example, Wi-Fi, which is a wireless LAN standard defined by IEEE802.11. Note that the communication method is not limited to Wi-Fi, and may include Bluetooth (registered trademark), an infrared communication method, or a wired USB communication method.

[0019] The ROM 110 stores a program for controlling the camera 100. When the power switch of the operation unit 108 is turned on, the CPU 113 loads the program stored in the ROM 110 into the RAM 111 and starts the operation of the camera 100.

[0020] The RAM 111 is a memory area that temporarily stores the RAW data transmitted from the image sensor 103, the image data generated by the image processing unit 105, and the programs used to operate the camera 100. The recording medium 112 is a non-volatile memory for recording the RAW data transmitted from the image sensor 103 and the image data generated by the image processing unit 105.

[0021] The CPU 113 determines the settings of the shooting parameters based on the subject detection results by the subject detection unit 106. The shooting parameters are parameters for processes such as AF processing, AE processing, white balance processing, brightness correction processing, color correction processing, and gradation correction processing. The CPU 113 transmits setting information for the determined shooting parameters to the image processing unit 105. The image processing unit 105 can use the received setting information for the shooting parameters to perform shooting processing on image data of subsequent frames.

[0022] The photographing process includes a process for acquiring RAW data and a development process. Specifically, the photographing process includes AF processing, AE processing, and development processes such as white balance processing, brightness correction processing, color correction processing, and gradation correction processing. CPU 113 can execute appropriate photographing processes by using the detection results of cloud server 200. However, some processes, such as AF processing and AE processing that require high processing speed, may be executed based on the detection results of camera 100.

[0023] The CPU 113 controls the processing of each component included in the camera 100 to realize still image shooting, live view, and video shooting. The CPU 113 also controls the AF processing and AE processing by the image processing unit 105 in accordance with the subject detection result by the subject detection unit 106.

[0024] The monitor 104 is a device for displaying a live view image, a menu screen, etc. to the user, and is, for example, a liquid crystal display (LCD). The monitor 104 is also touch operable, and the user can change various settings and perform various operations on the camera 100 by touching a predetermined position.

[0025] Next, the configuration of the cloud server 200 will be described. The subject detection unit 202 detects a subject appearing in the image data transmitted from the camera 100, and transmits the detected subject via the communication unit 203. The subject detection result is then transmitted to the camera 100.

[0026] The parameter determination unit 204 determines the settings of the shooting parameters used in the image processing unit 105 based on the subject detection result by the subject detection unit 202. The CPU 201 transmits, via the communication unit 203, to the camera 100, information on the subject detected by the subject detection unit 202 and setting information on the shooting parameters determined by the parameter determination unit 204 as detection results.

[0027] The subject tracking unit 107 identifies the position of the subject and tracks the subject based on the result of subject detection by the subject detection unit 106 or the subject detection unit 202. The subject tracking unit 107 can track the position of the subject using, for example, color information or luminance information of the image data generated by the image processing unit 105.

[0028] Referring to Figure 2, object recognition by the object detection unit 106 and the object detection unit 202 will be described. A technique called deep learning is known as a method for learning and recognizing objects in an image. A convolutional neural network (hereinafter referred to as CNN) is a representative deep learning technique.

[0029] A typical CNN is realized through multi-stage operations. In the convolutional layer of a CNN, local features of an image are spatially integrated through convolution operations and input to the neurons in the next hidden layer. Furthermore, features are compressed spatially through operations called pooling or subsampling. CNNs can acquire complex feature representations through multi-stage feature transformation, allowing them to perform highly accurate category recognition and object detection of objects in images based on features. In machine learning, represented by CNNs, image signals and corresponding training signals are trained as a set. As a result of training, a trained model is generated as processing parameters for object detection.

[0030] The object detection unit 106 can detect an object using, for example, a trained model based on a neocognitron, which is a type of CNN. The trained model is not limited to a neocognitron, and may be generated using a generative adversarial network (GAN) or a recurrent recurrent neural network (RNN). The trained model may also be generated by machine learning without using a neural network.

[0031] The process of detecting an object using CNN from input image data will be described with reference to Figure 2. Figure 2 shows the basic process flow of object detection using CNN. CNN includes multiple layers, and each layer includes two layers called a feature detection layer (S layer) and a feature integration layer (C layer). In the example of Figure 2, the input image input to the CNN is processed in order from the first layer to the nth layer.

[0032] The S layer corresponds to a convolutional layer, and the C layer corresponds to a pooling or subsampling layer. In CNN, the S layer first detects features of the input image based on the features detected in the previous layer. Next, the features detected in the S layer are integrated in the C layer and input to the next layer as the detection result of the current layer.

[0033] The S layer contains multiple feature detection cell planes, each of which detects a different feature. The C layer contains multiple feature integration cell planes, which pool or subsample the detection results from the previous feature detection cell plane. The final output layer (nth layer) is composed of S layers without using C layers.

[0034] A feature detection cell surface is composed of multiple feature detection neurons, each of which is connected to the C layer of the previous layer in a specific structure. A feature integration cell surface is composed of multiple feature integration neurons, each of which is connected to the S layer of the same layer in a specific structure.

[0035] By adjusting the connection coefficients of each feature integration neuron to an appropriate value, the CNN becomes able to detect a specific feature. The connection coefficients are adjusted through learning. In the feature detection layer (S layer) of the final layer (nth layer), the connection coefficients are corrected using, for example, the least squares method. In the feature detection layers (S layers) of the other layers (1st to n-1th layers), the connection coefficients are corrected using, for example, the backpropagation method. The connection coefficient correction method using the least squares method and the backpropagation method can be any known technique. For reference, see Alex Krizhevsky, et al. "ImageNet Classification with Deep Convolutional Neural Networks”, Advances in Neural Information Processing Systems 25 (NIPS'12), 2012.

[0036] As test patterns for learning, patterns to be detected and patterns not to be detected are prepared. Each test pattern has image data and a corresponding teacher signal. Image data corresponding to a pattern to be detected is given a teacher signal such that the output of a neuron corresponding to an area where the pattern to be detected exists in the feature detection cell plane of the final layer becomes 1. On the other hand, image data corresponding to a pattern not to be detected is given a teacher signal such that the output of a neuron corresponding to an area where the pattern not to be detected exists becomes -1.

[0037] The trained model (dictionary) of this embodiment can be prepared, for example, by training a test pattern using image data based on an image signal obtained by the image sensor 103. The trained model may be generated by a device other than the camera 100. The trained model is stored in the ROM 110. The subject detection unit 106 can detect subjects appearing in the image data using the trained model stored in the ROM 110.

[0038] Like the object detection unit 106, the object detection unit 202 of the cloud server 200 can detect objects using a trained model generated by a neocognitron, which is a type of CNN. The object detection unit 106 of the camera 100 is subject to constraints such as the circuit size and the number of trained models that can be stored in the ROM 110. On the other hand, the object detection unit 202 is not subject to the same constraints as the object detection unit 106 in terms of the circuit size and the number of trained models that can be stored, and is capable of detecting objects using a more complex network model. Therefore, the object detection unit 202 has higher object detection accuracy than the object detection unit 106 and can detect a wider variety of objects.

[0039] The detection accuracy here refers to the accuracy of two types of capabilities. One is the ability to detect whether a subject is included in image data and not make a false detection if it is not included. The other is the ability to detect more detailed features of the subject. For example, when detecting a dog, if the breed can be detected, the detection accuracy is higher than if the dog is simply detected as a dog.

[0040] Since the camera 100 is connected to the cloud server 200 via a network, the time it takes to send image data and obtain (receive) the detection results from the cloud server 200 is longer than the time it takes to obtain the detection results from the subject detection unit 106.

[0041] 3 shows the characteristics of object detection by the object detection unit 106 of the camera 100 and the object detection unit 202 of the cloud server 200. The object detection speed of the camera 100 is faster than that of the cloud server 200. The detection speed here is 00 and the time it takes to transmit the detection results from cloud server 200 to camera 100. The camera 100 can detect fewer types of subjects than the cloud server 200. The detection accuracy of camera 100 is lower than that of cloud server 200.

[0042] (Live View shooting) Live view shooting (LV shooting) of camera 100 will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating live view shooting processing. The LV shooting processing here is processing for displaying an image captured by camera 100 on monitor 104. The LV shooting processing is started, for example, when camera 100 is powered on.

[0043] In S101, the subject detection unit 106 (first acquisition means) performs subject detection processing using CNN from the image data generated by the image processing unit 105. The algorithm used for subject detection is not limited to CNN, and may be other deep learning algorithms such as R-CNN, YOLO, SSD, etc.

[0044] In S102, the CPU 113 determines whether or not the subject detection unit 106 has detected a subject. If the subject detection unit 106 has detected a subject, the process proceeds to S104. If the subject detection unit 106 has not detected a subject, the process proceeds to S103.

[0045] In S103, CPU 113 (first acquisition means) performs subject detection processing from image data without using CNN. For example, CPU 113 can detect a moving subject determined by a motion vector, a subject closer to camera 100 determined by using distance information, or a subject determined from color information or histogram information. CPU 113 may detect a subject based on information combining a motion vector, distance information, color information, and histogram information.

[0046] In S104, the CPU 113 controls the display of the monitor 104 so as to notify the user of the subject detected in S101 or S103. The CPU 113 can notify the user, for example, by surrounding the detected subject with a bounding box.

[0047] CPU 113 executes AF processing and AE processing based on the subject detection result (first detection result) in S101 or S103. Note that if CPU 113 has already acquired the subject detection result from cloud server 200, it may execute AF processing and AE processing based on the subject detection result by cloud server 200 (second detection result).

[0048] In S105, CPU 113 determines the subject detected in S101 or S103 as the main subject. If multiple subjects are detected, CPU 113 may determine the subject closest to camera 100, the subject closest to the center of the angle of view, or a subject of a preset type (person, animal, etc.) as the main subject. The process of S105 may be executed before S104.

[0049] In S106, CPU 113 determines whether to acquire the subject detection result from cloud server 200. If a predetermined condition is met, CPU 113 (second acquisition means) transmits image data to cloud server 200 and acquires the subject detection result by subject detection unit 202 from cloud server 200.

[0050] The predetermined condition is, for example, that the subject detection result has never been acquired from the cloud server 200 after the start of LV photography, or that a predetermined time has elapsed since the subject detection result was received from the cloud server 200. In addition, the predetermined condition is that the subject detection unit 106 The predetermined condition may be that a subject has been detected, or that a subject detected by cloud server 200 has disappeared from the angle of view, i.e., that subject tracking unit 107 has lost sight of the subject. Furthermore, the predetermined condition may be that a scene has changed. The scene change can be determined using, for example, color information or luminance information of the image data.

[0051] If the predetermined condition is met and it is determined that the subject detection result is to be acquired from cloud server 200, the process proceeds to S107. If it is determined that the subject detection result is not to be acquired from cloud server 200, the process proceeds to S110.

[0052] In S107, CPU 113 determines whether or not a detection result has been received from cloud server 200. Specifically, CPU 113 first transmits image data to cloud server 200 via communication unit 109. Next, subject detection unit 202 of cloud server 200 detects a subject from the received image data.

[0053] If a subject is detected, the parameter determination unit 204 determines the settings of the shooting parameters according to the detected subject. The CPU 201 of the cloud server 200 transmits information on the detected subject and setting information of the determined shooting parameters as the detection result to the camera 100 via the communication unit 203. If the subject detection unit 202 does not detect a subject, the CPU 201 notifies the camera 100 that a subject was not detected from the received image data.

[0054] If the camera 100 receives the detection result, the process proceeds to S108. If the camera 100 does not receive the detection result, or if the camera 100 receives a notification that the subject was not detected from the image data, the process proceeds to S110.

[0055] In S108, the CPU 113 controls the display of the monitor 104 so as to notify the user of the subject of the detection result (the subject detected by the subject detection unit 202) acquired from the cloud server 200 in S106 and S107. For example, the CPU 113 notifies the user by surrounding the subject of the detection result with a bounding box.

[0056] In S109, CPU 113 updates the main subject set in S105 to the subject of the detection result acquired from cloud server 200. Note that the process of S109 may be executed before S108.

[0057] In S110, CPU 113 executes a photographing process according to the main subject. The photographing process according to the main subject is, for example, an AF process and an AE process according to the position of the main subject, and various image processes for other photographing parameters. If a main subject is detected based on the detection result from cloud server 200 in S108, CPU 113 executes the photographing process using the setting information of the photographing parameters determined by parameter determination unit 204.

[0058] If no detection result is acquired from cloud server 200 in S106 and S107, CPU 113 executes the photographing process using the photographing parameter settings determined by camera 100 according to the current main subject. CPU 113 may also execute the photographing process using the photographing parameter setting information acquired from cloud server 200 immediately before.

[0059] In S111, subject tracking unit 107 performs tracking processing of the main subject. There are cases where a subject detected by subject detection unit 202 of cloud server 200 is not detected by camera 100. By tracking the subject detected by subject detection unit 202, CPU 113 can continue shooting processing according to the subject detected with high accuracy by cloud server 200.

[0060] In S112, the CPU 113 determines whether or not to end the LV photography. The CPU 113 can determine to end the LV photography when, for example, an operation to turn off the power or an operation to switch from the photography mode to the playback mode is received from the user. If the LV photography is to end, the LV photography process shown in Fig. 4 ends. If the LV photography is not to end, the process returns to S101.

[0061] (Notifying the user of the subject detection status) 5 and 6, the process of notifying the user of the detected subject in S104 and S108 of Fig. 4 will be described. Fig. 5 is a diagram showing an example in which the same type of subject as the subject detected by camera 100 is detected in cloud server 200. In the example of Fig. 5, it is assumed that subject detection unit 106 of camera 100 can detect dogs, and subject detection unit 202 of cloud server 200 can detect more detailed characteristics such as the dog's breed and coat color.

[0062] FIG. 5(A) shows a state in which a dog appears within the field of view. At the time the dog appears, camera 100 and cloud server 200 have not yet detected the subject. Camera 100's subject detection unit 106 is capable of subject detection at a speed slower than the period for capturing each frame in LV photography, and is therefore able to immediately detect the dog. Furthermore, the image data shown in FIG. 5(A) is sent to cloud server 200 when a predetermined condition is met.

[0063] 5(B) shows a state in which the dog detected by subject detection unit 106 is surrounded by a dotted rectangle 301, thereby notifying the user that a dog has been detected as a subject. The dotted rectangle indicates a detection result based on subject detection by subject detection unit 106, i.e., camera 100, and not based on detection by cloud server 200. On the other hand, when subject detection unit 202 of cloud server 200 receives image data from camera 100, it detects a subject from the image data (S107 in FIG. 4).

[0064] 5(C) shows a state in which the camera 100 is waiting for the detection result from the cloud server 200. The detection speed from sending image data from the camera 100 to the cloud server 200 to obtaining the detection result from the cloud server 200 is slower than the detection speed at which the subject detection unit 106 detects the subject from the image data. To notify the user that the detection result from the cloud server 200 is being obtained, the CPU 113 displays an icon (item) 302 indicating that the cloud server 200 is detecting the subject, as shown in FIG. 5(C). Because better photographic results can be provided by using the subject detection result from the cloud server 200, the CPU 113 displays the icon 302 to prompt the user to wait.

[0065] 5(D) shows a state in which the subject detection result is received from cloud server 200, and the subject detected by subject detection unit 202 is surrounded by solid-line rectangle 303 to notify the user that the subject has been set as the main subject. This allows the user to recognize that the subject has been detected based on the detection result by cloud server 200. In this way, by surrounding the subject based on the detection result by camera 100 with dotted-line rectangle 301 and the subject based on the detection result by cloud server 200 with solid-line rectangle 303, the user can identify which detection result the subject is based on. Note that the display style of the frame surrounding the subject is not limited to dotted-line and solid-line rectangles, and may be distinguished by other shapes, line types, or line colors as long as the user can identify which detection result the subject is based on.

[0066] When the subject detection unit 202 detects the subject as an Akita dog, the parameter determination unit 204 determines the settings of the shooting parameters by utilizing the characteristics of the coat color of the Akita dog in calculations such as white balance. By performing the photographing process using the parameter setting information, it is possible to achieve more accurate white balance and the like.

[0067] FIG. 5(E) is a diagram showing the subject detection state and the main subject in the states shown in FIGS. 5(A) to 5(D). In the state of FIG. 5(A), the subject has not been detected by either camera 100 or cloud server 200, and no main subject has been set. In the state of FIG. 5(B), the subject has been detected by camera 100 but not by cloud server 200, and the main subject has been set to a dog. In the state of FIG. 5(C), the subject has been detected by camera 100 but is currently being detected by cloud server 200, and the main subject remains set to a dog. In the state of FIG. 5(D), the subject has been detected by cloud server 200, and the main subject has been set to a dog detected as an Akita dog. Because cloud server 200 identifies the dog breed, CPU 113 can use the subject detection results by cloud server 200 to set appropriate shooting parameters according to the subject.

[0068] Fig. 6 is a diagram showing an example in which a subject different from the subject detected by camera 100 is detected in cloud server 200. In the example of Fig. 6, it is assumed that subject detection unit 106 of camera 100 can detect people but does not have a dictionary (trained model) for detecting birds, and that subject detection unit 202 of cloud server 200 can detect both people and birds.

[0069] Fig. 6(A) shows a state in which a bird and a person appear within the angle of view. At the time when the bird and the person appear, the camera 100 and the cloud server 200 have not detected the subject. In S101 of Fig. 4, the subject detection unit 106 detects the subject from the image data. Furthermore, the image data shown in Fig. 6(A) is transmitted to the cloud server 200 when a predetermined condition is satisfied.

[0070] 6(B) shows a state in which the user is notified that a person has been detected as a subject by surrounding the person detected by subject detection unit 106 with dotted-line rectangular area 401. Meanwhile, when subject detection unit 202 of cloud server 200 receives image data from camera 100, it detects a subject from the image data (S107 in FIG. 4).

[0071] Fig. 6(C) shows a state in which the detection result from cloud server 200 is being waited for. To notify the user that the detection result from cloud server 200 is being waited for, CPU 113 displays icon 402 indicating that subject detection is being performed by cloud server 200, as shown in Fig. 6(C).

[0072] 6(D) shows a state in which the subject detection result is received from cloud server 200, and the subject detected by subject detection unit 202 is surrounded by solid-line rectangle 403, thereby notifying the user that the subject has been set as the main subject. In the example of FIG. 6(D), subject detection unit 202 of cloud server 200 detects a person and a bird as subject candidates, but because the person is facing sideways, it detects the bird as the main subject.

[0073] When multiple subjects are detected, the main subject may be determined based on the orientation or size of the detected subjects or parts of the subjects (such as a person's face), etc. Alternatively, the main subject may be determined based on the distance from camera 100, the position within the angle of view, the type of subject, etc.

[0074] FIG. 6(E) is a diagram showing the subject detection state and the main subject in the state shown in FIG. 6(A) to FIG. 6(D). In the state of FIG. 6(A), the subject has not been detected by either camera 100 or cloud server 200, and no main subject has been set. In the state of FIG. 6(B), the subject has been detected by camera 100 but not by cloud server 200, and the main subject is a person. In the state of FIG. 6(C), the subject has been detected by camera 100 and is being detected by cloud server 200, and the main subject remains set to a person. In the state of FIG. 6(D), the subject has been detected by cloud server 200, and the main subject has been changed to a bird. Because cloud server 200 can detect a bird that is not detected by camera 100 as a subject, CPU 113 can use the subject detection result by cloud server 200 to set appropriate shooting parameters according to the subject.

[0075] The subject detection status may be notified to the user in other ways than by a dotted or solid rectangle. For example, the subject detection status may be notified by changing the color of the line surrounding the subject or by displaying an icon in the area of ​​or around the subject.

[0076] (still image capture) 7, still image capture by the camera 100 will be described. Still image capture begins when the shutter 102 is pressed during LV capture and the release switch SW2 is turned on.

[0077] 7 is a flowchart illustrating a still image capturing process. When the camera 100 does not detect the subject with high accuracy, the CPU 113 performs a capturing process using the detection result from the cloud server 200.

[0078] In S201, CPU 113 determines whether or not a subject detection result by subject detection unit 202 has been acquired from cloud server 200 during LV shooting. If a detection result from cloud server 200 has been acquired, the process proceeds to S204. If a detection result from cloud server 200 has not been acquired, the process proceeds to S202.

[0079] Even if the detection results have already been obtained from the cloud server 200, if a predetermined condition is met, such as if a predetermined time has passed since the results were obtained, the CPU 113 may transmit image data to the cloud server 200 to obtain the detection results.

[0080] In S202, CPU 113 executes a photographing process based on the subject detection result by camera 100. The generated image data is stored in recording medium 112. In S203, CPU 113 waits until the subject detection result is received from cloud server 200.

[0081] Note that CPU 113 may execute AF processing and AE processing based on the subject detection result by camera 100, and wait until receiving the subject detection result from cloud server 200. In this case, CPU 113 can execute imaging processing other than AF processing and AE processing based on the received detection result from cloud server 200. CPU 113 can also wait until receiving the subject detection result from cloud server 200, and then execute imaging processing, including AF processing and AE processing.

[0082] In S204, CPU 113 executes the photographing process based on the subject detection result by subject detection unit 202 of cloud server 200. In this case, the image data stored in recording medium 112 in S202 is overwritten with the image data generated by the photographing process in S204. Before overwriting with the image data generated in S204, CPU 113 may display a message to ask the user whether or not to allow the overwriting. Note that the image data generated in S204 may be stored as image data separate from the image data stored in recording medium 112 in S202.

[0083] Furthermore, the setting information of the shooting parameters determined based on the result of subject detection by the subject detection unit 202 may be stored in the file of the image data stored in the recording medium 112 in S202. When playing back the image data on the monitor 104, the CPU 113 may display the image data by applying the setting information of the shooting parameters stored in the same file.

[0084] Furthermore, if the subject detected by the camera 100 and the cloud server 200 are the same, the CPU 113 may use the subject detection result by the camera 100 in S204.

[0085] Furthermore, even if the subject is the same, if the shooting parameter settings determined by the camera 100 differ from the settings by the parameter determination unit 204, the CPU 113 may execute the shooting process using the shooting parameter settings determined by the parameter determination unit 204. In this way, the CPU 113 first executes the shooting process using the shooting parameter settings determined by the camera 100, and upon receiving the detection results from the cloud server 200, can execute the shooting process using the shooting parameter settings determined based on the received detection results.

[0086] 7 illustrates the capture of still images, but the present invention can also be applied to the capture of moving images. As with live view capture processing, camera 100 can record moving images that have been captured and processed in accordance with the detected objects with higher accuracy by using the subject detection results obtained by camera 100 and cloud server 200.

[0087] (Processing for each shooting parameter) 1.AF processing The following describes the advantages of using the subject detection results from camera 100 and cloud server 200 in AF processing. When photographing a moving subject, it is important to immediately adjust the focus after the subject appears within the angle of view and to continue adjusting the focus as the subject moves. For example, when photographing a car with camera 100 capable of detecting cars, subject detection unit 106 immediately performs AF processing on the entire area of ​​the detected car or on an area including the detected position.

[0088] Because the subject detection unit 106 of the camera 100 has sufficient performance for detecting cars, the CPU 113 can immediately focus on an area including the detected car. However, when photographing a car, the area to focus on changes depending on the type of car or the direction of the car. Therefore, the camera 100 transmits image data to the cloud server 200 and obtains the subject detection result from the subject detection unit 202.

[0089] The subject detection unit 202 of the cloud server 200 identifies the vehicle type from the received image data. The parameter determination unit 204 determines a notable area (for example, a headlight area) that has been set in advance according to the identified vehicle type as the AF area. The cloud server 200 transmits information about the subject (the vehicle whose vehicle type has been identified) detected by the subject detection unit 202 and the AF area (AF processing setting information) determined by the parameter determination unit 204 to the camera 100 as the detection results.

[0090] By switching the AF area based on the received detection result, CPU 113 can perform AF processing with higher accuracy than when AF processing is performed based on the detection result of subject detection unit 106. By using the subject detection results from camera 100 and cloud server 200 in this way, camera 100 can achieve both high detection speed and high detection accuracy.

[0091] 2. Brightness correction processing, gradation correction processing The following describes the advantages of using the subject detection results from camera 100 and cloud server 200 in brightness correction processing and gradation correction processing. It is assumed that subject detection unit 106 of camera 100 is simply capable of detecting people, and subject detection unit 202 of cloud server 200 is also capable of detecting people's race, sex, age, and so on.

[0092] When using the subject detection results from subject detection unit 106 of camera 100, it may be difficult for camera 100 to achieve optimal brightness suited to the person because the person's race, gender, age, etc. are not specified. As described in FIG. 7 , when the subject detection results from subject detection unit 202 of cloud server 200 are acquired and used, camera 100 can achieve optimal brightness suited to the person's race, gender, age, etc. For example, if cloud server 200 specifies that the person is a fair-skinned woman, camera 100 can perform brightness correction processing based on the detection results from cloud server 200 to brighten the woman's face.

[0093] There are also advantages to using the detection results from both camera 100 and cloud server 200. For example, when photographing a person in backlight, if exposure conditions are determined to achieve the desired brightness without using the subject detection results, the gain applied in image processing will be large, which may increase noise.

[0094] On the other hand, when the subject detection result by the subject detection unit 106 is obtained, the camera 100 can capture a still image under exposure conditions that are closer to the optimal brightness. In this case, the gain applied in image processing is suppressed compared to when the detection result by the subject detection unit 106 is not used, and therefore an increase in noise is suppressed. By using not only the detection result by the cloud server 200 but also the detection result by the camera 100 itself, the camera 100 can suppress noise.

[0095] 3. White balance processing The following describes the advantages of using the subject detection results from camera 100 and cloud server 200 in white balance processing. If the subject detection unit 106 of camera 100 has lower detection accuracy than cloud server 200, it is likely to erroneously determine that subjects such as blue sky, soil, or brick are white. Therefore, camera 100 transmits image data to cloud server 200 and obtains the subject detection results from cloud server 200.

[0096] The subject detection unit 202 of the cloud server 200 detects subjects that are likely to be erroneously determined as white in the image data received from the camera 100. The parameter determination unit 204 determines the settings for white balance processing by excluding evaluation values ​​such as the integral value of an area that includes subjects that are likely to be erroneously determined as white. The CPU 201 transmits setting information for the white balance processing to the camera 100.

[0097] Image processing unit 105 of camera 100 can apply more accurate white balance processing to image data by using the setting information for white balance processing received from cloud server 200. Note that camera 100 may obtain information on areas of the subject that are likely to be erroneously determined as white from cloud server 200, and use CPU 113 to determine the setting for white balance processing.

[0098] The subject detection result by subject detection unit 106 of camera 100 is used when LV shooting starts, when the main subject changes, when the scene changes (for example, when shooting changes from indoors to outdoors), etc. By using the subject detection result by subject detection unit 106, camera 100 can quickly perform white balance processing even when it has not received the detection result from cloud server 200.

[0099] 4. Color correction processing The following describes the advantages of using the subject detection results from camera 100 and cloud server 200 in color correction processing. Subject detection unit 202 of cloud server 200 can detect landmarks such as famous buildings and landscapes by storing a dictionary for detecting these landmarks in the recording medium of cloud server 200.

[0100] For example, if the subject detection unit 202 detects the ocean in Hawaii as the subject, the parameter determination unit 204 determines the settings for the color correction process so as to vividly express the emerald green of the ocean and the color of the sky. By using the setting information for the color correction process received from the cloud server 200, the image processing unit 105 of the camera 100 can apply more accurate color correction process to the image data.

[0101] The subject detection result by subject detection unit 106 of camera 100 is used when LV shooting starts, when the main subject changes, when the scene changes, etc. By using the subject detection result by subject detection unit 106, camera 100 can quickly perform color correction processing even when it has not received the detection result from cloud server 200.

[0102] According to the above-described embodiment, camera 100 can accurately recognize a subject by transmitting image data to cloud server 200 and acquiring the subject detection result from cloud server 200. Furthermore, camera 100 can suppress a decrease in subject recognition speed by using the subject detection result from camera 100 until the subject detection result is received from cloud server 200.

[0103] The above-described embodiment shows an example in which the camera 100 uses the subject detection result from the cloud server 200, which has higher detection accuracy, but is not limited to this. Even if the detection accuracy of the camera 100 is not as high as that of the cloud server 200, the camera 100 may use the subject detection result from another electronic device, such as a smartphone, as long as the detection accuracy is higher than that of the camera 100.

[0104] Furthermore, camera 100 may acquire subject detection results from multiple electronic devices or cloud server 200, depending on the type of subject dictionary held by each of the electronic devices and cloud server 200. For example, camera 100 may acquire subject detection results for people from a smartphone, and acquire subject detection results for landmarks, landscapes, or the like from cloud server 200. By acquiring subject detection results for people from a smartphone that has a faster response speed than cloud server 200, camera 100 can prevent a decrease in the person recognition speed.

[0105] While the present invention has been described above with reference to preferred embodiments, it is not limited to these embodiments and various modifications and variations are possible within the scope of the present invention. Furthermore, the features described in the embodiments can be combined as appropriate.

[0106] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0107] 100: camera unit, 106: subject detection unit, 109: communication unit, 113: CPU 200: Cloud server

Claims

1. An imaging means; a first acquisition means for acquiring a first detection result that is a result of detecting a subject from image data captured by the imaging means; a second acquisition means for transmitting the image data to an external image processing device and acquiring a second detection result, which is a result of the image processing device detecting a subject, from the image data; a control means for executing an imaging process using imaging parameter settings determined based on at least one of the first detection result and the second detection result; and the second detection result includes information about the subject detected by the image processing device, The control means performing an imaging process using the imaging parameter settings determined based on the first detection result until the second detection result is obtained from the image processing device; When the second detection result is acquired from the image processing device, a photographing process is performed using the photographing parameter settings determined based on the information on the subject included in the second detection result. An imaging device characterized by:

2. The camera further includes a display control unit that notifies a user of the subject in the first detection result and the subject in the second detection result.

2. The imaging device according to claim 1.

3. The display control means notifies the user so that the user can distinguish whether the subject is based on the first detection result or the second detection result.

3. The imaging device according to claim 2.

4. The display control means displays an item indicating that the second detection result is being acquired until the second detection result is received.

4. The imaging device according to claim 2, wherein the imaging device is a lens.

5. The photographing parameters include AF processing (autofocus processing), AE processing (autoextract processing), The parameter includes at least one of the following processing parameters: image processing, white balance processing, brightness correction processing, color correction processing, and gradation correction processing.

5. The imaging device according to claim 1, wherein the first and second lenses are arranged parallel to each other.

6. When the control means has not received the second detection result from the image processing device, the control means determines settings of the AF processing and the AE processing among the shooting parameters based on the first detection result.

6. The imaging device according to claim 5.

7. When the second detection result has already been acquired from the image processing device during live view photography, the control means executes photography processing using settings for the AF processing and the AE processing determined based on the second detection result.

7. The imaging device according to claim 6.

8. The second detection result has a higher subject detection accuracy than the first detection result.

8. The imaging device according to claim 1, wherein the imaging device is a lens.

9. The first acquisition means detects the object using a trained model prepared for each type of object.

9. The imaging device according to claim 1, wherein the imaging device is a lens.

10. The second detection result includes information about the subject detected using a trained model prepared for each type of subject.

10. The imaging device according to claim 1, wherein the imaging device is a lens.

11. The second acquisition means transmits the image data to the image processing device and acquires the second detection result when a predetermined condition is satisfied.

11. The imaging device according to claim 1.

12. The predetermined condition is any one of the following: the second detection result has never been received from the image processing device after live view shooting was started; a predetermined time has elapsed since the second detection result was received; the first acquisition means has detected a new subject; the subject of the second detection result has disappeared from the angle of view; and the scene of the image data has changed.

12. The imaging device according to claim 11.

13. The computer a first acquisition step of acquiring a first detection result that is a result of detecting a subject from captured image data; a second acquisition step of transmitting the image data to an external image processing device and acquiring a second detection result, which is a result of the image processing device detecting the subject from the image data; a control step of executing an imaging process using imaging parameter settings determined based on at least one of the first detection result and the second detection result; and the second detection result includes information about the subject detected by the image processing device, In the control step, performing an imaging process using the imaging parameter settings determined based on the first detection result until the second detection result is obtained from the image processing device; When the second detection result is acquired from the image processing device, a photographing process is performed using the photographing parameter settings determined based on the information on the subject included in the second detection result.

10. A method for controlling an imaging device, comprising:

14. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 12.

15. An image processing device capable of communicating with the imaging device according to any one of claims 1 to 12, a receiving means for receiving image data captured by the imaging device; a detection means for detecting a subject from the image data; a determination means for determining settings of the photographing parameters based on the object detected by the detection means; a transmitting means for transmitting information about the subject detected by the detecting means to the imaging device as the second detection result; 1. An image processing device comprising:

16. An imaging system including an imaging device and an image processing device, The imaging device is An imaging means; a first acquisition means for acquiring a first detection result that is a result of detecting a subject from image data captured by the imaging means; a second acquisition means for transmitting the image data to an external image processing device and acquiring a second detection result, which is a result of the image processing device detecting a subject, from the image data; a control means for executing an imaging process using imaging parameter settings determined based on at least one of the first detection result and the second detection result; The image processing device includes: a receiving means for receiving image data captured by the imaging device; a detection means for detecting a subject from the image data; a determination means for determining settings of the photographing parameters based on the object detected by the detection means; a transmitting means for transmitting information about the subject detected by the detecting means to the imaging device as the second detection result, The control means executes a photographing process using the settings of the photographing parameters determined based on the first detection result until the second detection result is obtained from the image processing device, and when the second detection result is obtained from the image processing device, executes a photographing process using the settings of the photographing parameters determined based on information about the subject as the second detection result. An imaging system characterized by:

Images