Imaging device, parameter adjustment method, program, and recording medium

Abstract

An imaging device according to the present invention comprises: a sensor unit that obtains a captured image; an AI processing unit that performs AI processing on the captured image by using an AI model; a signal processing unit that adjusts the image quality of the image input to the AI processing unit; and a control unit that performs parameter adjustment of the signal processing unit on the basis of the result of the AI processing by the AI processing unit.

Description

Imaging device, parameter adjustment method, program, and recording medium

[0001] The present technology relates to an imaging device equipped with an AI (Artificial Intelligence) processing unit that performs AI processing using an AI model on an input image, and a parameter adjustment method, program, and recording medium for the imaging device.

[0002] As a processing target for a captured image, there is a technology that performs processing using an AI (Artificial Intelligence) model such as a DNN (Deep Neural Network) (hereinafter referred to as "AI processing"). For example, Patent Document 1 listed below discloses a technology for an imaging device that performs AI processing such as object recognition processing on a captured image. Specifically, Patent Document 1 discloses a technology that performs AI processing on a captured image within a sensor unit of a digital camera.

[0003] International Publication No. 2018 / 051809

[0004] Here, when performing AI processing on a captured image, it is conceivable to perform preprocessing for adjusting the image quality of the captured image as preprocessing for the AI ​​processing. Conventionally, such preprocessing has been performed as image quality adjustment processing for viewing, in which people view images, but there is no guarantee that preprocessing suitable for viewing will also be preprocessing suitable for AI processing. In other words, there is no guarantee that it will contribute to improving the AI ​​processing accuracy, such as recognition accuracy.

[0005] Furthermore, when performing AI processing in an imaging device, the ability to adapt to changes in the imaging environment should also be considered as preprocessing. If the image quality adjustment does not adapt well to changes in the imaging environment, there will be an increased chance that AI processing will be performed on images that are inappropriate for the environment, which will result in a decrease in the accuracy of the AI ​​processing.

[0006] This technology was developed in consideration of the above circumstances, and aims to improve the accuracy of AI processing by improving the speed at which it can respond to environmental changes while realizing image quality adjustment processing suitable for AI processing pre-processing.

[0007] The imaging device according to the present technology includes a sensor unit for acquiring a captured image, an AI processing unit for performing AI processing using an AI model on the captured image, a signal processing unit for adjusting the image quality of an input image to the AI ​​processing unit, and a control unit for adjusting parameters of the signal processing unit based on the AI ​​processing results by the AI ​​processing unit. By adjusting the parameters of the signal processing unit based on the AI ​​processing results as described above, it is possible to perform preprocessing suitable for the AI ​​processing. Furthermore, according to the above configuration, the signal processing unit and the control unit are provided within the same imaging device, so it is possible to quickly adjust the parameters based on the AI ​​processing results.

[0008] FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment. FIG. 2 is an explanatory diagram of an example of dynamic range adjustment. FIG. 3 is an explanatory diagram of advantages of the configuration of the first embodiment. FIG. 4 is a flowchart showing an example of processing procedures for implementing the parameter adjustment method according to the first embodiment. FIG. 5 is a block diagram showing an example of the configuration of an imaging device according to a second embodiment. FIG. 6 is a flowchart showing an example of processing procedures for implementing the parameter adjustment method according to the first embodiment. FIG. 7 is a diagram showing advantages of separately performing development processing (image quality adjustment processing) and AI processing in multiple systems. FIG. 8 is a block diagram showing an example of the configuration of an imaging device according to a third embodiment. FIG. 9 is a flowchart showing an example of processing procedures for implementing the parameter adjustment method according to the third embodiment. FIG. 10 is a block diagram showing an example of the configuration of an imaging device according to another example of the third embodiment. FIG. 11 is a flowchart showing an example of processing procedures in an imaging device according to another example. FIG. 12 is an explanatory diagram of an imaging scene assumed in a fourth embodiment. FIG. 13 is an explanatory diagram of subject designation by a user. FIG. 14 is a block diagram showing an example of a system configuration for implementing the parameter adjustment method according to the fourth embodiment. FIG. 14 is a flowchart showing example processing procedures on the imaging device side and the information processing device side for implementing the parameter adjustment method according to the fourth embodiment.

[0009] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. First embodiment> (1-1. Configuration example of imaging device) (1-2. Parameter adjustment method as first embodiment) <2. Second embodiment> <3. Third embodiment> <4. Fourth embodiment> <5. Modified example> <6. Summary of embodiments> <7. Present technology>

[0010] 1 is a block diagram showing an example of the configuration of an imaging device 1 according to a first embodiment of the present technology. As shown in the figure, the imaging device 1 includes an image sensor 2, an AI (artificial intelligence) signal processing unit 3, a camera control unit 4, and a communication unit 5.

[0011] The image sensor 2 is an example of a sensor unit that obtains a captured image. In this specification, "capturing" broadly refers to obtaining image data by capturing a subject using a light-receiving element. The light-receiving element here is not limited to those that receive visible light, but may also include those that receive invisible light. Furthermore, the term "image data" here collectively refers to data consisting of multiple pixel data. The pixel data may include not only data indicating the amount of light received from the subject using a predetermined number of gradation values, but also various other data related to the subject, such as data indicating the distance to the subject, data indicating polarization information, and data indicating temperature. In other words, the "image data" obtained by "capturing" may include data as a gradation image indicating the gradation value of the amount of light received for each pixel, data as a distance image indicating information on the distance to the subject for each pixel, data as a polarization image indicating polarization information for each pixel, data as a thermal image indicating information on temperature for each pixel, and so on.

[0012] The image sensor 2 is configured as a gradation sensor that obtains the above-mentioned gradation image, and is configured, for example, as a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. The image sensor 2 in this example is configured to obtain a color image of R (red), G (green), and B (blue), and specifically has a sensor structure in which R pixels formed with color filters that selectively receive R light, G pixels formed with color filters that selectively receive G light, and B pixels formed with color filters that selectively receive B light are two-dimensionally arranged according to a predetermined arrangement rule, such as a Bayer array.

[0013] The AI ​​signal processing unit 3 has a development processing unit 6, an AI processing unit 7, a control unit 8, and a communication I / F (interface) 9. The development processing unit 6 performs development processing on the captured image as a RAW image obtained by the image sensor 2. The development processing here includes at least processing to convert the RAW image into an RGB color image, such as demosaic processing on the RAW image in a Bayer array.

[0014] The AI ​​processing unit 7 performs AI processing, which is processing using an AI model, on the captured image. Specifically, in this example, the AI ​​processing unit 7 performs AI processing on the captured image after development processing by the development processing unit 6.

[0015] The development processing unit 6 has an image quality adjustment function as an image signal processing function. Image quality adjustment here includes adjustment of at least one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction. Brightness adjustment is realized, for example, by applying a common offset value to the R, G, and B luminance values.

[0016] 2A and 2B are explanatory diagrams of examples of dynamic range adjustment. An example of dynamic range adjustment with 10-bit input and 8-bit output is shown here. FIG. 2A shows an example of the maximum dynamic range, FIG. 2B shows an example in which the dynamic range is narrowed toward the low-luminance side, and FIG. 2C shows an example in which the dynamic range is narrowed toward the high-luminance side. By performing such dynamic range adjustment, it is possible to increase the resolution of brightness in dark areas at the expense of the resolution of brightness in bright areas (as in FIG. 2B), or conversely, increase the resolution of brightness in bright areas at the expense of the resolution of brightness in dark areas (as in FIG. 2C).

[0017] 1 , the captured image after image quality adjustment by the development processing unit 6 is input to the AI ​​processing unit 7 as a target image for AI processing. In other words, the development processing unit 6 adjusts the image quality of the input image to the AI ​​processing unit 7.

[0018] The development processing unit 6 also has an image size adjustment function as an adjustment function for the input image to the AI ​​processing unit 7. Specifically, the development processing unit 6 has a downsampling function from the image size of the RAW image (e.g., 12 MP (megapixels)) to the Input Tensor size in the AI ​​processing unit 7 (e.g., 640 × 480 pixels).

[0019] Here, the development processing unit 6 in this example performs development processing, including image quality adjustment, in multiple separate systems. Specifically, the development processing unit 6 has a first processing unit 6a, a second processing unit 6b, and a third processing unit 6c for performing development processing, including image quality adjustment, in three separate systems. In this example, the first processing unit 6a to the third processing unit 6c are configured to simultaneously obtain images (developed images) after development processing and whose image quality has been adjusted in multiple systems. Note that the method for obtaining developed images whose image quality has been adjusted in multiple systems is not limited to the method of providing multiple processing units in parallel as described above, and it is also possible to adopt a method in which a single processing unit performs development processing (including image quality adjustment) in a time-division manner.

[0020] Furthermore, in this example, the AI ​​processing unit 7 is configured to be able to execute AI processing separately for multiple systems. Specifically, the AI ​​processing unit 7 in this example has a first AI unit 7a, a second AI unit 7b, and a third AI unit 7c for individually performing AI processing on the developed images of each system obtained in the development processing unit 6. The first AI unit 7a performs AI processing on the developed image obtained by the first processing unit 6a, the second AI unit 7b performs AI processing on the developed image obtained by the second processing unit 6b, and the third AI unit 7c performs AI processing on the developed image obtained by the third processing unit 6c.

[0021] Specific examples of AI processing in the AI ​​processing unit 7 include object detection processing for detecting specific subjects such as people, animals, vehicles, ships, and airplanes in a captured image, and object recognition processing for identifying the type of specific subject. Examples of object recognition processing include class identification processing for identifying which of predetermined classes a specific subject belongs to, such as identifying attributes such as gender and age for a specific subject as a person, and face authentication processing for identifying whether a person's facial features match pre-specified facial features.

[0022] In the AI ​​processing unit 7, it is conceivable to use an AI model that has a neural network structure such as a DNN (Deep Neural Network) and has undergone machine learning as deep learning as the AI ​​model used for AI processing. Note that the AI ​​model is not limited to a model with a neural network structure, and it is also possible to use a model that does not have a neural network structure, such as a Vision Transformer, as long as it is an AI model that has undergone machine learning.

[0023] In the AI ​​processing unit 7, the first AI unit 7a to the third AI unit 7c are each capable of outputting an AI processing result indicating the result of the AI ​​processing and a captured image as an input image that was the subject of the AI ​​processing. Here, the information on the AI ​​processing result includes, for example, if the AI ​​processing is the object detection processing described above, information indicating the presence area of ​​the specific subject within the image, such as a bounding box of the detected specific subject. Also, if the AI ​​processing is an object recognition processing, the information on the AI ​​processing result includes information indicating the identification result of the specific subject, such as information indicating the class of the specific subject. The information on the AI ​​processing result also includes score information indicating the likelihood of the inference performed as the AI ​​processing, such as the likelihood of the identification result.

[0024] It is also possible to adopt a configuration in which only a single processing unit is provided for the AI ​​processing unit 7, and input images from multiple systems are processed by the single processing unit in a time-division manner.

[0025] Information on the AI ​​processing result obtained by the AI ​​processing unit 7 is output via the communication I / F 9 to the camera control unit 4 provided outside the AI ​​signal processing unit 3. Here, when the AI ​​processing unit 7 outputs a captured image that has been the subject of AI processing, the captured image is output to the camera control unit 4 via the communication I / F 9 together with the AI ​​processing result.

[0026] The control unit 8 is configured with a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and various functional operations are realized by the CPU executing processes in accordance with programs stored in the ROM. Specifically, the control unit 8 adjusts parameters of the development processing unit 6 based on the AI ​​processing results by the AI ​​processing unit 7. In this example, the control unit 8 adjusts parameters of the first processing unit 6a, the second processing unit 6b, and the third processing unit 6c in the development processing unit 6 individually based on the AI ​​processing results of the first AI unit 7a for the first processing unit 6a, the AI ​​processing results of the second processing unit 6b, and the AI ​​processing results of the third AI unit 7c for the third processing unit 6c, but details will be described later.

[0027] The control unit 8 is also capable of performing data communication with the camera control unit 4 via the communication I / F 9 .

[0028] The camera control unit 4 is configured with a microcomputer having a CPU, ROM, and RAM, and the CPU executes processing in accordance with a program stored in the ROM, thereby performing overall control of the imaging device 1. For example, the camera control unit 4 controls the execution of imaging operations by the image sensor 2. Furthermore, the camera control unit 4 can control the execution of operations (development processing and AI processing) of the AI ​​signal processing unit 3 by issuing instructions to the control unit 8.

[0029] A communication unit 5 is also connected to the camera control unit 4. The communication unit 5 is configured to be capable of performing wired or wireless data communication with an external device of the imaging device 1. The communication unit 5 may be configured to have a communication function via a network such as the Internet or a LAN (Local Area Network). The camera control unit 4 is capable of performing data communication with the external device via the communication unit 5. For example, the camera control unit 4 is capable of transmitting the AI ​​processing results obtained by the AI ​​signal processing unit 3 and the captured images output by the AI ​​signal processing unit 3 to the external device via the communication unit 5.

[0030] Here, the imaging device 1 has an AE (Auto Exposure) function. The AE function is realized by adjusting an iris in an imaging optical system (not shown), the shutter speed of an electronic shutter in the image sensor 2, and ISO sensitivity. Control for realizing the AE function may be performed by the camera control unit 4 or the control unit 8. The control for realizing the AE function is performed so that image brightness is adjusted in accordance with a predetermined photometry mode (overall metering, area metering, spot metering, etc.) based on the result of detection of an image captured by the image sensor 2.

[0031] As described above, the imaging device 1 in this embodiment has an AI processing function for captured images, and it is conceivable that such an imaging device 1 can be applied to, for example, various surveillance camera applications. Examples of such applications include surveillance cameras for indoors in stores, offices, homes, etc., surveillance cameras for monitoring outdoors in parking lots, city streets, etc. (including traffic surveillance cameras, etc.), surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation), and surveillance cameras for monitoring the interior and exterior of vehicles.

[0032] For example, when used as a surveillance camera in a store, multiple imaging devices 1 may be placed at predetermined locations within the store, allowing users to check the customer demographics (such as gender and age group) and behavior (traffic patterns) within the store. In this case, analysis using AI processing results may be performed to generate information on the customer demographics of the customers, their traffic patterns within the store, and congestion status at checkout registers (e.g., waiting time information at the checkout register). Alternatively, when used as a traffic surveillance camera, multiple imaging devices 1 may be placed at various locations near roads to recognize information about passing vehicles, such as their license plate numbers (vehicle numbers), vehicle colors, and vehicle models. Furthermore, when using a traffic surveillance camera in a parking lot, imaging devices 1 may be placed to monitor each parked vehicle, monitoring for suspicious individuals around each vehicle and, if a suspicious individual is detected, notifying the user of the suspicious individual's presence and their attributes (such as gender, age group, and clothing). Furthermore, it may be possible to monitor available spaces in a city or parking lot, thereby notifying the user of available parking spaces.

[0033] (1-2. Parameter Adjustment Method of First Embodiment) Here, as described above, in the imaging device 1 of this embodiment, the development processing unit 6 has first processing unit 6a to third processing unit 6c and performs development processing on RAW images in multiple systems, and the AI ​​processing unit 7 has first AI unit 7a to third AI unit 7c and performs AI processing individually on each captured image obtained by the development processing of these multiple systems. Under this configuration, the control unit 8 in this embodiment sets different image quality parameters to each of the first processing unit 6a to third processing unit 6c and causes them to perform image quality adjustment processing.

[0034] With the above configuration, the development processing unit 6 in the first embodiment performs image quality adjustment processing using different image quality parameters on the captured image as a RAW image, thereby obtaining multiple image quality-adjusted images. The AI ​​processing unit 7 then performs AI processing on each of the multiple image quality-adjusted images individually.

[0035] With the configuration of the first embodiment as described above, even if multiple subjects are imaged with different image quality and the AI ​​processing unit 7 cannot detect or recognize all of the subjects with a single image quality adjustment, the development processing unit 6 can obtain an image quality adjusted image with image quality suitable for detecting or recognizing each subject, making it possible for the AI ​​processing unit 7 to detect or recognize each subject.

[0036] A specific example will be described with reference to Fig. 3. Fig. 3 shows an example of subjects that are successfully detected when a human face detection process is performed as the AI ​​process, and three subjects (human subjects) as subjects S11, S12, and S13 are captured in a captured image, and different brightness adjustments are performed for each subject as image quality adjustments. The subjects that are successfully detected are those surrounded by a detection frame Dt in the figure.

[0037] 3B illustrates an image whose brightness has not been adjusted in the development processing unit 6 (i.e., an image whose brightness is set by the AE function). Hereinafter, the brightness of the image in FIG. 3B will be referred to as "reference brightness." Here, the reference brightness is taken as an example of brightness when the brightness has not been adjusted in the development processing unit 6, but the reference brightness may literally be a reference brightness, or may be brightness that has been adjusted by a predetermined amount in the development processing unit 6.

[0038] 3B, the brightness of subject S13 is suitable for face detection processing (face detection is possible), while the brightness of subject S12 is darker than that of subject S13, making face detection impossible. Furthermore, the brightness of subject S11 is brighter than that of subject S13, making face detection impossible. Therefore, for this image with standard brightness, only subject S13 is detected by the AI ​​processing of the AI ​​processing unit 7.

[0039] The image shown in Fig. 3A is an image whose brightness has been adjusted to be darker than the image with the reference brightness shown in Fig. 3B. In the image of Fig. 3A, the brightness of subject S11, which appeared brightest in the image of Fig. 3B, becomes suitable for face detection processing, the brightness of subject S13 decreases to a level at which face detection is impossible, and the brightness of subject S12 becomes darker than the brightness of subject S13, making face detection impossible. Therefore, for the image of Fig. 3A, only subject S11 is detected by the AI ​​processing of the AI ​​processing unit 7.

[0040] The image shown in Fig. 3C is an image whose brightness has been adjusted to be brighter than the image with the reference brightness shown in Fig. 3B. In the image of Fig. 3A, the brightness of subject S12, which appeared darkest in the image of Fig. 3B, becomes suitable for face detection processing, the brightness of subject S13 increases to a level at which face detection is impossible, and the brightness of subject S11 becomes brighter than the brightness of subject S13, making face detection impossible. Therefore, in the image of Fig. 3C, only subject S12 is detected by the AI ​​processing of the AI ​​processing unit 7.

[0041] In this way, if the development processing unit 6 performs image quality adjustment processing on the captured image using different image quality parameters to obtain multiple image quality-adjusted images, and the AI ​​processing unit 7 performs AI processing individually on these multiple image quality-adjusted images, it becomes possible for the AI ​​processing unit 7 to detect each subject even in cases where there are multiple subjects to be detected in the captured image and the AI ​​processing unit 7 cannot detect all of the subjects with a single image quality adjustment due to differences in brightness among the subjects, thereby improving the accuracy of the AI ​​processing.

[0042] The control unit 8 in this embodiment sets different image quality parameters for the first processing unit 6 a to the third processing unit 6 c and causes them to perform image quality adjustment processing as described above, while also adjusting the image quality parameters for the first processing unit 6 a to the third processing unit 6 c based on the results of the individual AI processing performed by the first AI unit 7 a to the third AI unit 7 c. This makes it possible to detect subjects with different brightness levels while also being able to respond to changes in the imaging environment.

[0043] Specifically, the control unit 8 adjusts the image quality parameters of the first processing unit 6a based on score information obtained as a result of AI processing by the first AI unit 7a, adjusts the image quality parameters of the second processing unit 6b based on score information obtained as a result of AI processing by the second AI unit 7b, and adjusts the image quality parameters of the third processing unit 6c based on score information obtained as a result of AI processing by the third AI unit 7c. Specifically, as image quality parameter adjustment for each frame of a captured image, processing is performed to adjust the brightness parameter so that the score value indicated by the score information tends to increase. At this time, if the score value decreases in response to an adjustment to darken the brightness, it is possible to switch to an adjustment to brighten the brightness, and conversely, if the score value decreases in response to an adjustment to brighten the brightness, it is possible to switch to an adjustment to darken the brightness.

[0044] FIG. 4 is a flowchart showing an example of a specific processing procedure to be executed by the control unit 8 (CPU) to realize the parameter adjustment method of the first embodiment described above. Specifically, this is an example of a processing procedure for adjusting parameters in the development processing (image quality adjustment processing) of each system based on the AI ​​processing results of each system. Here, for each system, from the first processing unit 6a to the third processing unit 6c, the adjustable brightness range is defined as the aforementioned reference brightness range, a range brighter than the reference brightness range, and a range darker than the reference brightness range. In this example, the control unit 8 adjusts the parameters of each system within the brightness adjustment range defined for each system. It is also assumed here that the parameter adjustment processing is performed for each frame of a captured image. That is, the control unit 8 repeatedly executes the processing shown in FIG. 4 for each frame.

[0045] 4, in step S101, the control unit 8 inputs the AI ​​processing results (score information) from the first AI unit 7a to the third AI unit 7c.

[0046] In step S102 following step S101, the control unit 8 performs parameter calculation processing based on the AI ​​processing result. Specifically, for each system, brightness parameters are calculated so that the score value indicated by the score information tends to increase, as exemplified above.

[0047] In step S103 following step S102, the control unit 8 performs a parameter adjustment process by setting the parameters for each system calculated in step S102 to the processing unit of the corresponding system in the development processing unit 6 (any of the first processing unit 6a to the third processing unit 6c).

[0048] After executing the process of step S103, the control unit 8 ends the series of processes shown in FIG.

[0049] It should be noted that the AI ​​processing performed by the AI ​​processing unit 7 is not limited to the face detection processing exemplified above, but may be various other processing. For example, it may be possible to perform face recognition processing on an image of a detected face area.

[0050] In the case of face recognition processing, not only the brightness of the face but also the color tone and contours of the face affect the accuracy of AI processing. Therefore, as the parameter adjustment process shown in Fig. 4, it is also conceivable to adjust color-related parameters (the aforementioned saturation and hue) and sharpness-related parameters in addition to brightness based on the AI ​​processing results.

[0051] 2. Second Embodiment Next, a second embodiment will be described. In the second embodiment, brightness is adjusted not only by adjusting the parameters of the development processing unit 6 but also by adjusting exposure.

[0052] 5 is a block diagram showing an example of the configuration of an image capture device 1A according to the second embodiment. In the following description, parts that are the same as parts that have already been described will be assigned the same reference numerals and descriptions thereof will be omitted.

[0053] 1 in that an AI signal processing unit 3A is provided instead of the AI ​​signal processing unit 3. The AI ​​signal processing unit 3A is different from the AI ​​signal processing unit 3 in that a control unit 8A is provided instead of the control unit 8.

[0054] The control unit 8A differs from the control unit 8 in that, with regard to brightness adjustment based on the AI ​​processing results, it not only adjusts the brightness parameters in the development processing unit 6 but also adjusts the exposure. For convenience of explanation, this example shows exposure adjustment as adjustment of the shutter speed and ISO sensitivity in the image sensor 2, but of course exposure adjustment including the aperture value can also be used.

[0055] A specific example of the processing procedure of the control unit 8A is shown in the flowchart of Fig. 6. In this case as well, brightness adjustment based on the AI ​​processing result is performed for each frame, and the control unit 8A repeatedly executes the processing shown in Fig. 6 for each frame.

[0056] The control unit 8A performs processing to input the AI ​​processing results for each system through the processing of step S101 described above. Then, in response to executing the input processing of step S101, the control unit 8A calculates parameters and exposure values ​​based on the AI ​​processing results in step S110. Here, although the brightness parameters can be set individually for each system, the exposure value is set for a single image sensor 2, so the control unit 8A determines an exposure value that is common to each system.

[0057] In response to the execution of the calculation process in step S110, the control unit 8A executes a parameter and exposure adjustment process in step S111. That is, the brightness parameters calculated for each system in step S110 are set in the corresponding processing units in the development processing unit 6, and the exposure values ​​calculated in step S110 (shutter speed and ISO sensitivity in this example) are set in the image sensor 2. Here, the exposure values ​​calculated in step S110 are set with priority over the exposure values ​​calculated by the AE function described above.

[0058] After executing the adjustment process in step S111, the control unit 8A ends the series of processes shown in FIG.

[0059] In the explanation so far, an example has been given in which development processing (image quality adjustment processing) and AI processing are performed separately in multiple systems, but by performing development processing and AI processing separately in multiple systems in this way, in addition to the aforementioned advantage of being able to detect or recognize multiple subjects with different brightness in each system, it can be said that there is the following advantage: That is, the advantage is that the target subject can be detected or recognized in at least one of the systems.

[0060] A specific example will be described with reference to Fig. 7. Fig. 7 shows an example of a captured image obtained in a backlit environment, with Fig. 7B illustrating an image with the aforementioned reference brightness, Fig. 7A illustrating an image with brightness darker than the reference brightness, and Fig. 7C illustrating an image with brightness brighter than the reference brightness. Here, of subjects S21 and S22 present in the captured image, subject S22 is assumed to be the subject to be detected.

[0061] For example, as shown in the figures, there may be cases where the target subject (S22) cannot be detected in the images with the brightness of Figure 7A or 7B, but can be detected in the image with the brightness of Figure 7C. In other words, even in cases where the target subject cannot be detected by a single brightness adjustment, it becomes possible to detect the target subject.

[0062] In particular, by including not only brightness adjustment but also exposure adjustment by the development processing unit 6 as in the second embodiment, the range of brightness adjustment is widened, making it possible to make the target subject more easily detectable.

[0063] 3. Third Embodiment In the third embodiment, parameters of the development processing unit 6 are adjusted based on the results of AI processing of a RAW image.

[0064] Fig. 8 is a block diagram showing an example configuration of an imaging device 1B according to a third embodiment. The imaging device 1B differs from the imaging device 1 shown in Fig. 1 in that an AI signal processing unit 3B is provided instead of the AI ​​signal processing unit 3, and a camera control unit 4B is provided instead of the camera control unit 4. In the example of Fig. 8, the AI ​​signal processing unit 3B and the camera control unit 4B are configured on separate chips.

[0065] Compared to the AI ​​signal processing unit 3, the AI ​​signal processing unit 3B differs in that it has a RAW signal processing unit 10 instead of the development processing unit 6, an AI processing unit 11 instead of the AI ​​processing unit 7, and a control unit 8B instead of the control unit 8.

[0066] The RAW signal processing unit 10 performs image quality adjustment on the captured image as a RAW image obtained by the image sensor 2. As with the development processing unit 6, the image quality adjustment by the RAW signal processing unit 10 also includes adjustment of at least any of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction.

[0067] The AI ​​processing unit 11 uses an AI model that has been machine-trained using RAW format images, rather than RGB images, as learning input data as an AI model for performing AI processing.

[0068] Furthermore, the camera control unit 4B differs from the camera control unit 4 in that it is equipped with a development function (including an image quality adjustment function) for RAW images and an AI processing function for developed images. Specifically, the camera control unit 4B is equipped with a development processing unit 41 that performs development processing on RAW images input via the AI ​​signal processing unit 3B, and an AI processing unit 42 that performs AI processing on developed images output from the development processing unit 41.

[0069] The camera control unit 4B also includes a CPU 40. The CPU 40 adjusts parameters of the development processing unit 41 based on instructions from the control unit 8B in the AI ​​signal processing unit 3B, details of which will be described later.

[0070] 8, as in the first embodiment, there are multiple systems of parameter adjustment processing, and parameter adjustment is performed for each system. To this end, in the camera control unit 4B, the development processing unit 41 has a first processing unit 41a, a second processing unit 41b, and a third processing unit 41c, and the AI ​​processing unit 42 has a first AI unit 42a that performs AI processing on images developed by the first processing unit 41a, a second AI unit 42b that performs AI processing on images developed by the second processing unit 41b, and a third AI unit 42c that performs AI processing on images developed by the third processing unit 41c.

[0071] In the third embodiment, the processing of each system of the development processing and AI processing can also be configured to be performed in a time-division manner using a single piece of hardware.

[0072] In addition, in this example, the image quality adjustment processing and AI processing for RAW images in the AI ​​signal processing unit 3B are also performed in multiple systems, and as shown, the RAW signal processing unit 10 has a first RAW processing unit 10a, a second RAW processing unit 10b, and a third RAW processing unit 10c, and the AI ​​processing unit 11 has a first AI unit 11a that performs AI processing on images after image quality adjustment by the first RAW processing unit 10a, a second AI unit 11b that performs AI processing on images after image quality adjustment by the second RAW processing unit 10b, and a third AI unit 11c that performs AI processing on images after image quality adjustment by the third RAW processing unit 10c.

[0073] It should be noted that the image quality adjustment and AI processing for these RAW images can also be performed in a time-sharing manner using a single piece of hardware.

[0074] In the imaging device 1B, the AI ​​processing unit 11 performs the same AI processing as the AI ​​processing unit 42. Specifically, if the AI ​​processing unit 42 performs face detection processing, the AI ​​processing unit 11 also performs face detection processing, and so on.

[0075] 8, the control unit 8B of the AI ​​signal processing unit 3B and the CPU 40 of the camera control unit 4B cooperate to realize the following processing: That is, image quality parameters are derived based on the AI ​​processing results of the image quality adjusted RAW image obtained by having the RAW signal processing unit 10 perform image quality adjustment using candidate parameters, and the derived image quality parameters are set in the development processing unit 41.

[0076] Specifically, the control unit 8B functions as an analysis unit that analyzes the relationship between the candidate parameters and the AI ​​processing results for the image quality adjusted RAW image. The CPU 40 then functions as a derivation setting unit that derives image quality parameters to be set in the development processing unit 41 based on the analysis results obtained by functioning as the analysis unit, and sets the derived image quality parameters in the development processing unit 41. In this example, since the processing related to parameter adjustment is performed for each system as in the first embodiment, the control unit 8B performs processing as the analysis unit for each system, and the CPU 40 also performs processing as the derivation setting unit for each system. Note that the advantages of processing multiple systems are the same as in the first embodiment.

[0077] Here, according to the configuration of FIG. 8 , the analysis unit and the derivation setting unit are configured on separate chips. By adopting such a configuration, for example, in a case where a customer is a manufacturer of an imaging device 1B that prepares its own chip (camera control unit 4B) having a derivation setting unit and an AI processing unit, a chip having an analysis unit (AI signal processing unit 3B) can be sold to the customer. By adopting such a configuration in which a chip having an analysis unit is sold to a customer, even if the customer does not have the know-how to perform image quality adjustments for AI processing, the analysis results of the analysis unit (e.g., information that increasing brightness or contrast will improve the AI ​​processing results) can be provided, and the customer can improve the accuracy of AI processing by preparing a chip having a derivation setting unit.

[0078] A specific example of the processing will be described with reference to the flowchart in Fig. 9. In the figure, the processing indicated as "AI signal processing unit side" is processing executed by the control unit 8B, and the processing indicated as "camera control unit side" is processing executed by the CPU 40. The processing shown in Fig. 9 is repeatedly executed for each frame of a captured image.

[0079] First, in step S121, the control unit 8B inputs the AI ​​processing results, i.e., the AI ​​processing results (AI processing results for RAW images) by the first AI unit 11a to the third AI unit 11c.

[0080] In step S122 following step S121, the control unit 8B generates analysis result information based on the AI ​​processing results. In this example, when executing the process of step S121, the control unit 8B causes each processing unit of the RAW signal processing unit 10 to perform image quality adjustment with candidate parameters set, and in step S121, inputs the AI ​​processing results from each processing unit of the AI ​​processing unit 11 for the image after the image quality adjustment. In the process of step S122, the control unit 8B analyzes the relationship between the candidate parameters and the AI ​​processing score values ​​for each system in multiple past frames. For example, in the case of a brightness parameter, the control unit 8B obtains the AI ​​processing score values ​​when the brightness is changed within a predetermined range over multiple past frames, and generates analysis result information, such as, for example, that increasing the brightness improves the AI ​​processing results, based on the analysis results of how the score values ​​change in response to changes in brightness. Here, the analysis in step S122 can also be performed in the form of assigning multiple types of parameters and identifying the parameters that contribute to an increase in the score value.

[0081] In step S123 following step S122, the control unit 8B transmits the analysis result information to the CPU 40 in the camera control unit 4B. After executing the process of step S123, the control unit 8B ends the series of processes shown in FIG.

[0082] On the camera control unit 4B side, the CPU 40 waits for reception of analysis result information in step S201. When the analysis result information is received, the process proceeds to step S202, where the CPU 40 derives parameters based on the analysis result information. For example, if the analysis result information is information suggesting that a specific parameter be changed in a specific manner to increase the score value, the CPU 40 derives a parameter based on the value of the specific parameter changed in the specific manner. The CPU 40 in this example performs this parameter derivation process for each system based on the analysis result information for each system received from the control unit 8B.

[0083] In step S203 following step S202, the CPU 40 performs parameter adjustment processing by setting the parameters derived in step S202 to the corresponding processing units in the development processing unit 41. After executing the processing of step S203, the CPU 40 ends the series of processing shown in FIG.

[0084] Here, in the third embodiment, the CPU 40 can not only adjust parameters in accordance with analysis information based on the results of AI processing of the RAW image as described above, but also adjust parameters of the development processing unit 41 based on the results of AI processing by the AI ​​processing unit 42. For example, it is conceivable that, in frames subsequent to the frame in which parameters of the development processing unit 41 have been derived and set in accordance with analysis information based on the results of AI processing of the RAW image, image quality parameters may be derived based on the results of AI processing by the AI ​​processing unit 42, and the derived image quality parameters may be set in the development processing unit 41.

[0085] As a result, parameter adjustments of the development processing unit 41 are not only made based on the results of AI processing of the RAW image, but also based on the results of AI processing of the developed image, thereby further improving the accuracy of AI processing.

[0086] 10 is a block diagram showing an example configuration of an image capture device 1C as another example according to the third embodiment. In the image capture device 1C as another example, parameters are derived based on the results of AI processing of a RAW image on the AI ​​signal processing unit side.

[0087] The imaging device 1C differs from the imaging device 1B in that it has an AI signal processing unit 3C instead of the AI ​​signal processing unit 3B, and in that it has a camera control unit 4 instead of a camera control unit 4B.

[0088] Compared to the AI ​​signal processing unit 3B, the AI ​​signal processing unit 3C is similar in that it has a RAW signal processing unit 10 and an AI processing unit 11, but differs in that it has the development processing unit 6 and AI processing unit 7 described in the first and second embodiments, and that it has a control unit 8C instead of the control unit 8B.

[0089] In another example of imaging device 1C, control unit 8C derives image quality parameters based on the results of AI processing of the RAW image, and sets the derived image quality parameters in development processing unit 6. In other words, this is an example in which parameter adjustment for development processing based on the results of AI processing of the RAW image is performed entirely within AI signal processing unit 3C.

[0090] A specific example of the processing procedure of the control unit 8C will be described with reference to Fig. 11. The processing shown in Fig. 11 is also repeatedly executed for each frame. First, in step S131, the control unit 8C inputs the AI ​​processing results of the RAW image. That is, the AI ​​processing results of each system obtained by the AI ​​processing unit 11 are input.

[0091] In step S132 following step S131, the control unit 8C performs parameter derivation processing based on the AI ​​processing results. In this example, it is not necessary to generate analysis result information based on the AI ​​processing results of the RAW image. Instead, a method is adopted in which parameters suitable for AI processing by the AI ​​processing unit 11 (AI processing for the same task as the AI ​​processing unit 7) are derived based on the AI ​​processing results of the RAW image, and the derived parameters are set in the development processing unit 6. In step S131, parameters suitable for AI processing by the AI ​​processing unit 11 are derived for each system based on the AI ​​processing results of the RAW image input in step S131. For example, the parameter derivation here can be performed using a method of deriving parameters so that the score value tends to increase, similar to the parameter derivation in the first embodiment.

[0092] In step S133 following step S132, the control unit 8C performs parameter adjustment processing for the development processing based on the derived parameters. That is, processing is performed to set the parameters derived for each system in step S132 to the processing units of the corresponding systems in the development processing unit 6.

[0093] Here, in the third embodiment, it is possible to derive appropriate image quality parameters in the development processing unit 41 based on the AI ​​processing results for the RAW image before development processing. In other words, it is possible to derive appropriate image quality parameters and perform development processing using those image quality parameters within a single frame period, making it possible to follow changes in the imaging environment without causing frame delays. By improving the ability to follow changes in the imaging environment, it is possible to improve the accuracy of the AI ​​processing.

[0094] Although the parameter adjustment in the third embodiment is described above as an example in which adjustment is performed for each system, it is also possible to configure the parameter adjustment to be performed for only a single system.

[0095] 10 , the AI ​​processing unit 7 that performs AI processing on developed images and the AI ​​processing unit 11 that performs AI processing on RAW images are configured using separate hardware. However, it is also possible to provide a single piece of hardware, such as a single DSP (Digital Signal Processor), to perform AI processing on RAW images and AI processing on developed images in a time-sharing manner. In this case, different AI models are used for the AI ​​processing on RAW images and the AI ​​processing on developed images (because the input data used for learning differs between RAW images and developed images), and therefore the AI ​​model must be switched. By switching between AI models in this way, multiple AI processes can be performed on one captured frame.

[0096] 4. Fourth Embodiment In the fourth embodiment, when the AI ​​processing by the AI ​​processing unit 7 is processing for detecting or recognizing a specific subject, the user is allowed to specify the subject to be detected or recognized by the AI ​​processing unit 7, and parameters are adjusted to be suitable for the specified subject.

[0097] An overview of the parameter adjustment technique according to the fourth embodiment will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is an explanatory diagram of an image capture scene assumed in the fourth embodiment, with Fig. 12B illustrating an image with the aforementioned reference brightness, Fig. 12A illustrating an image whose brightness has been adjusted to be darker than the image with the reference brightness, and Fig. 12C illustrating an image whose brightness has been adjusted to be brighter than the image with the reference brightness. In this example, the captured image includes two people, subjects S31 and S32.

[0098] Here, it is assumed that parameter adjustment for each system is performed as in the first embodiment. Then, with such parameter adjustment for each system performed, a case is assumed in which, as shown in the figures, only subject S31 of subjects S31 and S32 is detected by AI processing in the images of each system shown in Figures 12A to 12C. It is assumed that such a case occurs when, for example, imaging is performed in a dark place, such as at night, resulting in a significant lack of brightness for subject S32.

[0099] In such a case, if the subject that the user wants to detect is not subject S31 but subject S32, parameter adjustment based on the results of AI processing performed in the imaging device will not be able to detect the subject that should actually be detected.

[0100] Therefore, in the fourth embodiment, an external device for an imaging device allows a user to specify a subject to be detected (or recognized) on a display screen, as shown in Fig. 13. In the figure, the frame marked "St" indicates the specified subject. Then, parameters of the development processing unit 6 are adjusted so that image quality suited to the specified subject is achieved.

[0101] 14 is a block diagram showing an example of a system configuration for realizing the parameter adjustment method according to the fourth embodiment described above. In the fourth embodiment, an information processing device 50 is used together with an imaging device 1D shown in the diagram.

[0102] The imaging device 1D differs from the imaging device 1 of the first embodiment in that an AI signal processing unit 3D is provided instead of the AI ​​signal processing unit 3, and the AI ​​signal processing unit 3D differs from the AI ​​signal processing unit 3 in that a control unit 8D is provided instead of the control unit 8. The control unit 8D performs processing to adjust the image quality parameters of the development processing unit 6 to image quality parameters corrected in accordance with correction information (correction values) to be described later that is transmitted from the information processing device 50, and this will be described in detail later.

[0103] The information processing device 50 includes a calculation unit 51, a communication unit 52, a display unit 53, and an operation unit 54. The information processing device 50 may take the form of, for example, a personal computer (PC), a tablet terminal, a smartphone, or the like.

[0104] The calculation unit 51 is configured with a microcomputer having, for example, a CPU, ROM, RAM, etc., and the CPU executes processing in accordance with a program stored in the ROM or a memory other than the ROM (not shown), thereby controlling the entire information processing device 50 and performing various calculation processes.

[0105] The communication unit 52 performs wired or wireless data communication with an external device. In this system, data communication can be performed between the calculation unit 51 and the camera control unit 4 via the communication unit 52 and the communication unit 5 in the imaging device 1D. Furthermore, the calculation unit 51 can also perform data communication with the control unit 8D in the AI ​​signal processing unit 3D via the camera control unit 4.

[0106] The communication with the information processing device 50 may be performed via a network such as the Internet, or may be performed as device-to-device communication according to a predetermined wired communication method such as USB (Universal Serial Bus) or a predetermined wireless communication method such as Bluetooth (registered trademark).

[0107] The display unit 53 is configured as a display device having a display panel capable of displaying images, such as an LCD (Liquid Crystal Display) panel or an organic EL (Electro-Luminescence) panel, and displays various information to the user based on instructions from the calculation unit 51. In the figure, the display unit 53 is shown as being provided in the information processing device 50, but the display unit 53 may be a display device external to the information processing device 50 (separate from the information processing device 50).

[0108] The operation unit 54 comprehensively refers to devices that allow a user to input various operations to the information processing device 50. For example, the operation unit 54 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The operation unit 54 detects a user operation, and a signal corresponding to the detected operation is interpreted by the calculation unit 51.

[0109] 15 , an example of the procedure of processing to be executed on the imaging device 1D side and the information processing device 50 side in order to realize the parameter adjustment method of the fourth embodiment will be described. In the figure, processing shown as "imaging device side" is processing executed by the control unit 8D, and processing shown as "information processing device side" is processing executed by the calculation unit 51.

[0110] First, in step S301, the calculation unit 51 on the information processing device 50 side performs a data request process, requesting the control unit 8D to transmit the AI ​​processing results, image data, and image quality parameters. That is, the calculation unit 51 requests the control unit 8D to transmit the image data that was the subject of AI processing by the AI ​​processing unit 7, the AI ​​processing results indicating the results of the AI ​​processing performed on the image data, and the image quality parameters used by the development processing unit 41 to adjust the image quality of the image data. As can be seen from FIG. 14 , in this example, parameter adjustment in the imaging device 1D is performed for each system (three systems in this case as well) as in the first embodiment, and the calculation unit 51 in this example requests the control unit 8D to transmit the AI ​​processing results, image data, and image quality parameters for each system as the above-mentioned transmission request.

[0111] On the imaging device 1D side, the control unit 8D waits for the request of step S301 in step S141, and if the request is received, performs processing to transmit the AI ​​processing results, image data, and image quality parameter data (data for each system) to the calculation unit 51 in step S142.

[0112] On the information processing device 50 side, the calculation unit 51 waits in step S302 to receive the transmission data of step S142 above, and when the transmission data is received, the process proceeds to step S303 to perform an AI processing result screen display process. That is, the display unit 53 displays the image data received in step S303 and information indicating the detection area of ​​the subject obtained as a result of the AI ​​processing (e.g., a bounding box). The image data displayed here may be at least one of the image data for each system. By displaying the detection area information of the subject obtained as a result of the AI ​​processing together with the image data, the user can recognize whether the subject to be detected has been detected.

[0113] In step S304 following step S303, the calculation unit 51 waits for an object designation operation. If an object designation operation is performed, the processing proceeds to step S305, where the calculation unit 51 calculates correction values ​​for the image quality parameters to enable detection of the specified object. That is, the calculation unit 51 calculates the correction values ​​for the image quality parameters for each system based on the image quality parameters for each system and the image quality information of the image area of ​​the specified object. For example, if the image quality parameter is a brightness parameter, a target value for brightness to enable detection of the object is set in advance, and the difference between the brightness of the image area of ​​the specified object and the target value is calculated for each system, and the difference value for each system is determined as the correction value. For other parameters such as saturation, calculation of correction values ​​based on target values ​​may also be performed in a similar manner.

[0114] In step S306 following step S305, the calculation unit 51 performs a process of transmitting the calculated correction values ​​(correction values ​​for each system) to the control unit 8D. After executing the transmission process of step S306, the calculation unit 51 ends the series of processes shown in FIG. 15 .

[0115] The control unit 8D waits in step S143 to receive the correction value transmitted in step S306, and when the correction value is received, performs processing to store the correction value in step S144. The control unit 8D ends the series of processing shown in FIG. 15 in response to having executed the storage processing in step S144.

[0116] Thereafter, the control unit 8D performs parameter adjustment processing for each system based on the AI ​​processing result, using the correction values ​​for each system stored in step S144. This makes it possible to detect target subjects in each system in cases where a subject that is originally intended to be detected cannot be detected, such as when capturing an image in a dark place.

[0117] Note that, although the above explanation has been given on the assumption that the AI ​​processing by the AI ​​processing unit 7 is processing for detecting a specific subject (e.g., a person), the AI ​​processing by the AI ​​processing unit 7 may also be object recognition processing targeting a specific subject.

[0118] Also, in the fourth embodiment, the parameter adjustment process can be performed for only a single system, rather than for each system.

[0119] Here, the fourth embodiment described above can be said to be one in which the control unit 8D performs the following processes: a transmission process in which the AI ​​processing unit 7 transmits to an external device the captured image used for AI processing, the AI ​​processing result for the captured image, and the image quality parameters used by the development processing unit 6 to adjust the image quality of the captured image, and a reception process in which the external device receives correction information for the image quality parameters calculated based on the captured image, the AI ​​processing result, and the image quality parameters transmitted by the transmission process, for achieving image quality that enables detection or recognition of a user-specified subject in the captured image, and the parameters of the development processing unit 6 are adjusted based on the correction information.

[0120] In the fourth embodiment, it is also conceivable that the correction values ​​may be shared among multiple imaging devices 1D. Fig. 16 illustrates an example of a system configuration in this case. For example, a case is envisioned in which multiple imaging devices 1D capture images of the same location (e.g., the inside of a store or a parking lot) from different angles, such as in a surveillance camera application. In this case, as shown in the figure, communication is performed between one imaging device 1D and an information processing device 50, and a correction value is calculated in the information processing device 50 (calculation unit 51). The correction value is then transmitted to the other imaging devices 1D, and parameter adjustments are performed using the correction value in those imaging devices 1D as well.

[0121] Since each imaging device 1D captures an image of the same location, the correction values ​​can be shared among the imaging devices 1D. Since there is no need to calculate the correction values ​​for each imaging device 1D, the processing load on the information processing device 50 (the calculation unit 51) can be reduced.

[0122] 5. Modifications Note that embodiments are not limited to the specific examples described above, and various modified configurations are possible. For example, it is possible to perform AI processing using fusion data of an RGB image and a distance image obtained by a ToF (Time of Flight) sensor or the like as input data. In this case, signal processing can be adjusted based on the AI ​​processing results based on the RGB image and distance information based on the distance image, and reflected in the development processing. For example, when object detection processing is performed as AI processing, image quality can be adjusted to be specialized for a subject detected at a specific distance. When multiple subjects are detected, it is difficult to select a specific subject and perform image quality adjustment. Therefore, it is preferable to perform image quality adjustment using distance information as described above.

[0123] It is also possible to perform AI processing using fusion data of an RGB image and wavelength analysis information of a subject obtained by a multispectral camera as input data. In this case, signal processing can be adjusted based on the AI ​​processing results based on the RGB image and the wavelength analysis information, and reflected in the development process. For example, it is possible to adjust image quality to be specialized for a subject detected at a specific wavelength. When detecting a subject using an RGB camera, it is difficult to select a specific wavelength and perform image quality adjustment. Therefore, it is preferable to perform adjustment based on wavelength analysis information as described above. For example, a color matrix can be adapted to colors specialized for specific RGB wavelengths.

[0124] Furthermore, when detecting a subject using an RGB camera, it is difficult to detect the subject under extremely low illuminance. Therefore, it is possible to adjust the image quality based on subject information detected in the RGB image and the IR (near-infrared) image. For example, it is possible to adjust the dynamic range to a luminance that allows the subject to be detected in RGB based on the subject information detected in the IR image.

[0125] 6. Summary of the Embodiments As described above, the imaging device (1, 1A, 1B, 1C, 1D) according to the embodiment includes a sensor unit (image sensor 2) that acquires a captured image, an AI processing unit (7, 42) that performs AI processing using an AI model on the captured image, a signal processing unit (development processing unit 6, 41) that adjusts the image quality of the input image to the AI ​​processing unit, and a control unit (8, 8A, 8B, 8C, 8D) that adjusts parameters of the signal processing unit based on the AI ​​processing results by the AI ​​processing unit. By adjusting the parameters of the signal processing unit based on the AI ​​processing results as described above, preprocessing suitable for the AI ​​processing becomes possible. Furthermore, according to the above configuration, the signal processing unit and control unit are provided within the same imaging device, enabling rapid parameter adjustment based on the AI ​​processing results. Therefore, in preprocessing for AI processing, it is possible to improve the speed of tracking environmental changes while achieving image quality adjustment processing suitable for AI processing, thereby improving the accuracy of the AI ​​processing.

[0126] In addition, in an imaging device according to an embodiment, the signal processing unit performs image quality adjustment processing on the captured image using different image quality parameters to obtain multiple image quality-adjusted images, and the AI ​​processing unit performs AI processing on each of the multiple image quality-adjusted images individually. This allows the signal processing unit to obtain image quality-adjusted images with image quality suitable for detecting or recognizing each subject, even if multiple subjects are captured with different image qualities (e.g., brightness) and the AI ​​processing unit cannot detect or recognize all of the subjects with a single image quality adjustment (e.g., brightness adjustment). This allows the AI ​​processing unit to detect or recognize each subject. This improves the accuracy of AI processing.

[0127] Furthermore, the imaging device (1B, 1C) according to the embodiment includes a RAW signal processing unit (11) that performs image quality adjustment on the RAW image of the captured image. The signal processing unit (development processing unit 6, 41) has a development function for the RAW image. The control unit (CPU 40 and control unit 8B, or control unit 8C) derives image quality parameters based on the AI ​​processing results for the image quality-adjusted RAW image obtained by having the RAW signal processing unit perform image quality adjustment using candidate parameters, and sets the derived image quality parameters in the signal processing unit. This configuration enables the signal processing unit to derive appropriate image quality parameters for development processing based on the AI ​​processing results for the RAW image before development processing. In other words, it is possible to derive appropriate image quality parameters and perform development processing using the image quality parameters within a single frame period, enabling tracking of changes in the imaging environment without causing frame delays. By improving tracking of changes in the imaging environment, the accuracy of AI processing can be improved.

[0128] Furthermore, in the imaging device (1B) according to the embodiment, the control unit includes an analysis unit (control unit 8B) that analyzes the relationship between the candidate parameters and the AI ​​processing results for the image quality-adjusted RAW image, and a derivation setting unit (CPU 40) that derives image quality parameters to be set in the signal processing unit based on the analysis results by the analysis unit and sets the derived image quality parameters in the signal processing unit, with the analysis unit and the derivation setting unit being configured on separate chips. As described above, by configuring the analysis unit and the derivation setting unit on separate chips, for example, in a case where a customer is a manufacturer of imaging devices that prepares its own chips having a derivation setting unit and an AI processing unit, it is possible to sell the chip having the analysis unit to the customer. By selling the chip having the analysis unit to the customer in this way, even if the customer does not have the know-how to perform image quality adjustments for AI processing, the analysis results of the analysis unit (e.g., information that increasing the contrast will improve the AI ​​processing results) can be provided, and the customer can improve the accuracy of AI processing by preparing a chip having a derivation setting unit.

[0129] Furthermore, in the imaging device according to the embodiment, the control unit (CPU 40 and control unit 8B or control unit 8C) derives image quality parameters based on the AI ​​processing results for the developed image obtained by the development function of the signal processing unit, and sets the derived image quality parameters in the signal processing unit. As a result, the signal processing unit adjusts the parameters not only based on the AI ​​processing results for the RAW image, but also based on the AI ​​processing results for the developed image. This allows for further improvement in the accuracy of AI processing.

[0130] In addition, in an embodiment of an imaging device (same as 1D), the AI ​​processing is a process for detecting or recognizing a specific subject, and the control unit (same as 8D) performs a transmission process in which the captured image used by the AI ​​processing unit for the AI ​​processing, the AI ​​processing results for the captured image, and the image quality parameters used by the signal processing unit to adjust the image quality of the captured image to an external device, and a reception process in which the external device receives correction information for the image quality parameters calculated based on the captured image, the AI ​​processing results, and the image quality parameters transmitted by the transmission process to achieve an image quality that allows the user-specified subject in the captured image to be detected or recognized, and adjusts the parameters of the signal processing unit based on the correction information. This enables the detection or recognition of subjects that cannot be detected or recognized by the image quality adjustment function of the imaging device. Therefore, the processing accuracy of the AI ​​processing for detecting or recognizing a specific subject can be improved.

[0131] Furthermore, in the imaging device according to the embodiment, the image quality adjustment by the signal processing unit includes adjustment of at least one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction. This allows for appropriate image quality adjustment according to the task content of the AI ​​processing. Therefore, the accuracy of the AI ​​processing can be improved.

[0132] The parameter adjustment method as an embodiment is a parameter adjustment method for an imaging device that includes a sensor unit that obtains a captured image, an AI processing unit that performs AI processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, and is a parameter adjustment method that adjusts parameters of the signal processing unit based on the AI ​​processing result by the AI ​​processing unit. With such a parameter adjustment method, it is possible to obtain the same functions and effects as the imaging device as the above-mentioned embodiment.

[0133] Here, as an embodiment, a program that causes, for example, a CPU, a DSP, or a device including these, to realize the functions of the control unit 8, etc., described with reference to Figures 4, 6, 9, 11, 15, etc., can be considered. That is, the program of the embodiment is a program readable by a computer device in an imaging device that includes a sensor unit that obtains a captured image, an AI processing unit that performs AI processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, and causes the computer device to realize a function of adjusting parameters of the signal processing unit based on the AI ​​processing results by the AI ​​processing unit. Such a program allows the functions of the control unit 8, etc., described above, to be realized in a device that is an imaging device.

[0134] The above-described programs can be pre-recorded on a hard disk drive (HDD) or solid state drive (SSD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the programs can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a Magneto Optical (MO) disc, a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray Disc (registered trademark)), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN or the Internet.

[0135] Furthermore, such a program is suitable for widely providing the parameter adjustment method according to the embodiment, and allows various types of imaging devices to function as devices that implement the parameter adjustment method of the present disclosure.

[0136] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0137] 7. The Present Technology The present technology may also have the following configuration. (1) An imaging device comprising: a sensor unit that obtains a captured image; an AI processing unit that performs AI processing using an AI model on the captured image; a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit; and a control unit that adjusts parameters of the signal processing unit based on a result of the AI ​​processing by the AI ​​processing unit. (2) The imaging device described in (1) above, wherein the signal processing unit obtains a plurality of image quality adjusted images by performing image quality adjustment processing on the captured image using different image quality parameters, and the AI ​​processing unit performs AI processing individually on the plurality of image quality adjusted images. (3) The imaging device described in (1) or (2) above, comprising: a RAW signal processing unit that performs image quality adjustment on a RAW image of the captured image, the signal processing unit having a development function for the RAW image, and the control unit derives image quality parameters based on a result of AI processing on the image quality adjusted RAW image obtained by causing the RAW signal processing unit to perform image quality adjustment using candidate parameters, and sets the derived image quality parameters in the signal processing unit. (4) The imaging device according to (3), wherein the control unit has an analysis unit that analyzes the relationship between the candidate parameters and AI processing results for the image quality adjusted RAW image, and a derivation setting unit that derives image quality parameters to be set in the signal processing unit based on the analysis results by the analysis unit and sets the derived image quality parameters in the signal processing unit, and the analysis unit and the derivation setting unit are configured on separate chips. (5) The imaging device according to (3) or (4), wherein the control unit derives image quality parameters based on AI processing results for a developed image obtained by the development function of the signal processing unit, and sets the derived image quality parameters in the signal processing unit.(6) The imaging device according to any of (1) to (5), wherein the AI ​​processing is processing for detecting or recognizing a specific subject, and the control unit performs a transmission process of transmitting to an external device the captured image used by the AI ​​processing unit for the AI ​​processing, the AI ​​processing result for the captured image, and image quality parameters used by the signal processing unit for adjusting the image quality of the captured image, and a reception process of receiving correction information for the image quality parameters calculated by the external device based on the captured image, the AI ​​processing result, and the image quality parameters transmitted by the transmission process, for achieving image quality that enables detection or recognition of a user-specified subject in the captured image, and adjusts parameters of the signal processing unit based on the correction information. (7) The imaging device according to any of (1) to (6), wherein the image quality adjustment by the signal processing unit includes adjustment of at least any of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction. (8) A parameter adjustment method for an imaging device comprising: a sensor unit that obtains a captured image; an AI processing unit that performs AI processing using an AI model on the captured image; and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, wherein the parameter adjustment method adjusts the parameters of the signal processing unit based on the AI ​​processing result by the AI ​​processing unit. (9) A program readable by a computer device in an imaging device comprising: a sensor unit that obtains a captured image; an AI processing unit that performs AI processing using an AI model on the captured image; and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, wherein the program causes the computer device to realize a function of adjusting the parameters of the signal processing unit based on the AI ​​processing result by the AI ​​processing unit. (10) A recording medium on which a program readable by a computer device is recorded in an imaging device that includes a sensor unit that obtains a captured image, an AI processing unit that performs AI processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, the recording medium on which a program is recorded that causes the computer device to realize a function of adjusting parameters of the signal processing unit based on the AI ​​processing results by the AI ​​processing unit.

[0138] REFERENCE SIGNS LIST 1, 1A, 1B, 1C, 1D Imaging device 2 Image sensor 3, 3A, 3B, 3C, 3D AI signal processing unit 4, 4B Camera control unit 5 Communication unit 6 Development processing unit 6a First processing unit 6b Second processing unit 6c Third processing unit 7 AI processing unit 7a First AI unit 7b Second AI unit 7c Third AI unit 8, 8A, 8B, 8C, 8D Control unit 9 Communication unit 10 RAW signal processing unit 10a First RAW processing unit 10b Second RAW processing unit 10c Third RAW processing unit 11 AI processing unit 11a First AI unit 11b Second AI unit 11c Third AI unit 40 CPU 41 Development processing unit 41a First processing unit 41b Second processing unit 41c Third processing unit 42 AI processing unit 42a First AI unit 42b Second AI unit 42c Third AI unit 50 Information processing device 51 Calculation unit 52 Communication unit 53 Display unit 54 Operation unit

Claims

1. An imaging device comprising: a sensor unit that obtains an image; an AI processing unit that performs AI processing using an AI model on the image; a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit; and a control unit that adjusts parameters of the signal processing unit based on the results of the AI ​​processing by the AI ​​processing unit.

2. The imaging device of claim 1, wherein the signal processing unit obtains a plurality of image quality adjusted images by performing image quality adjustment processing on the captured image using different image quality parameters, and the AI ​​processing unit performs AI processing individually on the plurality of image quality adjusted images.

3. An imaging device as described in claim 1, further comprising a RAW signal processing unit that performs image quality adjustment on a RAW image of the captured image, the signal processing unit having a development function for the RAW image, and the control unit deriving image quality parameters based on AI processing results for the image quality adjusted RAW image obtained by causing the RAW signal processing unit to perform image quality adjustment using candidate parameters, and setting the derived image quality parameters in the signal processing unit.

4. The imaging device described in claim 3, wherein the control unit has an analysis unit that analyzes the relationship between the candidate parameters and AI processing results for the image quality adjusted RAW image, and a derivation setting unit that derives image quality parameters to be set in the signal processing unit based on the analysis results by the analysis unit and sets the derived image quality parameters in the signal processing unit, and the analysis unit and the derivation setting unit are configured on separate chips.

5. The imaging device according to claim 3, wherein the control unit derives image quality parameters based on AI processing results for the developed image obtained by the development function of the signal processing unit, and sets the derived image quality parameters in the signal processing unit.

6. The imaging device of claim 1, wherein the AI ​​processing is a process for detecting or recognizing a specific subject, and the control unit performs a transmission process for transmitting to an external device the captured image used by the AI ​​processing unit for the AI ​​processing, the AI ​​processing result for the captured image, and the image quality parameters used by the signal processing unit to adjust the image quality of the captured image, and a reception process for receiving correction information for the image quality parameters calculated by the external device based on the captured image, the AI ​​processing result, and the image quality parameters transmitted by the transmission process, for achieving an image quality in which a user-specified subject in the captured image can be detected or recognized, and adjusts parameters of the signal processing unit based on the correction information.

7. The imaging device according to claim 1, wherein the image quality adjustment by the signal processing unit includes adjustment of at least any one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction.

8. A parameter adjustment method for an imaging device equipped with a sensor unit that obtains a captured image, an AI processing unit that performs AI processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, the parameter adjustment method comprising: adjusting parameters of the signal processing unit based on the AI ​​processing results by the AI ​​processing unit.

9. A program readable by a computer device in an imaging device equipped with a sensor unit that obtains an image, an AI processing unit that performs AI processing using an AI model on the image, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, the program causing the computer device to realize a function of adjusting parameters of the signal processing unit based on the results of AI processing by the AI ​​processing unit.

10. A recording medium having recorded thereon a program readable by a computer device in an imaging device equipped with a sensor unit that obtains an image, an AI processing unit that performs AI processing using an AI model on the image captured, and a signal processing unit that adjusts the image quality of an input image to the AI ​​processing unit, the recording medium having recorded thereon a program that causes the computer device to realize a function of adjusting parameters of the signal processing unit based on the results of AI processing by the AI ​​processing unit.

Images