Imaging device, parameter adjustment method, program, and recording medium

Abstract

To improve, regarding pre-processing of AI processing, accuracy of the AI processing by improving a speed of following environmental changes while realizing an image quality adjustment processing suitable for the AI processing.SOLUTION: An imaging device according to the present technique includes: a sensor unit that obtains a captured image; an AI processing unit that performs AI processing on the captured image by using an AI model; a signal processing unit that adjusts image quality of the image input to the AI processing unit; and a control unit that performs parameter adjustment of the signal processing unit on the basis of a result of the AI processing by the AI processing unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present technology relates to an imaging device including an AI processing unit that performs AI processing using an AI (Artificial Intelligence) model on an input image, a parameter adjustment method, a program, and a recording medium in the imaging device.

Background Art

[0002] As a process for an imaging image, there is a technique for performing processing using an AI (Artificial Intelligence) model such as a DNN (Deep Neural Network) (hereinafter referred to as "AI processing"). For example, Patent Document 1 below discloses a technique for an imaging device that performs AI processing such as object recognition processing on an imaging image. Specifically, Patent Document 1 discloses a technique for performing AI processing on an imaging image within a sensor unit included in a digital camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, when performing AI processing on an imaging image, it is conceivable to perform preprocessing for image quality adjustment on the imaging image as preprocessing for the AI processing. Conventionally, as such preprocessing, image quality adjustment processing for viewing for a person to view an image has been performed, but there is no guarantee that the preprocessing suitable for viewing is the preprocessing suitable for AI processing. That is, there is no guarantee for improving AI processing accuracy such as recognition accuracy.

[0005] In addition, when performing AI processing in an imaging device, as preprocessing, the adaptability to changes in the imaging environment should also be considered. If the adaptability of image quality adjustment to changes in the imaging environment is not good, the chance of performing AI processing on images inappropriate for the environment increases, leading to a decrease in AI processing accuracy.

[0006] This technology has been made in view of the above circumstances, and aims to improve the AI processing accuracy by achieving an image quality adjustment process suitable for AI processing and improving the follow-up speed to environmental changes in the preprocessing of AI processing.

Means for Solving the Problems

[0007] The imaging device according to this technology includes a sensor unit that obtains an imaging image, an AI processing unit that performs AI processing, which is a process using an AI model on the imaging image, a signal processing unit that adjusts the image quality of the input image for the AI processing unit, and a control unit that adjusts the parameters of the signal processing unit based on the AI processing result by the AI processing unit. By adjusting the parameters of the signal processing unit based on the AI processing result as described above, it becomes possible to perform preprocessing suitable for AI processing. And according to the above configuration, since the signal processing unit and the control unit are provided in the same imaging device, it becomes possible to quickly perform parameter adjustment based on the AI processing result.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0009] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. First Embodiment> (1-1. Example of the Configuration of the Imaging Device) (1-2. Parameter Adjustment Method as the First Embodiment) <2. Second Embodiment> <3. Third Embodiment> <4. Fourth Embodiment> <5. Modification Example> <6. Summary of Embodiments> <7. This Technology>

[0010] <1. First Embodiment> (1-1. Configuration Example of Imaging Device) FIG. 1 is a block diagram showing a configuration example of an imaging device 1 as a first embodiment according to this technology. As shown in the figure, the imaging device 1 includes an image sensor 2, an AI (Artificial Intelligence) signal processing unit 3, a camera control unit 4, and a communication unit 5.

[0011] The image sensor 2 is an example of a sensor unit that obtains a captured image. Here, "imaging" as used in this specification broadly means obtaining image data by capturing a subject using a light-receiving element. The light-receiving element here is not limited to one that receives visible light, but may also include one that receives non-visible light. Also, the image data referred to here is a general term for data composed of a plurality of pixel data. As pixel data, it is not limited to data that indicates the amount of light received from the subject by a gradation value of a predetermined number of gradations, but various data related to the subject such as data indicating the distance to the subject, data indicating polarization information, data indicating temperature, etc. can be considered. In other words, the "image data" obtained by "imaging" includes data as a gradation image that indicates the gradation value of the amount of light received for each pixel, data as a distance image that indicates the distance information to the subject for each pixel, or data as a polarization image that indicates polarization information for each pixel, data as a thermal image that indicates temperature information for each pixel, etc.

[0012] The image sensor 2 is configured as a gradation sensor that obtains the above gradation image, and is configured as, for example, a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. The image sensor 2 in this example is configured to obtain color images of R (red), G (green), and B (blue). Specifically, an R pixel formed with a color filter that selectively receives R light, a G pixel formed with a color filter that selectively receives G light, and a B pixel formed with a color filter that selectively receives B light are two-dimensionally arranged according to a predetermined arrangement rule such as a Bayer array, etc., and a sensor structure is adopted.

[0013] The AI signal processing unit 3 includes a development processing unit 6, an AI processing unit 7, a control unit 8, and a communication I / F (interface) 9. The development processing unit 6 performs development processing on the captured image as a RAW image obtained by the image sensor 2. The development processing here includes at least processing such as demosaicing processing for a RAW image with a Bayer array and processing for converting the RAW image into an RGB color image.

[0014] The AI processing unit 7 performs AI processing, which is processing using an AI model on the captured image. Specifically, in this example, the AI processing unit 7 performs AI processing on the captured image after the development processing by the development processing unit 6.

[0015] Here, the development processing unit 6 has an image quality adjustment function as an image signal processing function. The image quality adjustment here includes at least adjustment of any one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction. The adjustment of brightness is realized, for example, as a process of giving an offset value common to the luminance values of R, G, and B.

[0016] Figure 2 is an explanatory diagram of an example of dynamic range adjustment. Here, an example of dynamic range adjustment with an input of 10 bits and an output of 8 bits is shown. In Figure 2A, an example with the maximum dynamic range is shown, in Figure 2B, an example with the dynamic range narrowed on the low-luminance side is shown, and in Figure 2C, an example with the dynamic range narrowed on the high-luminance side is shown respectively. By performing such dynamic range adjustment, it becomes possible to increase the brightness resolution in the dark part (in the case of Fig. 2B) at the expense of the brightness resolution in the bright part, or conversely, to increase the brightness resolution in the bright part (in the case of Fig. 2C) at the expense of the brightness resolution in the dark part.

[0017] In Fig. 1, the imaging image after image quality adjustment by the development processing unit 6 is input to the AI processing unit 7 as the target image for AI processing. That is, it can be said that the development processing unit 6 performs image quality adjustment on the input image for the AI processing unit 7.

[0018] In addition, the development processing unit 6 also has a function of adjusting the image size as a function of adjusting the input image to the AI processing unit 7. Specifically, the development processing unit 6 has a downsampling function from the image size of the RAW image (for example, 12MP (megapixel), etc.) to the Input Tensor size (for example, 640×480 pixels, etc.) in the AI processing unit 7.

[0019] Here, the development processing unit 6 in this example divides the development processing including image quality adjustment into a plurality of systems. Specifically, the development processing unit 6 has a first processing unit 6a, a second processing unit 6b, and a third processing unit 6c for dividing the development processing including image quality adjustment into three systems. In this example, the first processing unit 6a to the third processing unit 6c are configured to be able to obtain a plurality of images (developed images) after the developed processing with image quality adjustment simultaneously. Note that the method of obtaining a plurality of developed images with image quality adjustment is not limited to the method of providing a plurality of processing units in parallel as described above, and it is also conceivable to adopt a method in which a single processing unit performs development processing (including image quality adjustment) in a time-division manner.

[0020] Also, in this example, the AI processing unit 7 is also configured to be able to execute AI processing by dividing it into multiple systems. Specifically, the AI processing unit 7 in this example has a first AI unit 7a, a second AI unit 7b, and a third AI unit 7c for individually performing AI processing on the developed images of each system obtained by the development processing unit 6. The first AI unit 7a performs AI processing on the developed image obtained by the first processing unit 6a, the second AI unit 7b performs AI processing on the developed image obtained by the second processing unit 6b, and the third AI unit 7c performs AI processing on the developed image obtained by the third processing unit 6c.

[0021] Here, as specific examples of the AI processing in the AI processing unit 7, object detection processing for detecting specific subjects such as people, animals, vehicles, ships, airplanes, etc. in the captured image, object recognition processing for identifying the types of specific subjects, etc. can be cited. As the object recognition processing, for a specific subject as a person, class identification processing for identifying to which class among a predetermined set of classes the specific subject belongs, such as identifying attributes such as gender and age, and face authentication processing for identifying whether the features of a person's face match the pre-specified face features, etc. can be cited.

[0022] In the AI processing unit 7, as the AI model used for AI processing, for example, an AI model having a neural network structure by DNN (Deep Neural Network) or the like and performing machine learning as deep learning can be considered. Note that the AI model is not limited to a model having a neural network structure. For example, a model without a neural network structure such as a Vision Transformer can also be used, as long as it is an AI model that has been machine-learned.

[0023] In the AI processing unit 7, the first AI unit 7a to the third AI unit 7c are each capable of outputting an AI processing result indicating the result of the AI processing and a captured image as the input image for the AI processing. Here, the information on the AI processing result includes, for example, when the AI processing is the object detection processing described above, information indicating the existence area of the specific subject in the image, such as the bounding box of the detected specific subject, and when the AI processing is the object recognition processing, information indicating the identification result of the specific subject, such as information indicating the class of the specific subject. In addition, the information on the AI processing result also includes score information indicating the likelihood of the inference performed as the AI processing, such as the likelihood of the identification result.

[0024] Note that for the AI processing unit 7, it is also conceivable to adopt a configuration in which only a single processing unit is provided and a plurality of input images of different systems are time-division processed by the single processing unit.

[0025] The information on the AI processing result obtained by the AI processing unit 7 is output to the camera control unit 4 provided outside the AI signal processing unit 3 via the communication I / F 9. Here, when the AI processing unit 7 outputs the captured image that is the object of the AI processing, the captured image is output to the camera control unit 4 via the communication I / F 9 together with the AI processing result.

[0026] The control unit 8 is configured to include a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and realizes various functional operations by the CPU executing processing according to the program stored in the ROM. Specifically, the control unit 8 adjusts the parameters of the developing processing unit 6 based on the AI processing result by the AI processing unit 7. In this example, the control unit 8 adjusts the parameters of the first processing unit 6a, the second processing unit 6b, and the third processing unit 6c in the developing processing unit 6 individually based on the AI processing result of the first AI unit 7a for the first processing unit 6a, the AI processing result of the second AI unit 7b for the second processing unit 6b, and the AI processing result of the third AI unit 7c for the third processing unit 6c. Details will be described later.

[0027] In addition, the control unit 8 is capable of performing data communication with the camera control unit 4 via the communication I / F 9.

[0028] The camera control unit 4 is configured to include a microcomputer having a CPU, a ROM, and a RAM, and performs overall control of the imaging device 1 by executing processing according to a program stored in the ROM by the CPU. For example, the camera control unit 4 performs execution control of the imaging operation by the image sensor 2. In addition, the camera control unit 4 can perform execution control of the operations (development processing and AI processing) of the AI signal processing unit 3 by giving an instruction to the control unit 8.

[0029] In addition, a communication unit 5 is connected to the camera control unit 4. The communication unit 5 is configured to be capable of performing wired or wireless data communication with an external device of the imaging device 1. As the communication unit 5, for example, it can also be configured to have a communication function via a network such as the Internet or a LAN (Local Area Network). The camera control unit 4 is capable of performing data communication with an external device via the communication unit 5. For example, the camera control unit 4 can transmit the AI processing result obtained by the AI signal processing unit 3 or the captured image output by the AI signal processing unit 3 to the external device via the communication unit 5.

[0030] Here, the imaging device 1 has an AE (Auto Exposure) function. The AE function is realized by adjusting the aperture (iris) in the imaging optical system (not shown), the shutter speed of the electronic shutter in the image sensor 2, and the ISO sensitivity. The control for realizing the AE function may be performed by the camera control unit 4 or the control unit 8. The control for realizing the AE function is performed based on the result of detecting the captured image by the image sensor 2 so that the brightness adjustment of the image according to a predetermined photometry mode (overall photometry, area photometry, spot photometry, etc.) is realized.

[0031] As described above, the imaging device 1 in the present embodiment has an AI processing function for imaging images. Such an imaging device 1 can be applied, for example, to various uses of surveillance cameras. For example, it can be mentioned for surveillance cameras for indoor areas such as stores, offices, and houses, surveillance cameras for outdoor areas such as parking lots and streets (including traffic surveillance cameras, etc.), surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation), surveillance cameras for monitoring inside and outside vehicles, and other uses.

[0032] For example, in the case of the use of a surveillance camera in a store, it is conceivable to arrange a plurality of imaging devices 1 at predetermined positions within the store so that the user can confirm the customer layer (such as gender and age group) of the customers visiting the store and the behavior (flow line) within the store. In that case, as an analysis using the AI processing result, it is conceivable to generate information on the customer layer of these customers visiting the store, information on the flow line within the store, and information on the congestion state at the checkout register (for example, waiting time information at the checkout register). Alternatively, in the case of the use of a traffic surveillance camera, it is conceivable to arrange a plurality of imaging devices 1 at each position near the road so that information such as the number (license plate number) of the passing vehicle, the color of the car, and the type of the car can be recognized. Also, when a traffic surveillance camera is used in a parking lot, the imaging device 1 is arranged so that each parked vehicle can be monitored, and it is considered to monitor whether there are any suspicious persons around each vehicle. If there is a suspicious person, it is considered to notify the fact that there is a suspicious person and the attributes of the suspicious person (such as gender, age group, clothing, etc.). Furthermore, it is also considered to monitor the available spaces in the street and the parking lot and be able to notify the locations of the spaces where cars can be parked.

[0033] (1-2. Parameter adjustment method as the first embodiment) Here, as described above, in the imaging device 1 of the present embodiment, the development processing unit 6 includes a first processing unit 6a to a third processing unit 6c, and performs development processing on the RAW image in multiple systems. The AI processing unit 7 includes a first AI unit 7a to a third AI unit 7c, and performs AI processing individually on each of the captured images obtained by the multiple systems of development processing. Under this configuration, the control unit 8 in the present embodiment sets different image quality parameters for the first processing unit 6a to the third processing unit 6c respectively, and executes image quality adjustment processing.

[0034] With the above configuration, the development processing unit 6 in the first embodiment is configured to obtain a plurality of image quality adjustment images by performing image quality adjustment processing on the captured image as a RAW image using different image quality parameters respectively. Then, the AI processing unit 7 will perform AI processing individually on those multiple image quality adjustment images.

[0035] With the configuration as the first embodiment as described above, even when a plurality of subjects are imaged with different image qualities and the AI processing unit 7 cannot detect or recognize all the subjects with a single image quality adjustment, it is possible to obtain an image quality adjustment image with an image quality suitable for detecting or recognizing each subject by the development processing unit 6, and it is possible to ensure that the AI processing unit 7 can detect or recognize each subject.

[0036] A specific example will be described with reference to FIG. 3. In FIG. 3, it is a case where human face detection processing is performed as AI processing, and an example of a subject for which detection is successful when different brightness adjustments are performed as image quality adjustments in a case where three subjects (subjects as humans) S11, S12, and S13 are captured in the captured image is shown. The subject for which detection is successful is the subject surrounded by the detection frame Dt in the figure.

[0037] In FIG. 3B, an image with unadjusted brightness in the development processing unit 6 (that is, an image with the brightness set by the AE function) is illustrated. Hereinafter, the brightness of the image in FIG. 3B is referred to as "reference brightness". Here, as an example, the reference brightness is set to the brightness when the brightness in the development processing unit 6 is unadjusted. However, the reference brightness may be any brightness that serves as a reference, and may be a brightness adjusted by a predetermined amount in the development processing unit 6.

[0038] In the image with the reference brightness shown in FIG. 3B, the brightness of the subject S13 is suitable for face detection processing (face detection is possible), the brightness of the subject S12 is darker than the brightness of the subject S13, and face detection is impossible. Also, the brightness of the subject S11 is brighter than the brightness of the subject S13, and face detection is impossible. Therefore, for this image with the reference brightness, only the subject S13 will be detected by the AI processing of the AI processing unit 7.

[0039] The image shown in FIG. 3A is an image whose brightness is adjusted to be darker than the image with the reference brightness shown in FIG. 3B. In this image of FIG. 3A, the brightness of the subject S11, which was the brightest in the image of FIG. 3B, becomes suitable for face detection processing, the brightness of the subject S13 decreases to a brightness at which face detection is impossible, and the brightness of the subject S12 becomes darker than the brightness of the subject S13 and face detection is made impossible. Therefore, for the image of FIG. 3A, only the subject S11 will be detected by the AI processing of the AI processing unit 7.

[0040] The image shown in FIG. 3C is an image whose brightness is adjusted to be brighter than the image with the reference brightness shown in FIG. 3B. In this image of FIG. 3A, the brightness of the subject S12, which was the darkest in the image of FIG. 3B, becomes suitable for face detection processing, the brightness of the subject S13 increases to a brightness at which face detection is impossible, and the brightness of the subject S11 becomes brighter than the brightness of the subject S13 and face detection is made impossible. Therefore, for the image of FIG. 3C, only the subject S12 will be detected by the AI processing of the AI processing unit 7.

[0041] In this way, if the image processing unit 6 performs image quality adjustment processing using different image quality parameters for each captured image to obtain a plurality of image quality adjusted images, and the AI processing unit 7 performs AI processing individually on these plurality of image quality adjusted images, even in a case where there are a plurality of subjects to be detected in the captured image and there is a difference in brightness among these subjects, making it impossible for the AI processing unit 7 to detect all the subjects with a single image quality adjustment, it is possible to ensure that the AI processing unit 7 can detect each subject. Therefore, it is possible to improve the accuracy of AI processing.

[0042] The control unit 8 in the present embodiment sets different image quality parameters for the first processing unit 6a to the third processing unit 6c as described above to execute image quality adjustment processing, and at the same time adjusts the respective image quality parameters of the first processing unit 6a to the third processing unit 6c based on the individual AI processing results of the first AI unit 7a to the third AI unit 7c. Thereby, it is possible to make it possible to detect each subject with different brightnesses and also to cope with changes in the imaging environment.

[0043] Specifically, the control unit 8 adjusts the image quality parameters of the first processing unit 6a based on the score information obtained as the AI processing result of the first AI unit 7a, adjusts the image quality parameters of the second processing unit 6b based on the score information obtained as the AI processing result of the second AI unit 7b, and adjusts the image quality parameters of the third processing unit 6c based on the score information obtained as the AI processing result of the third AI unit 7c. Specifically, as the image quality parameter adjustment for each frame of the captured image, a process of adjusting the brightness parameter is performed so that the score value indicated by the score information tends to increase. At this time, if the score value decreases for an adjustment to darken the brightness, it is conceivable to switch to an adjustment to brighten the brightness, and conversely, if the score value decreases for an adjustment to brighten the brightness, to switch to an adjustment to darken the brightness.

[0044] FIG. 4 is a flowchart showing an example of a specific processing procedure that the control unit 8 (CPU) should execute to implement the parameter adjustment method as the first embodiment described above. Specifically, it is an example of a processing procedure for performing parameter adjustment in the development process (image quality adjustment process) for each system based on the AI processing results of each system. Here, in each system from the first processing unit 6a to the third processing unit 6c, as the adjustable range of brightness, the range of the reference brightness described above, a range brighter than the range of the reference brightness, and a dark range are defined. The control unit 8 in this example performs parameter adjustment for each system within the adjustable range of brightness for each system thus defined. Also, here, it is assumed that the parameter adjustment process is performed for each frame of the captured image. That is, the control unit 8 repeatedly executes the process shown in FIG. 4 for each frame.

[0045] In FIG. 4, at step S101, the control unit 8 inputs the AI processing results. That is, in this example, the AI processing results (score information) by the first AI unit 7a to the third AI unit 7c respectively are input.

[0046] At step S102 following step S101, the control unit 8 performs parameter calculation processing based on the AI processing results. Specifically, for each system, brightness parameters are calculated so that the score value indicated by the score information tends to increase as exemplified above.

[0047] At step S103 following step S102, as the parameter adjustment process, the control unit 8 performs a process of setting the parameters for each system calculated at step S102 to the corresponding processing unit (any one of the first processing unit 6a to the third processing unit 6c) of each system in the development processing unit 6.

[0048] In response to executing the process of step S103, the control unit 8 ends the series of processes shown in FIG. 4.

[0049] Note that the AI processing in the AI processing unit 7 is not limited to the face detection processing exemplified above, and various considerations are possible. For example, it is also conceivable to perform face authentication processing on the image of the detected face area.

[0050] In the case of face authentication processing, not only the brightness of the face but also the color tone of the face, the contour of the face, etc. affect the AI processing accuracy. Therefore, as the parameter adjustment processing shown in FIG. 4, it is also conceivable to adjust parameters related to color (saturation and hue described above) and sharpness based on the AI processing result in addition to brightness.

[0051] <2. Second Embodiment> Next, the second embodiment will be described. In the second embodiment, for the adjustment of brightness, not only the parameter adjustment of the development processing unit 6 but also exposure adjustment is used in combination.

[0052] FIG. 5 is a block diagram showing a configuration example of the imaging device 1A as the second embodiment. In the following description, the same parts as those already described will be denoted by the same reference numerals and the description thereof will be omitted.

[0053] The imaging device 1A is different from the imaging device 1 shown in FIG. 1 in that an AI signal processing unit 3A is provided instead of the AI signal processing unit 3. The AI signal processing unit 3A is different from the AI signal processing unit 3 in that a control unit 8A is provided instead of the control unit 8.

[0054] The control unit 8A is different from the control unit 8 in that, regarding the adjustment of brightness based on the AI processing result, not only the parameter adjustment of brightness in the development processing unit 6 but also exposure adjustment is performed. Here, for the sake of explanation, as an example of exposure adjustment, the adjustment of the shutter speed and ISO sensitivity in the image sensor 2 is performed. However, of course, exposure adjustment including the aperture value can also be performed.

[0055] An example of a specific processing procedure of the control unit 8A is shown in the flowchart of FIG. 6. Also in this case, the brightness adjustment based on the AI processing result is performed for each frame, and the control unit 8A repeatedly executes the processing shown in FIG. 6 for each frame.

[0056] The control unit 8A performs a process of inputting the AI processing result for each system by the process of step S101 described above. Then, in response to executing the input process of step S101, the control unit 8A calculates parameters and exposure values based on the AI processing result in step S110. Here, although the brightness parameters can be individually set for each system, since the exposure value is set for the single image sensor 2, the control unit 8A obtains an exposure value common to each system.

[0057] In response to executing the calculation process of step S110, the control unit 8A executes parameter and exposure adjustment processing in step S111. That is, the brightness parameters calculated for each system in step S110 are set in the corresponding processing units in the developing processing unit 6, and the exposure values (shutter speed and ISO sensitivity in this example) calculated in step S110 are set in the image sensor 2. Here, the exposure value calculated in step S110 is set preferentially over the exposure value calculated by the AE function described above.

[0058] The control unit 8A ends the series of processes shown in FIG. 6 in response to executing the adjustment process of step S111.

[0059] Here, in the description so far, an example in which development processing (image quality adjustment processing) and AI processing are individually performed for a plurality of systems has been given. However, by individually performing development processing and AI processing for a plurality of systems in this way, in addition to the advantage that a plurality of subjects with different brightnesses can be detected or recognized for each system as described above, the following advantages can be said. That is, there is an advantage that the target subject can be detected or recognized in at least one of the systems.

[0060] A specific example will be described with reference to FIG. 7. FIG. 7 shows an example of a captured image obtained in a backlight environment. FIG. 7B illustrates an image of the aforementioned reference brightness, FIG. 7A illustrates an image with a brightness darker than the reference brightness, and FIG. 7C illustrates an image with a brightness brighter than the reference brightness. Here, it is assumed that among the subjects S21 and S22 existing in the captured image, the subject S22 is the subject to be detected.

[0061] For example, as shown in the figure, there may be a case where the target subject (S22) cannot be detected in the images of the brightness of FIG. 7A and FIG. 7B, but can be detected in the image of the brightness of FIG. 7C. That is, even in a case where the target subject cannot be detected by a single brightness adjustment, the target subject can be detected.

[0062] In particular, by including not only the brightness adjustment by the development processing unit 6 but also the exposure adjustment as in the second embodiment, the adjustment range of the brightness can be widened, and it can be achieved that the target subject is more easily detected.

[0063] <3. Third Embodiment> The third embodiment performs parameter adjustment of the development processing unit 6 based on the AI processing result of the RAW image.

[0064] FIG. 8 is a block diagram showing a configuration example of the imaging device 1B as the third embodiment. The imaging device 1B is different from the imaging device 1 shown in FIG. 1 in that an AI signal processing unit 3B is provided instead of the AI signal processing unit 3, and a camera control unit 4B is provided instead of the camera control unit 4. Here, in the example of FIG. 8, it is assumed that the AI signal processing unit 3B and the camera control unit 4B are configured on separate chips.

[0065] The AI signal processing unit 3B is different in that instead of the development processing unit 6, a RAW signal processing unit 10 is provided, instead of the AI processing unit 7, an AI processing unit 11 is provided, and instead of the control unit 8, a control unit 8B is provided, as compared with the AI signal processing unit 3.

[0066] The RAW signal processing unit 10 performs image quality adjustment on the captured image as a RAW image obtained by the image sensor 2. Similar to the case of the development processing unit 6, the image quality adjustment by this RAW signal processing unit 10 also includes at least adjustment of any one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction.

[0067] The AI processing unit 11 uses, as an AI model for performing AI processing, an AI model that has been opportunistically learned using a RAW - format image rather than an RGB image as learning input data.

[0068] Also, the camera control unit 4B is different from the camera control unit 4 in that it has a development function (including an image quality adjustment function) for RAW images and an AI processing function for developed images. Specifically, the camera control unit 4B includes a development processing unit 41 that performs development processing on the RAW image input via the AI signal processing unit 3B, and an AI processing unit 42 that performs AI processing on the developed image output from the development processing unit 41.

[0069] Also, the camera control unit 4B includes a CPU 40. The CPU 40 adjusts the parameters of the development processing unit 41 based on an instruction from the control unit 8B in the AI signal processing unit 3B. Details will be described later.

[0070] In the example of FIG. 8, similar to the case of the first embodiment, a plurality of parameter adjustment processing lines are provided, and parameter adjustment is performed for each line. For this purpose, in the camera control unit 4B, the development processing unit 41 has a first processing unit 41a, a second processing unit 41b, and a third processing unit 41c. Also, the AI processing unit 42 has a first AI unit 42a that performs AI processing on the developed image by the first processing unit 41a, a second AI unit 42b that performs AI processing on the developed image by the second processing unit 41b, and a third AI unit 42c that performs AI processing on the developed image by the third processing unit 41c.

[0071] Note that also in the third embodiment, the processing of each system of development processing and AI processing can adopt a configuration in which a single piece of hardware is used to perform the processing in a time-division manner.

[0072] Also, in this example, the image quality adjustment processing and AI processing for the RAW image in the AI signal processing unit 3B are also divided into a plurality of systems and performed. As shown in the figure, the RAW signal processing unit 10 has a first RAW processing unit 10a, a second RAW processing unit 10b, and a third RAW processing unit 10c. Also, the AI processing unit 11 has a first AI unit 11a that performs AI processing on the image after image quality adjustment by the first RAW processing unit 10a, a second AI unit 11b that performs AI processing on the image after image quality adjustment by the second RAW processing unit 10b, and a third AI unit 11c that performs AI processing on the image after image quality adjustment by the third RAW processing unit 10c.

[0073] Note that also for the processing of each system of image quality adjustment and AI processing for these RAW images, a configuration in which a single piece of hardware is used to perform the processing in a time-division manner can be adopted.

[0074] In the imaging device 1B, the AI processing unit 11 performs the same AI processing as the AI processing unit 42. Specifically, if the AI processing unit 42 performs face detection processing, the AI processing unit 11 also performs face detection processing, etc. The AI processing unit 11 performs AI processing with the same task as the AI processing unit 42.

[0075] In the example of FIG. 8, through the cooperation of the control unit 8B of the AI signal processing unit 3B and the CPU 40 in the camera control unit 4B, the following processing is realized. That is, based on the AI processing result for the image quality adjusted RAW image obtained by causing the RAW signal processing unit 10 to perform image quality adjustment using candidate parameters, image quality parameters are derived, and the derived image quality parameters are set in the development processing unit 41.

[0076] Specifically, the control unit 8B has a function as an analysis unit that analyzes the relationship between the above candidate parameters and the AI processing result for the image quality adjusted RAW image. Then, the CPU 40 has a function as a derivation setting unit that derives the image quality parameters to be set in the development processing unit 41 based on the analysis result obtained by the function as the above analysis unit, and sets the derived image quality parameters in the development processing unit 41. In this example, in order to perform the processing related to parameter adjustment for each system in the same manner as in the first embodiment, the control unit 8B performs the processing as the above analysis unit for each system, and the CPU 40 also performs the processing as the above derivation setting unit for each system. The merits of performing the processing for a plurality of systems are the same as in the case of the first embodiment.

[0077] Here, according to the configuration of FIG. 8, the analysis unit and the derivation setting unit are configured on separate chips. By adopting such a configuration, for example, when the manufacturer of the imaging device 1B that prepares the chip (camera control unit 4B) having the derivation setting unit and the AI processing unit by itself is regarded as the customer, the chip (AI signal processing unit 3B) having the analysis unit can be sold to the customer. By adopting the form of selling the chip having the analysis unit to the customer, even if the customer does not have the know-how on what kind of image quality adjustment should be performed for AI processing, the analysis result of the analysis unit (for example, information such as the AI processing result will be improved if the brightness or contrast is increased) can be provided, so that the customer side can improve the accuracy of AI processing by preparing the chip having the derivation setting unit.

[0078] Referring to the flowchart of FIG. 9, a specific processing example will be described. In the figure, the process indicated as "on the AI signal processing unit side" is the process executed by the control unit 8B, and the process indicated as "on the camera control unit side" is the process executed by the CPU 40. The process shown in FIG. 9 is repeatedly executed for each frame of the captured image.

[0079] First, in step S121, the control unit 8B inputs the AI processing result. That is, each AI processing result (AI processing result for the RAW image) by the first AI unit 11a to the third AI unit 11c is input.

[0080] In step S122 following step S121, the control unit 8B generates analysis result information based on the AI processing result. In this example, when executing the process of step S121, the control unit 8B causes each processing unit of the RAW signal processing unit 10 to execute image quality adjustment with candidate parameters set respectively, and in step S121, the AI processing results by each processing unit of the AI processing unit 11 are input for the image after the image quality adjustment. In the process of step S122, the control unit 8B analyzes the relationship between the candidate parameters and the AI processing score values in a plurality of past frames for each system. For example, for the brightness parameter, the score value of the AI processing when the brightness is changed within a predetermined range over a plurality of past frames is acquired, and from the analysis result of the change mode of the score value with respect to the change in brightness, for example, analysis result information such as that the AI processing result becomes better when the brightness is increased is generated. Here, as the analysis in step S122, it is also conceivable to perform analysis in a form of varying a plurality of types of parameters and specifying the parameters that contribute to the increase in the score value among them.

[0081] In step S123 following step S122, the control unit 8B transmits the analysis result information to the CPU 40 in the camera control unit 4B. In response to executing the process of step S123, the control unit 8B ends the series of processes shown in FIG. 9.

[0082] On the camera control unit 4B side, the CPU 40 waits for the reception of the analysis result information in step S201. When the analysis result information is received, the process proceeds to step S202 to derive parameters based on the analysis result information. For example, when the analysis result information is information suggesting that a specific parameter is to be changed in a specific manner to increase the score value, for the specific parameter, a parameter with a value changed in the specific manner is derived. The CPU 40 in this example performs such parameter derivation processing for each system based on the analysis result information for each system received from the control unit 8B.

[0083] In step S203 following step S202, the CPU 40 performs, as parameter adjustment processing, the process of setting the parameters derived in step S202 to the corresponding processing units in the developing processing unit 41. In response to executing the process of step S203, the CPU 40 ends the series of processes shown in FIG. 9.

[0084] Here, in the third embodiment, the CPU 40 can not only perform parameter adjustment according to the analysis information based on the AI processing result of the RAW image as described above, but also perform parameter adjustment of the developing processing unit 41 based on the AI processing result by the AI processing unit 42. For example, in the frames after the next frame of the frame in which parameter derivation and parameter setting of the developing processing unit 41 are performed according to the analysis information based on the AI processing result of the RAW image, derivation of image quality parameters based on the AI processing result by the AI processing unit 42, and setting of the derived image quality parameters to the developing processing unit 41 can be considered.

[0085] As a result, as parameter adjustment of the developing processing unit 41, not only adjustment based on the AI processing result of the RAW image but also adjustment based on the AI processing result of the developed image is performed, and further improvement in the accuracy of AI processing can be achieved.

[0086] FIG. 10 is a block diagram showing a configuration example of the imaging device 1C as another example in the third embodiment. As an alternative example, the imaging device 1C performs parameter derivation based on the AI processing result of the RAW image on the AI signal processing unit side.

[0087] The imaging device 1C is different from the imaging device 1B in that an AI signal processing unit 3C is provided instead of the AI signal processing unit 3B, and a camera control unit 4 is provided instead of the camera control unit 4B.

[0088] The AI signal processing unit 3C is similar to the AI signal processing unit 3B in that it includes a RAW signal processing unit 10 and an AI processing unit 11, but is different in that it includes a development processing unit 6 and an AI processing unit 7 described in the first and second embodiments, and in that it includes a control unit 8C instead of the control unit 8B.

[0089] In the imaging device 1C as an alternative example, the control unit 8C performs a process of deriving image quality parameters based on the AI processing result of the RAW image and a process of setting the derived image quality parameters in the development processing unit 6. That is, this is an example in which parameter adjustment of the development process based on the AI processing result of the RAW image is completed within the AI signal processing unit 3C.

[0090] Referring to FIG. 11, a specific processing procedure example of the control unit 8C will be described. The process shown in FIG. 11 is also repeatedly executed for each frame. First, in step S131, the control unit 8C inputs the AI processing result of the RAW image. That is, the AI processing results of each system obtained by the AI processing unit 11 are input.

[0091] In step S132 following step S131, the control unit 8C performs a parameter derivation process based on the AI processing result. In this example, generation of analysis result information based on the AI processing result of the RAW image is not essential, and a method is adopted in which parameters suitable for AI processing (AI processing of the same task as the AI processing unit 7) by the AI processing unit 11 are derived based on the AI processing result of the RAW image, and the derived parameters are set in the development processing unit 6. In step S131, based on the AI processing results of the RAW images input in step S131, parameters suitable for the AI processing by the AI processing unit 11 are derived for each system. For example, as the parameter derivation here, similar to the parameter derivation in the first embodiment, a method of deriving parameters so that the score value tends to increase can be considered.

[0092] In step S133 following step S132, the control unit 8C performs parameter adjustment processing for the development processing based on the derived parameters. That is, the parameters derived for each system in step S132 are set in the corresponding system processing units in the development processing unit 6.

[0093] Here, in the third embodiment, it is possible to derive appropriate image quality parameters in the development processing unit 41 based on the AI processing results of the RAW images before the development processing. That is, within a single frame period, it is possible to derive appropriate image quality parameters and perform development processing using the image quality parameters, and it is possible to follow changes in the imaging environment without causing frame delay. By improving the followability to changes in the imaging environment, the accuracy of the AI processing can be improved.

[0094] Note that in the above, an example of performing parameter adjustment for each system in the parameter adjustment of the third embodiment has been given, but a configuration in which parameter adjustment is performed only for a single system can also be adopted.

[0095] Also, in the configuration example shown in FIG. 10, an example of using separate hardware for the AI processing unit 7 that performs AI processing on the developed images and the AI processing unit 11 that performs AI processing on the RAW images has been given. However, it is also conceivable to provide only a single hardware such as a single DSP (Digital Signal Processor) as the hardware that executes the AI processing, and adopt a configuration in which the AI processing on the RAW images and the AI processing on the developed images are executed by the hardware in a time-division manner. At this time, different AI models will be used for the AI processing of RAW images and the AI processing of developed images (because the input data used for learning is different between RAW images and developed images), so the AI model will be switched. By switching the AI model in this way, multiple AI processes can be performed on one captured frame.

[0096] <4. Fourth Embodiment> In the fourth embodiment, when the AI processing by the AI processing unit 7 is the processing of detecting or recognizing a specific subject, the user is made to specify the subject to be detected or recognized by the AI processing unit 7, and the parameters are adjusted to be suitable for the specified subject.

[0097] With reference to FIGS. 12 and 13, the outline of the parameter adjustment method as the fourth embodiment will be described. FIG. 12 is an explanatory diagram of the imaging scene assumed in the fourth embodiment. FIG. 12B is an image in which the brightness of the above-mentioned reference brightness image is adjusted darker than that of FIG. 12A, and FIG. 12C is an image in which the brightness of the reference brightness image is adjusted brighter. Here, as an example, the captured image shows two persons, subjects S31 and S32.

[0098] Here, it is assumed that parameter adjustment is performed for each system in the same manner as in the first embodiment. And under the condition that such parameter adjustment is performed for each system, in the images of each system from FIG. 12A to FIG. 12C as shown in the figure, a case is assumed in which only the subject S31 among the subjects S31 and S32 is detected by AI processing. Such a case is assumed to occur, for example, due to imaging in a dark place such as at night, where the brightness of the subject S32 is significantly insufficient.

[0099] In such a case, if the subject that the user wants to detect is subject S32 instead of subject S31, in the parameter adjustment based on the AI processing result performed in the imaging device, it will not be possible to detect the subject that should originally be the detection target.

[0100] Therefore, in the fourth embodiment, in an external device of the imaging device, as illustrated in FIG. 13, the user is made to specify on the display screen the subject to be detected (or recognized). In the figure, the frame indicated as "St" is the frame indicating the specified subject. Then, the parameter adjustment of the development processing unit 6 is performed so as to obtain an image quality suitable for the subject specified in this way.

[0101] FIG. 14 is a block diagram showing an example of a system configuration for realizing the parameter adjustment method as the fourth embodiment described above. In the fourth embodiment, together with the imaging device 1D shown in the figure, the information processing device 50 is used.

[0102] The imaging device 1D is different in that an AI signal processing unit 3D is provided instead of the AI signal processing unit 3 as compared with the imaging device 1 of the first embodiment, and the AI signal processing unit 3D is different in that a control unit 8D is provided instead of the control unit 8 as compared with the AI signal processing unit 3. The control unit 8D performs a process of adjusting the image quality parameter of the development processing unit 6 to an image quality parameter corrected according to correction information (correction value) described later transmitted from the information processing device 50 side, which will be described in detail later.

[0103] The information processing device 50 includes an arithmetic unit 51, a communication unit 52, a display unit 53, and an operation unit 54. As the device form of the information processing device 50, for example, forms such as a PC (personal computer), a tablet terminal, a smartphone, etc. are conceivable.

[0104] The calculation unit 51 is configured to include a microcomputer having, for example, a CPU, a ROM, a RAM, etc. The CPU executes processing according to a program stored in the ROM or a memory other than the ROM (not shown), thereby performing overall control of the information processing apparatus 50 and various calculation processes.

[0105] The communication unit 52 performs wired or wireless data communication with an external device. In this system, data communication can be performed between the calculation unit 51 and the camera control unit 4 via this communication unit 52 and the communication unit 5 in the imaging device 1D. Further, the calculation unit 51 can also perform data communication with the control unit 8D in the AI signal processing unit 3D through the camera control unit 4.

[0106] Note that the communication with the information processing apparatus 50 can be considered to be, for example, communication via a network such as the Internet. Alternatively, this communication can also be performed as device - to - device communication according to a predetermined wired communication method such as USB (Universal Serial Bus) or a predetermined wireless communication method such as Bluetooth (registered trademark).

[0107] The display unit 53 is configured as a display device having a display panel capable of image display, such as an LCD (Liquid Crystal Display) panel or an organic EL (Electro - Luminescence) panel, and performs various information displays for the user based on an instruction from the calculation unit 51. In the figure, an example is shown where the display unit 53 is provided in the information processing apparatus 50, but the display unit 53 may be an externally attached (separate from the information processing apparatus 50) display device.

[0108] The operation unit 54 comprehensively shows devices for the user to perform various operation inputs on the information processing apparatus 50. For example, as the operation unit 54, various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed. An operation of the user is detected by the operation unit 54, and a signal corresponding to the detected operation is interpreted by the calculation unit 51.

[0109] Referring to the flowchart of FIG. 15, an example of the procedure of the processing to be executed on the imaging device 1D side and the information processing device 50 side to implement the parameter adjustment method as the fourth embodiment will be described. In the figure, the processing indicated as "on the imaging device side" is the processing executed by the control unit 8D, and the processing indicated as "on the information processing device side" is the processing executed by the arithmetic unit 51.

[0110] First, on the information processing device 50 side, in step S301, the arithmetic unit 51 performs a process of requesting the control unit 8D to transmit the AI processing result, image data, and image quality parameters as a data request process. That is, a request is made to transmit each data of the image data targeted for AI processing by the AI processing unit 7, the AI processing result indicating the result of the AI processing performed on the image data, and the image quality parameters used by the developing processing unit 41 for image quality adjustment of the image data. As can be understood by referring to FIG. 14 above, in this example, the parameter adjustment in the imaging device 1D is performed for each system (also three systems in this case) in the same manner as in the first embodiment, and the arithmetic unit 51 in this example makes a request to transmit the AI processing result, image data, and image quality parameters for each system as the above transmission request.

[0111] On the imaging device 1D side, the control unit 8D waits for the request in step S301 in step S141, and when the request is received, in step S142, it performs a process of transmitting each data of the AI processing result, image data, and image quality parameters (data for each system) to the arithmetic unit 51.

[0112] On the side of the information processing apparatus 50, the arithmetic unit 51 waits for the reception of the transmission data in step S302 of the above step S142. When the transmission data is received, the process proceeds to step S303 to perform the AI processing result screen display process. That is, the display unit 53 is caused to display the image data received in step S303 and the information indicating the detection area of the subject obtained as the AI processing result (for example, a bounding box). The image data to be displayed here may be at least any one of the image data for each system. By displaying the detection area information of the subject obtained as the AI processing result together with the image data, the user can be made to recognize whether or not the subject to be detected has been detected.

[0113] In step S304 following step S303, the arithmetic unit 51 waits for a subject designation operation. When there is a subject designation operation, the process proceeds to step S305 to calculate a correction value for the image quality parameter for making the designated subject detectable. That is, based on the image quality parameter for each system and the image quality information of the image area of the designated subject, a correction value for the image quality parameter is calculated for each system. For example, when the image quality parameter is a brightness parameter, a target value of brightness for making the subject detectable is determined in advance. For each system, the difference between the brightness of the image area of the designated subject and the target value is calculated, and the value of the difference for each system is obtained as the correction value. It is conceivable to perform the calculation of the correction value based on the target value in the same manner for other parameters such as saturation.

[0114] In step S306 following step S305, the arithmetic unit 51 performs the process of transmitting the calculated correction value (correction value for each system) to the control unit 8D. The arithmetic unit 51 ends the series of processes shown in FIG. 15 in response to the execution of the transmission process in step S306.

[0115] The control unit 8D waits for the reception of the correction value transmitted in step S306 in step S143. When the correction value is received, the control unit 8D performs the process of storing the correction value in step S144. In response to having executed the storage process in step S144, the control unit 8D finishes a series of processes shown in FIG. 15.

[0116] Thereafter, the control unit 8D performs parameter adjustment processing for each system based on the AI processing result, using the correction value for each system stored in step S144. Thereby, it is possible to aim to detect the target subject in each system in a case where the subject that is originally desired to be detected, such as when imaging in a dark place, cannot be detected.

[0117] Note that, in the above description, the explanation has been made on the premise that the AI processing by the AI processing unit 7 is processing for detecting a specific subject (for example, a person), but the AI processing by the AI processing unit 7 may be object recognition processing targeting a specific subject.

[0118] Also, in the fourth embodiment as well, the parameter adjustment processing can be processing for only a single system instead of processing for each system.

[0119] Here, it can be paraphrased that the above-described fourth embodiment is one in which the control unit 8D performs the following processing. That is, a transmission process in which the AI processing unit 7 transmits the captured image used for AI processing, the AI processing result for the captured image, and the image quality parameters used by the image development processing unit 6 for image quality adjustment of the captured image to an external device, and a reception process in which the external device calculates correction information of the image quality parameters for making the image quality capable of detecting or recognizing the user-specified subject in the captured image based on the captured image, the AI processing result, and the image quality parameters transmitted by the transmission process, and the parameter adjustment of the image development processing unit 6 is performed based on the correction information.

[0120] Note that, in the fourth embodiment, it is also conceivable to share the correction value among a plurality of imaging devices 1D. FIG. 16 illustrates the system configuration in that case. For example, in the case of a surveillance camera application or the like, a case is assumed in which a plurality of imaging devices 1D image the same location (for example, inside a store, a parking lot, etc.) at different angles. In this case, communication is performed between one imaging device 1D and the information processing device 50 as shown in the figure, and a correction value is calculated in the information processing device 50 (arithmetic unit 51). Then, the correction value is also transmitted to other imaging devices 1D, and parameter adjustment using the correction value is also performed in those imaging devices 1D.

[0121] Since each imaging device 1D images the same location, the correction value can be shared among the imaging devices 1D. Since it is not necessary to calculate the correction value for each imaging device 1D, the processing load on the information processing device 50 (arithmetic unit 51) can be reduced.

[0122] <5. Modification Example> Note that the embodiment is not limited to the specific examples described above, and configurations as various modification examples can be adopted. For example, it is conceivable to perform AI processing using fusion data of an RGB image and a distance image obtained by a ToF (Time of Flight) sensor or the like as input data. In that case, signal processing can be adjusted based on the AI processing result based on the RGB image and the distance information based on the distance image and reflected in the development processing. For example, if object detection processing is performed as the AI processing, adjustment can be made to the image quality specialized for the subject detected at a specific distance. When a plurality of subjects are detected, it becomes difficult to select a specific subject and perform image quality adjustment. Therefore, it is preferable to perform image quality adjustment using the distance information as described above.

[0123] Also, it is conceivable to perform AI processing using fusion data of an RGB image and wavelength analysis information of a subject obtained by a multispectral camera as input data. In that case, signal processing can be adjusted based on the AI processing result based on the RGB image and the wavelength analysis information and reflected in the development processing. For example, it is conceivable to adjust the image quality specialized for the subject detected at a specific wavelength. When detecting a subject using an RGB camera, it is difficult to select a specific wavelength for image quality adjustment. Therefore, it is preferable to perform adjustment based on the wavelength analysis information as described above. For example, a color matrix can be adapted to a color specialized for a specific wavelength in RGB.

[0124] Also, when detecting a subject with an RGB camera, it is difficult to detect the subject under extremely low illuminance. Therefore, it is conceivable to perform image quality adjustment based on the subject information detected in the RGB image and the IR (near-infrared) image. For example, based on the subject information detected in the IR image, it is conceivable to adjust the dynamic range to a luminance at which the subject can be detected in RGB.

[0125] <6. Summary of Embodiments> As described above, the imaging device (the same as 1, 1A, 1B, 1C, 1D) as an embodiment includes a sensor unit (image sensor 2) that obtains an imaging image, an AI processing unit (the same as 7, 42) that performs AI processing which is processing using an AI model on the imaging image, a signal processing unit (development processing unit 6, 41) that adjusts the image quality of the input image to the AI processing unit, and a control unit (the same as 8, 8A, 8B, 8C, 8D) that adjusts the parameters of the signal processing unit based on the AI processing result by the AI processing unit. By adjusting the parameters of the signal processing unit based on the AI processing result as described above, it becomes possible to perform preprocessing suitable for the AI processing. And according to the above configuration, since the signal processing unit and the control unit are provided in the same imaging device, it becomes possible to quickly perform parameter adjustment based on the AI processing result. Therefore, regarding the preprocessing of the AI processing, it becomes possible to improve the follow-up speed to environmental changes while realizing an image quality adjustment process suitable for the AI processing, and to improve the accuracy of the AI processing.

[0126] In addition, in the imaging device according to the embodiment, the signal processing unit obtains a plurality of image quality adjustment images by performing image quality adjustment processing on the captured image using different image quality parameters respectively, and the AI processing unit performs AI processing individually on the plurality of image quality adjustment images. As a result, even when a plurality of subjects are captured with different image qualities (for example, brightness), and the AI processing unit cannot detect or recognize all the subjects with a single image quality adjustment (for example, brightness adjustment), it is possible to obtain an image quality adjustment image with an image quality suitable for detecting or recognizing each subject by the signal processing unit, and it is possible to ensure that the AI processing unit can detect or recognize each subject. Therefore, the accuracy of the AI processing can be improved.

[0127] In addition, in the imaging device (the same as 1B, 1C) according to the embodiment, a RAW signal processing unit (the same as 11) that performs image quality adjustment on the RAW image of the captured image is provided. The signal processing unit (the development processing units 6, 41) has a development function for the RAW image. The control unit (CPU 40 and the control unit 8B, or the control unit 8C) derives image quality parameters based on the AI processing result of the image quality adjustment RAW image obtained by causing the RAW signal processing unit to perform image quality adjustment with candidate parameters, and sets the derived image quality parameters in the signal processing unit. According to the above configuration, it is possible to derive appropriate image quality parameters in the development processing by the signal processing unit based on the AI processing result of the RAW image before the development processing. That is, within a single frame period, it is possible to derive appropriate image quality parameters and perform development processing using the image quality parameters, and it is possible to follow changes in the imaging environment without causing frame delay. By improving the followability to changes in the imaging environment, the accuracy of the AI processing can be improved.

[0128] Furthermore, in the imaging device (same as 1B) as an embodiment, the control unit includes an analysis unit (control unit 8B) that analyzes the relationship between candidate parameters and the AI processing results for the image quality adjustment RAW image, and a derivation setting unit (CPU 40) that derives the image quality parameters to be set in the signal processing unit based on the analysis results by the analysis unit and sets the derived image quality parameters in the signal processing unit. The analysis unit and the derivation setting unit are configured by separate chips. As described above, since the analysis unit and the derivation setting unit are configured by separate chips, for example, when a manufacturer of an imaging device that prepares a chip having the derivation setting unit and the AI processing unit by itself is regarded as a customer, it is possible to adopt a form of selling the chip having the analysis unit to the customer. By adopting such a form of selling the chip having the analysis unit to the customer, even if the customer does not have the know-how of what image quality adjustment should be performed for AI processing, by providing the analysis results of the analysis unit (for example, information such as that the AI processing results will be improved if the contrast is increased), the customer side can improve the accuracy of AI processing by preparing a chip having the derivation setting unit.

[0129] Furthermore, in the imaging device as an embodiment, the control unit (CPU 40 and control unit 8B, or control unit 8C) derives image quality parameters based on the AI processing results for the developed image obtained by the developing function of the signal processing unit, and sets the derived image quality parameters in the signal processing unit. Thereby, as parameter adjustment of the signal processing unit, not only adjustment based on the AI processing results of the RAW image but also adjustment based on the AI processing results of the developed image is performed. Therefore, further improvement in the accuracy of AI processing can be achieved.

[0130] In addition, in the imaging device (same as 1D) as an embodiment, the AI processing is a process of detecting or recognizing a specific subject, and the control unit (same as 8D) performs a transmission process of transmitting to an external device the captured image used by the AI processing unit for AI processing, the AI processing result for the captured image, and the image quality parameters used by the signal processing unit for image quality adjustment of the captured image, and a reception process of receiving correction information of the image quality parameters for making the image quality such that a user-specified subject in the captured image can be detected or recognized, which is calculated by the external device based on the captured image, the AI processing result, and the image quality parameters transmitted by the transmission process, and performs parameter adjustment of the signal processing unit based on the correction information. Thus, it is intended that even a subject that cannot be detected or recognized by the image quality adjustment function on the imaging device side can be detected or recognized. Therefore, for the AI processing of detecting or recognizing a specific subject, the processing accuracy can be improved.

[0131] Furthermore, in the imaging device as an embodiment, the image quality adjustment by the signal processing unit includes at least adjustment of any one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction. Thereby, it becomes possible to perform an appropriate type of image quality adjustment according to the task content of the AI processing. Therefore, the accuracy of the AI processing can be improved.

[0132] The parameter adjustment method as an embodiment is a parameter adjustment method in an imaging device including a sensor unit that obtains a captured image, an AI processing unit that performs AI processing which is a process using an AI model on the captured image, and a signal processing unit that performs image quality adjustment of the input image to the AI processing unit, and is a parameter adjustment method of performing parameter adjustment of the signal processing unit based on the AI processing result by the AI processing unit. Also by such a parameter adjustment method, the same operations and effects as those of the imaging device as the above-described embodiment can be obtained.

[0133] Here, as an embodiment, a program that realizes the functions of the control unit 8 and the like described with reference to FIGS. 4, 6, 9, 11, 15, etc. in a device such as a CPU, a DSP, or a device including these can be considered. That is, the program of the embodiment is a program readable by a computer device in an imaging device including a sensor unit that obtains an imaging image, an AI processing unit that performs AI processing, which is processing using an AI model on the imaging image, and a signal processing unit that adjusts the image quality of the input image to the AI processing unit, and causes the computer device to realize a function of adjusting the parameters of the signal processing unit based on the AI processing result by the AI processing unit. With such a program, the functions of the control unit 8 and the like described above can be realized in a device as an imaging device.

[0134] The above program can be pre-recorded in an HDD (Hard Disc Drive), an SSD (Solid State Drive) as a recording medium built in a device such as a computer device, or a ROM in a microcomputer having a CPU. Alternatively, it can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such a removable recording medium can be provided as so-called packaged software. Also, such a program can be installed from a removable recording medium to a personal computer or the like, or downloaded from a download site via a network such as a LAN or the Internet.

[0135] Also, according to such a program, it is suitable for providing a wide range of parameter adjustment methods as an embodiment. Imaging devices in various forms can be made to function as devices that implement the parameter adjustment method of the present disclosure.

[0136] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0137] <7. The present technology> The present technology can also adopt the following configuration. (1) A sensor unit that obtains a captured image, An AI processing unit that performs AI processing, which is processing using an AI model on the captured image, A signal processing unit that adjusts the image quality of the input image to the AI processing unit, A control unit that adjusts the parameters of the signal processing unit based on the AI processing result by the AI processing unit, and includes An imaging device. (2) The signal processing unit Obtains a plurality of image quality adjustment images by performing image quality adjustment processing using different image quality parameters on the captured image, The AI processing unit Performs AI processing individually on the plurality of image quality adjustment images The imaging device according to (1) above. (3) Includes a RAW signal processing unit that performs image quality adjustment on the RAW image of the captured image, The signal processing unit has a developing function for the RAW image, The control unit Derives image quality parameters based on the AI processing result of the image quality adjustment RAW image obtained by causing the RAW signal processing unit to perform image quality adjustment with candidate parameters, and sets the derived image quality parameters in the signal processing unit The imaging device according to (1) or (2) above. (4) The control unit An analysis unit that analyzes the relationship between the candidate parameter and the AI processing result for the image quality adjustment RAW image, and a derivation setting unit that derives the image quality parameter to be set in the signal processing unit based on the analysis result by the analysis unit and sets the derived image quality parameter in the signal processing unit, wherein the analysis unit and the derivation setting unit are configured by separate chips The imaging device according to (3) above (5) The control unit Derive an image quality parameter based on the AI processing result for the developed image obtained by the development function of the signal processing unit, and set the derived image quality parameter in the signal processing unit The imaging device according to (3) or (4) above (6) The AI processing is a process of detecting or recognizing a specific subject The control unit A transmission process of transmitting the captured image used by the AI processing unit for AI processing, the AI processing result for the captured image, and the image quality parameter used by the signal processing unit for image quality adjustment of the captured image to an external device, A reception process of receiving correction information of the image quality parameter for making the image quality capable of detecting or recognizing a user-specified subject in the captured image calculated by the external device based on the captured image, the AI processing result, and the image quality parameter transmitted by the transmission process, Perform parameter adjustment of the signal processing unit based on the correction information The imaging device according to any one of (1) to (5) above (7) The image quality adjustment by the signal processing unit includes at least adjustment of any one of brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction The imaging device according to any one of (1) to (6) above (8) A parameter adjustment method in an imaging device, comprising a sensor unit that obtains a captured image, an AI processing unit that performs AI processing, which is processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of the input image for the AI processing unit. Based on the AI processing result by the AI processing unit, the parameters of the signal processing unit are adjusted. Parameter adjustment method. (9) A computer-readable program in an imaging device, comprising a sensor unit that obtains a captured image, an AI processing unit that performs AI processing, which is processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of the input image for the AI processing unit. Based on the AI processing result by the AI processing unit, the computer device is caused to realize a function of adjusting the parameters of the signal processing unit. Program. (10) A recording medium on which a computer-readable program in an imaging device is recorded, the imaging device comprising a sensor unit that obtains a captured image, an AI processing unit that performs AI processing, which is processing using an AI model on the captured image, and a signal processing unit that adjusts the image quality of the input image for the AI processing unit. A recording medium on which a program for causing the computer device to realize a function of adjusting the parameters of the signal processing unit based on the AI processing result by the AI processing unit is recorded. Recording medium.

Explanation of reference numerals

[0138] 1, 1A, 1B, 1C, 1D Imaging device 2 Image sensor 3, 3A, 3B, 3C, 3D AI signal processing unit 4, 4B Camera control unit 5 Communication unit 6 Development processing unit 6a First processing unit 6b Second processing unit 6c Third processing unit 7 AI processing unit 7a First AI unit 7b Second AI Unit 7c Third AI Unit 8, 8A, 8B, 8C, 8D Control Unit 9 Communication Unit 10 RAW Signal Processing Unit 10a First RAW Processing Unit 10b Second RAW Processing Unit 10c Third RAW Processing Unit 11 AI Processing Unit 11a First AI Unit 11b Second AI Unit 11c Third AI Unit 40 CPU 41 Development Processing Unit 41a First Processing Unit 41b Second Processing Unit 41c Third Processing Unit 42 AI Processing Unit 42a First AI Unit 42b Second AI Unit 42c Third AI Unit 50 Information Processing Device 51 Arithmetic Unit 52 Communication Unit 53 Display Unit 54 Operation Unit

Claims

1. A sensor unit that obtains a captured image, An AI processing unit that performs AI processing, which is processing using an AI model on the captured image, A signal processing unit that adjusts the image quality of the input image to the AI processing unit, A control unit that adjusts the parameters of the signal processing unit based on the AI processing result by the AI processing unit, comprising An imaging device.

2. The signal processing unit Obtains a plurality of image quality adjustment images by performing image quality adjustment processing using different image quality parameters on the captured image, The AI processing unit Performs AI processing individually on the plurality of image quality adjustment images The imaging device according to claim 1.

3. Comprises a RAW signal processing unit that performs image quality adjustment on the RAW image of the captured image, The signal processing unit has a developing function for the RAW image, The control unit Derives image quality parameters based on the AI processing result of the image quality adjustment RAW image obtained by causing the RAW signal processing unit to perform image quality adjustment with candidate parameters, and sets the derived image quality parameters in the signal processing unit The imaging device according to claim 1.

4. The control unit Has an analysis unit that analyzes the relationship between the candidate parameters and the AI processing result of the image quality adjustment RAW image, and a derivation setting unit that derives the image quality parameters to be set in the signal processing unit based on the analysis result by the analysis unit and sets the derived image quality parameters in the signal processing unit, and the analysis unit and the derivation setting unit are constituted by separate chips The imaging device according to claim 3.

5. The control unit Derives image quality parameters based on the AI processing result of the developed image obtained by the developing function of the signal processing unit, and sets the derived image quality parameters in the signal processing unit The imaging device according to claim 3.

6. The AI processing is processing for detecting or recognizing a specific subject, The control unit A transmission process of transmitting to an external device the captured image used by the AI processing unit for AI processing, the AI processing result of the captured image, and the image quality parameters used by the signal processing unit for image quality adjustment of the captured image, A reception process of receiving correction information of the image quality parameters for making the image quality capable of detecting or recognizing a user-specified subject in the captured image calculated by the external device based on the captured image, the AI processing result, and the image quality parameters transmitted by the transmission process Adjust the parameters of the signal processing unit based on the correction information The imaging device according to claim 1

7. The image quality adjustment by the signal processing unit includes at least one of adjustments for brightness, contrast, gamma, dynamic range, saturation, hue, sharpness, and noise reduction The imaging device according to claim 1

8. A parameter adjustment method in an imaging device including a sensor unit that obtains a captured image, an AI processing unit that performs AI processing which is processing using an AI model on the captured image, and a signal processing unit that performs image quality adjustment on the input image for the AI processing unit, comprising Based on the AI processing result by the AI processing unit, adjust the parameters of the signal processing unit Parameter adjustment method

9. A computer-readable program in an imaging device including a sensor unit that obtains a captured image, an AI processing unit that performs AI processing which is processing using an AI model on the captured image, and a signal processing unit that performs image quality adjustment on the input image for the AI processing unit, comprising Cause the computer device to realize a function of adjusting the parameters of the signal processing unit based on the AI processing result by the AI processing unit Program

10. A recording medium on which is recorded a computer-readable program in an imaging device including a sensor unit that obtains a captured image, an AI processing unit that performs AI processing which is processing using an AI model on the captured image, and a signal processing unit that performs image quality adjustment on the input image for the AI processing unit, comprising A program that causes the computer device to realize a function of adjusting the parameters of the signal processing unit based on the AI processing result by the AI processing unit is recorded Recording medium

Images