Signal processing device and signal processing method

Abstract

To improve processing accuracy of image analysis processing using an AI model.SOLUTION: A signal processing device according to the present technique includes an AI processing unit that performs image analysis processing using an AI model with a non-demosaic image being a photographed image in a state of not being subjected to demosaic processing as input data for the photographed image obtained by a pixel array unit that is constructed by two-dimensionally arranging multiple pixel units in which multiple pixels that receive light of different wavelength bands are two-dimensionally arranged in a prescribed pattern.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present technology relates to a signal processing device that performs image analysis processing using an AI model, and a method thereof. [Background technology]

[0002] For example, a technology for performing image analysis processing such as object detection processing and object recognition processing on captured images using an AI (Artificial Intelligence) model having a neural network such as a CNN (Convolutional Neural Network) is becoming widespread. In such image analysis processing, it is common to use images with three colors, R (red), G (green), and B (blue), as input data for the AI ​​model (see, for example, Patent Document 1 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-170890 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, when it comes to applications where people will be viewing images, it is desirable to use 3-channel images (R, G, B) because this allows for full-color expression; however, when considering the performance of an AI model, using 3-channel images (R, G, B) as input data is not necessarily the best.

[0005] This technology was developed in consideration of the above circumstances, and aims to improve the processing accuracy of image analysis processing using AI models. [Means for solving the problem]

[0006] The signal processing device of the present technology includes an AI processing unit that performs image analysis processing using an AI model on an captured image obtained by a pixel array unit configured with a plurality of pixel units arranged two-dimensionally in a predetermined pattern, each pixel unit being made up of a plurality of pixels that receive light of different wavelength bands and arranged two-dimensionally. The captured image is obtained by using, as input data, a non-demosaic image, which is the captured image in a state that has not been demosaiced, as input data, the non-demosaic image, which is the captured image in a state that has not been demosaiced. Since demosaic processing involves spatial interpolation processing, deviations from the original pixel values ​​tend to occur, and when a demosaic-processed image is used as input data for image analysis processing using an AI model, the processing accuracy may decrease due to the deviation in pixel values. By using a non-demosaic image as input data for image analysis processing as described above, it is possible to prevent such a decrease in processing accuracy caused by demosaic processing. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of a schematic configuration of a camera device including a signal processing device according to a first embodiment. [Diagram 2] 2 is a diagram for explaining an example of the internal configuration of an image signal processing unit included in the signal processing device according to the first embodiment; FIG. [Diagram 3] FIG. 13 is a diagram showing the results of an experiment conducted on the change characteristics of image analysis processing accuracy when the type of input data to the AI ​​processing unit is changed. [Figure 4] 4A and 4B are explanatory diagrams of color separation images in the embodiment. [Diagram 5] FIG. 13 is a diagram for considering the cause of the decrease in analysis processing accuracy that occurs when a demosaiced image is used. [Figure 6] 1 is a flowchart showing an example of a specific processing procedure for implementing an analysis processing method according to a first embodiment. [Figure 7] FIG. 11 is a block diagram for explaining an example of the configuration of a camera device according to a second embodiment. [Figure 8] 13 is a flowchart showing an example of a specific processing procedure for implementing an analysis processing method as a modified example. [Figure 9] FIG. 11 is an explanatory diagram of a configuration example in which an AI processing unit is provided outside a sensor device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, with reference to the accompanying drawings, an embodiment of a signal processing device according to the present technology will be described in the following order. <1. First embodiment> [1-1. Example of camera device configuration] [1-2. Analysis processing method as the first embodiment] [1-3. Processing Procedure] <2. Second embodiment> <3. Modifications> <4. Summary of the embodiment> <5. This Technology>

[0009] <1. First embodiment> [1-1. Example of camera device configuration] FIG. 1 is a block diagram showing an example of a schematic configuration of a camera device 10 including a signal processing device according to a first embodiment of the present technology. As shown in the figure, the camera device 10 includes a sensor unit 1, an optical system 11, a communication I / F (interface) 12, a camera control unit 13, an external sensor memory unit 14, and a communication unit 15.

[0010] In the camera device 10, the sensor unit 1 corresponds to the signal processing device of the first embodiment. The sensor unit 1 is configured as an image sensor such as a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. The sensor unit 1 is configured to be capable of performing not only an imaging function but also image analysis processing using an AI (Artificial Intelligence) model as image analysis processing on the captured image. The sensor unit 1 is formed with a plurality of pixels each having a light receiving element, each pixel receiving light of different wavelength bands, and is configured to be capable of obtaining a captured image as a color image.

[0011] In the camera device 10, the optical system 11 includes lenses such as a cover lens and a focus lens, and an iris mechanism. The optical system 11 guides light (incident light) from a subject and collects the light on the light receiving surface of the sensor unit 1.

[0012] The communication interface (I / F) 12 is a communication interface for performing data communication between the sensor unit 1 and the camera control unit 13.

[0013] The camera control unit 13 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and performs overall control of the camera device 10 by the CPU executing various processes according to programs stored in the ROM or programs loaded into the RAM. The camera control unit 13 can receive various types of data from the sensor unit 1 and transmit various types of data to the sensor unit 1 via the communication interface 12.

[0014] The sensor external memory unit 14 is connected to the camera control unit 13. The sensor external memory unit 14 is configured with a non-volatile storage device such as an SSD (Solid State Drive) or a flash memory device, and is used to store information used by the camera control unit 13 for various controls. The sensor external memory unit 14 can also be used to store various data obtained by the sensor unit 1, such as image data captured by the sensor unit 1. In the camera device 10 of this embodiment, the sensor external memory unit 14 stores first AI model setting data P1 and second AI model setting data P2, which will be described later.

[0015] In addition, a communication unit 15 is connected to the camera control unit 13 . The communication unit 15 is configured to be capable of performing wired or wireless data communication with an external device. The communication unit 15 can also be configured to have a network communication function, in which case the camera control unit 13 is capable of exchanging data with a predetermined device (e.g., a server device) on a predetermined network such as the Internet via the communication unit 15.

[0016] As shown in the figure, the sensor unit 1 has a pixel array unit 2, an image signal processing unit 3, a pre-processing unit 4, an AI processing unit 5, an internal sensor control unit 6, an internal sensor memory unit 7, an output data generation unit 8, and a communication interface (I / F) 9.

[0017] In the pixel array section 2, a plurality of pixels, each having a light receiving element (photoelectric conversion element) such as a photodiode, are arranged two-dimensionally in the horizontal and vertical directions. Specifically, the pixel array section 2 has pixel units Pu (see Figure 4A described later) each consisting of a plurality of pixels that receive light of different wavelength bands and are arranged two-dimensionally in a predetermined pattern, and these pixel units Pu are arranged in a two-dimensional manner.

[0018] In this embodiment, the pixel unit Pu is formed by arranging three types of pixels, an R pixel for receiving R (red) light, a G pixel for receiving G (green) light, and a B pixel for receiving B (blue) light, in a predetermined arrangement pattern. Specifically, the pixel unit Pu in this example is formed by arranging the R pixel, the G pixel, and the B pixel in a Bayer arrangement.

[0019] The pixel array section 2 also includes components for obtaining image data as digital data, such as a readout circuit that reads out the value of each pixel (received light value) and an ADC (Analog to Digital Converter) for digitally sampling the pixel values ​​as analog signals.

[0020] An imaging signal processor (ISP) 3 receives image data (captured image data) obtained by the pixel array unit 2 and performs various types of image signal processing. The internal configuration of the image signal processing unit 3 will be explained later.

[0021] The pre-processing unit 4 inputs the image data after the image signal processing by the image signal processing unit 3, and performs image signal processing as pre-processing for the image analysis processing by the AI ​​processing unit 5. Specifically, the pre-processing unit 4 in this example is configured to be capable of performing at least image resizing processing.

[0022] The AI ​​processing unit 5 performs image analysis processing using an AI model, using the image data output from the pre-processing unit 4 as input data. The AI ​​processing unit 5 is configured by, for example, a DSP (Digital Signal Processor), and is capable of switching the AI ​​model used in the image analysis process by switching processing parameters. The AI ​​processing unit 5 of this embodiment is configured to be capable of performing image analysis processing using an AI model having a neural network such as a CNN (Convolutional Neural Network).

[0023] Here, as an example, the camera device 10 in this example is placed in a commercial facility such as a supermarket or a department store, and performs image analysis processing with people as target subjects. Specifically, it is assumed that the camera device 10 performs object detection processing with people as target subjects. In this case, an AI model that has been machine-learned to perform object detection processing with a person as the target subject is used as the AI ​​model in the AI ​​processing unit 5. Here, the object detection processing includes processing to specify an area in which the target subject exists, as a so-called bounding box.

[0024] The sensor internal control unit 6 is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, and the like, and controls the overall operation of the sensor unit 1. For example, the intra-sensor control unit 6 controls the operation of the pixel array unit 2. Specifically, it controls the start / stop of the operation. The intra-sensor control unit 6 also controls the operation of the image signal processing unit 3, the pre-processing unit 4, and the AI ​​processing unit 5. Regarding the operation control of the image signal processing unit 3 and the pre-processing unit 4, the intra-sensor control unit 6 is capable of controlling processing parameters of various processes. As part of the control of the AI ​​processing unit 5, the intra-sensor control unit 6 is capable of controlling switching of the AI ​​model.

[0025] An in-sensor memory unit 7 is connected to the in-sensor control unit 6. The in-sensor memory unit 7 is configured with a non-volatile storage device such as a flash memory device, and is used to store information used by the in-sensor control unit 6 for various controls.

[0026] The output data generation unit 8 receives the analysis processing result by the AI ​​processing unit 5 and the image data output by the image signal processing unit 3, and generates output data to be output outside the sensor unit 1. The output data generation unit 8 generates the output data based on an instruction from the intra-sensor control unit 6. For example, it is possible to switch between using both the analysis processing result by the AI ​​processing unit 5 and the image data as output data, or using only the analysis processing result by the AI ​​processing unit 5 as output data, based on an instruction from the intra-sensor control unit 6.

[0027] The communication interface 9 is a communication interface that enables data output from within the sensor unit 1 to outside the sensor unit 1, and data input from outside the sensor unit 1 to the sensor unit 1, and performs data communication with the aforementioned communication interface 12 in accordance with a specified communication data format. Through this communication interface 9, the above-mentioned output data can be output to the outside of the sensor unit 1 (to the camera control unit 13 in this example). In addition, the internal sensor control unit 6 is capable of performing data communication with the camera control unit 13 via the communication interface 9.

[0028] FIG. 2 is a diagram for explaining an example of the internal configuration of the image signal processing unit 3. As shown in FIG. 2 shows an example of the internal configuration of the image signal processing unit 3, as well as the pre-processing unit 4, the AI ​​processing unit 5, and the internal sensor control unit 6 shown in FIG.

[0029] Image data as RAW data is input to the image signal processing unit 3 from the pixel array unit 2 shown in Fig. 1. The RAW data referred to here means image data obtained by reading out the values ​​of each pixel in raster order, that is, image data in a state in which the pixel arrangement according to the Bayer array is maintained in this example.

[0030] As shown in the figure, the image signal processing unit 3 has a black level correction unit 31, a gain adjustment unit 33, a demosaic processing unit 34, a color correction unit 35, a gamma correction unit 36, and a dewarp processing unit 37, and is capable of sequentially performing shading correction processing, gain adjustment processing, demosaic processing, color correction processing, gamma correction processing, and dewarp processing on image data as RAW data input from the pixel array unit 2.

[0031] Here, the gain adjustment process by the gain adjustment unit 33 includes overall gain adjustment for adjusting the luminance distribution of the entire image regardless of the color of the pixels, and AWB (Auto White Balance) processing, which is a gain adjustment process for each color.

[0032] In addition, the demosaic processing by the demosaic processing unit 34 is a process that generates image data as an R image, a G image, and a B image with the same number of pixels as the input image data by performing spatial interpolation processing for each of the R, G, and B colors from the input image data in a Bayer array state.

[0033] The color correction unit 35 performs color correction processing by linear matrix processing on the image data after the demosaic processing. Furthermore, the dewarp processor 37 performs at least lens distortion correction processing as the dewarp processing.

[0034] Here, the image signal processing unit 3 in this embodiment also includes an image organizing unit 38 and a selector 39, which will be described later.

[0035] [1-2. Analysis processing method as the first embodiment] As mentioned above, when it comes to applications where people will be viewing images, using 3-channel images (R, G, B) is desirable as it allows for full-color expression; however, for image analysis processing using an AI model, using 3-channel images (R, G, B) as input data is not necessarily optimal in terms of analysis processing accuracy.

[0036] In view of this, in this embodiment, a method is proposed in which a non-demosaic image, which is a captured image that has not been subjected to demosaic processing, is used as input data to the AI ​​processing unit 5. Because demosaic processing involves spatial interpolation, there is a tendency for deviations from the original pixel values ​​to occur. When a demosaic image is used as input data for image analysis processing using an AI model, the accuracy of the image analysis processing may decrease due to the deviations in pixel values. In contrast, by using a non-demosaic image as input data for the AI ​​processing unit 5 as described above, it is possible to prevent a decrease in processing accuracy caused by such demosaic processing, and to improve the processing accuracy of image analysis processing using an AI model.

[0037] In addition, in this embodiment, based on the scene determination result for the scene to be imaged, the input data of the AI ​​processing unit 5 is switched between a demosaic image, which is an image after demosaic processing by the demosaic processing unit 34, and a non-demosaic image.

[0038] Here, when performing image analysis processing using an AI model within a sensor device as in this embodiment, resources (memory amount and computing power) that can be allocated to the image analysis processing tend to become insufficient. For example, in practical applications, image analysis processing using an AI model, such as object detection processing, is required to maintain accuracy even if a certain degree of change occurs in the scene captured by the camera. For example, the scene captured by the camera may change over time, such as light / dark scenes such as day and night, or scenes in which subjects to be analyzed, such as people, are relatively far away or nearby, and it is desirable to maintain a certain level of analysis processing accuracy even in the face of such scene changes.

[0039] In order to maintain accuracy in the face of scene changes, one possible approach is to train the AI ​​model using machine learning techniques that use images of each scene to be handled as input data for training, thereby creating an AI model that can absorb differences in scenes.

[0040] However, in order to absorb such scene differences, the AI ​​model would require a relatively large number of filter coefficients and a large number of network hierarchies, which would increase the resources required to realize the AI ​​model. For this reason, the technique of absorbing scene changes in an AI model is applicable when the image analysis process is executed by a computer device with relatively abundant resources, specifically, a computer device external to the camera device 10; however, it is difficult to apply this technique when the image analysis process is executed within the camera device 10 with limited resources, particularly when the image analysis process is executed within the sensor device (sensor unit 1).

[0041] For this reason, in this embodiment, a method is adopted in which the input data to the AI ​​processing unit 5 is switched between a demosaiced image and a non-demosaic image based on the scene determination result as described above.

[0042] FIG. 3 shows the results of an experiment conducted on the change characteristics of the image analysis processing accuracy when the type of input data to the AI ​​processing unit 5 is changed. Specifically, the experimental results in Figure 3 show the results when object detection processing was performed with a human body as the target subject as image analysis processing using an AI model. <1> from <6> 4 shows the measurement results of the evaluation value of the image analysis processing accuracy when the image data at each take-out position indicated by a is used as input data. The evaluation value here is the percentage of people that were accurately detected at 90% or more when object detection processing was performed multiple times with human bodies as the target subject.

[0043] Here, a dark scene means a scene in which the target subject appears dark, and can be defined as, for example, a scene in which the luminance value of the target subject is equal to or lower than a certain value, whereas a bright scene means a scene in which the target subject appears bright, and can be defined as, for example, a scene in which the luminance value of the target subject exceeds the above-mentioned certain value.

[0044] As can be seen with reference to FIG. <1> The extraction position of is a position immediately before input to the image signal processing unit 3, <2> The extraction position is a position between the shading correction unit 32 and the gain adjustment unit 33. <3> The extraction position is a position between the gain adjustment unit 33 and the demosaic processing unit 34. these <1> from <3> The extraction positions of both are the positions before demosaicing. <1> from <3> As image data for the extraction position, image data as a color-separated image described below is used in correspondence with the use of an AI model having a CNN as the AI ​​model.

[0045] FIG. 4 is an explanatory diagram of a color separation image in the embodiment. 4A shows a pixel arrangement (in this example, a Bayer arrangement) in the pixel array section 2. As shown in the figure, in this example, a pixel unit Pu is configured by four pixels of R, G, G, and B being arranged in a predetermined arrangement pattern according to the Bayer format.

[0046] The color separation image referred to here means an image formed by collecting pixel values ​​for pixels at the same position within each pixel unit Pu and arranging them in different regions on the same image plane. 4B illustrates a color-separated image generated in the case of the Bayer array. As shown in the figure, in the case of the Bayer array, pixel values ​​of R pixels, G pixels, G pixels, and B pixels in each pixel unit Pu that are located at the same position in the pixel unit are collected and arranged in different regions on the same image plane to generate a color-separated image having an image region in which the pixel values ​​of the R pixels in each pixel unit Pu are arranged, an image region in which the pixel values ​​of one G pixel in each pixel unit Pu are arranged, an image region in which the pixel values ​​of the other G pixel in each pixel unit Pu are arranged, and an image region in which the pixel values ​​of the B pixels in each pixel unit Pu are arranged.

[0047] By using the color separation image as described above as input data for the AI ​​processing unit 5, the format of the input data to the AI ​​processing unit becomes an input data format suitable for the configuration of CNN, thereby improving the accuracy of image analysis processing.

[0048] Let us return to the explanation given in FIG. <4> from <6> The extraction position of is the extraction position after demosaicing, <4> is the position between the demosaic processing unit 34 and the color correction unit 35, <5> is the position between the color correction unit 35 and the gamma correction unit 36, <6> is a position between the gamma correction unit 36 ​​and the dewarping unit 37.

[0049] According to the results in Fig. 3, in bright scenes, the input data <1> from <6> The evaluation score was over 90% for any data type. This shows that in the case of bright scenes, the accuracy of object detection processing tends not to depend on the data type.

[0050] On the other hand, in the case of a dark scene, <4> In other words, when the demosaiced image is used as input data, a significant drop in processing accuracy occurs. According to the experiment, <1> from <3> When the data type before demosaic is used, the evaluation value remains above 60%, whereas after demosaic, <4> <5> For this type of data, the evaluation value is significantly below 60%. <6> When using the data type (i.e., the demosaiced image after gamma correction), a slight increase in the evaluation value can be confirmed, but it is still less than 60%. In dark scenes, <2> In the case of using the data type above, that is, the image after shading correction and before gain adjustment, the evaluation value is the highest, with an evaluation value of about 75%.

[0051] As shown in the experimental results above, the decrease in accuracy of image analysis processing when using a demosaiced image is believed to be due to the fact that the deviation from the original pixel values ​​that accompanies the spatial interpolation processing used in the demosaicing process becomes more noticeable in dark scenes.

[0052] Furthermore, when resources are limited, the input data to the AI ​​processing unit 5 is resized (e.g., thinned out) to a predetermined size in the pre-processing unit 4 so as to reduce the amount of input data to the AI ​​processing unit 5. However, this is also thought to result in a decrease in processing accuracy when using a demosaiced image.

[0053] As a specific example, assuming that the amount of input data to the AI ​​processing unit 5 is limited to 1280 x 960 pixels, when a non-demosaic image is used as shown in Figure 5, an image with a resolution (information density) of 1280 x 960 pixels can be used as input data, but when a demosaiced image is used, the resolution of each of the R image, B image, and G image that serve as input data to the AI ​​processing unit 5 is reduced to 739 x 554 pixels. Such a decrease in resolution is also thought to be one of the factors that leads to a decrease in processing accuracy when a demosaiced image is used as input data.

[0054] In consideration of the above, in this embodiment, the scene to be imaged is determined by determining whether it is a dark scene or not, and if it is determined to be a dark scene, a non-demosaic image is input as input data to the AI ​​processing unit 5. This makes it possible to suppress degradation of image analysis processing accuracy in dark scenes when it is not possible to use an AI model capable of absorbing scene differences due to resource issues, thereby suppressing scene-dependent degradation of image analysis processing accuracy.

[0055] In this example, the input data for a dark scene is <2> The non-demosaic image after shading correction shown in the figure is used. Specifically, a color separation image of the non-demosaic image obtained between the shading correction unit 32 and the gain adjustment unit 33 is used as input data for a dark scene.

[0056] For this reason, in the camera device 10 of this embodiment, an image editing section 38 is provided in the image signal processing section 3. As shown in Figure 2, the image organization unit 38 inputs the non-demosaic image after shading correction output from the shading correction unit 32, and rearranges the pixel values ​​using the method described above in Figure 3, thereby generating a color separation image of the non-demosaic image after shading correction.

[0057] Furthermore, in this embodiment, as a result of the determination as to whether or not the scene is a dark scene, if it is determined that the scene is not a dark scene, the demosaiced image is input as input data to the AI ​​processing unit 5. Specifically, in this example, if it is determined that the scene is not a dark scene, the demosaiced image output from the dewarp processor 37 is provided as input data to the AI ​​processor 5 .

[0058] In order to enable switching of input data according to such a dark scene / bright scene, the image signal processing unit 3 is provided with a selector 39. The selector 39 inputs the non-demosaic image that has been converted into a color-separated image in the image organization unit 38 and the demosaiced image output from the dewarping processing unit 37, and outputs the image specified by the sensor control unit 6 to the pre-processing unit 4.

[0059] In this example, the determination as to whether or not the scene is a dark scene is performed based on the demosaiced image by the intra-sensor control unit 6. Specifically, in this example, the intra-sensor control unit 6 determines whether or not the scene is a dark scene based on the demosaiced image output by the dewarp processing unit 37. Then, the sensor control unit 6 determines whether or not the scene is a dark scene, and if it determines that the scene is not a dark scene, it causes the selector 39 to output (select) the demosaiced image, and if it determines that the scene is a dark scene, it causes the selector 39 to output (select) the non-demosaic image.

[0060] In addition, in this embodiment, the sensor control unit 6 controls so that when a demosaiced image is used as input data for the AI ​​processing unit 5, an AI model trained using the demosaiced image as learning input data is used as the AI ​​model, and when a non-demosaic image is used as input data for the AI ​​processing unit 5, an AI model trained using the non-demosaic image as learning input data is used as the AI ​​model.

[0061] In this example, the setting data of the AI ​​processing unit 5 for realizing the former AI model, i.e., an AI model trained using demosaiced images as learning input data, is stored in the sensor-external memory unit 14 shown in FIG. 1 as first AI model setting data P1, and the setting data of the AI ​​processing unit 5 for realizing the latter AI model, i.e., an AI model trained using non-demosaic images as learning input data, is stored in the sensor-external memory unit 14 as second AI model setting data P2. Here, the first AI model setting data P1 and the second AI model setting data P2 are data including parameters as filter coefficients used in filter processing such as convolution processing in CNN, and various parameters related to the structure of the neural network.

[0062] When the intra-sensor control unit 6 determines that the scene is not a dark scene, it instructs the camera control unit 13 via the communication interface 9 to read the first AI model setting data P1 from the sensor external memory unit 14 and transfer it to itself. Then, by setting parameters of the AI ​​processing unit 5 according to the transferred first AI model setting data P1, the AI ​​processing unit 5 is made ready to execute image analysis processing by an AI model trained using the demosaiced image as learning input data. In addition, when the sensor internal control unit 6 determines that it is a dark scene, it issues an instruction to the camera control unit 13 via the communication interface 9 to read the second AI model setting data P2 from the sensor external memory unit 14 and transfer it to itself, and sets parameters of the AI ​​processing unit 5 according to the transferred second AI model setting data P2, thereby enabling the AI ​​processing unit 5 to perform image analysis processing using an AI model trained using the non-demosaic image as learning input data.

[0063] [1-3. Processing Procedure] An example of a specific processing procedure to be executed by the intra-sensor control unit 6 in order to realize the analysis processing method as the first embodiment described above will be described with reference to the flowchart in FIG. 6 is executed by the CPU in the sensor control unit 6 based on a program stored in a predetermined storage device such as a ROM in the sensor control unit 6. However, for the sake of explanation, the process will be described below as being executed by the sensor control unit 6. In this example, the internal sensor control unit 6 starts the process shown in FIG. 6 upon startup.

[0064] First, in step S101, the intra-sensor control unit 6 determines whether or not a scene determination execution condition is satisfied. That is, it determines whether or not a predetermined condition is satisfied that indicates that a scene determination process (in this example, a process of determining whether or not the scene is a dark scene) in step S103 described later should be executed. The scene determination process in step S103 may be performed periodically at a predetermined time interval, for example, in which case the scene determination condition may be the passage of a certain period of time. In addition, the conditions for executing scene determination may include an external instruction being given, or a simple brightness detection using an illuminance sensor or the like resulting in a change in brightness of a predetermined amount or more, and are not limited to specific conditions.

[0065] If it is determined in step S101 that the scene determination execution conditions are not met, the sensor control unit 6 proceeds to step S102 and determines whether the processing has ended, that is, whether a predetermined condition has been met that indicates that the series of processing steps shown in Figure 6 should be ended, such as a power outage. If it is determined in step S102 that the processing has not ended, the internal sensor control unit 6 returns to step S101. That is, depending on the processing in steps S101 and S102, a loop process is formed in which the process waits for either the satisfaction of a scene determination execution condition or the satisfaction of a processing end condition.

[0066] In step S101, if it is determined that the scene determination execution condition is satisfied, the intra-sensor control unit 6 proceeds to step S103 and executes scene determination processing. That is, in this example, it is determined whether or not it is a dark scene based on the demosaiced image output from the dewarp processing unit 37. Specifically, it is determined whether or not the average luminance value of the area of ​​the target subject is equal to or less than a predetermined luminance value. At this time, it is considered that the area of ​​the target subject is specified based on area information of the target subject obtained as a result of the object detection processing by the AI ​​processing unit 5, for example. Note that, in a state in which the object detection processing by the AI ​​processing unit 5 has not started, it is considered that the area of ​​the moving object detected by the moving object detection processing by, for example, the difference detection processing between frames, is specified as the area of ​​the target subject.

[0067] In step S104 following step S103, the intra-sensor control unit 6 determines whether or not it is a dark scene. That is, it determines whether or not the determination result of the determination process in step S103 indicates that it is a dark scene.

[0068] If it is determined in step S104 that the scene is not a dark scene, the intra-sensor control unit 6 proceeds to step S105 and issues an instruction to select a demosaiced image. That is, the intra-sensor control unit 6 issues an instruction to the selector 39 to select and output the demosaiced image input from the dewarp processing unit 37.

[0069] Then, in step S106 following step S105, the intra-sensor control unit 6 performs a setting process for the first AI model. That is, it instructs the camera control unit 13 via the communication interface 9 to read the first AI model setting data P1 from the extra-sensor memory unit 14 and transfer it to itself, and sets parameters for the AI ​​processing unit 5 according to the transferred first AI model setting data P1. As a result, if it is determined that the scene is not a dark scene, the AI ​​processing unit 5 performs image analysis processing using an AI model that has been trained using the demosaiced image as learning input data, with the demosaiced image as input data.

[0070] After executing the setting process in step S106, the internal sensor control unit 6 returns to step S101.

[0071] Furthermore, if it is determined in the previous step S104 that the scene is a dark scene, the intra-sensor control unit 6 proceeds to step S107 and issues an instruction to select a non-demosaic image, i.e., an instruction to the selector 39 to select and output a non-demosaic image based on the color separation image input from the image organization unit 38.

[0072] Furthermore, in step S108 following step S107, the intra-sensor control unit 6 performs a setting process for the second AI model. That is, it instructs the camera control unit 13 via the communication interface 9 to read the second AI model setting data P2 from the sensor extra-memory unit 14 and transfer it to itself, and sets parameters for the AI ​​processing unit 5 according to the transferred second AI model setting data P2. As a result, if it is determined that the scene is dark, the AI ​​processing unit 5 performs image analysis processing using an AI model that has been trained using the non-demosaic image made from color separation image as input data and the non-demosaic image made from color separation image as learning input data.

[0073] After executing the setting process in step S108, the internal sensor control unit 6 returns to step S101.

[0074] Moreover, if it is determined in the previous step S102 that the processing has ended, the internal sensor control unit 6 ends the series of processing steps shown in FIG.

[0075] Regarding the scene determination process, the determination of whether or not the scene is a dark scene may be executed by the AI ​​processing unit 5 by providing a function of determining whether or not the scene is a dark scene as a part of the AI ​​model in the AI ​​processing unit 5. In this case, the intra-sensor control unit 6 controls the selector 39 (and controls switching of the AI ​​model) based on the result of the process of determining whether or not the scene is a dark scene executed in the AI ​​processing unit 5 in this manner.

[0076] <2. Second embodiment> Next, a second embodiment will be described. In the second embodiment, it is assumed that the AI ​​model used in the AI ​​processing unit 5 will be re-learned, and non-demosaic images are stored.

[0077] FIG. 7 is a block diagram illustrating an example of the configuration of a camera device 10A as a second embodiment having a function of storing non-demosaic images. 7, for convenience of illustration, the optical system 11, the pixel array unit 2, the in-sensor memory unit 7, and the output data generation unit 8 are omitted, but the camera device 10A of the second embodiment also has these optical system 11, pixel array unit 2, in-sensor memory unit 7, and output data generation unit 8, just like the camera device 10. Also, while the first AI model setting data P1 and the second AI model setting data P2 are omitted from the sensor off-memory unit 14, the first AI model setting data P1 and the second AI model setting data P2 are stored in the sensor off-memory unit 14 in the camera device 10A as well, just like the camera device 10.

[0078] In the following description, parts that are similar to parts that have already been described will be given the same reference numerals and description thereof will be omitted.

[0079] Camera device 10A differs from camera device 10 shown in FIG. 1 in that sensor unit 1A is provided instead of sensor unit 1. Sensor unit 1A differs from sensor unit 1 in that an internal sensor control unit 6A is provided instead of internal sensor control unit 6.

[0080] The intra-sensor control unit 6A differs from the intra-sensor control unit 6 in that it performs processing for storing non-demosaic images in a memory within the camera device 10A, specifically in the extra-sensor memory unit 14 in this example. As shown in the figure, in the sensor unit 1A, a non-demosaic image based on a color-separated image generated by an image editing unit 38 can be input to not only a selector 39 but also a communication interface 9. The sensor internal control unit 6A transmits a non-demosaic image based on the color separation image to the camera control unit 13 via the communication interface 9, and also instructs the camera control unit 13 via the communication interface 9 to store the non-demosaic image in the sensor external memory unit 14. This makes it possible to store in the sensor external memory unit 14 captured images that are non-demosaic images that can be obtained in the actual usage environment of the camera device 10A.

[0081] Furthermore, the intra-sensor control unit 6A performs a process of transmitting the non-demosaic images stored in the sensor extra-memory unit 14 to the outside of the camera device 10A. Specifically, based on an instruction given from an external device (e.g., a server device, etc.) of the camera device 10A, the intra-sensor control unit 6A instructs the camera control unit 13 to perform a process of transmitting the non-demosaic images stored in the sensor extra-memory unit 14 to the external device. This makes it possible to transmit a non-demosaic image to be used for re-learning to the external device in response to a case where re-learning of an AI model used in the AI ​​processing unit 5 is performed in the external device.

[0082] In the above example, a configuration is illustrated in which the sensor control unit 6A instructs the camera control unit 13 to execute the transmission process for the accumulated non-demosaic images, but it is also possible that the camera control unit 13 performs the transmission process for the accumulated non-demosaic images based on instructions from outside.

[0083] In addition, in the above example, the storage unit that stores the non-demosaic images is the sensor external memory unit 14, but the storage unit may also be a memory within the sensor unit 1A, such as the sensor internal memory unit 7. Here, if the sensor unit 1A has a communication unit capable of directly communicating with an external device of the camera device 10A, the intra-sensor control unit 6A may be the entity that executes the transmission process of the stored non-demosaic images.

[0084] <3. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, it is also possible to change the signal processing parameters of the image signal processing unit 3 based on the scene determination result. As an example, it is possible to change the parameters of the gain adjustment process as the above-mentioned overall gain adjustment by the gain adjustment unit 33 and the parameters of the gamma correction process by the gamma correction unit 36 ​​based on the determination result of whether or not the scene is dark. Specifically, when the target subject is a dark scene as a dark scene, gamma correction and overall gain adjustment are performed using parameters that tend to prevent blackout, and when the target subject is a bright scene (when not a dark scene), gamma correction and overall gain adjustment are performed using parameters that tend to prevent whiteout.

[0085] Here, the parameter adjustment according to dark / bright scenes as described above is an effective method for performing object detection processing as image analysis processing using an AI model. When performing segmentation processing such as semantic segmentation as image analysis processing using an AI model, the segmentation processing involves class identification for each block, such as for each pixel, so performing gamma correction or overall gain adjustment may result in a deterioration of processing accuracy in bright and dark blocks. Therefore, when performing segmentation processing as image analysis processing using an AI model, it is possible to use captured images that have not yet undergone gamma correction or overall gain adjustment as input data.

[0086] FIG. 8 is a flowchart showing an example of a specific processing procedure for implementing the analysis processing method as a modified example in which the signal processing parameters of the image signal processor 3 are changed based on the scene determination result as described above. Here, the processing shown in FIG. 8 may be performed by either the internal sensor control unit 6 or the internal sensor control unit 6A, but here it will be described as being executed by the internal sensor control unit 6.

[0087] The process shown in FIG. 8 differs from the process shown in FIG. 6 in that steps S110 and S111 are added. Specifically, in this case, the intra-sensor control unit 6 performs a signal processing parameter adjustment process in step S110 in response to having executed the first AI model setting process in step S106. In this example, the signal processing parameter adjustment process in step S110 performs a control process such that the above-mentioned parameters that tend to prevent whiteout are set in the gain adjustment unit 33 and the gamma correction unit 36 ​​in response to a case where it is determined that the scene is not dark. In this case, the intra-sensor control unit 6 performs a signal processing parameter adjustment process in step S111 in response to executing the second AI model setting process in step S108. Specifically, the signal processing parameter adjustment process in step S111 in this example performs a control process such that the above-mentioned parameters that tend to prevent blackout are set in the gain adjustment unit 33 and the gamma correction unit 36 ​​in response to the determination that the scene is a dark scene.

[0088] In this case, the intra-sensor control unit 6 returns the process to step S101 regardless of whether the signal processing parameter adjustment process has been performed in step S110 or S111.

[0089] In the above, we have given an example of changing parameters related to overall gain adjustment and gamma correction based on the scene determination result, but it is also possible to change other parameters based on the scene determination result, such as AWB parameters.

[0090] In addition, while the above example illustrates a configuration in which the AI ​​processing unit 5 is provided within the sensor device, it is also possible to consider a configuration in which the AI ​​processing unit 5 is provided outside the sensor device, as in the camera device 10B shown in FIG. 9. Specifically, in camera device 10B, instead of sensor unit 1 (or sensor unit 1A), a sensor unit 1B having only a pixel array unit 2 and a communication interface 9 is provided. In this case, image signal processing unit 3, pre-processing unit 4, and AI processing unit 5 are provided outside sensor unit 1B. Also, in camera device 10B, instead of camera control unit 13, a camera control unit 13B is provided.

[0091] As shown in the figure, in camera device 10B, a captured image as RAW data obtained by pixel array unit 2 is input to image signal processing unit 3 via communication interface 9 and communication interface 12. Then, image data selected by selector 39 of image signal processing unit 3 is input to AI processing unit 5 via pre-processing unit 4.

[0092] The camera control unit 13B controls the selector 39 based on the scene determination result, similar to the intra-sensor control unit 6. Also, the camera control unit 13B performs the AI ​​model setting process of the AI ​​processing unit 5, similar to the intra-sensor control unit 6, based on the first AI model setting data P1 and the second AI model setting data P2 stored in the sensor extra-memory unit 14. Furthermore, when realizing the accumulation of non-demosaic images described in the second embodiment, the camera control unit 13B performs processing to store the non-demosaic images obtained by the image signal processing unit 3 in a memory within the camera device 10B, such as the sensor external memory unit 14. In this case, the camera control unit 13B can also perform processing to transmit the accumulated non-demosaic images to an external device.

[0093] In addition, in the above, an example was given in which a color-separated image was used as the non-demosaic image to be input to the AI ​​processing unit 5, but a non-demosaic image in a data format other than a color-separated image can also be provided as input data to the AI ​​processing unit 5.

[0094] In addition, in the above example, the input data switching of the AI ​​processing unit 5 is performed by switching between a demosaiced image and a non-demosaic image based on the result of determining whether or not the scene is dark. However, it is also possible to switch between a demosaiced image and a non-demosaic image based on criteria other than the result of determining whether or not the scene is dark. For example, when the target subject is small and high resolution is required, the input data may be switched to a non-demosaic image, and in a scene where the target subject is large, the input data may be switched to a demosaiced image.

[0095] Also, it is possible to adopt a configuration in which three or more scenes are judged as the scene judgment, and the extraction position of the input data of the AI ​​processing unit 5 is changed for each scene. For example, it is possible to use a non-demosaic image after shading correction as the input data for a first scene, a non-demosaic image after AWB processing as the input data for a second scene, and a demosaic image as the input data for a third scene.

[0096] In addition, in the above example, the pixel array unit 2 is configured to receive and separate only three wavelengths of R, G, and B. However, the present technology can also be suitably applied to a case where a pixel array unit configured to receive and separate light of four or more wavelength bands is used, such as a pixel array unit in a multispectral camera.

[0097] In addition, if it is assumed that the AI ​​processing unit 5 will switch between multiple AI models with different analysis tasks, it may be possible to determine whether or not to convert the input data to the AI ​​processing unit 5 into a non-demosaic image depending on the type of AI model (i.e., the type of analysis task) used by the AI ​​processing unit 5.

[0098] <5. Summary of the embodiment> As described above, the signal processing device (sensor unit 1, 1A, camera device 10B) as an embodiment is equipped with an AI processing unit (ibid. 5) that performs image analysis processing using an AI model on an image obtained from a pixel array unit (ibid. 2) configured with a plurality of pixel units arranged two-dimensionally in a predetermined pattern, each pixel unit being made up of a plurality of pixels that receive light in different wavelength bands. The image obtained from the pixel array unit (ibid. 2) is ... Since demosaic processing involves spatial interpolation processing, deviations from the original pixel values ​​tend to occur, and when a demosaic-processed image is used as input data for image analysis processing using an AI model, the processing accuracy may decrease due to the deviation in pixel values. By using a non-demosaic image as input data for image analysis processing as described above, it is possible to prevent such a decrease in processing accuracy caused by demosaic processing. Therefore, it is possible to improve the processing accuracy of image analysis processing using an AI model.

[0099] In addition, the signal processing device as an embodiment is equipped with a demosaic processing unit (34) that performs demosaic processing on the captured image obtained by the pixel array unit, and a control unit (intra-sensor control unit 6, 6A, camera control unit 13B) that controls the input data of the AI ​​processing unit to be switched between a demosaic image, which is an image after demosaic processing by the demosaic processing unit, and a non-demosaic image, based on the scene determination result for the scene to be captured. As described with reference to Fig. 3, it has been confirmed by experiments that, depending on the scene, the accuracy of image analysis processing may be improved by using a non-demosaic image rather than a demosaiced image. In other words, even if an AI model that does not have the function of absorbing scene differences is used, the accuracy of image analysis processing may be improved by adopting a technique of using a non-demosaic image rather than a demosaiced image in a certain scene. By switching the input data of the AI ​​processing unit between a demosaiced image and a non-demosaic image based on the scene determination result as described above, it is possible to use a non-demosaic image as input data in certain scenes where the accuracy of image analysis processing is improved when a non-demosaic image is used as input data. Therefore, even if it is not possible to use an AI model that can absorb scene differences due to resource issues, it is possible to suppress deterioration in the accuracy of image analysis processing that is dependent on the scene.

[0100] Furthermore, in a signal processing device as an embodiment, the control unit controls so that when a demosaiced image is used as input data, an AI model trained using the demosaiced image as learning input data is used as the AI ​​model, and when a non-demosaic image is used as input data, an AI model trained using the non-demosaic image as learning input data is used as the AI ​​model. This makes it possible to execute appropriate image analysis processing according to the scene as image analysis processing using an AI model.

[0101] Furthermore, in the signal processing device of the embodiment, the determination of the scene to be imaged is a determination of whether or not it is a dark scene, and the control unit inputs a non-demosaic image as input data when it is determined to be a dark scene. As described with reference to FIG. 3, it has been confirmed that in dark scenes, the accuracy of image analysis processing can be improved by using a non-demosaic image rather than a demosaiced image. Therefore, with the above configuration, even if it is not possible to use an AI model that can absorb scene differences due to resource issues, it is possible to suppress a decrease in the accuracy of scene-dependent image analysis processing.

[0102] In addition, the signal processing device as an embodiment is provided with an image signal processing unit (3) that includes a demosaic processing unit and performs image signal processing on a captured image, and the control unit changes the signal processing parameters of the image signal processing unit based on the scene determination result (see Figure 8). This makes it possible to perform image signal processing suitable for the non-demosaic image when it is determined that the scene is a specific one and image analysis processing is performed using a non-demosaic image as input data, and to perform image signal processing suitable for the demosaiced image when it is determined that the scene is not a specific one and image analysis processing is performed using a demosaiced image as input data. Therefore, the accuracy of the image analysis process can be improved.

[0103] Furthermore, in the signal processing device as an embodiment, the AI ​​model is an AI model having a neural network as a CNN, and is equipped with an image organizing unit (38) that generates a color-separated image, which is an image formed by collecting pixel values ​​for each pixel having the same position within the pixel unit from each pixel unit and arranging them in different areas on the same image plane, and the AI ​​processing unit performs image analysis processing using the color-separated image as input data. This makes the format of the input data to the AI ​​processing unit suitable for the construction of a CNN. Therefore, the accuracy of the image analysis process can be improved.

[0104] Furthermore, in the signal processing device according to the embodiment, the AI ​​processing unit performs image analysis processing using the non-demosaic image after shading correction as input data. As explained with reference to Figure 3, in scenes where a non-demosaic image should be used as input data, using a non-demosaic image after shading correction can improve the accuracy of analysis processing compared to using a non-demosaic image before shading correction. Therefore, according to the above configuration, it is possible to improve the accuracy of analysis processing in response to specific scenes such as dark scenes.

[0105] Moreover, the signal processing device (sensor unit 1A, camera device 10B) according to the embodiment includes a storage unit (sensor internal memory unit 7, sensor external memory unit 14) for storing non-demosaic images. The non-demosaic images stored in the storage unit can be used to re-learn the AI ​​model used in the AI ​​processing unit. By making it possible to re-learn the AI ​​model, it is possible to improve the accuracy of image analysis processing by the AI ​​processing unit.

[0106] Furthermore, the signal processing device as the embodiment includes a transmission processing unit (camera control unit 13, intra-sensor control unit 6A, camera control unit 13B) that performs processing to transmit the non-demosaic image stored in the storage unit to the outside of the device. This makes it possible to transmit non-demosaic images to be used for re-learning to an external device external to the signal processing device, in response to the case where re-learning of an AI model is performed in the external device. Therefore, it is possible to ensure that the AI ​​model is retrained appropriately.

[0107] Furthermore, the signal processing device (sensor unit 1, 1A) according to the embodiment is configured as a sensor device having a pixel array unit. This makes it possible to improve the accuracy of image analysis processing for a sensor device configured to be able to perform image analysis processing using an AI model.

[0108] In addition, the signal processing method as an embodiment is a signal processing method that performs image analysis processing using an AI model on an captured image obtained by a pixel array section configured by a plurality of pixel units arranged two-dimensionally in a predetermined pattern, each pixel unit being made up of a plurality of pixels that receive light in different wavelength bands and arranged two-dimensionally. The captured image is obtained by using as input data a non-demosaic image, which is an captured image in a state that has not been demosaiced. With such a signal processing method, it is possible to obtain the same functions and effects as those of the signal processing device according to the above embodiment.

[0109] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0110] <6. This Technology> The present technology can also be configured as follows. (1) The image processing device includes an AI processing unit that performs image analysis processing using an AI model on an image obtained by a pixel array unit configured by arranging a plurality of pixel units in a two-dimensional manner, each pixel unit being configured by arranging a plurality of pixels receiving light of different wavelength bands in a two-dimensional manner in a predetermined pattern, using as input a non-demosaic image, which is an image that has not been demosaiced. Signal processing device. (2) a demosaic processing unit that performs demosaic processing on an image obtained by the pixel array unit; and a control unit that controls input data of the AI ​​processing unit to be switched between a demosaic image that is an image after demosaic processing by the demosaic processing unit and the non-demosaic image based on a scene determination result for a scene to be imaged. The signal processing device according to (1) above. (3) The control unit is When the demosaiced image is used as the input data, an AI model trained using the demosaiced image as learning input data is used as the AI ​​model, and when the non-demosaic image is used as the input data, an AI model trained using the non-demosaic image as learning input data is used as the AI ​​model. The signal processing device according to (2) above. (4) The determination of the scene to be imaged is a determination of whether the scene is a dark scene, The control unit is When the scene is determined to be a dark scene, the non-demosaic image is input as the input data. The signal processing device according to (2) or (3). (5) an image signal processing unit that includes the demosaic processing unit and performs image signal processing on the captured image, The control unit is A signal processing parameter of the image signal processing unit is changed based on the scene determination result. A signal processing device according to any one of (2) to (4). (6) The AI ​​model is an AI model having a neural network as a CNN, an image organizing unit that generates a color separation image, which is an image formed by collecting pixel values ​​of pixels at the same position in each pixel unit from each of the pixel units and arranging them in different regions on the same image plane; The AI ​​processing unit performs the image analysis process using the color separation image as input data. A signal processing device according to any one of (1) to (5). (7) The AI ​​processing unit performs the image analysis process using the non-demosaic image after shading correction as input data. A signal processing device according to any one of (1) to (6). (8) a storage unit for storing the non-demosaic image; A signal processing device according to any one of (1) to (7). (9) a transmission processing unit for performing processing for transmitting the non-demosaic image stored in the storage unit to an outside of the device; The signal processing device according to (8) above. (10) A sensor device including the pixel array unit A signal processing device according to any one of (1) to (9). (11) For an image captured by a pixel array section configured by arranging a plurality of pixel units, each of which is configured by arranging a plurality of pixels that receive light of different wavelength bands in a two-dimensional manner in a predetermined pattern, an image analysis process is performed using an AI model with a non-demosaic image, which is the captured image in a state that has not been demosaiced, as input data. Signal processing methods. [Explanation of symbols]

[0111] 10, 10A, 10B Camera device 1,1A,1B Sensor section 2. Pixel array section 3. Image signal processing section 4 Pretreatment section 5 AI Processing Unit 6,6A Sensor internal control unit 7. Sensor internal memory 8. Output Data Generation Unit 9 Communication Interface (I / F) 11 Optical system 12 Communication Interface (I / F) 13,13B Camera control unit 14 Sensor external memory section 15 Communications Department P1 First AI model setting data P2 Second AI model setting data 31 Black level correction section 32 Shading correction section 33 Gain adjustment section 34 Demosaic Processing Unit 35 Color Correction Section 36 Gamma correction section 37 Dewarping section 38 Image Editing Department 39 Selector Pu pixel unit

Claims

1. The image processing device includes an AI processing unit that performs image analysis processing using an AI model on an image obtained by a pixel array unit configured by arranging a plurality of pixel units in a two-dimensional manner, the pixel units being each configured by arranging a plurality of pixels receiving light of different wavelength bands in a two-dimensional manner in a predetermined pattern, using as input data a non-demosaic image, which is an image obtained without being demosaiced. Signal processing device.

2. a demosaic processing unit that performs demosaic processing on an image obtained by the pixel array unit; A control unit is provided that controls input data of the AI ​​processing unit to be switched between a demosaic image, which is an image after demosaic processing by the demosaic processing unit, and the non-demosaic image based on a scene determination result for a scene to be imaged. The signal processing device according to claim 1 .

3. The control unit is When the demosaiced image is used as the input data, an AI model trained using the demosaiced image as learning input data is used as the AI ​​model, and when the non-demosaic image is used as the input data, an AI model trained using the non-demosaic image as learning input data is used as the AI ​​model. The signal processing device according to claim 2 .

4. The determination of the scene to be imaged is a determination of whether the scene is a dark scene, The control unit is When the scene is determined to be a dark scene, the non-demosaic image is input as the input data. The signal processing device according to claim 2 .

5. an image signal processing unit that includes the demosaic processing unit and performs image signal processing on the captured image, The control unit is A signal processing parameter of the image signal processing unit is changed based on the scene determination result. The signal processing device according to claim 2 .

6. The AI ​​model is an AI model having a neural network such as a CNN, an image organizing unit that generates a color separation image, which is an image formed by collecting pixel values ​​of pixels at the same position in each pixel unit from each of the pixel units and arranging them in different regions on the same image plane; The AI ​​processing unit performs the image analysis process using the color separation image as input data. The signal processing device according to claim 1 .

7. The AI ​​processing unit performs the image analysis process using the non-demosaic image after shading correction as input data. The signal processing device according to claim 1 .

8. a storage unit for storing the non-demosaic image; The signal processing device according to claim 1 .

9. a transmission processing unit for performing processing for transmitting the non-demosaic image stored in the storage unit to an outside of the device; The signal processing device according to claim 8.

10. A sensor device including the pixel array unit The signal processing device according to claim 1 .

11. For an image captured by a pixel array section configured by arranging a plurality of pixel units, each of which is configured by arranging a plurality of pixels that receive light of different wavelength bands in a two-dimensional manner in a predetermined pattern, an image analysis process is performed using an AI model as input data, using a non-demosaic image, which is the captured image in a state that has not been demosaiced. Signal processing methods.

Images