Signal processing device and signal processing method

Abstract

Processing accuracy of image analysis processing using an AI model is improved by using an image processing system including circuitry configured to acquire image data captured by an image sensor, process the image data to generate a non-demosaiced image, and transmit the non-demosaiced image to the image sensor. And selectively outputting the non-demosaiced image to an artificial intelligence model trained to perform image analysis.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Japanese priority patent application JP2023-094237, filed on June 7, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This technology relates to a signal processing apparatus and method for performing image analysis processing using an AI model. Background Technology

[0004] For example, techniques that use artificial intelligence (AI) models, including neural networks such as convolutional neural networks (CNNs), to perform image analysis processing (such as object detection processing or object recognition processing) on ​​captured images have been widely used.

[0005] In such image analysis processing, a 3ch image of red (R), green (G) and blue (B) is typically used as input data for the AI ​​model (see, for example, patent document PTL 1 below).

[0006] [List of Citations]

[0007] [Patent Literature]

[0008] PTL 1: JP 2011-170890 A Summary of the Invention

[0009] [Technical Issues]

[0010] In human image viewing, it is desirable to use 3ch images of R, G, and B to represent full color. However, using 3ch images of R, G, and B as input data may not be optimal for AI model performance.

[0011] In view of the above, this technology is proposed, with the aim of improving the processing accuracy of image analysis using AI models.

[0012] [Solution to the problem]

[0013] An image processing system includes circuitry configured to acquire image data captured by an image sensor, process the image data to generate a non-demosaic image, and selectively output the non-demosaic image to an artificial intelligence model trained to perform image analysis.

[0014] Because demosaicing involves spatial interpolation, it often results in deviations from the original pixel values. Furthermore, when a demosaiced image is used as input data for image analysis in an AI model, this pixel value deviation can reduce processing accuracy. By using a non-demosaiced image as input data for the image analysis process described above, the decrease in processing accuracy caused by this demosaicing process can be prevented. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating a schematic configuration example of a camera device including a signal processing apparatus according to the first embodiment.

[0016] Figure 2 This is a diagram illustrating an example of the internal configuration of an image signal processing unit included in a signal processing apparatus according to a first embodiment.

[0017] Figure 3 This is a graph illustrating the experimental results showing the changing characteristics of image analysis processing accuracy when the type of input data to the AI ​​processing unit changes.

[0018] Figure 4 This is an explanatory diagram of a color separation image according to an embodiment.

[0019] Figure 5 This is a diagram used to account for factors that reduce the accuracy of analysis processing when using demosaiced images.

[0020] Figure 6 This is a flowchart illustrating a specific processing example for implementing the analysis and processing method according to the first embodiment.

[0021] Figure 7 This is a block diagram used to describe a configuration example of a camera device according to the second embodiment.

[0022] Figure 8 This is a flowchart illustrating a specific processing example for implementing the analysis processing method based on the variant example.

[0023] Figure 9 This is an illustrative diagram of an exemplary configuration in which the AI ​​processing unit is located outside the sensor device. Detailed Implementation

[0024] In the following description, embodiments of the signal processing apparatus according to the present technology will be described in the following order with reference to the accompanying drawings.

[0025] <1. First Implementation Method>

[0026] (1-1. Example of camera setup configuration)

[0027] (1-2. Analysis and processing method according to the first embodiment)

[0028] (1-3. Processing procedure)

[0029] <2. Second Implementation Method>

[0030] <3. Variations>

[0031] <4. Overview of Implementation Methods>

[0032] <5. This technology>

[0033] <1. First Implementation Method>

[0034] (1-1. Example of camera setup configuration)

[0035] Figure 1 This is a block diagram illustrating a schematic configuration example of a camera device 10 including a signal processing apparatus according to the present technology.

[0036] As shown in the figure, the camera device 10 includes an optical system 11, a communication interface (I / F) 12, a camera control unit 13, a sensor external memory unit 14, a communication unit 15, and a sensor unit 1.

[0037] In the camera device 10, the sensor unit 1 corresponds to the signal processing device according to the first embodiment.

[0038] The sensor unit 1 is configured as, for example, an image sensor, such as a charge-coupled device (CCD) type image sensor or a complementary metal-oxide-semiconductor (CMOS) type image sensor.

[0039] The sensor unit 1 is configured to not only perform imaging functions, but also to perform image analysis processing using an artificial intelligence (AI) model, as image analysis processing of the captured images.

[0040] Furthermore, in sensor unit 1, multiple pixels that receive light of different wavelengths are formed into pixels that each have a light receiving element, and a captured image as a color image can be obtained.

[0041] In the camera device 10, the optical system 11 includes lenses such as cover lenses and focusing lenses, as well as an aperture (iris) mechanism. Light from the subject (incident light) is guided by the optical system 11 and focused onto the light-receiving surface of the sensor unit 1.

[0042] The communication interface (I / F) 12 is a communication interface used to perform data communication between the sensor unit 1 and the camera control unit 13.

[0043] The camera control unit 13 includes, for example, a microcomputer, which includes a central processing unit (CPU), a read-only memory (ROM), and a random access memory (RAM), and performs overall control of the camera device 10 by the CPU performing various types of processing according to a program stored in the ROM or a program loaded into the RAM.

[0044] The camera control unit 13 can receive various data from the sensor unit 1 and send various data to the sensor unit 1 through the communication interface 12.

[0045] The external sensor memory unit 14 is connected to the camera control unit 13. The external sensor memory unit 14 includes, for example, a non-volatile storage device such as a solid-state drive (SSD) or flash memory device, and is used to store information for various controls of the camera control unit 13. In addition, the external sensor memory unit 14 can also be used to store various data obtained by the sensor unit 1, such as image data captured by the sensor unit 1.

[0046] In the camera device 10 of this embodiment, the first AI model setting data P1 and the second AI model setting data P2 are stored in the sensor external memory unit 14, which will be described later.

[0047] Furthermore, the communication unit 15 is connected to the camera control unit 13. The communication unit 15 is configured to perform wired or wireless data communication with external devices. The communication unit 15 can also be configured to have network communication capabilities, and in this case, for example, the camera control unit 13 can exchange data with a predetermined device (e.g., a server device) on a predetermined network such as the Internet via the communication unit 15.

[0048] As shown in the figure, the sensor unit 1 includes a pixel array unit 2, an image signal processing unit 3, a preprocessing unit 4, an AI processing unit 5, an in-sensor control unit 6, an in-sensor memory unit 7, an output data generation unit 8, and a communication interface (I / F) 9.

[0049] In pixel array unit 2, for example, multiple pixels are arranged in two dimensions in the horizontal and vertical directions, and each pixel has a light receiving element (photoelectric conversion element) such as a photodiode.

[0050] Specifically, pixel array unit 2 includes pixel unit Pu (see description later). Figure 4 A), wherein multiple pixels that receive light of different wavelengths are arranged in a predetermined pattern in two dimensions, and multiple pixel units Pu are arranged in two dimensions.

[0051] In this embodiment, a pixel unit Pu is formed by arranging three types of pixels—R pixels that receive R (red) light, G pixels that receive G (green) light, and B pixels that receive B (blue) light—in a predetermined array pattern. Specifically, in this example, the pixel unit Pu is formed by arranging the R pixels, G pixels, and B pixels in a Bayer array.

[0052] The pixel array unit 2 also includes configurations for acquiring image data as digital data, such as readout circuitry for reading the value of each pixel (received light value) and analog-to-digital converter (ADC) for digitally sampling the pixel values ​​as analog signals.

[0053] The image signal processing unit (ISP: imaging signal processor) 3 receives image data (captured image data) obtained through the pixel array unit 2 and performs various types of image signal processing.

[0054] Note that the internal configuration of the image signal processing unit 3 will be described again later.

[0055] The preprocessing unit 4 receives the image data after image signal processing by the image signal processing unit 3, and performs image signal processing as preprocessing for image analysis processing by the AI ​​processing unit 5. Specifically, the preprocessing unit 4 in this example is configured to at least perform image scaling processing.

[0056] AI processing unit 5 uses the image data output from preprocessing unit 4 as input data and uses an AI model to perform image analysis and processing.

[0057] AI processing unit 5 includes, for example, a digital signal processor (DSP), and can switch the AI ​​model used for image analysis processing by switching processing parameters.

[0058] The AI ​​processing unit 5 in this example is configured to perform image analysis processing using an AI model that includes a neural network, such as a convolutional neural network (CNN).

[0059] Here, as an example, the camera device 10 of this example is deployed in a commercial facility such as a supermarket or department store to perform image analysis processing with customers as the target subject. Specifically, it is assumed that object detection processing with people as the target subject is performed.

[0060] In this context, the AI ​​model in AI processing unit 5 is a machine trained to perform object detection processing on human subjects. Here, object detection processing includes identifying regions containing target objects as so-called bounding boxes.

[0061] The sensor internal control unit 6 includes a microcomputer, which includes, for example, a CPU, ROM, RAM, etc., and integrates the control of the operation of the sensor unit 1.

[0062] For example, the control unit 6 within the sensor controls the operation of the pixel array unit 2. Specifically, it controls the start / stop of operations, etc.

[0063] Furthermore, the control unit 6 within the sensor also controls the operation of the image signal processing unit 3, the preprocessing unit 4, and the AI ​​processing unit 5. Regarding the operation control of the image signal processing unit 3 and the preprocessing unit 4, the control unit 6 within the sensor can control the processing parameters for various types of processing.

[0064] Furthermore, under the control of the AI ​​processing unit 5, the control unit 6 within the sensor can perform switching control of the AI ​​model.

[0065] The sensor-internal memory unit 7 is connected to the sensor-internal control unit 6. The sensor-internal memory unit 7 includes, for example, a non-volatile storage device such as a flash memory device, and is used to store information for various controls performed by the sensor-internal control unit 6.

[0066] The output data generation unit 8 receives the analysis and processing results from the AI ​​processing unit 5 and the image data output from the image signal processing unit 3, and generates output data to be output to the outside of the sensor unit 1. The output data generation unit 8 generates the output data based on instructions from the sensor-internal control unit 6. For example, according to instructions from the sensor-internal control unit 6, it can switch whether to use both the analysis and processing results of the AI ​​processing unit 5 and the image data as output data, or only the analysis and processing results of the AI ​​processing unit 5 as output data.

[0067] Communication interface 9 is a communication interface that enables data to be output from the inside of sensor unit 1 to the outside of sensor unit 1 and data to be input from the outside of sensor unit 1 to the inside of sensor unit 1, and performs data communication with the aforementioned communication interface 12 according to a predetermined communication data format.

[0068] The output data can be output to the outside of the sensor unit 1 (in this example, the camera control unit 13) via the communication interface 9.

[0069] In addition, the sensor control unit 6 can communicate with the camera control unit 13 via the communication interface 9.

[0070] Figure 2 This is a diagram illustrating an example of the internal configuration of the image signal processing unit 3.

[0071] Notice, Figure 2 It shows Figure 1 The internal configuration example of the preprocessing unit 4, AI processing unit 5, sensor-in-control unit 6, and image signal processing unit 3 shown.

[0072] Image data as raw (RAW) data from Figure 1 The pixel array unit 2 shown is input to the image signal processing unit 3. The RAW data referred to here is image data obtained by reading the value of each pixel in raster order, that is, in this example, image data that maintains the pixel array state of the Bayer array.

[0073] As shown in the figure, the image signal processing unit 3 includes a black level correction unit 31, a gain adjustment unit 33, a demosaic processing unit 34, a color correction unit 35, a gamma correction unit 36, and a distortion correction processing unit 37. It can sequentially perform shadow correction processing, gain adjustment processing, demosaic processing, color correction processing, gamma correction processing, and distortion correction processing on the image data input as RAW data from the pixel array unit 2.

[0074] Here, the gain adjustment processing of the gain adjustment unit 33 includes: overall gain adjustment, which adjusts the brightness distribution of the entire image regardless of the color of the pixels; and automatic white balance (AWB) processing, which is gain adjustment processing for each color.

[0075] Furthermore, the demosaicing processing unit 34 performs the following process: by performing spatial interpolation processing on each of the R, G, and B colors according to the input image data in the Bayer array state, image data is generated as R image, G image, and B image with the same number of pixels as the input image data.

[0076] The color correction unit 35 performs color correction processing on the de-mosaiced image data through linear matrix processing.

[0077] In addition, the distortion correction unit 37 performs at least lens distortion correction processing as distortion correction processing.

[0078] Here, the image signal processing unit 3 in this embodiment also includes an image organization unit 38 and a selector 39, which will be described again later.

[0079] (1-2. Analysis and processing method according to the first embodiment)

[0080] As mentioned above, in applications where people appreciate images, it is desirable to use 3ch images of R, G, and B colors because full color representation can be performed. However, in image analysis and processing using AI models, using 3ch images of R, G, and B colors as input data is not necessarily optimal in terms of analysis and processing accuracy.

[0081] In view of this, this embodiment proposes a method for using unde-mosaic images as input data for AI processing unit 5, wherein the unde-mosaic image is an image captured in a state without de-mosaic processing.

[0082] Since demosaicing involves spatial interpolation, it often results in deviations from the original pixel values. Furthermore, when demosaiced images are used as input data for image analysis processing by an AI model, the accuracy of the image analysis processing may be reduced due to these pixel value deviations.

[0083] On the other hand, by using unde-mosaiced images as input data to the AI ​​processing unit 5 as described above, the decrease in processing accuracy caused by such de-mosaicing can be prevented, and the processing accuracy of image analysis processing using the AI ​​model can be improved.

[0084] Furthermore, in this embodiment, the input data of the AI ​​processing unit 5 switches between de-mosaic images and non-de-mosaic images. The de-mosaic image is the image after de-mosaic processing by the de-mosaic processing unit 34, and the non-de-mosaic image is based on the scene determination result of the scene to be imaged.

[0085] Here, when image analysis processing using an AI model is performed in a sensor device as described in this embodiment, the resources (storage and computing power) available for image analysis processing are often insufficient.

[0086] For example, in image analysis processing using AI models (such as object detection processing), it is practically required to maintain accuracy even when the scene captured by the camera undergoes some degree of change. For instance, the scene captured by the camera (such as a bright / dark scene like day and night, a scene where the subject to be analyzed, such as a person, is relatively far away, or a scene where the subject to be analyzed, such as a person, is nearby) can change over time. It is expected that the accuracy of the analysis processing will be maintained above a certain level in response to such changes in the scene.

[0087] To maintain accuracy regarding scene changes, one could envision using images captured for each scene as input data to train an AI model, thereby performing machine training and creating an AI model capable of incorporating differences between scenes.

[0088] However, in order to incorporate the differences between scenarios in this way, a relatively large number of filter coefficients and a relatively large number of network layers are requested as AI models, and the resources required to implement the AI ​​models increase.

[0089] Therefore, the method of incorporating scene changes into the AI ​​model can be applied when the entity performing image analysis processing is a computing device with relatively abundant resources, specifically, a computer device outside the camera device 10. However, when image analysis processing is performed in the camera device 10 with limited resources, specifically, when image analysis processing is performed in the sensor device (sensor unit 1), it is difficult to apply the method of incorporating scene changes into the AI ​​model.

[0090] Therefore, in this embodiment, the method of switching the input data of the AI ​​processing unit 5 between de-mosaic images and non-de-mosaic images based on the scene determination results described above is used.

[0091] Figure 3 Experimental results are shown regarding the changes in image analysis processing accuracy as the type of input data to AI processing unit 5 changes.

[0092] Specifically, Figure 3 The experimental results show the outcome of performing object detection processing with the human body as the target subject as image analysis processing using an AI model, and demonstrate the results in... Figure 2 Zhongyou <1> to <6> The image data at each extraction location is used as input data for each scene, both dark and bright, to measure the accuracy of the image analysis processing.

[0093] Here, the evaluation value is obtained as the percentage of people that can be accurately detected when the object detection process with the human body as the target subject is performed multiple times.

[0094] Here, a dark scene refers to a scene where the subject appears dark, and can be defined, for example, as a scene where the subject's brightness value is equal to or less than a specific value. A bright scene refers to a scene where the subject appears bright, and can be defined, for example, as a scene where the subject's brightness value exceeds a specific value.

[0095] refer to Figure 2 It can be seen that the extraction location <1> It is the position immediately preceding the input image signal processing unit 3, where the position is extracted. <2> It is located between the shadow correction unit 32 and the gain adjustment unit 33. Extraction location. <3> It is the position between the gain adjustment unit 33 and the de-mosaic processing unit 34.

[0096] Extraction location <1> to <3> These are all positions before mosaic removal. In this experiment, the image data described later as the color-separated image was used as the extraction positions corresponding to those extracted using an AI model with a CNN as the AI ​​model. <1> to <3> Image data.

[0097] Figure 4 This is an explanatory diagram of a color separation image according to an embodiment.

[0098] Figure 4 Figure A shows the pixel array (in this example, a Bayer array) in pixel array unit 2. As shown, in this example, pixel unit Pu is formed by arranging four pixels R, G, G, and B into a predetermined array pattern according to the Bayer format.

[0099] Here, color-separated image refers to an image formed by collecting pixel values ​​from each pixel unit Pu at the same pixel unit location and setting the pixel values ​​in different regions on the same image plane.

[0100] Figure 4 B illustrates a color-separated image generated in the case of a Bayer array. In the illustrated Bayer array, for each pixel unit Pu, the pixel values ​​of pixels having the same pixel unit position (R, G, G, and B pixels) are collected and set in different regions on the same image plane, thereby generating a color-separated image. This color-separated image includes an image region in which the pixel values ​​of the R pixels in each pixel unit Pu are arranged; an image region in which the pixel values ​​of one G pixel in each pixel unit Pu are arranged; an image region in which the pixel values ​​of another G pixel in each pixel unit Pu are arranged; and an image region in which the pixel values ​​of the B pixels in each pixel unit Pu are arranged.

[0101] By using the color-separated image as described above as the input data to the AI ​​processing unit 5, the format of the input data to the AI ​​processing unit is suitable for the configuration of CNN, and the accuracy of image analysis and processing can be improved.

[0102] return Figure 3 Describe it.

[0103] Extraction location <4> to <6> It's the extraction location after removing the mosaic. <4> It is located between the de-mosaic processing unit 34 and the color correction unit 35. <5> It is located between the color correction unit 35 and the gamma correction unit 36. <6> It is located between the gamma correction unit 36 ​​and the distortion correction unit 37.

[0104] according to Figure 3As a result, in bright scenes, regardless of... <1> to <6> The accuracy of the input data depends on its data type, with an evaluation value exceeding 90%. This demonstrates that, in bright scenes, the accuracy of subject detection processing does not depend on the data type.

[0105] On the other hand, in dark scenes, in extracting location <4> Subsequently, it was confirmed that the evaluation value decreased significantly. That is, when the demosaic image was used as input data, the processing accuracy decreased significantly.

[0106] According to the experiment, the data type before mosaic removal <1> to <3> In this case, the evaluation value remains above 60%, while in the data type after de-mosaicing... <4> <5> In this case, the evaluation value is significantly lower than 60%. When using data types... <6> In the case of (i.e., the de-mosaic image after gamma correction), the evaluation value can be confirmed to increase slightly, but less than 60%.

[0107] In dark scenes, use data types <2> In this case, the image after shadow correction and before gain adjustment has the highest evaluation value, which is approximately 75%.

[0108] As in the experimental results above, when using de-mosaic images, the degradation in accuracy in image analysis and processing is considered to be due to the significant deviation from the original pixel values ​​caused by spatial interpolation processing as a de-mosaic process in dark scenes.

[0109] Furthermore, when resources are limited, the input data to the AI ​​processing unit 5 is adjusted (e.g., reduced) to a predetermined size in the preprocessing unit 4 to reduce the amount of input data to the AI ​​processing unit 5. This is to account for the potential decrease in processing accuracy when using de-mosaic images.

[0110] As a specific example, for instance, given that the input data volume of AI processing unit 5 is limited to 1280×960 pixels, when using a non-de-mosaic image, such as Figure 5 As shown, an image with a resolution (information density) of 1280×960 pixels can be used as input data, while when using a de-mosaic image, the resolution of each of the R, B, and G images, which are input data to the AI ​​processing unit 5, is reduced to 739×554 pixels.

[0111] The reduced resolution in this approach is also considered a reason for the decreased processing accuracy when de-mosaic images are used as input data.

[0112] Considering the above points, in this embodiment, as a method for determining the scene to be imaged, a method is used in which it is determined whether the scene is a dark scene, and if the scene is determined to be a dark scene, the unde-mosaic image is input as input data to the AI ​​processing unit 5.

[0113] Therefore, when AI models that can incorporate differences between scenes cannot be used due to resource constraints, the decrease in image analysis processing accuracy in dark scenes can be suppressed, and the decrease in image analysis processing accuracy dependent on the scene can also be suppressed.

[0114] In this example, when indicated as <2> The unde-mosaic image after shadow correction is used as input data in dark scenes. Specifically, in dark scenes, the color-separated image of the unde-mosaic image obtained between the shadow correction unit 32 and the gain adjustment unit 33 is used as input data.

[0115] Therefore, in the camera device 10 of this embodiment, the image signal processing unit 3 is provided with an image organization unit 38.

[0116] like Figure 2 As shown, the image organization unit 38 receives the shadow-corrected, unde-mosaic image output from the shadow correction unit 32, and uses a reference... Figure 3 The above method rearranges pixel values ​​to generate a color-separated image of the non-demosaic image after shadow correction.

[0117] Furthermore, in this embodiment, if it is determined that the scene is not a dark scene as a result of the determination of whether the scene is a dark scene, the de-mosaic image is input as input data to the AI ​​processing unit 5.

[0118] Specifically, in this example, when it is determined that the scene is not dark, the de-mosaic image output from the distortion processing unit 37 is provided as input data for the AI ​​processing unit 5.

[0119] In order to enable this input data switching based on dark / bright scenes, the image signal processing unit 3 is provided with a selector 39.

[0120] Selector 39 receives inputs of a non-demosaic image, which is a color-separated image in image organization unit 38, and inputs of a demosaic image output from distortion correction processing unit 37, in order to output the image indicated by control unit 6 within the sensor to preprocessing unit 4.

[0121] In this example, the sensor-in-control unit 6 determines whether it is a dark scene based on the de-mosaic image. Specifically, in this example, the sensor-in-control unit 6 determines whether it is a dark scene based on the de-mosaic image output by the distortion correction processing unit 37.

[0122] Then, based on the result of determining whether it is a dark scene, the control unit 6 inside the sensor causes the selector 39 to output (select) a de-mosaic image if it is determined that the scene is not a dark scene, and causes the selector 39 to output (select) a non-de-mosaic image if it is determined that the scene is a dark scene.

[0123] Furthermore, in this embodiment, the control unit 6 within the sensor performs control such that when the de-mosaic image is used as input data for the AI ​​processing unit 5, the AI ​​model trained using the de-mosaic image as input data for training is used as the AI ​​model, and when a non-de-mosaic image is used as input data for the AI ​​processing unit 5, the AI ​​model trained using the non-de-mosaic image as input data for training is used as the AI ​​model.

[0124] In this example, the setup data of the AI ​​processing unit 5 used to implement the previous AI model (i.e., the AI ​​model trained using de-mosaic images as input data for training) is stored as the first AI model setup data P1. Figure 1 The sensor external memory unit 14 shown contains the setting data of the AI ​​processing unit 5, which is used to implement the next AI model (i.e., the AI ​​model trained using non-demosaic images as input data for training). The setting data is stored as the second AI model setting data P2 in the sensor external memory unit 14.

[0125] Here, the first AI model setup data P1 and the second AI model setup data P2 are data that include parameters such as filter coefficients used in filtering processes such as convolution in CNN, as well as various parameters related to the structure of the neural network.

[0126] If the scene is determined not to be a dark scene, the sensor-in-control unit 6 instructs the camera control unit 13 to read the first AI model setting data P1 from the sensor-external memory unit 14 via the communication interface 9, and transmits the first AI model setting data P1 to the sensor-in-control unit. Then, by executing the parameter settings of the AI ​​processing unit 5 according to the transmitted first AI model setting data P1, the AI ​​processing unit 5 can perform image analysis processing using an AI model trained with de-mosaic images as input data for training.

[0127] Furthermore, in the case of a dark scene, the sensor-in-control unit 6 instructs the camera control unit 13 to read the second AI model setting data P2 from the sensor-external memory unit 14 via the communication interface 9, and transmits the second AI model setting data P2 to the sensor-in-control unit. The AI ​​processing unit 5 sets the parameters according to the transmitted second AI model setting data P2, so that the AI ​​processing unit 5 can perform image analysis processing through the AI ​​model trained using non-de-mosaic images as input data for training.

[0128] (1-3. Processing procedure)

[0129] Reference Figure 6 The flowchart describes a specific processing example of the analysis and processing method as described in the first embodiment above, which is executed by the control unit 6 within the sensor.

[0130] Notice, Figure 6 The processing shown is executed by the CPU in the sensor-in-control unit 6 based on a program stored in a predetermined storage device such as the ROM of the sensor-in-control unit 6. However, in the following description, for illustrative purposes, the entity executing the processing is referred to as the sensor-in-control unit 6.

[0131] In this example, the control unit 6 within the sensor starts in response to activation. Figure 6 The processing is shown in the diagram.

[0132] First, in step S101, the control unit 6 within the sensor determines whether the scene determination execution conditions are met. That is, it determines whether the predetermined conditions are met as conditions for performing the scene determination process in step S103 (described later, in this example, the process of determining whether the scene is a dark scene).

[0133] For example, the scene determination process in step S103 can be executed periodically at predetermined time intervals. In this case, the scene determination conditions only need to pass through a certain period of time.

[0134] Note that the scene determination execution conditions are the received external instructions, the results of simple brightness detection using a light intensity sensor, etc., and the change in brightness is above a specified amount, etc., and the scene determination execution conditions are not limited to specific conditions.

[0135] If it is determined in step S101 that the scenario determination execution conditions are not met, the sensor-internal control unit 6 proceeds to step S102 and determines whether the processing has ended, i.e., whether the predetermined action conditions have been met. Figure 6 The conditions under which a series of processes should end (such as power outage) are shown in the diagram.

[0136] If it is determined in step S102 that the process has not ended, the control unit 6 inside the sensor returns to step S101.

[0137] That is, through the processing of steps S101 and S102, a loop process is formed to establish the execution conditions for the waiting scenario or the processing termination conditions.

[0138] If the scene determination execution conditions are met in step S101, the control unit 6 within the sensor proceeds to step S103 and performs the scene determination process. That is, in this example, it is determined whether the scene is a dark scene based on the de-mosaic image output from the distortion correction processing unit 37. Specifically, it is determined whether the average brightness value of the target subject's region is equal to or less than a predetermined brightness value. At this time, it is conceivable to identify the target subject's region based on, for example, the target subject's region information obtained as a result of the object detection processing of the AI ​​processing unit 5. Note that in the state where the object detection processing of the AI ​​processing unit 5 has not started, for example, it is conceivable to identify the region of a moving body detected by motion detection processing such as inter-frame difference detection processing as the region of the target subject.

[0139] In step S104, following step S103, the control unit 6 within the sensor determines whether it is a dark scene. That is, based on the result of the determination process in step S103, it determines whether a determination result representing a dark scene has been obtained.

[0140] If it is determined in step S104 that the scene is not dark, the control unit 6 within the sensor proceeds to step S105 and issues an instruction to select a demosaic image. That is, the selector 39 is instructed to selectively output the demosaic image input from the distortion correction processing unit 37.

[0141] Then, in step S106 following step S105, the sensor-internal control unit 6 proceeds with the setting process of the first AI model. That is, the control unit instructs the camera control unit 13 to read the first AI model setting data P1 from the sensor external memory unit 14 and transmit it to the control unit via the communication interface 9, and performs parameter setting of the AI ​​processing unit 5 according to the transmitted first AI model setting data P1.

[0142] Therefore, when it is determined that the scene is not a dark scene, the AI ​​processing unit 5 uses an AI model trained with the de-mosaic image as input data and the de-mosaic image as training input data to perform image analysis processing.

[0143] In response to the execution of the setting process in step S106, the control unit 6 within the sensor returns to step S101.

[0144] Furthermore, in step S104 above, if it is determined that the scene is a dark scene, the control unit 6 in the sensor proceeds to step S107 and instructs the selector 39 to select a non-demosaic image, that is, to select and output a non-demosaic image by means of the color separation image input from the image organization unit 38.

[0145] Furthermore, in step S108 following step S107, the sensor-internal control unit 6 performs the setting process for the second AI model. That is, the control unit instructs the camera control unit 13 to read the second AI model setting data P2 from the sensor external memory unit 14 and transmit it to the control unit via the communication interface 9, and performs parameter setting of the AI ​​processing unit 5 according to the transmitted second AI model setting data P2.

[0146] Therefore, when the scene is determined to be a dark scene, the AI ​​processing unit 5 uses an AI model to perform image analysis processing. The AI ​​model is trained using the non-demosaic image of the color-separated image as input data and the non-demosaic image of the color-separated image as input data for training.

[0147] In response to the execution of the setting process in step S108, the control unit 6 within the sensor returns to step S101.

[0148] Furthermore, if the above process is determined to be complete in step S102, the sensor internal control unit 6 will terminate. Figure 6 The series of processes shown in the figure.

[0149] Note that regarding scene determination processing, it is also conceivable that the AI ​​processing unit 5 could perform the determination of whether a scene is a dark scene by providing a function to determine whether the scene is a dark scene as part of the AI ​​model in the AI ​​processing unit 5. In this case, the control unit 6 within the sensor would execute the control of the selector 39 (and the switching control of the AI ​​model) based on the result of the determination processing of whether the scene is a dark scene as described above by the AI ​​processing unit 5.

[0150] <2. Second Implementation Method>

[0151] Next, the second embodiment will be described.

[0152] In the second embodiment, it is assumed that the AI ​​model used in the AI ​​processing unit 5 is retrained to accumulate unde-mosaic images.

[0153] Figure 7 This is a block diagram illustrating a configuration example of a camera device 10A with non-demosaic image accumulation function according to a second embodiment.

[0154] Note that, although for ease of illustration, in Figure 7The illustrations of the optical system 11, pixel array unit 2, sensor internal memory unit 7, and output data generation unit 8 are omitted. However, like the camera device 10, the camera device 10A of the second embodiment also includes the optical system 11, pixel array unit 2, sensor internal memory unit 7, and output data generation unit 8. Although the description of the first AI model setting data P1 and the second AI model setting data P2 in the sensor external memory unit 14 is omitted, the first AI model setting data P1 and the second AI model setting data P2 are also stored in the sensor external memory unit 14 of the camera device 10A, just like in the camera device 10.

[0155] In the following description, the same reference numerals are given to parts similar to those already described, and their descriptions will be omitted.

[0156] Camera device 10A and in Figure 1 The difference in the camera device 10 shown is that a sensor unit 1A is provided instead of sensor unit 1.

[0157] The difference between sensor unit 1A and sensor unit 1 is that sensor internal control unit 6A is set to replace sensor internal control unit 6.

[0158] The difference between the in-sensor control unit 6A and the in-sensor control unit 6 is that the in-sensor control unit 6A performs the processing for accumulating the non-demosaic image in the memory of the camera device 10A (specifically, in this example, in the external sensor memory unit 14).

[0159] As shown in the figure, in sensor unit 1A, the non-demosaic image of the color separation image generated by image organization unit 38 can be input not only to selector 39, but also to communication interface 9.

[0160] The sensor-in-control unit 6A causes the camera control unit 13 to transmit the non-de-mosaic image of the color separation image via the communication interface 9, and instructs the camera control unit 13 to store the non-de-mosaic image in the sensor-external memory unit 14 via the communication interface 9.

[0161] Therefore, captured images of non-de-mosaic images that can be obtained in the actual use environment of the camera device 10A can be accumulated in the sensor external memory unit 14.

[0162] Furthermore, the sensor-in-control unit 6A performs the process of transferring the unde-mosaiced image accumulated in the sensor-external memory unit 14 to the outside of the camera device 10A. Specifically, the sensor-in-control unit 6A instructs the camera control unit 13 to perform the process of transferring the unde-mosaiced image accumulated in the sensor-external memory unit 14 to the external device based on instructions from an external device (e.g., a server device, etc.) of the camera device 10A.

[0163] Therefore, in cases where the retraining of the AI ​​model used by the AI ​​processing unit 5 is performed by an external device, the unde-mosaic image used for retraining can be transmitted to the external device.

[0164] Note that although the above describes the configuration of the sensor-in-sensor control unit 6A instructing the camera control unit 13 to perform transmission processing on the accumulated non-demosaic image, it is also conceivable that the camera control unit 13 performs transmission processing on the accumulated non-demosaic image based on instructions from an external source.

[0165] Furthermore, in the above description, the accumulation unit for accumulating non-demosaic images is described as an example of the external memory unit 14, but the accumulation unit can be a memory in the sensor unit 1A, such as the internal memory unit 7.

[0166] Here, if the sensor unit 1A includes a communication unit that can communicate directly with an external device of the camera device 10A, the entity that performs the processing of the accumulated non-demosaic images can be the sensor-internal control unit 6A.

[0167] <3. Variations>

[0168] Note that the implementation methods are not limited to the specific examples described above, and can be configured in various variations.

[0169] For example, it is conceivable to change the signal processing parameters of the image signal processing unit 3 based on the scene determination result. As an example, it is conceivable to change the parameters of the gain adjustment processing of the overall gain adjustment of the gain adjustment unit 33 and the parameters of the gamma correction processing of the gamma correction unit 36 ​​based on the determination result regarding whether it is a dark scene. Specifically, it is conceivable that when the target subject has a dark scene, parameters for preventing black compression are used to perform gamma correction and overall gain adjustment, and when the target subject is bright and forms a bright scene (in the case of a non-dark scene), parameters for preventing overexposure are used to perform gamma correction and overall gain adjustment.

[0170] Here, adjusting the parameters for dark / bright scenes as described above is an effective method when object detection processing is performed as image analysis processing using an AI model.

[0171] In image analysis processing using AI models, segmentation processes such as semantic segmentation are performed, with category recognition occurring for each block, such as for each pixel. Therefore, processing accuracy may degrade in blocks of bright or dark areas when gamma correction or overall gain adjustment is performed.

[0172] Therefore, when segmentation processing is performed as image analysis processing using an AI model, it is conceivable that images captured without gamma correction or overall gain adjustment would be used as input data.

[0173] Figure 8 This is a flowchart illustrating a specific processing example for implementing the analysis and processing method as a variant, in which the signal processing parameters of the image signal processing unit 3 are changed based on the scenario determination results described above.

[0174] Here, Figure 8 The processing shown can be performed by the sensor-in-sensor control unit 6 or the sensor-in-sensor control unit 6A, but here the processing will be described as being performed by the sensor-in-sensor control unit 6.

[0175] Figure 8 The processing shown is Figure 6 The difference in the process shown is the addition of steps S110 and S111 as shown in the attached figures.

[0176] Specifically, in this case, the sensor-in-control unit 6 performs signal processing parameter adjustment processing in step S110 in response to the setting process of the first AI model in step S106. In this example, the signal processing parameter adjustment processing in step S110 corresponds to the case that the scene is determined to be not a dark scene, and the control processing is performed to set parameters in the gain adjustment unit 33 and the gamma correction unit 36 ​​to prevent overexposure.

[0177] Furthermore, in this case, the sensor-in-control unit 6 performs signal processing parameter adjustment processing in step S111 in response to the second AI model setting process performed in step S108. Specifically, the signal processing parameter adjustment processing in step S111 of this example corresponds to the case where the scene is determined to be a dark scene, and the control processing is performed to set the aforementioned parameters for preventing black compression in the gain adjustment unit 33 and the gamma correction unit 36.

[0178] In this case, even if any signal processing parameter adjustment is performed in steps S110 and S111, the control unit 6 inside the sensor will return to step S101.

[0179] Note that the above description provides an example of how parameters related to overall gain adjustment and gamma correction can be changed based on scene determination results. However, other parameters, such as AWB parameters, can be considered if they change based on scene determination results.

[0180] Furthermore, while the above description has illustrated an example of a configuration where the AI ​​processing unit 5 is located within the sensor device, it is also conceivable to have a configuration where the AI ​​processing unit 5 is located outside the sensor device, such as in... Figure 9 The camera device 10B shown is included.

[0181] Specifically, in the camera device 10B, a sensor unit 1B comprising only the pixel array unit 2 and the communication interface 9 is provided instead of the sensor unit 1 (or sensor unit 1A). In this case, the image signal processing unit 3, the preprocessing unit 4, and the AI ​​processing unit 5 are disposed outside the sensor unit 1B. Furthermore, the camera device 10B includes a camera control unit 13B instead of the camera control unit 13.

[0182] As shown in the figure, in the camera device 10B, the captured image, which is RAW data obtained by the pixel array unit 2, is input to the image signal processing unit 3 via the communication interface 9 and the communication interface 12. Then, the image data selected by the selection unit 39 of the image signal processing unit 3 is input to the AI ​​processing unit 5 via the preprocessing unit 4.

[0183] Similar to the in-sensor control unit 6, the camera control unit 13B controls the selector 39 based on the scene determination result. Furthermore, like the in-sensor control unit 6, the camera control unit 13B performs AI model setting processing of the AI ​​processing unit 5 based on the first AI model setting data P1 and the second AI model setting data P2 stored in the external sensor memory unit 14.

[0184] Furthermore, in implementing the non-demosaic image accumulation described in the second embodiment, the camera control unit 13B performs the process of storing the non-demosaic image obtained by the image signal processing unit 3 in a memory (such as the external sensor memory unit 14) in the camera device 10B. Additionally, in this case, the camera control unit 13B may also perform the process of transmitting the accumulated non-demosaic image to an external device.

[0185] Furthermore, the above description illustrates an example of using a color-separated image as a non-demosaic image to input into the AI ​​processing unit 5, but non-demosaic images in data forms other than color-separated images can also be used as input data for the AI ​​processing unit 5.

[0186] Furthermore, in the above description regarding the switching of input data for AI processing unit 5, an example was described of switching between depixelated and non-depixelated images based on a determination of whether the scene is dark. However, it is also conceivable to switch between depixelated and non-depixelated images based on criteria other than the determination of whether the scene is dark.

[0187] For example, one could imagine switching the input data to a non-de-mosaic image when the target subject is imaged at a small size and resolution is required, and switching the input data to a de-mosaic image when the target subject is imaged at a large size.

[0188] Furthermore, regarding scene determination, it is conceivable to determine three or more scenes, and to change the extraction location of the input data of the AI ​​processing unit 5 for each scene. For example, the input data could be a non-demosaic image after shadow correction in the first scene, a non-demosaic image after AWB processing in the second scene, and a demosaic image in the third scene.

[0189] Furthermore, the above description has illustrated the case where pixel array unit 2 is configured to receive only the three wavelengths R, G, and B individually. However, this technology can also be suitably applied to pixel array units configured to receive four or more wavelengths of light individually (such as pixel array units in a multispectral camera).

[0190] Furthermore, in the AI ​​processing unit 5, assuming the switching and use of multiple AI models with different analysis tasks, it is also conceivable to determine whether to set the input data of the AI ​​processing unit 5 as a non-demosaic image based on the type of AI model used by the AI ​​processing unit 5 (i.e., the type of analysis task).

[0191] <5. Overview of Implementation Methods>

[0192] As described above, the signal processing apparatus (sensor unit 1, sensor unit 1A, camera device 10B) of this embodiment includes an AI processing unit (5), which uses an AI model with a non-de-mosaic image as input data to perform image analysis processing. The non-de-mosaic image is an image captured without de-mosaic processing. The captured image is obtained by a pixel array unit (2), which is composed of a plurality of pixel units arranged in two dimensions. Each pixel unit includes a plurality of pixels that receive light of different wavelengths. The plurality of pixels are arranged in two dimensions in a predetermined pattern.

[0193] Since demosaicing involves spatial interpolation, deviations from the original pixel values ​​often occur. When a demosaiced image is used as input data for image analysis processing in an AI model, this pixel value deviation can reduce processing accuracy. By using a non-demosaiced image as input data for the image analysis processing described above, the decrease in processing accuracy due to demosaicing can be prevented. Therefore, the processing accuracy of image analysis processing using an AI model can be improved.

[0194] Furthermore, the signal processing apparatus according to the embodiment includes: a demosaic processing unit (34) that performs demosaic processing on the captured image obtained by the pixel array unit; and a control unit (in-sensor control unit 6, in-sensor control unit 6A, camera control unit 13B) that performs control to switch the input data of the AI ​​processing unit between a demosaic image and a non-demosaic image based on the scene determination result of the scene to be imaged, wherein the demosaic image is the image after demosaic processing by the demosaic processing unit.

[0195] For reference Figure 3 As described, experiments have confirmed that replacing demosaic images with non-demosaic images can improve the accuracy of image analysis processing, depending on the scene. That is, even when a model lacking the ability to incorporate differences between scenes is used as an AI model, the accuracy of image analysis processing can be improved by employing the method of replacing demosaic images in a specific scene with non-demosaic images.

[0196] As described above, by switching the input data of the AI ​​processing unit between de-mosaic and non-de-mosaic images based on the scene determination results, the non-de-mosaic image can be used as input data in a specific scene, thereby improving the accuracy of image analysis and processing when the non-de-mosaic image is used as input data.

[0197] Therefore, even when AI models capable of incorporating differences between scenes cannot be used due to resource constraints, the degradation of image analysis processing accuracy based on the scene can be suppressed.

[0198] Furthermore, in the signal processing apparatus according to the embodiment, the control unit performs control such that when the demosaic image is the input data, an AI model trained using the demosaic image as input data for training is used as an AI model, and when a non-demosaic image is the input data, an AI model trained using the non-demosaic image as input data for training is used as an AI model.

[0199] Therefore, appropriate image analysis processing based on the scene can be performed as image analysis processing using AI models.

[0200] Furthermore, in the signal processing apparatus according to the embodiment, determining the scene to be imaged involves determining whether the scene is a dark scene, and if the scene is determined to be a dark scene, the control unit receives a non-de-mosaic image as input data.

[0201] For reference Figure 3 It has been confirmed that in dark scenes, using a non-demosaic image instead of a demosaic image can improve the accuracy of image analysis and processing.

[0202] Therefore, based on the above configuration, even if an AI model capable of incorporating differences between scenes cannot be used due to resource constraints, the degradation of image analysis processing accuracy based on the scene can be suppressed.

[0203] Furthermore, the signal processing apparatus according to the embodiment includes an image signal processing unit (3), which includes a demosaic processing unit and performs image signal processing on the captured image, and the control unit changes the signal processing parameters of the image signal processing unit based on the scene determination result (see [link]). Figure 8 ).

[0204] Therefore, when the scene is determined to be a specific scene and image analysis processing using a non-demosaic image as input data is performed, image signal processing suitable for a non-demosaic image can be performed; and when the scene is determined to be not a specific scene and image analysis processing using a demosaic image as input data is performed, image signal processing suitable for a demosaic image can be performed.

[0205] Therefore, the accuracy of image analysis and processing can be improved.

[0206] Furthermore, in the signal processing apparatus according to the embodiment, the AI ​​model is an AI model having a neural network such as CNN. The signal processing apparatus includes an image organization unit (38) that collects pixel values ​​of individual pixels having positions within the same pixel unit from each pixel unit and generates a color separation image formed by different regions arranged on the same image plane. The AI ​​processing unit uses the color separation image as input data to perform image analysis processing.

[0207] Therefore, the format of the input data to the AI ​​processing unit is an input data format suitable for the configuration of CNN.

[0208] Therefore, the accuracy of image analysis and processing can be improved.

[0209] Furthermore, in the signal processing apparatus according to the embodiment, the AI ​​processing unit uses the unde-mosaiced image after shadow correction as input data to perform image analysis processing.

[0210] For reference Figure 3 In scenarios where non-demosaic images should be used as input data, using non-demosaic images after shadow correction can improve the accuracy of analysis and processing compared to using non-demosaic images before shadow correction.

[0211] Therefore, based on the above configuration, the accuracy of analysis and processing can be improved for specific scenarios such as dark scenes.

[0212] Furthermore, the signal processing apparatus (sensor unit 1A and camera device 10B) according to the embodiment includes an accumulation unit (sensor internal memory unit 7 and sensor external memory unit 14) for accumulating non-demosaic images.

[0213] The unde-mosaic images accumulated in the accumulation unit can be used to retrain the AI ​​model used in the AI ​​processing unit.

[0214] Because AI models can be retrained, the accuracy of image analysis and processing by AI processing units can be improved.

[0215] Furthermore, the signal processing apparatus according to the embodiment includes a transmission processing unit (camera control unit 13, sensor internal control unit 6A, camera control unit 13B), which performs processing to transmit the non-de-mosaic image accumulated in the accumulation unit to the outside of the apparatus.

[0216] Therefore, in cases where the AI ​​model is retrained by an external device of the signal processing unit, the unde-mosaic image used for retraining can be sent to the external device.

[0217] Therefore, it is possible to properly retrain the AI ​​model.

[0218] Furthermore, the signal processing apparatus (sensor unit 1, sensor unit 1A) according to the embodiment is configured as a sensor apparatus including a pixel array unit.

[0219] Therefore, for sensor devices configured to perform image analysis processing using AI models, the accuracy of image analysis processing can be improved.

[0220] Furthermore, the signal processing method according to the embodiment includes: performing image analysis processing using an AI model with a non-demosaic image as input data, the non-demosaic image being an image captured in a state without demosaic processing, the captured image being obtained through a pixel array unit, the pixel array unit including a plurality of pixel units arranged in two dimensions, each pixel unit including a plurality of pixels arranged in two dimensions in a predetermined pattern, the plurality of pixels receiving light in different wavelength bands.

[0221] This signal processing method can achieve functions and effects similar to those of the signal processing device described in the above embodiments.

[0222] Note that the effects described in this specification are merely illustrative and not limited, and other effects may be provided.

[0223] <6. This technology>

[0224] This technology can also be configured as follows. (1)

[0226] An image processing system, comprising: The circuit is configured as Acquire image data captured by the image sensor. Process the image data to generate a non-demosaic image, and The unde-mosaiced image is selectively output to an artificial intelligence model trained to perform image analysis. (2)

[0228] According to the image processing system of (1), the unde-mosaic image is an image captured in a state without de-mosaic processing. (3)

[0230] According to the image processing system of (2), the unde-mosaic image is an image captured without de-mosaic processing and in a state after shadow correction. (4)

[0232] The image processing system according to (1) also includes: The memory is configured to store one or more undemosaic images. (5)

[0234] According to the image processing system of (4), the circuit is further configured as follows: The artificial intelligence model is retrained using one or more undemosaic images stored in memory. (6)

[0236] According to the image processing system in (1), the artificial intelligence model is a convolutional neural network. (7)

[0238] According to the image processing system of (1), the circuit is further configured as follows: Determine whether the scene corresponding to the acquired image data is a dark scene. (8)

[0240] According to the image processing system of (7), the scene is a dark scene when the brightness value of the target subject is equal to or less than a predetermined brightness value. (9)

[0242] According to the image processing system of (7), the circuit is further configured as follows: In response to determining that the scene is the dark scene, the non-de-mosaic image is selected. (10)

[0244] According to the image processing system of (9), the non-demosaic image is a color-separated image of the non-demosaic image. (11)

[0246] According to the image processing system of (9), the circuit is further configured as follows: Obtain artificial intelligence model settings data. Based on the acquired AI model settings data, the parameters of the AI ​​model are set. Among them, based on parameter settings, the artificial intelligence model is an artificial intelligence model trained using unde-mosaic images as input for training. (12)

[0248] According to the image processing system of (10), the artificial intelligence model setting data includes filter coefficients used in the convolution process of the convolutional neural network and the structure of the convolutional neural network. (13)

[0250] According to the image processing system of (7), the circuit is further configured as follows: If it cannot be determined whether the scene corresponding to the acquired image data is a dark scene, select the non-de-mosaic image. (14)

[0252] According to the image processing system of (7), the circuit is further configured as follows: In response to determining that the scene is not the dark scene, the demosaic image is selectively output to the artificial intelligence model trained to perform image analysis. (15)

[0254] The image processing system according to (1) also includes: The image sensor is configured to capture the image data. (16)

[0256] An image processing system, comprising: An image sensor is configured to capture image data; The processor is configured as Acquire image data captured by the image sensor. Process image data to generate unde-mosaic images, and Selectively output unde-mosaiced images to an artificial intelligence model trained to perform image analysis; and The communication interface is configured to output the image analysis results of the trained artificial intelligence model to the camera processing circuitry. (17)

[0258] An image processing method, comprising: Acquire image data captured by the image sensor; Processing image data to generate unde-mosaic images; and Unde-mosaic images are selectively output to an artificial intelligence model trained to perform image analysis. (18)

[0260] The method according to (17) further includes: Determine whether the scene corresponding to the acquired image data is a dark scene. (19)

[0262] The method according to (17) further includes: In response to determining that the scene is a dark scene, a non-demosaic image is selected, where the non-demosaic image is a color-separated image of the non-demosaic image. (20)

[0264] The method according to (19) further includes: Obtain artificial intelligence model settings data. Based on the acquired artificial intelligence model setting data, the parameters of the artificial intelligence model are set. Among them, based on parameter settings, the artificial intelligence model is an artificial intelligence model trained using unde-mosaic images as input for training.

[0265] Those skilled in the art will understand that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents.

[0266] [List of Reference Numbers]

[0267] Camera devices 10, 10A, and 10B

[0268] Sensor units 1, 1A, and 1B

[0269] 2-pixel array unit

[0270] 3 Image Signal Processing Unit

[0271] 4 Preprocessing Unit

[0272] 5 AI processing units

[0273] 6, 6A Sensor Internal Control Unit

[0274] 7. Sensor Internal Memory Unit

[0275] 8 Output Data Generation Unit

[0276] 9. Communication Interface (I / F)

[0277] 11 Optical System

[0278] 12. Communication Interface (I / F)

[0279] 13, 13B Camera Control Unit

[0280] 14 Sensor External Memory Unit

[0281] 15 Communication Units

[0282] P1 First AI Model Setup Data

[0283] P2 Second AI Model Settings Data

[0284] 31 Black Level Correction Unit

[0285] 32 Shadow Correction Units

[0286] 33 Gain Adjustment Unit

[0287] 34 De-mosaic processing units

[0288] 35 color correction units

[0289] 36 Gamma correction units

[0290] 37 Distortion Removal Unit

[0291] 38 Image Organization Units

[0292] 39 Selector

[0293] Pu pixel unit.

Claims

1. An image processing system comprising: circuitry configured to acquire image data captured by an image sensor, process the image data to generate a non-demosaiced image, and selectively output the non-demosaiced image to an artificial intelligence model trained to perform image analysis.

2. The image processing system of claim 1, wherein, the non-demosaiced image is a captured image in a state without demosaicing processing.

3. The image processing system of claim 2, wherein, the non-demosaiced image is the captured image in a state without demosaicing processing and after shading correction.

4. The image processing system according to claim 1, further comprising: a memory configured to store one or more non-demosaiced images.

5. The image processing system of claim 4, wherein, the circuitry is further configured to: retrain the artificial intelligence model using the one or more non-demosaiced images stored in the memory.

6. The image processing system of claim 1, wherein, the artificial intelligence model is a convolutional neural network.

7. The image processing system of claim 1, wherein, the circuitry is further configured to: determine whether a scene corresponding to the acquired image data is a dark scene.

8. The image processing system of claim 7, wherein, the scene is the dark scene in a case where a luminance value of a target subject is equal to or less than a predetermined luminance value.

9. The image processing system of claim 7, wherein, the circuitry is further configured to: select the non-demosaiced image in response to determining that the scene is the dark scene.

10. The image processing system of claim 9, wherein, the non-demosaiced image is a color-separated image of the non-demosaiced image.

11. The image processing system of claim 9, wherein, the circuitry is further configured to: acquire artificial intelligence model setting data, perform parameter setting of the artificial intelligence model according to the acquired artificial intelligence model setting data, wherein, based on the parameter setting, the artificial intelligence model is an artificial intelligence model trained with the non-demosaiced image as an input for training.

12. The image processing system of claim 10, wherein, the artificial intelligence model setting data includes filter coefficients used in a convolution process of a convolutional neural network and a structure of the convolutional neural network.

13. The image processing system of claim 7, wherein, the circuitry is further configured to: select the non-demosaiced image in a case where it is not possible to determine whether a scene corresponding to the acquired image data is the dark scene.

14. The image processing system of claim 7, wherein, the circuitry is further configured to: selectively output a demosaiced image to the artificial intelligence model trained to perform image analysis in response to determining that the scene is not the dark scene.

15. The image processing system according to claim 1, further comprising: the image sensor is configured to capture the image data.

16. An image processing system comprising: an image sensor configured to capture image data; a processor configured to acquire the image data captured by the image sensor, process the image data to generate a non-demosaiced image, and selectively output the non-demosaiced image to an artificial intelligence model trained to perform image analysis; and a communication interface configured to output an image analysis result of the trained artificial intelligence model to camera processing circuitry.

17. An image processing method comprising: acquiring image data captured by an image sensor; processing the image data to generate a non-demosaiced image; and selectively outputting the non-demosaiced image to an artificial intelligence model trained to perform image analysis.

18. The method according to claim 17, further comprising: ​ determine whether a scene corresponding to the acquired image data is a dark scene.

19. The method of claim 18, further comprising: in response to determining that the scene is the dark scene, selecting the non-demosaiced image, wherein the non-demosaiced image is a color-separated image of the non-demosaiced image.

20. The method of claim 19, further comprising: acquiring artificial intelligence model setting data, based on the acquired artificial intelligence model setting data, performing a parameter setting of the artificial intelligence model, wherein, based on the parameter setting, the artificial intelligence model is an artificial intelligence model trained with the non-demosaiced image as an input for training.

Images