Information processing device, information processing method, and imaging device

Abstract

The present disclosure relates to an information processing device, an information processing method, and an imaging device, which make it possible to perform DNN processing with a lower data amount and lower power consumption. A RAW format conversion processing unit converts the format of a RAW format image outputted from a sensor unit such that the data amount is reduced on the basis of a pixel array of the sensor unit, thereby acquiring a sensing RAW format image. A sensing processing unit executes sensing processing by performing DNN processing on the sensing RAW format image. This technology can be applied to, for example, an imaging device that recognizes a subject by DNN processing.

Description

Information processing device, information processing method, and imaging device

[0001] The present disclosure relates to an information processing device, an information processing method, and an imaging device, and in particular to an information processing device, an information processing method, and an imaging device that are capable of performing DNN processing with a smaller amount of data and lower power consumption.

[0002] In recent years, the accuracy of recognition functions using deep learning has been improving, and systems that perform various processes using deep neural networks (DNNs) (hereinafter referred to as DNN processing) in downstream processing using application processors or graphics processing units (GPUs) or on the cloud side have become widespread.

[0003] Patent Document 1 discloses an image processing device that uses Bayer array images, as well as 1-channel images of the Bayer array and images separated for each color channel, as training images and teacher images used in deep learning learning processes.

[0004] Japanese Patent Application Laid-Open No. 2022-008037

[0005] In conventional camera systems equipped with sensing functions using DNN processing, images that have undergone signal processing using an image sensor or DSP (Digital Signal Processor) are passed to subsequent processing for DNN processing, resulting in a large amount of image data being input to the DNN processing and a large amount of power consumption required for DNN processing.However, depending on the location where the camera system performs sensing, it is expected that it will be used in environments where stable communication facilities and power supplies are not guaranteed, and in order to realize sensing functions using DNN processing even in such environments, there is a demand for reducing the data volume and power consumption of DNN processing.

[0006] The present disclosure has been made in consideration of these circumstances, and makes it possible to perform DNN processing with a smaller amount of data and lower power consumption.

[0007] An information processing device according to one aspect of the present disclosure includes a RAW format conversion processing unit that converts the format of a RAW format image output from a sensor unit in which multiple pixels are arranged according to a predetermined pixel array to reduce the amount of data based on the pixel array, thereby obtaining an image in sensing RAW format, and a sensing processing unit that executes sensing processing by performing DNN processing on the image in sensing RAW format.

[0008] An information processing method according to one aspect of the present disclosure includes an information processing device converting the format of a RAW format image output from a sensor unit in which multiple pixels are arranged according to a predetermined pixel array to reduce the amount of data based on the pixel array, thereby obtaining an image in sensing RAW format, and performing sensing processing by performing DNN processing on the image in sensing RAW format.

[0009] An imaging device according to one aspect of the present disclosure includes a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array, a RAW format conversion processing unit that converts the format of a RAW format image output from the sensor unit to reduce the amount of data based on the pixel array, thereby obtaining an image in a sensing RAW format, and a sensing processing unit that executes sensing processing by performing DNN processing on the image in the sensing RAW format.

[0010] In one aspect of the present disclosure, a RAW format image output from a sensor unit in which multiple pixels are arranged according to a predetermined pixel array is converted in format to reduce the amount of data based on the pixel array, an image in sensing RAW format is obtained, and sensing processing is performed by performing DNN processing on the image in sensing RAW format.

[0011] FIG. 1 is a block diagram showing a configuration example of a first embodiment of an imaging device to which the present technology is applied. FIG. 2 is a diagram explaining YCGM conversion processing. FIG. 3 is a diagram explaining Bayer (1ch) conversion processing. FIG. 4 is a diagram explaining features of images in a plurality of types of sensing RAW formats. FIG. 5 is a block diagram showing a configuration example of a second embodiment of an imaging device to which the present technology is applied. FIG. 6 is a flowchart explaining image recognition processing. FIG. 7 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied. FIG. 8 is a diagram showing an example of use in which an image sensor is used.

[0012] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0013] <First Configuration Example of Imaging Apparatus> A configuration example of a first embodiment of an imaging apparatus to which the present technology is applied will be described with reference to FIGS. 1 to 4 .

[0014] FIG. 1 is a block diagram showing a first example of the configuration of an imaging device.

[0015] As shown in FIG. 1, the imaging device 11 includes a sensor unit 21, an image processing unit 22, a RAW format conversion processing unit 23, and a sensing processing unit 24.

[0016] The sensor unit 21 is configured with a plurality of pixels arranged in an array, and in the example shown in Fig. 1, the pixels are arranged to receive light of colors according to a Bayer array. As shown in the figure, in the Bayer array, of four pixels in a 2 x 2 array, a red pixel R is arranged in the upper left, a green pixel Gr is arranged in the upper right, a green pixel Gb is arranged in the lower left, and a blue pixel B is arranged in the lower right. The sensor unit 21 then supplies an image in a format (hereinafter referred to as a RAW format) that retains the color arrangement of the light received by each pixel to the image processing unit 22 and the RAW format conversion processing unit 23.

[0017] The image processing unit 22 performs image processing to demosaic the RAW format image supplied from the sensor unit 21, and outputs an RGB image obtained as a result of the processing. For example, the RGB image is an image in which a red image, a green image, and a blue image each have the same size as the RAW format image.

[0018] The RAW format conversion processing unit 23 performs RAW format conversion processing on the RAW format image supplied from the sensor unit 21, converting the format so as to reduce the amount of data based on the pixel array of the sensor unit 21, and supplies the sensing RAW format image obtained as a result of the processing to the sensing processing unit 24. For example, in the RAW format conversion processing, the RAW format conversion processing unit 23 converts the RAW format image into a sensing RAW format that has a smaller amount of data than an RGB image and that suppresses degradation of sensitivity, resolution, or color reproduction from the RAW format.

[0019] With reference to FIG. 2 , a YCGM conversion process, which is an example of a RAW format conversion process performed by the RAW format conversion processing unit 23, will be described. The RAW format image input to the RAW format conversion processing unit 23 is Bayer array data according to the pixel arrangement of the sensor unit 21, as shown on the left side of FIG. 2 . The RAW format conversion processing unit 23 converts the Bayer array data constituting the RAW format image into complementary color array data in which yellow, cyan, green, and magenta pixels are arranged, through YCGM conversion processing. In other words, when YCGM conversion processing is used as the RAW format conversion processing, the sensing RAW format image output from the RAW format conversion processing unit 23 becomes complementary color array data in which, in a 2×2 array of four pixels, a yellow pixel Y is arranged in the upper left, a cyan pixel C is arranged in the upper right, a green pixel G is arranged in the lower left, and a magenta pixel M is arranged in the lower right, as shown on the right side of FIG. 2 .

[0020] The sensing processing unit 24 performs DNN processing on the image in the sensing RAW format supplied from the RAW format conversion processing unit 23, thereby executing sensing processing to recognize the subject captured in the image captured by the sensor unit 21. Then, the sensing processing unit 24 outputs a sensing result obtained as a result of performing the sensing processing on the image in the sensing RAW format (for example, if a person is captured in the image, the fact that the person has been detected from the image).

[0021] The imaging device 11 configured as described above can convert the RAW format into the sensing RAW format, thereby reducing the amount of image data on which DNN processing is performed in the sensing processing unit 24, thereby reducing power consumption in the sensing processing unit 24. In this case, the sensing RAW format suppresses degradation in sensitivity, resolution, or color reproduction compared to the RAW format, so even for images with a small amount of data, it is possible to avoid a decrease in recognition accuracy due to DNN processing in the sensing processing unit 24 and achieve recognition accuracy equivalent to that of conventional methods. Alternatively, by using the sensing RAW format, it is possible to improve the recognition accuracy due to DNN processing in the sensing processing unit 24 compared to when other images with a small amount of data are used.

[0022] In other words, it is generally known that DNN processing improves recognition accuracy by increasing the amount of input data, and that reducing the amount of input data tends to decrease recognition accuracy.In contrast, the imaging device 11 can reduce the amount of input data without sacrificing recognition accuracy by adopting a sensing RAW format, which has less data than RGB images and suppresses degradation of sensitivity, resolution, or color reproduction from the RAW format.

[0023] In this way, the imaging device 11 can perform DNN processing with a smaller amount of data and lower power consumption, and is suitable for use in environments where, for example, communication facilities and power supplies are not stable. For example, the complementary color array data described above with reference to Figure 2 can reduce the data amount to one-third of that of an RGB image, and can achieve lower power consumption compared to performing DNN processing using an RGB image.

[0024] Furthermore, the imaging device 11 can also reduce power consumption by reducing the circuit scale compared to conventional devices. Also, the imaging device 11 has a different path for inputting an image from the sensor unit 21 to the sensing processing unit 24 via the RAW format conversion processing unit 23 and a path for outputting an image from the imaging device 11 to the outside via the image processing unit 22, so that the image in the sensing RAW format cannot be seen from the outside. In other words, by providing the RAW format conversion processing unit 23, the imaging device 11 can be configured so that the image output from the imaging device 11 to the outside is not affected.

[0025] 3, a description will be given of Bayer (1ch) conversion processing, which is an example of RAW format conversion processing by the RAW format conversion processing unit 23. Note that, since the amount of data of images used in DNN processing is generally reduced by reduction processing, here, a description will be given of Bayer (1ch) conversion processing that reduces the image size to 1 / 2.

[0026] In the Bayer (1ch) conversion process, Bayer array data such as that shown on the left side of Fig. 3 is converted into Bayer (1ch) data in which the image size is reduced in consideration of the pixel centroids, as shown on the right side of Fig. 3. In other words, when the Bayer (1ch) conversion process is used as the RAW format conversion process, the image in the sensing RAW format output from the RAW format conversion processing unit 23 becomes Bayer (1ch) data with an image size half that of the RAW format, in which, of the four pixels in a 2 × 2 array, a red pixel R' is arranged at the upper left, a green pixel Gr' is arranged at the upper right, a green pixel Gb' is arranged at the lower left, and a blue pixel B' is arranged at the lower right, as shown on the right side of Fig. 3.

[0027] Specifically, in the Bayer (1ch) data, the red pixel R'(0,0) located at the upper left corner is obtained by multiplying the red pixel R(0,0) of the Bayer array data by 9 as a coefficient that takes into account the pixel center of gravity, multiplying the red pixel R(1,0) of the Bayer array data by 3 as a coefficient that takes into account the pixel center of gravity, multiplying the red pixel R(0,1) of the Bayer array data by 3 as a coefficient that takes into account the pixel center of gravity, and multiplying the red pixel R(1,1) of the Bayer array data by 1 as a coefficient that takes into account the pixel center of gravity, and dividing the resulting values ​​by 16 corresponding to these coefficients. Similarly, the green pixel Gr', the green pixel Gb', and the blue pixel B' can be obtained using coefficients that take into account the pixel center of gravity.

[0028] In this way, by using the Bayer (1ch) conversion process that reduces the image size taking the pixel centroid into consideration, it is possible to convert a RAW format image (Bayer array data) into a sensing RAW format image (Bayer (1ch) data) without the need for demosaic or reBayer processing. Therefore, it is possible to reduce the amount of image data that is subjected to DNN processing in the sensing processing unit 24, thereby reducing the power consumption of the sensing processing unit 24 and avoiding a decrease in recognition accuracy due to DNN processing.

[0029] With reference to FIG. 4, the characteristics of images in a plurality of types of sensing RAW formats acquired through the RAW format conversion process by the RAW format conversion processing unit 23 will be described.

[0030] FIG. 4 shows Gray data, Bayer (1ch) data, complementary color array data, RGBW data, and RYYB data as examples of sensing RAW format images acquired by the RAW format conversion processing by the RAW format conversion processing unit 23.

[0031] The gray data is acquired by a RAW format conversion process that converts the format of the Bayer array data so that all pixels contain only brightness values. The color resolution of the gray data (resolution of brightness only) is the same as the image data of each RGB image. Gray data has very high sensitivity, high resolution, and is unable to reproduce colors, making it advantageous for the sensing process of the sensing processing unit 24 when capturing images under low illumination.

[0032] As described above with reference to FIG. 3, the Bayer (1ch) data is acquired by a RAW format conversion process that reduces the image size by taking into account the pixel centroid. The color resolution of the Bayer (1ch) data is half that of the red image data of the RGB image in the horizontal and vertical directions for red, half that of the green image data of the RGB image in the diagonal directions for green, and half that of the blue image data of the RGB image in the horizontal and vertical directions for blue. Bayer (1ch) data has the characteristics of very low sensitivity, very low resolution, and very high color reproduction, making it advantageous for the sensing process of the sensing processing unit 24 when performing color recognition.

[0033] As described above with reference to FIG. 2 , the complementary color array data is obtained by a RAW format conversion process (YCGM conversion process) that converts the format of the Bayer array data so that, in a 2×2 array of four pixels, a yellow pixel Y is arranged in the upper left, a cyan pixel C is arranged in the upper right, a green pixel G is arranged in the lower left, and a magenta pixel M is arranged in the lower right. The color resolution of the complementary color array data is half that of the red image data of the RGB image only in the diagonal direction for red, the same as the green image data of the RGB image in the horizontal and vertical directions for green, and the same as the blue image data of the RGB image in the vertical direction for blue. Furthermore, the complementary color array data has the characteristics of high sensitivity, high resolution, and moderate color reproduction, making it generally advantageous for the sensing process of the sensing processing unit 24.

[0034] The RGBW data is acquired by a RAW format conversion process (RGBW conversion process) that converts the format of the Bayer array data so that, in a 2x2 array of four pixels, a red pixel R is arranged in the upper left, a green pixel G in the upper right, a white pixel W in the lower left, and a blue pixel B in the lower right. The color resolution of the RGBW data is such that, for red, the vertical resolution is the same as that of the red image data of the RGB image, for green, the diagonal resolution is half that of the green image data of the RGB image, and for blue, the horizontal resolution is the same as that of the blue image data of the RGB image. The RGBW data is generally advantageous for the sensing process of the sensing processing unit 24 due to its characteristics of high sensitivity, high resolution, and moderate color reproduction.

[0035] The RYYB data is obtained by a RAW format conversion process (RYYB conversion process) that converts the format of the Bayer array data so that, in a 2x2 array of four pixels, a red pixel R is arranged in the upper left, a yellow pixel Y is arranged in the upper right, a yellow pixel Y is arranged in the lower left, and a blue pixel B is arranged in the lower right. The color resolution of the RYYB data is the same as that of the red image data of the RGB image in the vertical and horizontal directions for red, half that of the green image data of the RGB image in the diagonal directions for green, and half that of the blue image data of the RGB image in the vertical and horizontal directions for blue. The RYYB data has moderate sensitivity, moderate resolution, and good color reproduction, making it advantageous for the sensing process of the sensing processing unit 24 when capturing images under high illuminance.

[0036] In this way, there are scenes in which each type of sensing RAW format image is advantageous, and it is preferable to set the RAW format conversion process of the RAW format conversion processing unit 23 so that an image of a type of sensing RAW format that is suitable for the imaging environment in which the imaging device 11 is used is used.

[0037] It should be noted that the images in the multiple types of sensing RAW formats shown in FIG. 4 are merely examples, and the imaging device 11 can be configured to use images in other types of sensing RAW formats.

[0038] <Second Configuration Example of Imaging Apparatus> A configuration example of a second embodiment of an imaging apparatus to which the present technology is applied will be described with reference to Figs. 5 and 6 .

[0039] Fig. 5 is a block diagram showing a second configuration example of an imaging device 11A. Note that in the imaging device 11A shown in Fig. 5, components common to those in the imaging device 11 of Fig. 1 are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0040] 5, the imaging device 11A has a common configuration with the imaging device 11 in Fig. 1 in that it includes a sensor unit 21 and an image processing unit 22. On the other hand, the imaging device 11A has a different configuration from the imaging device 11 in Fig. 1 in that it includes a RAW format conversion processing unit 23A, a sensing processing unit 24A, and an imaging environment recognition unit 25.

[0041] The imaging environment recognition unit 25 recognizes the imaging environment when the sensor unit 21 captures an image. For example, the imaging environment recognition unit 25 can be configured to include an illuminance sensor that detects illuminance, and recognizes whether the imaging environment when the sensor unit 21 captures an image is high illuminance or low illuminance. The imaging environment recognition unit 25 then notifies the RAW format conversion processing unit 23A and the sensing processing unit 24A, respectively, of whether the imaging environment is high illuminance or low illuminance.

[0042] The RAW format conversion processing unit 23A switches the sensing RAW format of the image to be supplied to the sensing processing unit 24A depending on the imaging environment when the sensor unit 21 captures the image, in accordance with a notification from the imaging environment recognition unit 25. For example, the RAW format conversion processing unit 23A can be configured to include a high-illuminance environment converter 31 and a low-illuminance environment converter 32, and can switch between the high-illuminance environment converter 31 and the low-illuminance environment converter 32 depending on the imaging environment.

[0043] The high-illuminance environment conversion unit 31 performs a RAW format conversion process to convert the RAW format supplied from the sensor unit 21 into, for example, a high-resolution sensing RAW format, and supplies the high-resolution sensing RAW format image to the sensing processing unit 24A.

[0044] The conversion unit 32 for low-light environments performs a RAW format conversion process to convert the RAW format image supplied from the sensor unit 21 into a high-sensitivity sensing RAW format, and supplies the high-sensitivity sensing RAW format image to the sensing processing unit 24A.

[0045] Therefore, when the RAW format conversion processing unit 23A is notified by the imaging environment recognition unit 25 that the imaging environment is high illuminance, it performs RAW format conversion processing using the high illuminance environment conversion unit 31. On the other hand, when the RAW format conversion processing unit 23A is notified by the imaging environment recognition unit 25 that the imaging environment is low illuminance, it performs RAW format conversion processing using the low illuminance environment conversion unit 32.

[0046] The sensing processing unit 24A is configured to have a DNN 41 for high-light environments and a DNN 42 for low-light environments, and can switch between using the DNN 41 for high-light environments and the DNN 42 for low-light environments in accordance with notification from the imaging environment recognition unit 25.

[0047] The DNN 41 for high-illuminance environments performs sensing processing on the high-resolution sensing RAW format image supplied from the conversion unit 31 for high-illuminance environments using DNN processing suitable for high-illuminance imaging environments.

[0048] The DNN 42 for low-light environments performs sensing processing on the image in the high-sensitivity sensing RAW format supplied from the conversion unit 32 for low-light environments using DNN processing suitable for low-light imaging environments.

[0049] Therefore, when the sensing processing unit 24A is notified by the imaging environment recognition unit 25 that the imaging environment has high illuminance, it executes sensing processing by DNN processing performed by the DNN for high illuminance environment 41. On the other hand, when the sensing processing unit 24A is notified by the imaging environment recognition unit 25 that the imaging environment has low illuminance, it executes sensing processing by DNN processing performed by the DNN for low illuminance environment 42.

[0050] In addition, as a method for the sensing processing unit 24A to change the DNN processing, in addition to switching between the DNN 41 for high-light environments and the DNN 42 for low-light environments, a method such as automatically downloading from an externally mounted memory may also be used.

[0051] The imaging device 11A configured as described above can adaptively (automatically) switch the RAW format conversion process depending on the imaging environment when the sensor unit 21 captures an image, and can perform sensing processing using DNN processing suitable for that imaging environment.

[0052] For example, during the daytime when illumination is sufficient, the imaging device 11A can prioritize resolution and convert the RAW format into a high-resolution sensing RAW format, and perform sensing processing using DNN processing suitable for images in the high-resolution sensing RAW format. On the other hand, during the nighttime when illumination is low, the imaging device 11A can prioritize sensitivity and convert the RAW format into a high-sensitivity sensing RAW format, and perform sensing processing using DNN processing suitable for images in the high-sensitivity sensing RAW format. This allows the imaging device 11A to follow changes in the imaging environment when the sensor unit 21 captures an image, prevent a decrease in recognition accuracy due to DNN processing, and ensure robustness to environmental changes.

[0053] FIG. 6 is a flowchart illustrating the image recognition process executed in the imaging device 11A.

[0054] For example, the process starts when an image in RAW format is output from the sensor unit 21 in the imaging device 11A. In step S11, the RAW format conversion processing unit 23A acquires the image in RAW format output from the sensor unit 21.

[0055] In step S12, the image capturing environment recognition unit 25 determines whether the image capturing environment when the image was captured by the sensor unit 21 was high illuminance or low illuminance.

[0056] If the imaging environment recognition unit 25 determines in step S12 that the imaging environment is high illuminance, the process proceeds to step S13.

[0057] In step S13, the RAW format conversion processing unit 23A performs RAW format conversion processing on the RAW format image acquired in step S11 using the high-illumination environment conversion unit 31. Then, the RAW format conversion processing unit 23A supplies the high-resolution sensing RAW format image obtained as a processing result of the RAW format conversion processing using the high-illumination environment conversion unit 31 to the sensing processing unit 24A.

[0058] In step S14, the sensing processing unit 24A performs sensing processing on the sensing RAW format image supplied from the RAW format conversion processing unit 23A in step S13 using the DNN 41 for high-illuminance environments, and obtains the sensing results.

[0059] On the other hand, if the imaging environment recognition unit 25 determines in step S12 that the imaging environment is low illuminance, the process proceeds to step S15.

[0060] In step S15, the RAW format conversion processing unit 23A performs RAW format conversion processing on the RAW format image acquired in step S11 using the low-illuminance environment conversion unit 32. Then, the RAW format conversion processing unit 23A supplies the high-sensitivity sensing RAW format image obtained as a processing result of the RAW format conversion processing using the low-illuminance environment conversion unit 32 to the sensing processing unit 24A.

[0061] In step S16, the sensing processing unit 24A performs sensing processing on the sensing RAW format image supplied from the RAW format conversion processing unit 23A in step S15 using the DNN 42 for low-light environments, and obtains the sensing result.

[0062] After the processing of step S14 or step S16, the processing proceeds to step S17, and the sensing processing unit 24A outputs the sensing result acquired in step S14 or step S16, and then the image recognition processing ends.

[0063] By performing the image recognition process as described above, the imaging device 11A can switch between the RAW format conversion process and the sensing process, triggered by a change in the imaging environment (illuminance) when an image is captured by the sensor unit 21. This allows the imaging device 11A to prevent a decrease in recognition accuracy due to DNN processing even if the imaging environment changes, and ensures robustness to environmental changes.

[0064] The imaging device 11A may switch between the RAW format conversion process and the sensing process using a change in the imaging environment other than illuminance as a trigger. For example, the imaging device 11A can determine the necessity (importance) of color discrimination from the white balance detection value, and switch between the RAW format conversion process and the sensing process between a sensing RAW format with high color resolution and a sensing RAW format with high sensitivity.

[0065] <Example of Computer Configuration> Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0066] FIG. 7 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0067] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) 104 are interconnected by a bus 105. An input / output interface 106 is further connected to the bus 105, and the input / output interface 106 is connected to the outside.

[0068] In a computer configured as described above, the CPU 101 performs the above-described series of processes by loading programs stored in, for example, the ROM 102 and EEPROM 104 into the RAM 103 via the bus 105 and executing the programs. In addition, the programs executed by the computer (CPU 101) can be written in advance in the ROM 102, or can be installed or updated in the EEPROM 104 from outside via the input / output interface 106.

[0069] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0070] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0071] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0072] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0073] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0074] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0075] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0076] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0077] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0078] <Example of Use of Imaging Device> FIG. 8 is a diagram showing an example of use of the image sensor included in the imaging device 11 described above.

[0079] The image sensor included in the imaging device 11 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0080] ・Devices for taking images for viewing purposes, such as digital cameras and mobile devices with camera functions. ・Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping, and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. ・Devices for home appliances such as TVs, refrigerators, and air conditioners that take images of user gestures and operate the device according to those gestures. ・Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. ・Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. ・Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. ・Devices for sports purposes, such as action cameras and wearable cameras for sports, etc. ・Devices for agricultural purposes, such as cameras to monitor the condition of fields and crops.

[0081] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) An information processing device comprising: a RAW format conversion processing unit that converts the format of a RAW format image output from a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array so as to reduce the amount of data based on the pixel array, thereby acquiring an image in a sensing RAW format; and a sensing processing unit that executes sensing processing by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format. (2) The information processing device described in (1) above, in which the RAW format conversion processing unit acquires the image in the sensing RAW format by converting Bayer array data constituting the image in the RAW format into complementary color array data in which yellow, cyan, green, and magenta pixels are arranged in a 2x2 array. (3) The information processing device described in (1) above, in which the RAW format conversion processing unit acquires the image in the sensing RAW format by converting the Bayer array data constituting the image in the RAW format into Bayer (1ch) data in which the image size is reduced in consideration of pixel centroids. (4) The information processing device according to (1), wherein the RAW format conversion processing unit acquires the sensing RAW format image by converting Bayer array data constituting the RAW format image into RGBW data in which red, green, white, and blue pixels are arranged in a 2 x 2 array. (5) The information processing device according to (1), wherein the RAW format conversion processing unit acquires the sensing RAW format image by converting Bayer array data constituting the RAW format image into RYYB data in which red, yellow, yellow, and blue pixels are arranged in a 2 x 2 array.(6) The information processing device according to any of (1) to (5) above, further comprising an imaging environment recognition unit that recognizes an imaging environment when the sensor unit captures an image, wherein the RAW format conversion processing unit switches the sensing RAW format according to the imaging environment, and the sensing processing unit switches to the DNN processing suitable for the imaging environment and executes the sensing processing. (7) The information processing device according to (6) above, wherein, when the imaging environment is high illuminance, the RAW format conversion processing unit acquires an image in the sensing RAW format with high resolution, and the sensing processing unit executes the sensing processing using the DNN processing suitable for the imaging environment of high illuminance. (8) The information processing device according to (6) above, when the imaging environment is low illuminance, the RAW format conversion processing unit acquires an image in the sensing RAW format with high sensitivity, and the sensing processing unit executes the sensing processing using the DNN processing suitable for the imaging environment of low illuminance. (9) An information processing method including: an information processing device converting a format of a RAW format image output from a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array so as to reduce the amount of data based on the pixel array, thereby obtaining an image in a sensing RAW format; and executing sensing processing by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format. (10) An imaging device including: a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array, a RAW format conversion processing unit converting a format of the RAW format image output from the sensor unit so as to reduce the amount of data based on the pixel array, thereby obtaining an image in the sensing RAW format, and a sensing processing unit that executes sensing processing by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format.

[0082] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0083] DESCRIPTION OF SYMBOLS 11 Imaging device, 21 Sensor unit, 22 Image processing unit, 23 RAW format conversion processing unit, 24 Sensing processing unit, 25 Imaging environment recognition unit, 31 Conversion unit for high-illuminance environment, 32 Conversion unit for low-illuminance environment, 41 DNN for high-illuminance environment, 42 DNN for low-illuminance environment

Claims

1. An information processing apparatus comprising: a RAW format conversion processing unit that converts an image in RAW format output from a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array into a format for reducing the data amount based on the pixel array to obtain an image in a sensing RAW format; and a sensing processing unit that executes a sensing process by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format.

2. The information processing apparatus according to claim 1, wherein the RAW format conversion processing unit obtains the image in the sensing RAW format by converting Bayer array data constituting the image in the RAW format into complementary color array data in which yellow, cyan, green, and magenta pixels are arranged in a 2×2 array.

3. The information processing apparatus according to claim 1, wherein the RAW format conversion processing unit obtains the image in the sensing RAW format by converting Bayer array data constituting the image in the RAW format into Bayer(1ch) data with a reduced image size in consideration of pixel centroid.

4. The information processing apparatus according to claim 1, wherein the RAW format conversion processing unit obtains the image in the sensing RAW format by converting Bayer array data constituting the image in the RAW format into RGBW data in which red, green, white, and blue pixels are arranged in a 2×2 array.

5. The information processing apparatus according to claim 1, wherein the RAW format conversion processing unit obtains the image in the sensing RAW format by converting Bayer array data constituting the image in the RAW format into RYYB data in which red, yellow, yellow, and blue pixels are arranged in a 2×2 array.

6. The information processing apparatus according to claim 1, further comprising an imaging environment recognition unit that recognizes an imaging environment when the sensor unit captures an image, wherein the RAW format conversion processing unit switches the sensing RAW format according to the imaging environment, and the sensing processing unit switches to the DNN processing suitable for the imaging environment and executes the sensing process.

7. When the imaging environment has high illuminance, the RAW format conversion processing unit acquires an image in the high-resolution sensing RAW format, and the sensing processing unit executes the sensing processing using the DNN processing suitable for the imaging environment with high illuminance. The information processing apparatus according to claim 6.

8. When the imaging environment has low illuminance, the RAW format conversion processing unit acquires an image in the high-sensitivity sensing RAW format, and the sensing processing unit executes the sensing processing using the DNN processing suitable for the imaging environment with low illuminance. The information processing apparatus according to claim 6.

9. An information processing method including: converting a format of an image in a RAW format output from a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array to reduce a data amount based on the pixel array, and acquiring an image in a sensing RAW format; and executing sensing processing by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format.

10. An imaging apparatus including: a sensor unit in which a plurality of pixels are arranged according to a predetermined pixel array; a RAW format conversion processing unit that converts a format of an image in a RAW format output from the sensor unit to reduce a data amount based on the pixel array, and acquires an image in a sensing RAW format; and a sensing processing unit that executes sensing processing by performing DNN (Deep Neural Network) processing on the image in the sensing RAW format.

Images