Information processing system, information processing method, and information processing program

Abstract

This information processing system comprises: a data generation unit that generates data for training a neural network (NW) to be trained, the NW performing temporal computation and spatial computation; a training unit that, using the training data generated by the data generation unit, performs machine learning on the NW being trained; and a NW generation unit that, on the basis of the NW being trained, generates an executing NW that performs spatial computation, the executing NW outputting output data when inputted with temporally computed image data.

Description

Information processing system, information processing method, and information processing program

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

[0002] For example, Patent Document 1 discloses a technique for reconstructing a space-time domain from data of a single coded image captured using a coded shutter function for each pixel.

[0003] Special Publication No. 2014-511531

[0004] Dejia Xu, Peihao Wang, Yifan Jiang, Zhiwen Fan, Zhangyang Wang, "Signal Processing for Implicit Neural Representations", online, December 13, 2022, Cornell University, retrieved February 22, 2023, Internet<URL:https: / / arxiv.org / pdf / 2210.08772.pdf>

[0005] The processing of image data may include temporal operations such as convolution in the time direction. Conventionally, temporal operations have been performed on time-series image data, but this involves handling large amounts of data, which increases the calculation cost.

[0006] One aspect of the present disclosure reduces the computational cost required for time operations.

[0007] An information processing system according to one aspect of the present disclosure includes a data generation unit that generates learning data for a learning target network (neural network) that performs temporal and spatial operations, a learning unit that uses the learning data generated by the data generation unit to perform machine learning on the learning target network, and a network generation unit that generates an execution network based on the learning target network, which performs spatial operations and outputs output data when time-calculated image data is input.

[0008] An information processing method according to one aspect of the present disclosure includes generating training data for a training target network (neural network) that performs temporal and spatial operations, using the generated training data to train the training target network by machine learning, and generating an execution network based on the training target network that performs spatial operations and outputs output data when time-calculated image data is input.

[0009] An information processing program according to one aspect of the present disclosure causes a computer to perform the following processes: generating training data for a training target network (neural network) that performs temporal and spatial operations; using the generated training data to train the training target network by machine learning; and generating an execution network based on the training target network, which performs spatial operations and outputs output data when time-calculated image data is input.

[0010] An information processing system according to one aspect of the present disclosure includes a data generation unit that generates learning data for a learning target network (neural network) that performs temporal and spatial operations; a learning unit that uses the learning data generated by the data generation unit to perform machine learning on the learning target network; a network generation unit that generates an execution network that performs spatial operations based on the learning target network and outputs output data when time-calculated image data is input; a pixel array unit that includes a plurality of pixels, each of which outputs a pixel signal corresponding to the amount of charge accumulated during an exposure period; a drive unit that drives the pixel array unit so as to obtain one frame's worth of image data of a scene including a subject; and a processing unit that processes the image data using the execution network, wherein the drive unit drives the pixel array unit so that the pixels are exposed in an exposure period pattern that includes a plurality of exposure periods during one frame period.

[0011] 1 is a diagram illustrating an example of a schematic configuration of a sensor 1 according to an embodiment; FIG. 1 is a diagram illustrating an example of multiple exposure imaging; FIG. 2 is a diagram illustrating an example of a code C; FIG. 2 is a diagram illustrating an example of a pixel group; FIG. 3 is a diagram illustrating an example of a pixel group; FIG. 4 is a diagram illustrating an example of a code C; FIG. 4 is a diagram illustrating an example of a code C; FIG. 5 is a diagram illustrating an example of a code C; FIG. 6 is a diagram illustrating an example of a code C; FIG. 7 is a diagram illustrating an example of a code C; FIG. 8 is a diagram illustrating an example of a code C; FIG. 9 is a diagram illustrating an example of a code C; FIG. 10 is a diagram illustrating an example of a code C; FIG. 11 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 12 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 13 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 14 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 15 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 16 is a diagram illustrating an example of a characteristic of an exposure period pattern P; FIG. 17 is a diagram illustrating an example of a characteristic of an exposure period pattern P; 1 is a diagram showing an example of a kernel weight W. FIG. 2 is a diagram showing an example of a kernel weight W. FIG. 3 is a diagram showing an example of a trigger. FIG. 4 is a diagram showing an example of switching of an exposure period pattern P in the middle of one frame period T. FIG. 5 is a diagram showing an example of a mode change. FIG. 6 is a diagram showing an example of switching of processing by a processing unit 6 in response to switching of an exposure period pattern P. FIG. 7 is a diagram showing an example of switching of processing by a processing unit 6 in response to switching of an exposure period pattern P. FIG. 8 is a diagram showing an example of switching of processing by a processing unit 6 in response to switching of an exposure period pattern P. FIG. 9 is a diagram showing an example of a schematic configuration of a sensor system 1A according to an embodiment. FIG. 10 is a diagram showing an example of a schematic configuration of an information processing system 100 according to an embodiment.1 is a diagram illustrating an example of a schematic configuration of a learning target NW 130. A diagram illustrating an example of machine learning by a learning unit 13. A diagram illustrating an example of a generation method of input data 121. A diagram illustrating an example of machine learning by the learning unit 13. A diagram illustrating an example of a generation method of input data 121. A diagram illustrating an example of machine learning by the learning unit 13. A diagram illustrating an example of filter regularization. A diagram illustrating an example of data expansion. A diagram illustrating an example of data expansion. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of generation of an execution NW 60 by a NW generation unit 14. A diagram illustrating an example of processing (information processing method, sensing method) executed in an information processing system 100. A diagram illustrating an example of processing (information processing method, sensing method) executed in an information processing system 100. A diagram illustrating a modified example of a NW. A diagram illustrating a modified example of a NW. A diagram illustrating a modified example of a NW. A diagram illustrating an example of a hardware configuration of an apparatus.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same elements are designated by the same reference numerals, and redundant description will be omitted.

[0013] The present disclosure will be described in the following order: 1. Sensor embodiment 2. Sensor system embodiment 3. Information processing system embodiment 4. NW variation 5. Example of device hardware configuration 6. Conclusion

[0014] 1 is a diagram showing an example of a schematic configuration of a sensor 1 according to an embodiment. In the following description, the sensor 1 is assumed to be an image sensor (solid-state imaging device) such as a CMOS image sensor or a CCD image sensor.

[0015] The object imaged by the sensor 1 is illustrated and referred to as a scene 8. The scene 8 may include one or more objects 9. The sensor 1 images the scene 8 and generates data (image data) of the image. In the following description, the scene 8 and the object 9 may also be referred to as the scene 8, etc. or the object 9, etc.

[0016] The sensor 1 includes a pixel array unit 2, a drive unit 4, a processing unit 6, and a storage unit 7. Note that some elements may not be included in the sensor 1, but may be provided outside the sensor 1. This will be described again later with reference to FIG. 45 .

[0017] The pixel array section 2 includes a plurality of pixels 3. The width and height of the pixel array section 2 are referred to as width Wd and height Ht, and their directions are indicated by arrows. The plurality of pixels 3 are arranged side by side in the width Wd direction and height Ht direction (array direction). Light from a scene 8 is incident on each pixel 3 in the pixel array section 2.

[0018] Each pixel 3 outputs a pixel signal according to the amount of charge accumulated during the exposure period. The pixel 3 includes a photoelectric conversion unit, a transistor, etc. The photoelectric conversion unit is, for example, a photodiode, and generates charge according to the amount of light incident during the exposure period. The transistor is used to extract the charge accumulated in the photoelectric conversion unit and convert it into a voltage signal (pixel signal), or to read out the pixel signal. The photoelectric conversion unit, transistor, etc. are also called a pixel circuit. Various known pixel circuits may be used.

[0019] The driver 4 drives the pixel array unit 2. More specifically, the driver 4 drives each transistor in the pixel circuit of each pixel 3 so that exposure is performed at a desired timing in each pixel 3 and pixel signals are generated and read out. For example, a drive signal is supplied from the driver 4 to a control terminal (gate, etc.) of a corresponding transistor.

[0020] In this embodiment, the driver 4 drives the pixel array unit 2 so that the sensor 1 can perform multiple exposure imaging. Multiple exposure imaging will be described with reference to FIG.

[0021] 2 is a diagram showing an example of multiple exposure imaging. The horizontal axis represents time t. The period from time t10 to time t20 is the period during which exposure is possible, and this period is referred to as one frame period T. The pattern of exposure periods that one frame period T has is referred to as exposure period pattern P. The exposure period pattern P can be specified by, for example, the start time, length, and end time of each exposure period in the exposure period pattern P.

[0022] In multiple-exposure imaging, an exposure period pattern P includes multiple exposure periods. The pixels 3 exposed in the exposure period pattern P output pixel signals corresponding to charges accumulated over the multiple exposure periods in the exposure period pattern P. In the example shown in FIG. 2 , the pixel signals are output and read out between time t30 and time t40. The image obtained from the pixel signals thus read out is an image obtained by superimposing images corresponding to the multiple exposure periods in one frame period T.

[0023] Returning to Fig. 1, when the sensor 1 performs multiple-exposure imaging, the drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P that includes multiple exposure periods during one frame period T. Note that imaging other than multiple-exposure imaging, i.e., single-exposure imaging, is also possible. When the sensor 1 performs single-exposure imaging, the drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P that includes only one exposure period during one frame period T (the length of which may be the same as one frame period T or may be shorter than that). Unless otherwise specified, it is assumed that the sensor 1 performs multiple-exposure imaging.

[0024] An exposure period pattern P including multiple exposure periods within one frame period T weights the image in one frame period T in the time direction. That is, the exposure period pattern P can correspond to the weight of the kernel (filter) of the time operation. An example of the time operation is (operation of) convolution in the time direction. Unless otherwise specified, the time operation is assumed to be convolution in the time direction. Within a consistent range, the terms time operation and convolution in the time direction may be interpreted as appropriate. In order to obtain an exposure period pattern P for realizing a desired convolution kernel in the time direction, in this embodiment, a code obtained by binarizing the weight of the convolution kernel in the time direction is used. This code is referred to as code C. The following description will also refer to FIG. 3 .

[0025] 3 is a diagram showing an example of code C. Code C is described by two values, "1" (exposure on) and "0" (exposure off). "1" indicates a period during which exposure is performed, i.e., an exposure period. "0" indicates a period during which exposure is not performed, i.e., a non-exposure period. The illustrated code C can define the exposure period pattern P of FIG. 2 described above.

[0026] 1 , the driver 4 includes a code generator 41 that generates the above-described code C. The driver 4 drives the pixel array 2 in accordance with the code C generated by the code generator 41, i.e., so that the pixels 3 are exposed in the exposure period pattern P defined by the code C.

[0027] Information on the code that is the original form of the code C is prepared in advance as a base filter 40 (which can also be called a base code) and stored in the storage unit 7. For example, the code generation unit 41 generates the desired code C by modifying the base filter 40. An example of the modification is changing the duty ratio of "1" and "0" described by the base filter 40. Note that the code generation unit 41 may generate the code C by directly using the base filter 40.

[0028] When the driver 4 drives the pixel array 2 so that the sensor 1 performs multiple-exposure imaging, each pixel 3 outputs a pixel signal corresponding to the charge accumulated over multiple exposure periods within one frame period T. The pixel signal from each pixel 3 is processed by a signal processing circuit or the like (not shown), thereby obtaining image data for one frame of the scene 8. This image data is referred to as image data 5. The image data 5 is image data obtained by superimposing images corresponding to each of the multiple exposure periods within one frame period T. Note that, similar to the pixel array 2, the width and height of the image data 5 are illustrated as width Wd and height Ht.

[0029] In one embodiment, the driver 4 drives the pixel array unit 2 so as to obtain multiple pieces of image data 5 for the same frame. In the example shown in Fig. 1, four pieces of image data 5 are obtained at a time. To enable the image data 5 to be distinguished from one another, they are referred to as image data 5-1, image data 5-2, image data 5-3, image data 5-4, etc. in the drawings. When there is no need to distinguish between them, they are simply referred to as image data 5.

[0030] In order to obtain a plurality of image data 5, a plurality of pixels 3 in the pixel array unit 2 are grouped into a plurality of pixel groups. One pixel group includes at least one pixel 3. The following description will be made with reference to FIGS. 4 and 5.

[0031] 4 and 5 are diagrams showing examples of pixel groups. To distinguish between the pixels 3 in each pixel group, they are referred to as pixel 3-1, pixel 3-2, etc. Pixels 3 with the same reference numeral belong to the same pixel group.

[0032] In the example shown in FIG. 4, there are four pixel groups. The pixels 3 in each pixel group are shown as pixel 3-1 to pixel 3-4. A 2×2 pixel array (pixel unit) consisting of four pixels 3, each belonging to a different pixel group, is arranged side by side in the array direction (width Wd direction and height Ht direction). For example, the above-mentioned image data 5-1 is obtained based on the pixel signal from pixel 3-1. Image data 5-2 is obtained based on the pixel signal from pixel 3-2. Image data 5-3 is obtained based on the pixel signal from pixel 3-3. Image data 5-4 is obtained based on the pixel signal from pixel 3-4.

[0033] 5, there are 16 pixel groups. The pixels 3 in each pixel group are shown as pixel 3-1 to pixel 3-16. A 4×4 pixel array consisting of 16 pixels 3, each belonging to a different pixel group, is arranged side by side in the array direction.

[0034] The examples shown in FIGS. 4 and 5 above are not limiting, and various numbers of pixel groups may exist (the number of pixel groups may be 1), and the pixels 3 may be arranged in various ways. By changing the number of pixel groups, it is possible to adjust the image data 5 obtained at one time. By increasing the number of pixel groups, it is possible to obtain more image data 5. By reducing the number of pixel groups, it is possible to reduce the circuit size and simplify the configuration, for example.

[0035] Returning to Fig. 1 , the driver 4 drives the pixel array unit 2 so that one piece of image data 5 is obtained from one pixel group. For example, the driver 4 may drive the pixel array unit 2 so that at least the pixels 3 in the same pixel group are exposed with the same exposure period pattern P. The driver 4 may also drive the pixel array unit 2 so that the pixels 3 in different pixel groups are exposed with different exposure period patterns P. The driver 4 may also drive the pixel array unit 2 so that the pixels 3 in the same pixel group are exposed with different exposure period patterns P.

[0036] When the exposure period pattern P is to be different for each pixel group, the code generation unit 41 generates a plurality of codes C (the same number as the number of pixel groups), each of which defines a different exposure period pattern P. The drive unit 4 drives the pixel array unit 2 so that the pixels 3 of the corresponding pixel group are exposed with the exposure period pattern P defined by each code C generated by the code generation unit 41. This will be described with reference to FIGS. 6 and 7 as well.

[0037] 6 and 7 are diagrams showing examples of code C. In Fig. 6, code C-1 to code C-4 are shown as examples of multiple codes C generated by the code generation unit 41. Codes C-1 to C-4 define exposure period patterns P-1 to P-4.

[0038] The driver 4 drives the pixel array section 2 so that pixel 3-1 is exposed in the exposure period pattern P-1 defined by code C-1. This results in image data 5-1. The driver 4 drives the pixel array section 2 so that pixel 3-2 is exposed in the exposure period pattern P-2 defined by code C-2. This results in image data 5-2. The driver 4 drives the pixel array section 2 so that pixel 3-3 is exposed in the exposure period pattern P-3 defined by code C-3. This results in image data 5-3. The driver 4 drives the pixel array section 2 so that pixel 3-4 is exposed in the exposure period pattern P-4 defined by code C-4. This results in image data 5-4.

[0039] 7A shows a specific example of code C-1. 7B shows a specific example of code C-2. Codes C-1 and C-2 are different from each other, and therefore define different exposure period patterns P. Although not shown, this also applies to codes C-1 and C-2, as well as codes C-3 and C-4.

[0040] The exposure period pattern P defined by the code C can be given various characteristics. For example, by shortening the exposure period, blurring can be prevented even in a moving scene 8. By lengthening the exposure period, it is possible to handle a dark (low-illumination) scene 8. By combining and using a plurality of codes C with different arrangements of exposure periods and non-exposure periods, it is possible to prevent information about the scene 8 from being missed (lost) during one frame period T.

[0041] Returning to FIG. 1 , for example, the pixels 3 of each pixel group are exposed using an exposure period pattern P defined by the code C described above, and the corresponding image data 5, in this example, image data 5-1 to 5-4, is obtained. As described above, the code C is a binarized version of the weight of the convolution kernel in the time direction. The image data 5 obtained using the exposure period pattern P defined by such a code C is image data after (the calculation of) convolution in the time direction has been performed. Note that convolution in the time direction means the extraction of feature amounts in the time direction, and therefore the image data 5 can also be said to be image data from which feature amounts in the time direction have been extracted.

[0042] The processing unit 6 processes the image data 5. Any processing related to images may be performed. One example of the processing is image recognition processing. Examples of image recognition include recognition of the subject 9, etc., and recognition of the movement and motion of the subject 9, etc. Recognition may be interpreted as detection, identification, discrimination, etc., and these may be interpreted appropriately within a consistent range. Another example of processing is video generation processing.

[0043] In this embodiment, the processing unit 6 processes the image data 5 using a trained model, for example, a neural network (NW). The trained model is not limited to a NW. For example, a trained model that has similar functions to the NW described in this disclosure or that can be generated using similar machine learning may be used. Furthermore, this does not preclude the application of technologies such as trained models that may appear in the future. Hereinafter, the trained model will be described as a NW. The NW used by the processing unit 6 is illustrated and referred to as the execution NW 60. The execution NW 60 is generated in advance by machine learning and stored in the storage unit 7 so as to output output data when image data 5 is input. The output data of the execution NW 60 may include any data that can be obtained from the image data 5. For example, data indicating the results of the above-mentioned image recognition processing, data indicating the results of the video generation processing, etc., may be the output data of the execution NW 60. The execution NW 60 will be described with reference to FIG. 8 as well.

[0044] FIG. 8 is a diagram illustrating an example of the schematic configuration of the execution NW 60. The execution NW 60 is configured to perform spatial operations. The execution NW 60 may include an operation layer that performs spatial operations. Examples of operation layers include a convolution layer, a fully connected layer, and a self-attention layer. The convolution layer is a layer that performs convolution operations. The fully connected layer is a layer that connects all nodes to each other. The self-attention layer is a layer that extracts relationships between input data. Since those skilled in the art can understand more specific details of the convolution layer, fully connected layer, and self-attention layer, a description thereof will be omitted here. Unless otherwise specified, the spatial operation of the execution NW 60 is assumed to be (operation of) spatial convolution, and the operation layer is assumed to be a convolution layer. Within the scope of no contradiction, spatial operation and spatial convolution may be interpreted as appropriate. The computation layer of the execution network 60 is illustrated as a spatial convolution layer 62 .

[0045] The spatial convolution layer 62 performs convolution in the spatial direction of the image data 5. Specifically, the image data 5 is input to the spatial convolution layer 62 directly or via another layer. The spatial convolution layer 62 performs convolution on the input image data 5 in the spatial direction, more specifically, along the two-dimensional spatial axis directions of the width Wd direction and the height Ht direction, and extracts feature amounts in each direction.

[0046] The execution NW 60 does not need to include a convolution layer that performs convolution in the time direction. This is because, as described above, the image data 5 is image data after convolution in the time direction. The execution NW 60 can also be said to be a trained model that outputs output data when image data 5 that has been convolved in the time direction (time-operated) is input.

[0047] Returning to FIG. 1 , the processing unit 6 processes the image data 5 using the execution NW 60 described above. The processing unit 6 inputs the image data 5 to the execution NW 60. The execution NW 60 outputs output data. The processing unit 6 acquires the output data output by the execution NW 60. The processing unit 6 outputs the acquired output data or data based on it as the processing result.

[0048] As described above, the image data 5 input to the execution NW 60 is image data after convolution in the time direction. When this image data is input to the execution NW 60, convolution in the spatial direction is performed by the spatial convolution layer 62. As a result, in the sensor 1, convolution is performed on the image data 5 in all three-dimensional directions, that is, the time direction and the spatial direction (the width Wd direction and the height Ht direction).

[0049] It should be noted that the convolution in the time direction is realized by driving the pixel array unit 2 by the driver unit 4, not by the execution network 60 used by the processing unit 6. It can also be said that the convolution in the time direction is realized in hardware. The execution network 60 does not need to include a convolution layer for performing the convolution in the time direction. It can also be said that the convolution in the time direction does not need to be performed in software. As a result, the calculation cost required for the convolution in the time direction (temporal calculation) can be reduced. Power consumption can also be reduced.

[0050] <Examples of Generation of Code C> Several examples of generation of code C will be described with reference to Figs. 9 to 19. In the following description, the weight of the convolution kernel (filter) in the time direction will also be referred to as kernel weight W. The code C exemplified below may be included in the base filter 40, or may be generated by the code generation unit 41 based on the base filter 40.

[0051] 9 to 19 are diagrams showing examples of generating a code C. In each diagram, (A) shows a given kernel weight W (its scalar value). In each diagram, (B) shows the code C generated by binarizing the kernel weight W.

[0052] 9 to 12, the code C is generated using threshold processing. The threshold value set for the kernel weight W is referred to as the threshold value Wth. For example, a kernel weight W equal to or greater than the threshold value Wth is converted to "1" (exposure on), and a kernel weight W less than the threshold value Wth is converted to "0" (exposure off).

[0053] 9, the threshold value Wth is fixed. The kernel weight W can be binarized by a simple calculation.

[0054] In the examples shown in FIGS. 10 to 12, a dithering method is used. The value of the threshold Wth can vary over time (dither threshold). For example, even when the kernel weight W is constant, as shown in FIGS. 10A, 11A, and 12A, different codes C can be obtained depending on the magnitude of the kernel weight W and the set threshold Wth. Specifically, the code C shown in FIG. 10B describes consecutive "1"s. The code C shown in FIG. 11B generally describes "0"s and "1"s alternately. The code C shown in FIG. 12B describes these values ​​so that there are more "0"s than "1"s. For example, even when the kernel weight W is constant, it is possible to prevent consecutive "1"s or consecutive "0"s.

[0055] In the examples shown in Figures 13 and 14, "1" is written consecutively in a number corresponding to the magnitude (peak height) of the kernel weight W. For example, as shown in Figure 13A, if the kernel weight W at a certain time is very large, the code C obtained therefrom will write "1" over three consecutive periods including that time, as shown in Figure 13B. The kernel weight W shown in Figure 14A is even larger, and the code C obtained therefrom will write "1" over five consecutive periods, as shown in Figure 14B.

[0056] 15 to 19 show examples of sampling. The sampling period of the kernel weight W is referred to as the sampling period Ts1. The reciprocal of the sampling period Ts1 is the sampling frequency of the kernel weight W.

[0057] The sampling period of code C is referred to as sampling period Ts2. Sampling period Ts2 is the smallest unit of exposure period (exposure period) that can be defined by code C. The reciprocal of sampling period Ts2 is the sample frequency of code C, which is also the reciprocal of the exposure period (which can also be said to be the operating frequency of exposure).

[0058] In the example shown in Figure 15, the sampling period Ts1 of the kernel weight W is the same as the sampling period Ts2 of the code C (Ts1 = Ts2). In other words, the sampling frequency of the convolution kernel is the same as the operating frequency of the exposure. The kernel weight W is binarized by the threshold process described above.

[0059] In the examples shown in Figures 16 and 17, the sampling period Ts1 of the kernel weight W and the sampling period Ts2 of the code C are different from each other. In this case, the kernel weight W is resampled (subsampled) at the sampling period Ts2 of the code C and then binarized. In the example shown in Figure 16, the sampling period Ts1 of the kernel weight W is longer than the sampling period Ts2 of the code C (Ts1 > Ts2). In other words, the sampling frequency of the kernel weight W is lower than the operating frequency of the exposure. In the example shown in Figure 17, the sampling period Ts1 of the kernel weight W is shorter than the sampling period Ts2 of the code C (Ts1 < Ts2). In other words, the sampling frequency of the kernel weight W is higher than the operating frequency of the exposure.

[0060] In the example shown in Fig. 18, the kernel weights W are binarized without resampling. For example, as described above with reference to Figs. 13 and 14, the kernel weights W are binarized according to their peak heights.

[0061] 19, a continuous function is given that describes the kernel weight W. The kernel weight W described by the continuous function is sampled at a sampling period Ts2 of the code C and then binarized.

[0062] <Examples of Characteristics of Exposure Period Pattern P> As described above, the code C is generated to define exposure period patterns P having various characteristics. In other words, the exposure period pattern P is adjusted by the code C so that the exposure period pattern P has various characteristics. Some examples will be described with reference to FIGS. 20 to 28.

[0063] 20 to 28 are diagrams showing examples of the characteristics of the exposure period pattern P. FIG.

[0064] In the examples shown in Figures 20 and 21, the exposure period pattern P is adjusted so as to weight the exposure period to a portion of one frame period T. The exposure period pattern P shown in Figure 20 places weight on the exposure period in the first half of one frame period. The exposure period in the first half of one frame period T is longer than the exposure period in the second half. The exposure period pattern P shown in Figure 21 places weight on the exposure period in the second half of one frame period T. The exposure period in the second half of one frame period T is longer than the exposure period in the first half.

[0065] 20 and 21 may be generated according to the characteristics of the sensor 1. As an example, even if the total exposure period (the sum of the exposure periods in one frame period T) is the same, if the sensor 1 has a signal-to-noise (SN) ratio that improves when the exposure period in the first half of one frame period T is longer than the exposure period in the second half, the code C as shown in FIG. 20 may be generated. Conversely, if the sensor 1 has a signal-to-noise ratio that improves when the exposure period in the second half of one frame period T is longer than the exposure period in the first half, the code C as shown in FIG. 21 may be generated. This makes it easier to obtain image data 5 with a high SN ratio.

[0066] In the examples shown in Figures 22 and 23, the exposure period pattern P is adjusted to have a desired total exposure period. The exposure period pattern P illustrated in Figure 22 is adjusted to shorten the total exposure period. Code C is generated to describe more "0"s (exposure off) than "1"s (exposure on). The exposure period pattern P illustrated in Figure 23 is adjusted to lengthen the total exposure period. Code C is generated to describe more "1"s (exposure on) than "0"s (exposure off).

[0067] The code C defining the exposure period pattern P as shown in Figures 22 and 23 above may be generated according to the brightness of the scene 8, etc. As an example, if the scene 8, etc. is bright and the amount of light incident on the pixel 3 is large, the code C as shown in Figure 22 above may be generated. By suppressing saturation of the pixel 3 (photoelectric conversion unit), appropriate image data 5 can be obtained. If the scene 8, etc. is dark and the amount of light incident on the pixel 3 is small, the code C as shown in Figure 23 above may be generated. By lengthening the total exposure period and taking in a large amount of light, it becomes easier to obtain image data 5 with a high S / N ratio.

[0068] In the examples shown in FIGS. 24 and 25, the exposure period pattern P is adjusted to include exposure periods with a desired sparse density.

[0069] The exposure period pattern P illustrated in FIG. 24A is adjusted to include multiple exposure periods closely spaced (e.g., adjacent) on the time axis. Code C describes "1" (exposure on) over five consecutive periods. For example, when a moving scene 8 is captured, image data 5 such as that shown in FIG. 24B is obtained. Although the blurring makes it difficult to recognize the movement of the scene 8, the advantage is that there are fewer artifacts, such as those seen in FIG. 25 below.

[0070] The exposure period pattern P illustrated in FIG. 25A is adjusted to include multiple exposure periods sparsely (e.g., spaced apart) on the time axis. Code C describes "1" (exposure on) in five spaced periods. For example, when a scene 8 including a moving subject 9 is captured, image data 5 such as that shown in FIG. 25B is obtained. Although artifacts occur, this has the advantage of making it easier to recognize movement in the scene 8, etc.

[0071] For example, exposure period patterns P with different characteristics as shown in Figures 24 and 25 above may be used together (combined), making it possible to enjoy the advantages of each. That is, the pixels 3 of a certain pixel group may be exposed with the exposure period pattern P as shown in Figure 24 above, and the pixels 3 of another pixel group may be exposed with the exposure period pattern P as shown in Figure 25 above. Some further examples of combinations of exposure period patterns P will be described with reference to Figures 26 to 28.

[0072] 26 to 28 show examples of combinations of exposure period patterns P having different characteristics. An exposure period pattern P-1 defined by the code C-1 shown in (A) of each figure and an exposure period pattern P-2 defined by the code C-2 shown in (B) of each figure are used in combination.

[0073] 26, the exposure period of the exposure period pattern P-2 defined by the code C-2 overlaps on the time axis with the non-exposure period of the exposure period pattern P-1 defined by the code C-1. It can be said that the exposure period of the exposure period pattern P-2 fills at least a portion of the gap (non-exposure period) between the exposure periods of the exposure period pattern P-1. This reduces redundant information acquisition and allows for more information to be allocated to other exposure period patterns P.

[0074] In the example shown in Figure 27, the exposure period of exposure period pattern P-2 defined by code C-2 overlaps on the time axis with the exposure period of exposure period pattern P-1 defined by code C-1, and also overlaps with at least a portion of the non-exposure period of exposure period pattern P-1. This overlap can improve the reliability of information acquisition. It also makes it easier to recognize whether or not there is movement in scene 8, etc. This will be explained with reference to Figure 28.

[0075] For example, image data 5-1 shown in Figure 28(A) is obtained based on exposure period pattern P-1 shown in the same figure. Image data 5-2 shown in the same figure is obtained based on exposure period pattern P-2 shown in Figure 28(B). By obtaining these two sets of image data 5, image data 5-1 and 5-2, it becomes easier to recognize the movement of the subject 9, etc., than if only one of them were obtained.

[0076] <Examples of Kernel Weight W> There are various possible kernel weights W that are used to generate the code C. For example, a continuous function may be used to describe the kernel weight W. It is also possible to handcraft the kernel weight W. This will be described with reference to FIGS. 29 to 36.

[0077] 29 to 36 are diagrams showing examples of the kernel weight W. In the examples shown in Fig. 29 to 31, the kernel weight W is described by a continuous function with a constant value.

[0078] By using the dithering method described above, the kernel weight W as shown in Fig. 29(A) is binarized, and a code C that defines the exposure period pattern P as shown in Fig. 29(B) is generated. As described above, even when the kernel weight W is constant, it is possible to prevent consecutive "1"s or consecutive "0"s (to describe "1"s and "0"s in a balanced manner).

[0079] In the example shown in Fig. 30A, the kernel weight W is larger than the kernel weight W in Fig. 29 (shown by the dashed line in Fig. 30A). From this kernel weight W, a code C as shown in Fig. 30B is generated. In the exposure period pattern P defined by the code C, the exposure periods are denser than in the exposure period pattern P defined by the code C in Fig. 29B described above. In addition, the total exposure period is longer.

[0080] In the example shown in Fig. 31A, the kernel weight W is smaller than that shown in Fig. 29. From this kernel weight W, a code C as shown in Fig. 31B is generated. In the exposure period pattern P defined by the code C, the exposure periods are sparser than in the exposure period pattern P defined by the code C in Fig. 29B described above. In addition, the total exposure period is shorter.

[0081] The code C generated from the kernel weight W as shown in Figures 29 to 31 may be used differently depending on the brightness of scene 8, for example. The reason for this is the same as that shown in Figures 22 and 23, and therefore will not be repeated here. The characteristics of the code C that describes "1" and "0" in a balanced manner as shown in Figures 29 to 31 will be described with reference to Figure 32 as well.

[0082] 32 shows four types of codes Ci to Civ. Furthermore, their characteristics are exemplified by the signal-to-noise ratio, saturation characteristics, and moving object recognition. The saturation characteristics indicate the resistance of the pixel 3 (to saturation of its photoelectric conversion unit). The moving object recognition indicates the ease of recognizing the movement, motion, etc., of the subject 9, etc.

[0083] Code Ci specifies an exposure period pattern P for single-exposure imaging. It has a poor S / N ratio but excellent saturation characteristics and motion recognition. Code Cii specifies multiple exposure periods that are consecutive on the time axis over one frame period T. It has a very good S / N ratio but poor saturation characteristics and motion recognition. Code Ciii specifies multiple exposure periods that are consecutive on the time axis over only a portion of one frame period T (the first half in this example). By setting appropriate exposure periods, the S / N ratio, saturation characteristics, and motion recognition are all improved. Code CiV, like those described in Figures 29 to 31 above, describes a balanced number of "1"s and "0"s. It improves the S / N ratio, saturation characteristics, and motion recognition. With regard to motion recognition, there is less blurring compared to code Ciii, making it easier for the downstream processing unit 6 to recognize the scene 8 and subject 9 using the execution network 60, potentially resulting in better performance than code Ciii. However, artifacts may occur.

[0084] In the examples shown in FIGS. 33 and 34, the kernel weight W changes in value with a constant gradient.

[0085] In the example shown in Fig. 33A, the kernel weight W gradually decreases from the first half to the second half of one frame period T. From this kernel weight W, a code C as shown in Fig. 33B is generated. The weight of the exposure period is placed on the first half of one frame period T.

[0086] In the example shown in Fig. 34A, the kernel weight W gradually increases from the first half to the second half of one frame period T. From this kernel weight W, a code C as shown in Fig. 34B is generated. The weight of the exposure period is placed in the second half of one frame period T.

[0087] The code C generated from the kernel weight W as shown in Figures 33 and 34 may be used differently depending on, for example, the characteristics of the sensor 1. The reason for this is the same as that shown in Figures 20 and 21 described above, and therefore will not be described again here.

[0088] In the examples shown in Figures 35 and 36, the kernel weights W are varied to have a Gaussian distribution.

[0089] In the example shown in Fig. 35A, the kernel weight W has a relatively broad Gaussian distribution, is maximum at a certain time during one frame period T, and gradually decreases as the time moves away from that point. From this kernel weight W, a code C as shown in Fig. 35B is generated. The weight of the exposure period is placed near a certain time during one frame period T.

[0090] In the example shown in Fig. 36B, the kernel weight W has a relatively narrow Gaussian distribution, is maximum at a certain time during one frame period T, and rapidly decreases as the time moves away from that point. From this kernel weight W, a code C as shown in Fig. 35B is generated. The weight of the exposure period is placed near a certain time during one frame period T.

[0091] <Example of Switching Exposure Period Pattern P> Returning to FIG. 1 , in one embodiment, the driver 4 may drive the pixel array unit 2 so that the exposure period pattern P is switched in response to a trigger (condition). For example, the code generator 41 changes the code C to be generated in response to a trigger. The driver 4 drives the pixel array unit 2 in accordance with the exposure period pattern P defined by the changed code C. By switching the exposure period pattern P in response to various triggers, appropriate image data 5 can be obtained. Some examples of triggers will be described with reference to FIG. 37 .

[0092] 37 is a diagram showing examples of triggers, including a change in speed of the scene 8, a change in brightness of the scene 8, a change in speed of the subject 9, a change in brightness of the subject 9, a change in the state of the pixel 3, and a change in mode.

[0093] The speed and brightness changes of the scene 8, etc. may be estimated using sensing data from other sensors (not shown), or may be estimated from the obtained image data 5. Examples of other sensors include a LiDAR sensor, a radar sensor, an acceleration sensor, a gyro sensor, etc. The subject of estimation may be the driving unit 4, the processing unit 6, etc., or another element of the sensor 1 (not shown).

[0094] When a change in the speed of the scene 8 is used as a trigger, the drive unit 4 may drive the pixel array unit 2 so that the pixels 3 are exposed with an exposure period pattern P having a density that corresponds to the change in speed of the scene 8, etc. For example, if the speed is slow, the code C generated by the code generation unit 41 and therefore the exposure period pattern P may be switched so that the pixels 3 are exposed with an exposure period pattern P having a dense exposure period, as shown in FIG. 24 described above. If the speed is fast, the code C generated by the code generation unit 41 and therefore the exposure period pattern P may be switched so that the pixels 3 are exposed with an exposure period pattern P having a sparse exposure period, as shown in FIG. 25 described above. The specific method for determining whether the speed is large or small is not particularly limited, and may, for example, be a threshold value determination. The determination may be performed by the drive unit 4, the processing unit 6, etc., or another element of the sensor 1 not shown in the figure.

[0095] When a change in brightness of a scene 8 or the like is used as a trigger, the drive unit 4 may drive the pixel array unit 2 so that the pixel 3 is exposed by an exposure period pattern P having a total exposure period that corresponds to the change in brightness of the scene 8 or the like. For example, if the scene 8 or the like is dark, the code C generated by the code generation unit 41 and therefore the exposure period pattern P may be switched so that the pixel 3 is exposed by an exposure period pattern P with a short total exposure period, as shown in Figure 22 described above. If the scene 8 or the like is bright, the code C generated by the code generation unit 41 and therefore the exposure period pattern P may be switched so that the pixel 3 is exposed by an exposure period pattern P with a long total exposure period, as shown in Figure 23 described above.

[0096] An example of a change in the state of pixel 3 is a change in the amount of charge accumulated in the photoelectric conversion unit of pixel 3. This change can be detected, for example, by incorporating a detection transistor into pixel 3. The detection may be performed by the drive unit 4, the processing unit 6, or another element of the sensor 1 not shown in the figure. When such a change in the state of pixel 3 is used as a trigger, the drive unit 4 may switch the exposure period pattern P midway through one frame period T so that the photoelectric conversion unit of pixel 3 does not become saturated during one frame period T. This description will also be made with reference to Figure 38.

[0097] 38 is a diagram showing an example of switching of the exposure period pattern P in the middle of one frame period T. (B) of FIG. 38 schematically shows the amount of charge accumulated in the photoelectric conversion unit of the pixel 3 (amount of accumulated charge).

[0098] Initially, the code generation unit 41 of the drive unit 4 generates a code C-1 that defines an exposure period pattern P-1. At time t10, the drive unit 4 drives the pixel array unit 2 so that pixel 3 is exposed using the exposure period pattern P-1. As the exposure continues, charge accumulates in the photoelectric conversion unit of pixel 3, increasing the amount of accumulated charge. If exposure of pixel 3 using the exposure period pattern P-1 continues in this manner, the amount of accumulated charge will reach a saturation level before time t20, the end of one frame period T, as shown imaginarily by the dashed-dotted line in Figure 38 (B), and the photoelectric conversion unit of pixel 3 will saturate. This problem can occur when the exposure periods of the exposure period pattern P-1 are too close together or the total exposure period is too long for the brightness of scene 8, for example.

[0099] In order to prevent saturation of the photoelectric conversion units of the pixels 3 during one frame period T, the code generation unit 41 generates a code C-2 that specifies an exposure period pattern P-2 with a sparser exposure period (or a shorter total exposure period) than the exposure period pattern P-1. At time t15, the exposure period pattern P-1 is switched to the exposure period pattern P-2. The code generation unit 41 of the drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed by the drive unit 4 with the exposure period pattern P-2. As shown by the solid line in Figure 38 (B), the amount of accumulated charge does not reach a saturation level during one frame period T, and saturation of the photoelectric conversion units of the pixels 3 is avoided.

[0100] A method for predicting saturation midway through one frame period T (for example, at time t15) as described above will be described. The charge accumulation rate in the photoelectric conversion unit of pixel 3 is measured. The measurement may be performed by the drive unit 4, the processing unit 6, or another element of the sensor 1 not shown in the figure. The accumulation rate is the change in the amount of accumulated charge over time, and corresponds to the slope of the arrow indicated by the dashed line in Figure 38(B).

[0101] The kernel weight W that is the basis of the code C is adjusted according to the accumulation speed. The kernel weight W that is the basis of the code C-1 that defines the exposure period pattern P-1 (time t10 to time t15) is defined as kernel weight W1, and the kernel weight W that is the basis of the code C-2 that defines the exposure period pattern P-2 (time t15 to time t20) is defined as kernel weight W2. For example, the code C-2 that defines the exposure period pattern P-2 is generated from the kernel weight W2 calculated (adjusted) according to the following equation (1). (Equation 1) W2 = W1 × (maximum accumulation speed without saturation / measured accumulation speed)

[0102] Based on such kernel weight W2, a code C-2 that defines the above-mentioned exposure period pattern P-2 can be generated, and switching from exposure period pattern P-1 to exposure period pattern P-2 can be performed at an appropriate timing.

[0103] Returning to Fig. 37, an example of a change in mode is a change in the imaging mode of the sensor 1. This will be described with reference to Fig. 39 as well.

[0104] FIG. 39 is a diagram showing an example of a mode change. Examples of imaging modes of the sensor 1 include a sensing mode and a viewing mode. The imaging modes of the sensor 1 can be selected to transition between these modes. In the sensing mode, multiple exposure imaging is performed. The pixels 3 are exposed using the various exposure period patterns P described above, and image data 5 is obtained. The processing unit 6 processes the obtained image data 5 using the execution NW 60. In the viewing mode, single exposure imaging is performed. An image represented by the image data 5 is immediately displayed to the user via a display (not shown) or the like. The sensing mode and the viewing mode are changed, for example, by automatic selection or manual selection by user operation or the like.

[0105] In the sensing mode, the driver 4 drives the corresponding pixel 3 so as to perform multiple-exposure imaging. As explained above, image data 5 suitable for processing such as image recognition by the processor 6 is obtained. In the viewing mode, the driver 4 drives the corresponding pixel 3 so as to perform single-exposure imaging. Image data 5 showing an image that is easy for the user to view is obtained.

[0106] For example, the exposure period pattern P may be switched in response to the various triggers described above.

[0107] In one embodiment, the driver 4 may output information regarding the switching of the exposure period pattern P. Examples of the information include the charge accumulation rate described above, the switching time (time t15), and the exposure period pattern P-2 after the switching (which may be the code C-2 or the kernel weight W2 on which the code C-2 is based). Only simplified information indicating that the exposure period pattern P has been switched may be output. The processor 6 may process the image data 5 using the information output by the driver 4.

[0108] <Switching of Processing by Processing Unit 6 in Accordance with Switching of Exposure Period Pattern P> Returning to Fig. 1 , the processing of image data 5 by the processing unit 6 using the execution NW 60 may also be switched in response to the switching of the exposure period pattern P as described above. Some examples will be described with reference to Figs. 40 to 44 .

[0109] 40 to 44 are diagrams showing examples of switching of the processing by the processing unit 6 in response to switching of the exposure period pattern P. FIG.

[0110] 40 and 41, the execution NW 60 is a plurality of execution NWs 60, which are prepared in advance and stored in the storage unit 7. Each of the plurality of execution NWs 60 is a NW that has been machine-learned to process image data 5 obtained by exposing pixels 3 with different exposure period patterns P. In order to be able to distinguish between the execution NWs 60, they are referred to as execution NWs 60-1 to 60-3 and illustrated. When there is no particular need to distinguish between them, they are simply referred to as execution NWs 60.

[0111] The execution NW 60-1 processes image data 5-1 obtained by exposure period pattern P-1 defined by code C-1. The execution NW 60-2 processes image data 5-2 obtained by exposing pixel 3 with exposure period pattern P-2 defined by code C-2. The execution NW 60-3 processes image data 5-3 obtained by exposure period pattern P-3 defined by code C-3.

[0112] The processing unit 6 switches the execution NW 60 to be used depending on the exposure period pattern P used to obtain the image data 5 to be processed. Each image data 5 obtained by exposing the pixels 3 with a different exposure period pattern P can be appropriately processed.

[0113] Specifically, as shown in FIG. 41A, the processing unit 6 inputs image data 5-1 obtained by exposing the pixel 3 in the exposure period pattern P-1 to the execution NW 60-1.

[0114] When switching from exposure period pattern P-1 to exposure period pattern P-2, the processing unit 6 inputs image data 5-2 obtained by exposing pixel 3 in exposure period pattern P-2 to the execution NW 60-2, as shown in (B) of Figure 41.

[0115] When switching from exposure period pattern P-2 to exposure period pattern P-3, the processing unit 6 inputs image data 5-3 obtained by exposing pixel 3 in exposure period pattern P-3 to the execution NW 60-3, as shown in (C) of Figure 41.

[0116] In the example shown in FIG. 42, one execution NW 60 is machine-trained to process image data 5 obtained by a plurality of different exposure period patterns P. Even if the exposure period pattern P is switched, the processing unit 6 does not switch the execution NW 60 to be used. Specifically, as shown in (A) to (C) of FIG. 42, the processing unit 6 inputs image data 5-1 obtained by exposing pixel 3 with exposure period pattern P-1 to the execution NW 60. The processing unit 6 also inputs image data 5-2 obtained by exposing pixel 3 with exposure period pattern P-2 to the same execution NW 60. The processing unit 6 also inputs image data 5-3 obtained by exposing pixel 3 with exposure period pattern P-3 to the same execution NW 60. There is no need to prepare multiple execution NWs 60 as in FIGS. 40 and 41 described above.

[0117] In the examples shown in Figures 43 and 44, the execution NW 60 is machine-trained to process image data 5 obtained by exposing pixels 3 in a specific exposure period pattern P. In addition, a conversion network 64 that can be used together with this execution NW 60 is generated in advance and stored in the storage unit 7. There may be multiple conversion networks 64, and in Figure 43, two conversion networks 64 are exemplified as conversion network 64-2 and conversion network 64-3. When there is no particular distinction between these, they are simply referred to as conversion networks 64.

[0118] The conversion network 64 is generated so as to absorb the difference between the image data 5 obtained by exposing the pixel 3 in a specific exposure period pattern P and the image data 5 obtained by exposing the pixel 3 in another exposure period pattern P. Here, the specific exposure period pattern P is assumed to be exposure period pattern P-1. The conversion network 64-2 is generated so as to absorb the difference between the image data 5-1 obtained by exposing the pixel 3 in the exposure period pattern P-1 and the image data 5-2 obtained by exposing the pixel 3 in the exposure period pattern P-2. The conversion network 64-3 is generated so as to absorb the difference between the image data 5-1 obtained by exposing the pixel 3 in the exposure period pattern P-1 and the image data 5-3 obtained by exposing the pixel 3 in the exposure period pattern P-3.

[0119] The processing unit 6 inputs image data 5-1 obtained by exposing the pixel 3 in the exposure period pattern P-1 to the execution NW 60. On the other hand, the execution NW 60 inputs image data 5 obtained by an exposure period pattern P other than the exposure period pattern P-1 to the execution NW 60 via the corresponding conversion network 64.

[0120] Specifically, the processing unit 6 directly inputs image data 5-1 obtained by exposing the pixel 3 in the exposure period pattern P-1 to the execution NW 60, as shown in FIG.

[0121] When switching from exposure period pattern P-1 to exposure period pattern P-2, the processing unit 6 inputs the image data 5-2 obtained by exposing pixel 3 in exposure period pattern P-2 to the execution NW 60 via the conversion network 64-2, as shown in (B) of Figure 44.

[0122] When switching from exposure period pattern P-2 to exposure period pattern P-3, the processing unit 6 inputs the image data 5-3 obtained by exposure with exposure period pattern P-3 to the execution NW 60 via the conversion network 64-3, as shown in (C) of Figure 44.

[0123] In this way, by using a combination of the execution network 60 and the conversion network 64, it is possible to appropriately process each image data 5 obtained by exposing the pixels 3 with different exposure period patterns P. There is no need to prepare multiple execution networks 60 as in the previously described Figures 40 and 41.

[0124] 2. Embodiment of Sensor System Some elements of the sensor 1 described so far may be provided outside the sensor 1. This will be described with reference to FIG.

[0125] 45 is a diagram illustrating an example of a schematic configuration of a sensor system 1A according to an embodiment. The sensor system 1A differs from the sensor 1 (FIG. 1) described above in that some elements are provided outside the sensor 1.

[0126] In this example, the code generation unit 41, the processing unit 6, and the storage unit 7 are provided outside the sensor 1. The sensor 1 includes a pixel array unit 2 and a drive unit 4 (excluding the code generation unit 41). The sensor system 1A includes the sensor 1, the code generation unit 41, the processing unit 6, and the storage unit 7. For example, an application processor that uses the sensor 1 functions as the code generation unit 41 and the processing unit 6. The storage unit 7 may be a storage device such as a memory that stores information used by such an application processor.

[0127] By providing the code generation unit 41, the processing unit 6, and the storage unit 7 outside the sensor 1, it is possible to simplify the configuration of the sensor 1 and reduce the processing load on the sensor 1 compared to when they are provided inside the sensor 1. This can contribute to miniaturization and cost reduction of the sensor 1. Furthermore, since the processing unit 6, the code generation unit 41, and the storage unit 7 can be designed separately from the sensor 1, there is also the advantage of increasing the degree of freedom in design.

[0128] The elements provided outside the sensor 1 are not limited to the above example. For example, the code generation unit 41 may remain included in the drive unit 4 within the sensor 1, and only the processing unit 6 and the storage unit 7 may be provided outside the sensor 1.

[0129] 3. Embodiment of Information Processing System Fig. 46 is a diagram showing an example of a schematic configuration of an information processing system 100 according to an embodiment. The information processing system 100 includes an information processing device 10 and the sensor 1 described above. Note that the sensor 1 shown in Fig. 46 may be the sensor system 1A previously described with reference to Fig. 45.

[0130] The information processing device 10 generates an active NW 60. The information processing device 10 includes an acquisition unit 11, a data generation unit 12, a learning unit 13, a NW generation unit 14, and a storage unit 15.

[0131] The acquisition unit 11 acquires original data 110. The original data 110 is given data used to generate learning data 120, which will be described later. Examples of the original data 110 include time-series image data and CG (Computer Graphics) material data. The time-series image data is, for example, data of a series of images (natural images) obtained by capturing images of a real scene. The CG material data is material data that can be used to generate CG images of scenes including various subjects, and includes, for example, asset data such as buildings, cars, and people.

[0132] The manner in which the acquisition unit 11 acquires the original data 110 is not particularly limited. For example, the original data 110 may be acquired by a user of the information processing device 10 preparing (collecting, etc.) and inputting the original data 110. Any data that can be acquired via an external network (such as the Internet) may be acquired as the original data 110. The acquired original data 110 is stored in the storage unit 15.

[0133] The data generation unit 12 generates training data 120 using the original data 110. The training data 120 is training data to be used in subsequent machine learning, and is a set (data set) of pairs of input data 121 and supervised answer data 122. The generated training data 120 is stored in the storage unit 15.

[0134] The learning unit 13 uses the learning data 120 to perform machine learning on the NW. The NW that is the target of machine learning by the learning unit 13 is referred to as the learning target NW 130. The learning target NW 130 is stored in the storage unit 15. The learning target NW 130 and its machine learning will be described with reference to Figs. 47 and 48 .

[0135] FIG. 47 is a diagram showing an example of a schematic configuration of the learning target NW 130. The learning target NW 130 is configured to perform temporal and spatial operations. For example, the learning target NW 130 includes an operation layer that performs temporal and spatial operations. Unless otherwise specified, the operation layer of the learning target NW 130 is assumed to be a convolutional layer. FIG. 47 shows two types of configurations as examples of the configuration of the learning target NW 130.

[0136] In the example shown in (A) of FIG. 47, the learning network 130 includes a time axis convolution layer 131 and a spatial convolution layer 132 as an operation layer for performing temporal operations and an operation layer for performing spatial operations. The time axis convolution layer 131 performs convolution in the time direction. Features in the time direction are extracted. The spatial convolution layer 132 performs convolution in the spatial direction, similar to the spatial convolution layer 62 (FIG. 8) of the execution network 60 described above. Features in the spatial direction are extracted.

[0137] In the example shown in Fig. 47(B), the learning target network 130 includes a 3D calculation layer that performs temporal and spatial calculations as a calculation layer. An example of the 3D calculation layer is shown in the figure as a 3D convolution layer 133. The 3D convolution layer 133 has the functions of both the time axis convolution layer 131 and the spatial convolution layer 132 described above. Feature amounts in the time direction and the spatial direction are extracted.

[0138] 48 is a diagram showing an example of machine learning by the learning unit 13. In this example, the error backpropagation method is used. In the following description, the error may also be referred to as loss.

[0139] The learning unit 13 extracts (fetches) one pair of input data 121 and supervised data 122 from the learning data 120 generated by the data generation unit 12. In this example, the input data 121 is time-series image data and has three dimensions: the time direction and the spatial direction (two-dimensional spatial direction). The input data 121 is dense in the time direction so as to include time-series image data for the kernel size (number of taps) of the time axis convolution layer 131. Furthermore, each image data is dense in the two-dimensional spatial direction (width Wd direction and height Ht direction) so as to include data for the kernel size of the spatial convolution layer 132. The supervised data 122 is data that serves as a model for the output data of the learning target network 130. The supervised data 122 is also referred to as teacher data, etc.

[0140] The learning unit 13 inputs input data 121 of a pair extracted from the learning data 120 to the learning target NW 130. The learning unit 13 inputs the input data 121 to the learning target NW 130 so that the input data 121 is convolved in the time direction by the time axis convolution layer 131 of the learning target NW 130 and convolved in the spatial direction by the spatial convolution layer 132. In this example, the input data 121 is input directly or indirectly to the time axis convolution layer 131 so that the input data 121 is convolved in order by the time axis convolution layer 131 and the spatial convolution layer 132. In response to the input of the input data 121, the learning target NW 130 outputs output data.

[0141] The learning unit 13 compares the output data with the supervised data 122. Specifically, the learning unit 13 uses a loss function to calculate a loss value of the output data relative to the supervised data 122. Then, the learning unit 13 adjusts parameters of the learning target NW 130 including the time axis convolution layer 131 and the spatial convolution layer 132 so as to reduce the loss value (feedback). Specifically, the learning unit 13 updates the learning target NW 130 by backpropagation or the like using a differential gradient obtained from the loss function. A series of processes is repeated to machine-learn the learning target NW 130.

[0142] The generation of the input data 121 used in the machine learning of FIG. 48 will be described with reference to FIG. 49 as well.

[0143] 49 is a diagram showing examples of methods for generating the input data 121. Four types of generation methods are shown as examples.

[0144] In generation method 1, the data generation unit 12 uses the time-series image data included in the original data 110 as input data 121 as is.

[0145] In generation method 2, the data generation unit 12 uses CG material data included in the original data 110 and CG rendering technology to generate input data 121. Various known CG rendering engines and the like may be used.

[0146] In generation method 3, the data generation unit 12 generates input data 121 using time-series image data included in the original data 110 and Neural Radiance Fields (NeRF) technology. It is possible to generate image data of a scene 8 viewed from various angles. It is also possible to generate data from which blur has been removed. It is also possible to train the data of images that intentionally have blur. For example, blur can be imparted by generating and averaging multiple images in a time series or by using a blur kernel.

[0147] In generation method 4, the time-series image data included in the original data 110 is time-series image data with a frame rate lower than the frame rate (FPS) of the input data. The data generator 12 generates the input data 121 using such low-frame time-series image data and a frame interpolation technique. Various known frame interpolation techniques may be used. This is effective when the kernel size of the time-axis convolution layer 131 is large, for example. Because the original data 110 can be low-frame time-series image data, it is possible to reduce the data volume and simplify the corresponding hardware configuration.

[0148] 48 , in this example, the data generator 12 may generate the input data 121 according to the machine learning status (learning status) of the learning target NW 130. For example, the input data 121 including image data for which machine learning is insufficient is generated by the data generator 12. By generating the input data 121 by feeding back the machine learning status, more appropriate machine learning of the learning target NW 130 can be performed.

[0149] In the above description, the input data 121 is time-series image data having density in both the time and spatial directions. In one embodiment, the input data 121 may be image data convolved (time-operated) in the time direction. In this case, the input data 121 is image data having density only in the spatial direction. This will be described with reference to FIGS. 50 and 51 .

[0150] 50 is a diagram showing an example of machine learning by the learning unit 13. The data generating unit 12 generates image data after convolution in the time direction as input data 121. The input data 121 is data for a single image having density in the width Wd direction and the height Ht direction. The input data 121 here is generated in a differentiable manner, and a specific example will be described later.

[0151] The learning unit 13 inputs the input data 121 to the learning target NW 130 so that the input data 121 is not convolved in the time direction (no temporal operation) by the time axis convolution layer 131 of the learning target NW 130, but is convolved in the spatial direction (spatial operation) by the spatial convolution layer 132. In this example, the input data 121 is input directly or indirectly to the spatial convolution layer 132. In response to the input of the input data 121, the learning target NW 130 outputs output data.

[0152] The learning unit 13 compares the output data with the correct answer data 122 and calculates a loss value. Then, the learning unit 13 adjusts parameters of the learning target NW 130 including the temporal axis convolution layer 131 and the spatial convolution layer 132 (feedback) so as to reduce the loss value.

[0153] Unlike the previously described FIG. 48 , feedback to the time axis convolution layer 131 is performed via the data generator 12. This is because convolution in the time direction is essentially performed at the stage of generation of the input data 121 by the data generator 12. In the example shown in FIG. 50 , feedback to the time axis convolution layer 131 is performed via the spatial convolution layer 132, the input data 121, and the data generator 12. Because the input data 121 is generated by the data generator 12 in a differentiable manner, backpropagation to the time axis convolution layer 131 is possible by the error backpropagation method.

[0154] As in the case of FIG. 48 described above, the input data 121 may be generated depending on the machine learning status (learning status) of the learning target NW 130 .

[0155] The generation of the input data 121 used in the machine learning of FIG. 50 will be described with reference to FIG. 51 as well.

[0156] 51 is a diagram showing an example of a method for generating the input data 121. Two types of generation methods are exemplified.

[0157] In generation method 11, the data generation unit 12 generates input data 121 using CG material data included in the original data 110 and a differentiable CG rendering technique. Various known CG rendering engines may be used. As described above, using the differentiable CG rendering technique enables feedback to the time axis convolution layer 131 using the backpropagation method.

[0158] In generation method 12, the data generator 12 generates input data 121 using time-series image data included in the original data 110 and NeRF editing technology. NeRF editing is a technology for obtaining desired image data by rearranging the contents of an NeRF neural network in a black-box manner. For details, see, for example, Non-Patent Document 1. It is also possible to use an NeRF technology that edits the time-series image data, which is the original data 110, to obtain image data by convolving it in the time direction. A differentiable convolution operation may be used as the time-direction convolution in this NeRF technology. Using such an NeRF technology, as described above, feedback to the time axis convolution layer 131 is possible using the backpropagation method.

[0159] In one embodiment, some processing may be added to the machine learning shown in Figures 48 and 50 above. This will be explained with reference to Figure 52.

[0160] Fig. 52 is a diagram showing an example of machine learning by the learning unit 13. Compared to Fig. 48 and Fig. 50 described above, this differs in that it includes two processes, filter regularization and data augmentation. These processes themselves are well known, so the following description will mainly focus on the parts specific to this embodiment.

[0161] The filter regularization may be regularization of the kernel weights of the time axis convolution layer 131 of the learning target NW 130. For example, it is possible to adjust the kernel weights W (i.e., the weights of the exposure periods) at each time from the start (time t10) to the end (time t20) of one frame period T. This can be used for machine learning to process image data 5 obtained by an exposure period pattern P that places weight on the first half or the second half of one frame period T. The description will also refer to FIG. 53 .

[0162] FIG. 53 is a diagram showing an example of filter regularization. The kernel weight of the time axis convolution layer 131 of the learning target NW 130 is referred to as the kernel weight W, as in the previous sensor embodiment. A cost function (cost weighting function) that weights the kernel weight W for each time is defined. A loss value is calculated based on this kernel weight W and the cost function. Specifically, the loss value calculated by convolving the cost function with the kernel weight W is reflected in the loss value in FIG. 52. By updating the parameters of the learning target NW 130 so as to reduce this loss value, the kernel weight W from the start (time t10) to the end (time t20) of one frame period T is adjusted.

[0163] Returning to Fig. 52, data expansion will be described. By expanding the input data 121, more input data can be obtained. In this example, the data generator 12 expands the generated input data 121 to generate new input data. The new input data is referred to as input data 121a. Typically, the input data 121a is obtained by changing the color or size of the image indicated by the input data 121. Data expansion in accordance with the design of the sensor 1 (Fig. 46) is also possible, and this will be described with reference to Figs. 54 and 55.

[0164] 54 and 55 are diagrams showing examples of data extension.

[0165] 54, input data 121a is generated by shifting the motion period (subject motion period) of subject 9, etc. in input data 121. For example, as shown in (A) to (C) of Fig. 54, input data 121a different from input data 121 is generated by shifting the subject motion period little by little with respect to one frame period T.

[0166] In the example shown in FIG. 55 , the pattern of the convolution kernel in the time direction in the input data 121 is changed. The total period (corresponding to the total exposure period) over which the convolution kernel has weights is maintained. Various patterns can exist in both cases where the total period is long, as shown in FIG. 55 (A), and where the total period is short, as shown in FIG. 55 (B). For example, by randomly setting each period within one frame period T without changing the total period, input data 121a different from the input data 121 is generated. Even if the pattern is the same, input data 121a different from the input data 121 can be generated by changing the way the blur of scene 8, etc., shown in the input data 121 is generated.

[0167] Returning to FIG. 52 , for example, at least one of the filter regularity and data expansion processes described above may be added to the machine learning of the learning target NW 130 by the learning unit 13. Other processes may also be added. An example of other processes is a process for calculating additional loss values. For example, an evaluation network that outputs the inception loss may be separately provided, and additional loss values ​​may be calculated.

[0168] Returning to FIG. 46 , the learning target NW 130 that has been machine-learned by the learning unit 13 is stored in the storage unit 7 .

[0169] The NW generation unit 14 generates the execution NW 60 based on the learning target NW 130. The NW generation unit 14 may generate the execution NW 60 by converting a part of the learning target NW 130, or may generate the learning target NW 130 as the execution NW 60 as is. This will be described with reference to Figs. 56 to 60.

[0170] 56 to 60 are diagrams showing examples of generation of an execution NW 60 by the NW generation unit 14. In addition to the execution NW 60 of the sensor 1, the base filter 40 and the code generation unit 41 will also be described.

[0171] 56, the learning target NW 130 includes a time axis convolution layer 131 and a spatial convolution layer 132. As a configuration corresponding to this learning target NW 130, the base filter 40, the code generation unit 41, and the execution NW 60 in the sensor 1 (FIG. 46) or the sensor system 1A (FIG. 45) are shown.

[0172] The function of the time axis convolution layer 131 of the learning target NW 130 is incorporated into the base filter 40 and the code generator 41. That is, the function for generating the code C that defines the exposure period pattern P corresponding to the time axis convolution by the time axis convolution layer 131 is implemented in the sensor 1 ( FIG. 46 ) or in the sensor system 1A ( FIG. 45 ) as the base filter 40 and the code generator 41. For example, the base filter 40 and the code generator 41 are designed so that the code generator 41 can generate the code C that corresponds to the kernel weight W of the time axis convolution layer 131 as shown on the right side of FIG. 56 .

[0173] The function of the spatial convolutional layer 132 of the learning target NW 130 is incorporated into the spatial convolutional layer 62 of the execution NW 60. That is, the NW generation unit 14 generates the execution NW 60 by reflecting the spatial operation function in the spatial convolutional layer 132 in the spatial convolutional layer 62 of the execution NW 60 (reflecting it in the spatial operation of the execution NW 60). For example, the execution NW 60 is obtained by removing the time axis convolutional layer 131 from the learning target NW 130.

[0174] Here, the base filter 40 of the sensor 1 (or the code C that can be generated by the code generator 41) may be more limited than the time-direction convolution kernel (hereinafter also referred to as the filter) used during machine learning of the learning target NW 130. For example, the number of filters in the base filter 40 is less than the number of filters during machine learning. Also, while filters during machine learning can include positive and negative values, the base filter 40 can only include positive values. To address such issues, filters during machine learning may be merged or divided.

[0175] 57 shows an example of filter integration. The number of base filters 40 (or the number of codes C that the code generator 41 can generate) is smaller than the number of filters during machine learning. Therefore, the filters during machine learning are integrated (consolidated). For example, various known reparametrization techniques may be used.

[0176] Specifically, Fig. 57 shows filter a, filter b, filter c, and filter d as examples of filters used during machine learning. Filter a and filter b are integrated to obtain filter a+b. Furthermore, filter c and filter d are integrated to obtain filter c+d. The obtained filters a+b and c+d are used as base filters 40 or generated by the code generation unit 41. The number of filters used in sensor 1 can be reduced compared to the number of filters used during machine learning.

[0177] 58 shows an example of filter division. The filter during machine learning is assumed to include both positive and negative values. On the other hand, the filter that can be generated by the base filter 40 or the code generator 41 can only include positive values. For this reason, the filter during machine learning is divided into multiple filters that include only positive values.

[0178] Specifically, in FIG. 58, filter e and filter f are exemplified as filters used in machine learning. Filter e is divided, and filter e ＋ and filter e － For example, the filter e ＋corresponds to a filter obtained by dividing the positive value part of the filter e. － is a filter obtained by dividing the negative part of filter e and transforming it to include only positive values. Similarly, filter f is divided and filter f ＋ and filter f － is obtained.

[0179] The obtained filter e ＋ and filter e － and filter f ＋ and filter f － is used as the base filter 40 or generated by the code generator 41. ＋ and filter e － is used, for example, the filter e ＋ From the data after passing through the filter e － The data after passing through the filter f is subtracted. This subtraction may be a digital or analog operation. ＋ and filter f － The same is true for .

[0180] As shown in (B) of Fig. 47 described above, the training target NW 130 may include a 3D convolution layer 133. In this case, the NW generation unit 14 generates the execution NW 60 by reflecting the spatial convolution function in the 3D convolution layer 133 of the training target NW 130 in the spatial convolution layer 62 of the execution NW 60. This will be described with reference to Figs. 59 and 60.

[0181] 59 , the training target network 130 includes a 3D convolution layer 133. 140 decomposes the 3D convolution layer 133 of the training target network 130 into a time axis convolution layer 131 and a spatial convolution layer 132. For example, the 3D convolution layer 133 is decomposed by averaging the time channel. Of the obtained time axis convolution layer 131 and spatial convolution layer 132, the spatial convolution layer 132 is used as the spatial convolution layer 62 of the execution network 60.

[0182] In the example shown in FIG. 60, a time axis convolution layer 131 and a spatial convolution layer 132 are obtained from the 3D convolution layer 133 of the training target NW 130 by distillation of the NW. A NW including a time axis convolution layer and a spatial convolution layer (corresponding to the time axis convolution layer 131 and the spatial convolution layer 132 after distillation) is prepared separately from the training target NW 130. The same input data 121 is input to these two NWs. The time axis convolution layer and the spatial convolution layer of the prepared NW are adjusted so that the output data of each of the two NWs is the same, and are obtained as the time axis convolution layer 131 and the spatial convolution layer 132. The spatial convolution layer 132 of the obtained time axis convolution layer 131 and the spatial convolution layer 132 is used as the spatial convolution layer 62 of the execution NW 60.

[0183] 46 , the execution NW 60 obtained by the conversion by the NW generation unit 14 is stored in the storage unit 15. For example, if the learning target NW 130 includes the spatial convolution layer 132 but does not include the time axis convolution layer 131, the learning target NW 130 may be stored as is in the storage unit 15 as the execution NW 60.

[0184] The storage unit 15 stores information used by the information processing device 10. In addition to the above-described original data 110, learning data 120, learning target NW 130, and execution NW 60, examples of information stored in the storage unit 15 also include an information processing program 150. The information processing program 150 is a program (software, application, etc.) for causing a computer to function as the information processing device 10.

[0185] For example, the information processing device 10 described above can generate the execution NW 60. As described above, the sensor 1 acquires image data 5 and processes it using the execution NW 60. Some processing flows will be described with reference to Figs. 61 and 62.

[0186] 61 and 62 are diagrams showing examples of processes (information processing method, sensing method) executed in the information processing system 100. Descriptions of content that overlap with those described above will be omitted where appropriate.

[0187] 61, the process executed up to the generation of the working NW 60 is shown as step S1. The process of step S1 is executed by the information processing device 10. Several related processes are shown as steps S11 to S14.

[0188] In step S11, original data 110 is acquired. This process is executed by the acquisition unit 11 of the information processing device 10. In step S12, learning data 120 is generated. This process is executed by the data generation unit 12 of the information processing device 10. In step S13, the learning target NW 130 is subjected to machine learning. This process is executed by the learning unit 13 of the information processing device 10. In step S14, an execution NW 60 is generated based on the learning target NW 130. This process is executed by the NW generation unit 14 of the information processing device 10.

[0189] For example, the execution NW 60 generated as described above is installed in the sensor 1 or the sensor system 1A (stored in the storage unit 7) and used.

[0190] 62 shows steps S2 and S3 of the process executed when using the execution NW 60. The process of steps S2 and S3 is executed in the sensor 1 or the sensor 1A.

[0191] In step S2, image data 5 is obtained. Some related processes are shown as steps S21 and S22. In step S21, the pixel array section 2 is driven so that the pixels 3 are exposed in a desired exposure period pattern P. That is, the drive section 4 of the sensor 1 drives the pixel array section 2, which includes a plurality of pixels 3, each of which outputs a pixel signal corresponding to the amount of charge accumulated during an exposure period, so that image data 5 for one frame of a scene 8 including an object 9 is obtained. This driving includes driving the pixel array section 2 so that the pixels 3 are exposed in an exposure period pattern P including two or more exposure periods during one frame period. Further details are as described in the previous embodiment of the sensor 1. In step S22, image data 5 is obtained based on the pixel signals from each pixel 3.

[0192] In step S3, the image data 5 is processed using the execution NW 60. The processing unit 6 of the sensor 1 or the sensor system 1A processes the image data 5 obtained in the previous step S2 using the execution NW 60 generated in the previous step S1.

[0193] The inference phase (FIG. 61) and the learning phase (FIG. 62) may be phases that are performed at different times, places, etc. However, this does not prevent inference and learning from being performed in the same phase. For example, the execution NW 60 generated through the learning phase may be further subjected to machine learning while it is installed in and used by the sensor 1 or the sensor system 1A (during the inference phase). The learning data 120 used in the machine learning may be data obtained according to the usage environment of the sensor 1 or the sensor system 1A, thereby enabling customization (updating) of the execution NW 60 according to the user environment.

[0194] 4. Modified Examples of Network Several modified examples of the network will be described with reference to FIGS.

[0195] 63 to 65 are diagrams showing modified examples of the NW, and the explanation will be given here by taking the execution NW 60 as an example.

[0196] 63, not only the image data 5 but also auxiliary information is input to the execution NW 60. Examples of the input auxiliary information include information about the exposure period pattern P (for example, the total exposure period), information about the code C generated or switched by the code generation unit 41, and the like.

[0197] 64, the execution NW 60 outputs auxiliary information. Examples of the output auxiliary information include information that specifies an exposure period pattern P and information that specifies generation and switching of a code C by the code generation unit 41.

[0198] In the example shown in Figure 65, a recurrent connection may be used. Time series information is maintained across frames. Two consecutive frames are referred to as frame n and frame n+1. As shown in the figure, the internal state of the execution NW 60 in frame n is carried and input to the execution NW 60 in the next frame n+1.

[0199] Naturally, the same modifications as those for the execution NW 60 described above may also be applied to the learning target NW 130 .

[0200] 66 is a diagram showing an example of a hardware configuration of a device. For example, part of the sensor 1 described above, part of the sensor system 1A, and at least part of the information processing device 10 can be realized by a computer 1000 shown in FIG.

[0201] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected to each other via a bus 1050.

[0202] The CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs. An example of a program is the information processing program 150 described above with reference to FIG. 46. Another example of a program is a program for causing the computer 1000 to function as the processing unit 6 of the sensor 1 or the sensor system 1A.

[0203] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0204] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a program for the information processing method according to the present disclosure, which is an example of program data 1450.

[0205] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0206] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined computer-readable recording medium. Examples of the medium include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disc), tape media, magnetic recording media, and semiconductor memories.

[0207] When the computer 1000 functions as the information processing device 10 described above, the CPU 1100 of the computer 1000 can realize functions such as the acquisition unit 11, data generation unit 12, learning unit 13, and NW generation unit 14 by executing a program loaded on the RAM 1200. The program may be stored in the HDD 1400. When the computer 1000 functions as part of the sensor 1 or sensor system 1A described above, the CPU 1100 of the computer 1000 can realize functions of the processing unit 6 by executing a program loaded on the RAM 1200. The program (which may also be referred to as a sensing program, etc.) may be stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may acquire the program from another device via an external network 1550.

[0208] 6. Conclusion The techniques described above can be specified, for example, as follows. One of the techniques disclosed is a sensor 1. As described with reference to FIGS. 1 to 8 , the sensor 1 includes a pixel array section 2 including a plurality of pixels 3, each of which outputs a pixel signal according to the amount of charge accumulated during an exposure period, and a driver 4 that drives the pixel array section 2 so as to obtain one frame of image data 5 of a scene 8 including a subject 9. The driver 4 drives the pixel array section 2 so that the pixels 3 are exposed in an exposure period pattern P that includes a plurality of exposure periods during one frame period.

[0209] According to the sensor 1, the driver 4 drives the pixel array unit 2 so that the pixels 3 are exposed to an exposure period pattern P including a plurality of exposure periods during one frame period, thereby obtaining image data 5 that has been subjected to time calculation (for example, convolved in the time direction). By performing time calculation in this manner, it is possible to reduce the calculation cost required for the calculation. It is also possible to reduce power consumption.

[0210] 1 and 8, the image data 5 is input to the execution network 60, which performs spatial operations. For example, the execution network 60 includes an operation layer that performs spatial operations, and the operation layer may include at least one of a convolution layer (e.g., a spatial convolution layer 62), a fully connected layer, and a self-attention layer. Processing of the image data 5 that includes both temporal and spatial operations can be realized.

[0211] As described with reference to FIGS. 1 and 4 to 7 , the pixels 3 of the pixel array unit 2 may be grouped into a plurality of pixel groups, and the drive unit 4 may drive the pixel array unit 2 so that one image data 5 is obtained from one pixel group. The drive unit 4 may drive the pixel array unit 2 so that the pixels 3 of the same pixel group are exposed with the same exposure period pattern P. The drive unit 4 may drive the pixel array unit 2 so that the pixels 3 of different pixel groups are exposed with different exposure period patterns P. In this way, multiple image data 5 obtained by exposing the pixels 3 with different exposure period patterns P can be obtained.

[0212] 1 and 37 to 39 , the driver 4 drives the pixel array unit 2 so that the exposure period pattern P is switched in response to a trigger (condition), and the trigger may include at least one of a change in the speed of at least one of the scene 8 and the subject 9, a change in the brightness of at least one of the scene 8 and the subject 9, a change in the state of the pixel 3, and a change in mode. By switching the exposure period pattern P in response to various triggers, appropriate image data 5 can be obtained.

[0213] 1, 24, 25, 37, etc., the trigger may include a change in the speed of at least one of the scene 8 and the subject 9, and the drive unit 4 may drive the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P having a density that corresponds to the change in speed. This makes it possible to obtain image data 5 that makes it easy to recognize the scene 8, etc., even if the scene 8 is moving or in motion.

[0214] 1, 22, 23, 37, etc., the trigger may include a change in brightness of at least one of the scene 8 and the subject 9, and the drive unit 4 may drive the pixel array unit 2 so that the pixels 3 are exposed with an exposure period pattern P having a total exposure period according to the change in brightness. This makes it possible to obtain image data 5 that makes it easy to recognize the scene 8, etc., even if the scene 8, etc., is bright or dark.

[0215] 1, 37, 38, etc., the trigger includes a change in the state of the pixel 3, and the change in the state of the pixel 3 includes a change in the amount of charge accumulated in the photoelectric conversion unit of the pixel 3, and the drive unit 4 may switch the exposure period pattern P midway through one frame period T so that the photoelectric conversion unit of the pixel 3 does not become saturated during one frame period T. This makes it possible to prevent the pixel 3 from becoming saturated and obtain appropriate image data 5.

[0216] 1 , 37 , 39 , etc., the trigger includes a change in mode, and the mode may include a sensing mode in which the drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P including a plurality of exposure periods during one frame period T, and a viewing mode in which the drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P including only one exposure period during one frame period T. This makes it possible to obtain image data 5 convolved in the time direction in the sensing mode, and to obtain image data 5 suitable for display to a user in the viewing mode.

[0217] As described with reference to Figures 1, 3, 6, 7, 9 to 19, etc., the driver 4 may include a code generator 41 that generates a code C that defines the exposure period pattern P, and the driver 4 may drive the pixel array unit 2 in accordance with the code generated by the code generator 41. The code C may be a code that binarizes the kernel weight (kernel weight W) of the time calculation. This makes it possible to obtain image data 5 for which the desired time calculation has been performed.

[0218] 1 and 8, the sensor 1 may include a processing unit 6 that processes the image data 5 using an execution NW 60 that performs spatial operations. This makes it possible to process the image data 5, including both temporal and spatial operations, on the sensor 1.

[0219] 1, 40, 41, etc., the execution NW 60 may be a plurality of execution NWs, and the processing unit 6 may switch between the execution NWs to be used (e.g., the execution NW 60-1 to the execution NW 60-3) depending on the exposure period pattern used to obtain the image data 5 to be processed (e.g., the image data 5-1 to the image data 5-3). This allows the image data 5 obtained by exposing the pixels 3 with different exposure period patterns P to be appropriately processed.

[0220] As described with reference to FIGS. 1, 43, and 44, the processing unit 6 processes image data 5 (e.g., image data 5-1) obtained by exposing the pixel 3 in a specific exposure period pattern P (e.g., exposure period pattern P-1) using the execution NW 60, and image data (e.g., image data 5-2, image data 5-3) obtained by exposing the pixel 3 in an exposure period pattern other than the specific exposure period pattern P (e.g., exposure period pattern P-2, exposure period pattern P-3)) ​​using a conversion network 64 (e.g., conversion network 64-2, conversion network 64-3) that absorbs the differences in the exposure period pattern P and the execution NW 60. Even in this way, each image data 5 obtained by exposing the pixel 3 in a different exposure period pattern P can be appropriately processed. Furthermore, there is no need to prepare multiple execution NWs 60.

[0221] The sensor system 1A described with reference to Figures 1 to 8 and 45 is also one of the disclosed technologies. The sensor system 1A includes a sensor 1 that outputs image data 5 and a processing unit 6 that processes the image data 5 from the sensor 1. The sensor 1 includes a pixel array unit 2 including a plurality of pixels 3, each of which outputs a pixel signal corresponding to the amount of charge accumulated during an exposure period, a driving unit 4 that drives the pixel array unit 2 to obtain one frame of image data 5 of a scene 8 including a subject 9, and a processing unit 6 that processes the image data using an execution network 60. The driving unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed according to an exposure period pattern P including multiple exposure periods during one frame period T. The processing by the processing unit 6 includes inputting the image data 5 from the sensor 1 to the execution network 60, which performs spatial calculations. As described above, this sensor system 1A can also reduce the calculation cost and power consumption required for temporal calculations.

[0222] The sensing method described with reference to Figures 1 to 8 and 62 is also one of the disclosed techniques. The sensing method includes driving a pixel array unit 2 including a plurality of pixels 3, each of which outputs a pixel signal corresponding to the amount of charge accumulated during an exposure period, so as to obtain one frame's worth of image data 5 of a scene 8 including a subject 9 (steps S2 and S21). Driving includes driving the pixel array unit so that the pixels 3 are exposed in an exposure period pattern P including two or more exposure periods during one frame period T. As described above, this sensing method also makes it possible to reduce the calculation cost required for time calculations and reduce power consumption.

[0223] 1 to 8 and 46 to 60, etc., the information processing system 100 (particularly the information processing device 10) is also one of the disclosed technologies. The information processing system 100 includes a data generation unit 12 that generates learning data for a learning target NW 130 that performs temporal and spatial operations, a learning unit 13 that performs machine learning on the learning target NW 130 using the learning data 120 generated by the data generation unit 12, and a NW generation unit 14 that generates, based on the learning target NW 130, an execution NW 60 that performs spatial operations and outputs output data when time-calculated image data 5 is input. For example, the learning target network includes a computation layer that performs temporal and spatial computations, and the computation layer includes at least one of a convolution layer (e.g., a temporal axis convolution layer 131 and a spatial convolution layer 132, or a 3D convolution layer 133), a fully connected layer, and a self-attention layer, while the execution network includes a computation layer that performs spatial computations, and the computation layer may include at least one of a convolution layer (e.g., a spatial convolution layer 62), a fully connected layer, and a self-attention layer. Such an information processing system 100 can generate an execution network 60 that can be used in, for example, the sensor 1 or sensor system 1A described above.

[0224] As described with reference to Figures 48 and 49, the learning data 120 includes input data 121 to be input to the learning target NW 130. The input data 121 includes time-series image data. The learning unit 13 may input the input data 121 to the learning target NW 130 so that the learning target NW 130 performs temporal and spatial operations on the input data 121. For example, the data generator 12 may generate the input data 121 using given CG material data (original data 110) and CG renderer technology (generation method 2). The data generator 12 may generate the input data 121 using given time-series image data (original data 110) and NeRF technology. The data generator 12 may generate the input data 121 using given time-series image data (original data 110 (low-frame time-series image data)) and frame interpolation technology. For example, the learning data 120 including such input data 121 can be used to perform machine learning on the learning target NW 130 .

[0225] As described with reference to Figures 50 and 51, the learning data 120 includes input data 121 to be input to the learning target NW 130, and the input data 121 includes image data that has been subjected to time calculation. The learning unit 13 may input the input data 121 to the learning target NW 130 so that the input data 121 is not subjected to time calculation by the learning target NW 130 but is subjected to spatial calculation (e.g., so that the input data 121 is not convolved by the time axis convolution layer 131 but is convolved by the spatial convolution layer 132). For example, the data generation unit 12 may generate the input data 121 using CG material data (original data 110) and a differentiable CG renderer technique. The data generation unit 12 generates the input data 121 using given time-series image data (original data 110) and an NeRF technique edited to obtain image data obtained by convolving the time-series image data in the time direction. The time calculation in the edited NeRF technique may be performed using a differentiable calculation. For example, machine learning can be performed on the learning target NW 130 using the learning data 120 including such input data 121. Since the input data 121 is generated by the data generator 12 in a differentiable manner, machine learning using the backpropagation method is possible.

[0226] 48, 50, 52, 53, etc., the learning data 120 includes input data 121 to be input to the learning target NW 130 and supervised data 122 that pairs with the input data 121, and the learning unit 13 performs machine learning on the learning target NW 130 so that the loss value of the output data output by the learning target NW 130 to which the input data 121 has been input is reduced relative to the supervised data 122, and the machine learning by the learning unit 13 includes regularization (filter regularization) of the weight of the kernel of the time operation of the learning target NW 130, and the regularization may include calculation of a loss value based on the weight of the kernel of the time operation and a cost weighting function for one frame period T. For example, the data can be used for machine learning to process image data 5 obtained by an exposure period pattern P that places a weight on the first half or the second half of one frame period T.

[0227] 48, 50, 52, 54, 55, etc., the learning data 120 includes input data 121 to be input to the learning target NW 130, the data generator 12 extends the generated input data 121 to generate new input data 121a, and the extension by the data generator 12 may include at least one of shifting the operation period of the subject 9, etc. in the generated input data 121 and changing the kernel pattern of the time calculation in the generated input data 121. For example, such padding of the input data allows the learning target NW 130 to undergo machine learning using a larger amount of input data.

[0228] As described with reference to Figure 56 etc., the training target NW 130 includes a time axis calculation layer (e.g., time axis convolution layer 131) that performs time calculations and a spatial calculation layer (e.g., spatial convolution layer 132) that performs spatial calculations, and the NW generation unit 14 may generate the execution NW 60 by reflecting the spatial calculation function in the spatial calculation layer of the training target NW 130 in the spatial calculation of the execution NW 60. As described with reference to Figures 59 and 60 etc., the training target NW 130 includes a 3D calculation layer (e.g., convolution layer 133) that performs time calculations and spatial calculations, and the NW generation unit 14 may generate the execution NW 60 by reflecting the spatial calculation function in the 3D calculation layer of the training target NW 130 in the spatial calculation of the execution NW 60. For example, in this manner, the execution NW 60 can be generated based on the training target NW 130.

[0229] The information processing method described with reference to Figures 1 to 8 and Figures 46 to 61 is also one of the disclosed technologies. The information processing method includes generating learning data 120 for a learning target network that performs temporal and spatial operations (steps S1 and S12), using the generated learning data 120 to perform machine learning on a learning target network 130 (steps S1 and S13), and generating an execution network 60 based on the learning target network 130 that performs spatial operations and outputs output data when time-calculated image data 5 is input (steps S1 and S14). This information processing method can also generate an execution network 60 that can be used, for example, in the sensor 1 described above.

[0230] The information processing program 150 described with reference to Figures 1 to 8, 46 to 61, and 66 is also one of the disclosed technologies. The information processing program 150 causes the computer 1000 to execute the following processes: generating learning data 120 for a learning target NW 130 that performs temporal and spatial calculations; performing machine learning on the learning target NW 130 using the generated learning data 120; and generating an execution NW 60, based on the learning target NW 130, that performs spatial calculations and outputs output data when time-calculated image data 5 is input. Such an information processing program 150 can also generate an execution NW 60 that can be used in, for example, the sensor 1 or sensor system 1A described above.

[0231] 1 to 8 and 46 to 60, the information processing system 100 includes a data generation unit 12 that generates learning data 120 for a learning target network 130 that performs temporal and spatial operations, a learning unit 13 that performs machine learning on the learning target network 130 using the learning data 120 generated by the data generation unit 12, a network generation unit 14 that generates an execution network 60 that performs spatial operations based on the learning target network 130 and outputs output data when image data 5 convolved in the time direction is input, a pixel array unit 2 that includes a plurality of pixels 3 that each output a pixel signal corresponding to the amount of charge accumulated during an exposure period, a driver unit 4 that drives the pixel array unit 2 to obtain image data 5 for one frame of a scene 8 that includes a subject 9, and a processing unit 6 that processes the image data 5 using the execution network 60. The drive unit 4 drives the pixel array unit 2 so that the pixels 3 are exposed in an exposure period pattern P including a plurality of exposure periods during one frame period T. As described above, such an information processing system 100 can reduce the calculation cost required for time calculations and reduce power consumption.

[0232] The effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.

[0233] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0234] The present technology can also be configured as follows. (1) A sensor including: a pixel array unit including a plurality of pixels, each outputting a pixel signal according to an amount of charge accumulated during an exposure period; and a driver unit that drives the pixel array unit to obtain one frame of image data of a scene including a subject, wherein the driver unit drives the pixel array unit so that the pixels are exposed in an exposure period pattern including a plurality of exposure periods during one frame period. (2) The sensor according to (1), wherein the image data is input to an executive network (neural network) that performs spatial calculations. (3) The sensor according to (2), wherein the executive network includes an arithmetic layer that performs spatial calculations, and the arithmetic layer of the executive network includes at least one of a convolutional layer, a fully connected layer, and a self-attention layer. (4) The sensor according to any of (1) to (3), wherein the plurality of pixels of the pixel array unit are grouped into a plurality of pixel groups, and the driver unit drives the pixel array unit to obtain one piece of image data from one pixel group. (5) The sensor according to (4), wherein the driver drives the pixel array unit so that pixels of the same pixel group are exposed with the same exposure period pattern. (6) The sensor according to (4) or (5), wherein the driver drives the pixel array unit so that pixels of different pixel groups are exposed with different exposure period patterns. (7) The sensor according to any of (1) to (6), wherein the driver drives the pixel array unit so that the exposure period pattern is switched in response to a trigger, the trigger including at least one of: a change in speed of at least one of the scene and the subject; a change in brightness of at least one of the scene and the subject; a change in the state of the pixel; and a change in mode. (8) The sensor according to (7), wherein the trigger includes a change in speed of at least one of the scene and the subject; and the driver drives the pixel array unit so that the pixels are exposed with an exposure period pattern having a sparse / dense pattern according to the change in speed.(9) The sensor according to (7) or (8), wherein the trigger includes a change in brightness of at least one of the scene and the subject, and the drive unit drives the pixel array unit so that the pixel is exposed with an exposure period pattern having a total exposure period according to the change in brightness. (10) The sensor according to any of (7) to (9), wherein the trigger includes a change in a state of the pixel, and the change in the state of the pixel includes a change in an amount of charge accumulated in a photoelectric conversion unit of the pixel, and the drive unit switches the exposure period pattern midway through the one frame period so that the photoelectric conversion unit of the pixel is not saturated during the one frame period. (11) The sensor according to any one of (7) to (10), wherein the trigger includes a change in the mode, and the modes include: a sensing mode in which the driver drives the pixel array unit so that the pixels are exposed in an exposure period pattern including a plurality of exposure periods during one frame period; and a viewing mode in which the driver drives the pixel array unit so that the pixels are exposed in an exposure period pattern including only one exposure period during one frame period. (12) The sensor according to any one of (1) to (11), wherein the driver includes a code generator that generates a code that defines the exposure period pattern, and the driver drives the pixel array unit in accordance with the code generated by the code generator. (13) The sensor according to (12), wherein the code is a code obtained by binarizing a kernel weight of a temporal operation. (14) The sensor according to any one of (1) to (13), further comprising a processing unit that processes the image data using an execution network that performs spatial operation. (15) The sensor according to (14), wherein the execution network is a plurality of execution networks, and the processing unit switches the execution network to be used depending on an exposure period pattern used to obtain the image data to be processed.(16) The sensor according to (14), wherein the processing unit processes image data obtained by exposing the pixels in a specific exposure period pattern using the execution NW, and processes image data obtained by exposing the pixels in an exposure period pattern other than the specific exposure period pattern using a conversion network that absorbs differences in exposure period patterns and the execution NW. (17) An information processing system comprising: an information processing device that generates an execution NW that includes a spatial convolution layer that performs convolution in a spatial direction and that outputs output data when image data convolved in a time direction is input; and a sensor, wherein the sensor includes: a pixel array unit including a plurality of pixels that each output a pixel signal according to an amount of charge accumulated during an exposure period, a drive unit that drives the pixel array unit to obtain image data for one frame of a scene including a subject, and a processing unit that processes the image data using the execution NW, and the drive unit drives the pixel array unit so that the pixels are exposed with an exposure period pattern including a plurality of exposure periods during one frame period. (18) A sensing method comprising: driving a pixel array unit including a plurality of pixels, each outputting a pixel signal according to an amount of charge accumulated during an exposure period, so as to obtain image data for one frame of a scene including a subject, wherein the driving comprises driving the pixel array unit so that the pixels are exposed in an exposure period pattern including two or more exposure periods during one frame period. (101) An information processing system comprising: a data generation unit that generates training data for a training object NW (neural network) that performs temporal calculation and spatial calculation, a learning unit that performs machine learning on the training object NW using the training data generated by the data generation unit, and a network generation unit that generates an execution NW that performs spatial calculation based on the training object NW and outputs output data when time-calculated image data is input.(102) The information processing system according to (101), wherein the learning target network includes a calculation layer that performs temporal and spatial calculations, the calculation layer of the learning target network includes at least one of a convolutional layer, a fully connected layer, and a self-attention layer, and the execution network includes a calculation layer that performs spatial calculations, the calculation layer of the execution network includes at least one of a convolutional layer, a fully connected layer, and a self-attention layer. (103) The information processing system according to (101) or (102), wherein the learning data includes input data to be input to the learning target network, the input data includes time-series image data, and the learning unit inputs the input data to the learning target network so that the input data is subjected to temporal and spatial calculations by the learning target network. (104) The information processing system according to (103), wherein the data generation unit generates the input data using given CG material data and a CG renderer technology. (105) The information processing system according to (103) or (104), wherein the data generation unit generates the input data using given time-series image data and Neural Radiance Fields (NeRF) technology. (106) The information processing system according to any of (103) to (105), wherein the data generation unit generates the input data using given time-series image data and frame interpolation technology. (107) The information processing system according to (101) or (102), wherein the learning data includes input data to be input to the learning target network, the input data includes image data that has been time-calculated, and the learning unit inputs the input data to the learning target network such that the input data is not time-calculated by the learning target network but is spatially calculated. (108) The information processing system according to (107), wherein the data generation unit generates the input data using given CG material data and differentiable CG renderer technology.(109) The information processing system according to (107) or (108), wherein the data generation unit generates the input data using given time-series image data and NeRF technology edited to obtain image data obtained by time-operating the time-series image data, and the time operation in the edited NeRF technology is performed using a differentiable operation. (110) The information processing system according to any of (101) to (109), wherein the learning data includes input data to be input to the learning target NW and supervised data paired with the input data, and the learning unit trains the learning target NW by machine learning so that a loss value of output data output by the learning target NW to which the input data has been input is reduced compared to the supervised data, and the machine learning by the learning unit includes regularization of a kernel weight of the time operation of the learning target NW, and the regularization includes calculation of a loss value based on the kernel weight of the time operation and a cost weighting function for one frame period. (111) The information processing system according to any of (101) to (110), wherein the learning data includes input data to be input to the learning target network, and a data generation unit generates new input data by expanding the generated input data, and the expansion by the data generation unit includes at least one of: shifting a motion period of a subject in the generated input data, and changing a kernel pattern of a time operation in the generated input data. (112) The information processing system according to any of (101) to (111), wherein the learning target network includes a time axis operation layer that performs time operation and a spatial operation layer that performs spatial operation, and the network generation unit generates the execution network by reflecting a function of spatial operation in the spatial operation layer of the learning target network in the spatial operation of the execution network. (113) The information processing system according to any one of (101) to (111), wherein the learning target network includes a 3D calculation layer that performs temporal calculations and spatial calculations, and the network generation unit generates the execution network by reflecting the temporal calculation function in the 3D calculation layer of the learning target network in the spatial calculations of the execution network.(114) An information processing method comprising: generating learning data for a learning target NW (neural network) that performs temporal calculations and spatial calculations; subjecting the learning target NW to machine learning using the generated learning data; and generating, based on the learning target NW, an execution NW that performs spatial calculations and outputs output data when time-calculated image data is input. (115) An information processing program that causes a computer to execute the following processes: generating learning data for a learning target NW (neural network) that performs temporal calculations and spatial calculations; subjecting the learning target NW to machine learning using the generated learning data; and generating, based on the learning target NW, an execution NW that performs spatial calculations and outputs output data when time-calculated image data is input. (116) An information processing system comprising: a data generation unit that generates learning data for a learning target network (neural network) that performs temporal and spatial operations; a learning unit that uses the learning data generated by the data generation unit to perform machine learning on the learning target network; a network generation unit that generates an execution network that performs spatial operations based on the learning target network and outputs output data when time-operated image data is input; a pixel array unit that includes a plurality of pixels, each of which outputs a pixel signal according to the amount of charge accumulated during an exposure period; a drive unit that drives the pixel array unit so as to obtain image data for one frame of a scene including a subject; and a processing unit that processes the image data using the execution network, wherein the drive unit drives the pixel array unit so that the pixels are exposed in an exposure period pattern that includes a plurality of exposure periods during one frame period.

[0235] REFERENCE SIGNS LIST 1 Sensor 1A Sensor system 2 Pixel array unit 3 Pixel 4 Driving unit 40 Base filter 41 Code generation unit 5 Image data 6 Processing unit 60 Execution network 62 Spatial convolution layer 64 Conversion network 7 Memory unit 8 Scene 9 Object 10 Information processing device 11 Acquisition unit 110 Original data 12 Data generation unit 120 Learning data 121 Input data 121a Input data 122 Correct answer data 13 Learning unit 130 Learning target network 131 Time axis convolution layer 132 Spatial convolution layer 133 3D convolution layer 14 Network generation unit 15 Memory unit 150 Information processing program 100 Information processing system 1000 Computer 1050 Bus 1100 CPU 1200 RAM 1300 ROM 1400 HDD 1450 Program data 1500 Communication interface 1600 Input / output interface 1650 Input / output device

Claims

1. An information processing system comprising: a data generation unit that generates learning data for a learning target network (neural network) that performs temporal and spatial calculations; a learning unit that trains the learning target network by machine learning using the learning data generated by the data generation unit; and a network generation unit that generates an execution network based on the learning target network, which is a network that performs spatial calculations and outputs output data when time-calculated image data is input.

2. The information processing system of claim 1, wherein the learning target network includes an operation layer that performs temporal and spatial operations, the operation layer of the learning target network includes at least one of a convolutional layer, a fully connected layer, and a self-attention layer, the execution network includes an operation layer that performs spatial operations, and the operation layer of the execution network includes at least one of a convolutional layer, a fully connected layer, and a self-attention layer.

3. The information processing system of claim 1, wherein the learning data includes input data to be input to the learning target network, the input data includes time-series image data, and the learning unit inputs the input data to the learning target network so that the input data is subjected to temporal and spatial operations by the learning target network.

4. The information processing system according to claim 3, wherein the data generation section generates the input data using given CG material data and a CG rendering technique.

5. The information processing system according to claim 3, wherein the data generation unit generates the input data using given time-series image data and Neural Radiance Fields (NeRF) technology.

6. The information processing system according to claim 3, wherein the data generation unit generates the input data using given time-series image data and a frame interpolation technique.

7. The information processing system of claim 1, wherein the learning data includes input data to be input to the learning target network, the input data includes image data that has been time-operated, and the learning unit inputs the input data to the learning target network so that the input data is not time-operated by the learning target network but is spatially operated.

8. The information processing system according to claim 7, wherein the data generation section generates the input data by using given CG material data and a differentiable CG rendering technique.

9. The information processing system of claim 7, wherein the data generation unit generates the input data using given time-series image data and NeRF technology edited to obtain image data obtained by performing a time operation on the time-series image data, and the time operation in the edited NeRF technology is performed using a differentiable operation.

10. The information processing system of claim 1, wherein the learning data includes input data to be input to the learning target network and supervised data that pairs with the input data, the learning unit trains the learning target network by machine learning so that a loss value of output data output by the learning target network to which the input data has been input relative to the supervised data is reduced, the machine learning by the learning unit includes regularization of kernel weights of the time operation of the learning target network, and the regularization includes calculation of a loss value based on kernel weights of the time operation and a cost weighting function for one frame period.

11. The information processing system according to claim 1, wherein the learning data includes input data to be input to the learning target network, and a data generating unit extends the generated input data to generate new input data, and the extension by the data generating unit includes at least one of: shifting a motion period of a subject in the generated input data; and changing a kernel pattern of a time operation in the generated input data.

12. The information processing system of claim 1, wherein the learning target network includes a time axis calculation layer that performs time calculations and a spatial calculation layer that performs spatial calculations, and the network generation unit generates the execution network by reflecting the spatial calculation functions in the spatial calculation layer of the learning target network in the spatial calculations of the execution network.

13. The information processing system of claim 1, wherein the learning target network includes a 3D calculation layer that performs temporal and spatial calculations, and the network generation unit generates the execution network by reflecting the spatial calculation function in the 3D calculation layer of the learning target network in the spatial calculation of the execution network.

14. An information processing method comprising: generating learning data for a learning target network (neural network) that performs temporal and spatial operations; subjecting the learning target network to machine learning using the generated learning data; and generating, based on the learning target network, an execution network that performs spatial operations and outputs output data when time-calculated image data is input.

15. An information processing program that causes a computer to execute the following processes: generating learning data for a learning target network (neural network) that performs time and spatial calculations; training the learning target network through machine learning using the generated learning data; and generating an execution network based on the learning target network, which is a network that performs spatial calculations and outputs output data when time-calculated image data is input.

16. An information processing system comprising: a data generation unit that generates learning data for a learning target network (neural network) that performs temporal and spatial calculations; a learning unit that trains the learning target network by machine learning using the learning data generated by the data generation unit; a network generation unit that generates an execution network that performs spatial calculations based on the learning target network and outputs output data when time-calculated image data is input; a pixel array unit including a plurality of pixels that each output a pixel signal according to the amount of charge accumulated during an exposure period; a drive unit that drives the pixel array unit so as to obtain one frame's worth of image data of a scene including a subject; and a processing unit that processes the image data using the execution network, wherein the drive unit drives the pixel array unit so that the pixels are exposed in an exposure period pattern including a plurality of exposure periods during one frame period.

Images