Non-temporary computer-readable storage medium, image processing method, and image processing sysytem

Abstract

A non-temporary computer-readable storage medium according to an embodiment of the present invention, when executed by an information processing system, causes the information processing system to execute processing. The processing includes: receiving a coded image of an object captured by a lensless image sensor; and extracting information from the coded image by using a trained machine learning model.

Description

Non-transitory computer-readable storage medium, image processing method, and image processing system

[0001] The present disclosure relates to a non-transitory computer-readable storage medium, an image processing method, and an image processing system.

[0002] There is a lensless image sensor provided with an optical modulation element on which a known two-dimensional pattern including an amplitude modulation region or a phase modulation region of light is formed in front of the imaging element in the optical axis direction (see, for example, Patent Document 1).

[0003] The lensless image sensor projects light encoded by the effect of the characteristic two-dimensional pattern of the optical modulation element onto the imaging element, and generates a decoded image of the subject by decoding the effect of the mask through inverse calculations.

[0004] Meanwhile, there is a technology that uses AI (artificial intelligence) to perform image recognition and detection to extract desired information from images captured by a general image sensor, which captures an image by receiving light incident through a lens with an imaging element.

[0005] International Publication No. 2019 / 176349

[0006] However, the image information itself differs between images captured by a lensless image sensor and images captured by a general image sensor, so image recognition AI that extracts information from images captured by a general image sensor may not be able to extract the desired information from images captured by a lensless image sensor.

[0007] Therefore, the present disclosure proposes a non-transitory computer-readable storage medium, an image processing method, and an image processing system that are capable of extracting desired information from an image captured by a lensless image sensor.

[0008] A non-transitory computer-readable storage medium according to the present disclosure, when executed by an information processing system, causes the information processing system to perform a process that receives encoded images of an object captured by a lensless image sensor and extracts information from the encoded images using a trained machine learning model.

[0009] 19 is an explanatory diagram of the configuration and operation of a general lensless image sensor. FIG. 19 is a functional block diagram showing an example configuration of an information processing system according to an embodiment. FIG. 20 is a block diagram showing an example configuration of an information extraction unit according to an embodiment. FIG. 21 is a block diagram showing another example configuration of the information extraction unit according to an embodiment. FIG. 22 is a block diagram showing another example configuration of the information extraction unit according to an embodiment. FIG. 23 is a flowchart showing an example of processing performed by the information extraction unit shown in FIG. 5. FIG. 21 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. FIG. 22 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. FIG. 23 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. FIG. 24 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. FIG. 25 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. FIG. 26 is an explanatory diagram showing an example hardware configuration of an information processing system according to an embodiment. Fig. 22 is a functional block diagram showing another configuration example of the learning data generation device according to the embodiment. Fig. 23 is a flowchart showing an example of processing executed by the pairing processing unit shown in Fig. 21. Fig. 24 is an explanatory diagram of a learning device according to the embodiment.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] 1. Overview of Lensless Image Sensor First, an overview of a general lensless image sensor will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram of the configuration and operation of a general lensless image sensor.

[0012] The lensless image sensor shown in FIG. 1 (hereinafter referred to as "lensless sensor 1") is a so-called lensless camera that does not require a lens to focus light emitted from each point on a subject onto each corresponding point on the sensor.

[0013] 1, the lensless sensor 1 includes an imaging device 10 and a signal processing device 20. The imaging device 10 includes an optical modulation unit 11 and a pixel unit 12. The optical modulation unit 11 is provided before the pixel unit 12 in the optical axis direction. The optical modulation unit 11 includes a mask on which a known two-dimensional pattern including an amplitude modulation region or a phase modulation region of light is formed.

[0014] Amplitude modulation of light is achieved by using light-blocking materials or materials with different transmittances. Phase modulation of light is achieved by controlling the uneven structure within the mask surface, the refractive index distribution of the mask, and controlling the effective refractive index obtained by controlling the density of materials with multiple refractive indices on the order of a size less than the wavelength. Here, an example of amplitude modulation will be explained.

[0015] When imaging a subject T, the optical modulation unit 11 modulates light L incident from the direction of the subject using a known two-dimensional phase amplitude pattern, and guides the light to the downstream pixel unit 12 located a certain distance away. As a result, a light intensity distribution corresponding to the shape of the subject T and the shape of the two-dimensional pattern of the optical modulation unit 11 is projected onto the light receiving surface of the pixel unit 12.

[0016] The pixel unit 12 has a plurality of light receiving elements PD arranged in a two-dimensional matrix on its light receiving surface. The pixel unit 12 photoelectrically converts light L received by each light receiving element PD, thereby generating an encoded image DA in which an image of the subject T is encoded by the two-dimensional pattern of the optical modulation unit 11, and outputs the encoded image DA to the signal processing device 20. Because the encoded image DA is an image in which the light L incident via the optical modulation unit 11 is photoelectrically converted, the subject T is not recognizable to the naked eye.

[0017] The principle of capturing an encoded image DA by the pixel unit 12 is expressed by the following formula (1): Y=F×X+N (1) where, Y: light-receiving signal of each light-receiving element PD (one-dimensional data) X: scene vector (value (one-dimensional data) of light L incident on the optical modulation unit 11 during image capture) N: noise F: imaging matrix (matrix defined by the two-dimensional pattern of the optical modulation unit 11 and the distance between the mask and the image capture element) As can be seen from formula (1), the light-receiving signal Y of each light-receiving element PD is a signal obtained by adding noise N to light obtained by modulating the scene vector X during image capture by the imaging matrix F.

[0018] The signal processing device 20 performs a predetermined decoding process on the coded image DA to generate a two-dimensional image (hereinafter referred to as a "decoded image DB") that allows the subject T to be recognized in the same way as with a general camera. For example, the signal processing device 20 applies the inverse matrix F of the imaging matrix F to the light receiving signals Y of the light receiving elements PD. －１ and restore the scene vector X, thereby generating and outputting a decoded image DB.

[0019] In this way, the encoded image DA captured by the lensless sensor 1 is an image in which the subject T cannot be recognized with the naked eye, and the image information itself is different from that of an image captured by a normal camera that receives light from the subject that enters through a lens.

[0020] Furthermore, although the decoded image DB captured by the lensless sensor 1 is a two-dimensional image in which the subject T can be recognized, since it is an image generated from the encoded image DA, strictly speaking, the image information itself is different from that of an image captured by a normal camera.

[0021] Specific differences in image information include, for example, erroneous signals that appear in the decoded image DB when the decoding process acts on noise N contained in the received light signal Y, and blurring of the subject image due to the effects of regularization, etc., incorporated into the decoding process.

[0022] For this reason, general image recognition AI (Artificial Intelligence) may not be able to extract information desired by the user, such as position, contour, type, attributes such as gender and age, subject ID, or their reliability scores, from the image information of the encoded image DA or decoded image DB acquired by the lensless sensor 1.

[0023] Therefore, the information processing system according to the embodiment has a configuration that can acquire desired information from the image information acquired by the lensless sensor 1 .

[0024] 2. Configuration of Information Processing System According to Embodiment] Fig. 2 is a functional block diagram showing an example configuration of an information processing system 100 according to an embodiment. As shown in Fig. 2, the information processing system 100 includes an optical modulation unit 11, a pixel unit 12, and an information extraction unit 30.

[0025] The optical modulation unit 11 and pixel unit 12 are the same as the optical modulation unit 11 and pixel unit 12 of the general lensless sensor 1 shown in Fig. 1. The optical modulation unit 11 modulates light L incident from the direction of the subject using a known two-dimensional phase amplitude pattern, and guides the light to the pixel unit 12 located a certain distance downstream.

[0026] The pixel unit 12 photoelectrically converts the light L incident through the optical modulation unit 11, generates an encoded image DA in which the image of the subject T is encoded using the two-dimensional pattern of the optical modulation unit 11, and outputs it to the information extraction unit 30.

[0027] The pixel unit 12 is not limited to pixels of an image sensor equipped with a general RGB color filter. The pixel unit 12 may be pixels of a monochrome sensor that does not separate color information using a color filter, or pixels of an image sensor that acquires color information other than RGB. Furthermore, the pixel unit 12 may be pixels of a multispectral image sensor that acquires color information of two colors or four or more colors.

[0028] The pixel unit 12 may have a pixel structure that targets wavelength ranges other than the visible light range, such as ultraviolet (UV) and infrared (IR) light. The pixel unit 12 may be a pixel of an image sensor having a SPAD (Single Photon Avalanche Diode) structure. The pixel unit 12 may also be an event-based vision sensor that can detect luminance changes for each pixel.

[0029] The information extraction unit 30 includes a microcomputer having a central processing unit (CPU), read only memory (ROM), random access memory (RAM), etc., and various other circuits.

[0030] The information extraction unit 30 includes an AI that functions by the CPU executing a program stored in the ROM using the RAM as a work area. This AI is an information extraction AI dedicated to the lensless sensor that has been machine-learned to extract predetermined information from the image information of the encoded image DA acquired by the lensless sensor.

[0031] The information extraction unit 30 may be partially or entirely configured with hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] The information extraction AI includes a neural network such as a deep neural network (DNN), a convolutional neural network (CNN), a vision transformer, etc. The information extraction AI sequentially receives input of coded images DA (questions) from which predetermined information (answers) to be extracted are known.

[0033] The information extraction AI then extracts predetermined information from the encoded image DA, and updates the neural network coefficients to improve the degree of match (correct answer rate) between the extracted information and the answer information, thereby conducting machine learning. A specific example of a procedure for generating an image dataset that serves as a teaching material for such machine learning will be described later.

[0034] The information extraction unit 30 extracts information desired by the user (hereinafter referred to as "desired information") from the encoded image DA of the subject T input from the pixel unit 12, such as the position, contour, type, attributes such as gender and age of the subject T in the image, the subject ID, or their reliability scores.

[0035] Here, a description will be given of an example configuration of the information extraction unit 30. Fig. 3 is a block diagram showing an example configuration of the information extraction unit 30 according to the embodiment. As shown in Fig. 3, the information extraction unit 30 includes a subject detection unit 301, a subject type extraction unit 302, and a subject feature extraction unit 303.

[0036] The information extraction unit 30 also includes a subject detection unit 304, a subject type extraction unit 305, and a subject feature extraction unit 306. The information extraction unit 30 further includes a subject identity determination unit 307 and a subject ID assignment unit 308.

[0037] The object detection unit 301 receives an nth (n is a natural number) frame coded image DA as input. Meanwhile, the object detection unit 304 receives an n+1th frame coded image DA as input. The nth frame coded image DA and the n+1th frame coded image DA are coded images DA captured consecutively in time series.

[0038] The subject detection units 301 and 304 are processing units that detect the position and contour of a subject T from an encoded image DA input thereto. The subject detection unit 301 outputs information indicating the position and contour of the subject T detected from the encoded image DA, and the encoded image DA, to a subject type extraction unit 302, a subject feature extraction unit 303, and a subject identity determination unit 307, respectively, which are provided downstream.

[0039] In addition, the subject detection unit 304 outputs information indicating the position and contour of the subject T detected from the encoded image DA, and the encoded image DA, to the subsequent subject type extraction unit 305, subject feature extraction unit 306, and subject identity determination unit 307, respectively.

[0040] The subject type extraction units 302 and 305 are processing units that extract attributes of the subject T, such as the type, gender, and age of the subject T, from the encoded image DA input thereto. The subject type extraction units 302 and 305 output information indicating the extracted attributes of the subject T as desired information. Note that multiple subject type extraction units 302 and 305 may be provided for each piece of information to be extracted, such as the type, gender, and age of the subject T to be extracted.

[0041] The subject feature extraction units 303 and 306 are processing units that extract features (a vector of numerical values ​​indicating the features) of subject T from the input encoded image DA and the position and contour of subject T in the encoded image DA, and output them to the subject identity determination unit 307.

[0042] The subject identity determination unit 307 determines whether the subject T whose position and contour have been detected is the same subject T based on the encoded image DA input from the subject detection units 301 and 304, the position and contour of the subject T in the encoded image DA, and the features input from the subject feature extraction units 303 and 306.

[0043] That is, the subject identity determination unit 307 is a processing unit that determines whether or not the subject T detected from the nth frame of the coded image DA and the subject T detected from the n+1th frame of the coded image DA are the same subject T. The subject identity determination unit 307 outputs the identity determination result to the subject ID assignment unit 308.

[0044] The subject ID assignment unit 308 assigns a unique ID to the subject T in the nth frame of the encoded image DA and the subject T in the n+1th frame of the encoded image DA that have been determined to be identical by the subject identity determination unit 307, and outputs the ID as desired information.

[0045] In this way, the information processing system 100 is equipped with an information extraction AI that has been machine-learned specifically for image information captured by a lensless sensor, and therefore can extract accurate desired information from the image information acquired by the lensless sensor.

[0046] Furthermore, since the information extraction unit 30 extracts desired information from the metadata of the encoded image DA before image reconstruction, the decoded image DB in which subjects can be recognized with the naked eye is not leaked to the outside. Therefore, the information processing system 100 can protect privacy.

[0047] The information extraction unit 30 may also be configured as shown in Figures 4 and 5. Figures 4 and 5 are block diagrams showing other configuration examples of the information extraction unit 30 according to the embodiment. For example, as shown in Figure 4, the information extraction unit 30 may include image reconstruction units 311 and 312 provided in front of subject detection units 301 and 304, respectively.

[0048] The image reconstruction unit 311 receives an nth frame of coded image DA. The image reconstruction unit 312 receives an n+1th frame of coded image DA. The image reconstruction units 311 and 312 reconstruct a decoded image DB from the coded image DA received therein, and output the reconstructed decoded image DB to the subsequent object detection units 301 and 304, respectively. The subsequent processing is the same as the processing executed by the information extraction unit 30 shown in FIG. 3 , and therefore a redundant description will be omitted here.

[0049] 5, the information extraction unit 30 is provided with an image output determination unit 321 and an image reconstruction unit 322 following the subject type extraction unit 302, and with an image output determination unit 323 and an image reconstruction unit 324 following the subject type extraction unit 305. The rest of the configuration is the same as that of the information extraction unit 30 shown in FIG.

[0050] Therefore, only the image output determining units 321 and 323 and the image reconstructing units 322 and 324 will be described here, and redundant descriptions of the other components will be omitted.

[0051] The image output determination units 321 and 323 determine whether or not the attributes of the subject T, such as type, sex, and age, extracted by the preceding subject type extraction units 302 and 305, include attributes that the system operator wishes to identify as a monitoring target.

[0052] Then, when the image output determination units 321, 323 determine that the coded image DA has an attribute desired by the system operator, they output the coded image DA from which the attribute has been extracted to the downstream image reconstruction units 322, 324. Furthermore, the image output determination units 321, 323 output the coded image DA of a portion that includes a subject T with the attribute desired by the system operator to the downstream image reconstruction units 322, 324.

[0053] The image reconstruction units 322 and 324 reconstruct and output a decoded image DB from the input coded image DA, thereby reducing the processing load on the image reconstruction units 322 and 324 for reconstructing the decoded image DB from the coded image DA.

[0054] Fig. 6 is a flowchart showing an example of processing executed by the information extraction unit 30 shown in Fig. 5. As shown in Fig. 6, the information extraction unit 30 first detects a subject T from the encoded image DA (step S1). Then, the information extraction unit 30 determines and extracts the type of the subject T (step S2).

[0055] Thereafter, the information extraction unit 30 determines whether the type of subject T is a subject for image output, that is, whether the type is a subject desired by the system operator (step S3). If the information extraction unit 30 determines that the subject T is not a subject for image output (step S3, No), the process proceeds to step S4.

[0056] Furthermore, if the information extraction unit 30 determines that the image is to be output (step S3, Yes), it reconstructs a decoded image DB from the encoded image DA (step S5), and then the information extraction unit 30 crops the subject area containing the subject T from the reconstructed image, i.e., the reconstructed decoded image DB (step S6), and proceeds to step S4.

[0057] In step S4, the information extraction unit 30 outputs the extracted information (desired information) of the subject T and ends the process. The information extraction unit 30 executes the process shown in FIG. 6 every time the coded image DA is input from the pixel unit 12.

[0058] The optical modulation unit 11, the pixel unit 12, and the information extraction unit 30 included in the information processing system 100 may be mounted on a single piece of hardware, or may be distributed and mounted on multiple pieces of hardware that can communicate information with each other. Next, an example of the hardware configuration of the information processing system 100 will be described.

[0059] 7 to 11 are explanatory diagrams showing examples of the hardware configuration of an information processing system according to an embodiment. As shown in Fig. 7, the information processing system 100 may be mounted on a lensless sensor 101.

[0060] That is, the lensless sensor 101 according to the embodiment includes an optical modulation unit 11, a pixel unit 12, and an information extraction unit 30. With this configuration, the lensless sensor 101 can extract only accurate desired information from the image information of the coded image DA without outputting the decoded image DB to an external device.

[0061] Furthermore, the lensless sensor 101 may be configured to include only the processing related to the information extraction unit 30, and not include the processing related to image output as in a general image sensor. In this case, the lensless sensor 101 does not have an image output function, and the hardware guarantees that only desired metadata is output, which is preferable from the perspective of privacy protection.

[0062] 8 , the information processing system 100 according to the embodiment may be configured with a lensless sensor 102 and an information processing device 103. In this case, for example, the lensless sensor 102 includes an optical modulation unit 11 and a pixel unit 12.

[0063] On the other hand, the information processing device 103 includes an information extraction unit 30. With this configuration, by acquiring image information of the coded image DA from the existing lensless sensor 102, it is possible to extract accurate desired information from the image information.

[0064] 9 , the information processing system 100 may be configured with a lensless sensor 102, a data transfer unit 40, and a cloud 104. In this case, for example, the lensless sensor 102 includes an optical modulation unit 11 and a pixel unit 12.

[0065] The information extraction unit 30 is provided on the cloud 104. The data transfer unit 40 is a communication device capable of wireless or wired information communication with the information extraction unit 30 on the cloud 104 via a communication network such as the Internet.

[0066] Wireless communication includes third generation mobile communication systems (3G), fourth generation mobile communication systems (4G), Long Term Evolution (LTE), fifth generation mobile communication systems (5G), other mobile communication standards, wireless local area networks (LANs) such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), Z-wave (registered trademark), Ultra Wide Band (UWB), Thread (registered trademark), WiGig (registered trademark), near field communication (NFC), and other short-range wireless communication. Wired communication includes wired LANs such as Ethernet (registered trademark) and other wired communication.

[0067] The data transfer unit 40 transfers the image information of the encoded image DA output from the lensless sensor 102 to the information extraction unit 30 on the cloud 104. The information extraction unit 30 acquires the image information of the encoded image DA transferred from the data transfer unit 40, thereby being able to extract accurate desired information from the image information.

[0068] According to this configuration, the information extraction unit 30 provided on the cloud 104 can be shared by a plurality of users, thereby improving the versatility of the information processing system 100.

[0069] 10 , the information processing system 100 may be configured with a lensless sensor 105 having some of the functions of the information extraction unit 30, and an information processing device 106 having functions other than the part of the functions of the information extraction unit 30 of the lensless sensor 105. In this case, the lensless sensor 105 includes an optical modulation unit 11, a pixel unit 12, and a first information extraction unit 31. On the other hand, the information processing device 106 includes a second information extraction unit 32.

[0070] The first information extraction unit 31 extracts desired information that should be extracted in an earlier stage from the desired information extracted in multiple stages, and outputs the extracted information to the second information extraction unit 32 of the information processing device 106. The second information extraction unit 32 extracts desired information that should be extracted in a later stage from the desired information extracted in multiple stages.

[0071] For example, when the information extraction unit 30 extracts desired information from image information using a five-layer DNN (Deep Neural Network), the first information extraction unit 31 executes information extraction processing from the first layer, which is an input layer, to the second layer, which is an intermediate layer, while the second information extraction unit 32 executes information extraction processing from the third layer, which is an intermediate layer, to the fifth layer, which is an output layer.

[0072] With this configuration, the information processing system 100 can distribute some of the functions of the information extraction unit 30 and the remaining functions between the lensless sensor 105 and the information processing device 106. Therefore, with the information processing system 100, it is possible to reduce the processing load of each device compared to when the functions of the information extraction unit 30 are included in either the lensless sensor 105 or the information processing device 106.

[0073] Although the case where the information extraction unit 30 extracts the desired information from the encoded image DA generated by the lensless sensor has been described above, this is merely an example. The information extraction unit 30 may also be configured to extract the desired information from the decoded image DB generated by the lensless sensor.

[0074] 11, the information processing system 110 includes an optical modulation unit 11, a pixel unit 12, an image reconstruction unit 13, and an information extraction unit 33. The optical modulation unit 11 and the pixel unit 12 shown in FIG. 11 are the same as the optical modulation unit 11 and the pixel unit 12 shown in FIG.

[0075] 11 is the same as the signal processing device 20 shown in Fig. 1. That is, the image reconstruction unit 13 is a processing unit that generates a decoded image DB from an encoded image DA input from a pixel unit. The image reconstruction unit 13 outputs the generated decoded image DB to the information extraction unit 33.

[0076] Here, the image data set used by the information extraction unit 33 as the learning material for machine learning is different from the image data set used by the information extraction unit 30 shown in Figures 2 to 9. The information extraction AI of the information extraction unit 33 sequentially accepts input of a decoded image DB (question) from which predetermined information (answer) to be extracted is known.

[0077] The information extraction AI then extracts predetermined information from the decoded image DB and performs machine learning by updating the neural network coefficients so as to improve the degree of match (correct answer rate) between the extracted information and the answer information. The information extraction unit 33 then extracts desired information from the decoded image DB input from the image reconstruction unit 13.

[0078] In this way, the information extraction unit 33 is equipped with an information extraction AI that has been machine-trained to extract specific information from the decoded image DB generated by the lensless sensor, and therefore can extract accurate desired information from the decoded image DB.

[0079] In an embodiment involving image reconstruction processing, reconstruction processing may be applied that is designed not to include specific information in the reconstructed image. For example, when extracting information for the purpose of object detection, human face information is unnecessary, and from the viewpoint of privacy protection, it is preferable that such information not be included in the reconstructed image.

[0080] For this reason, the image reconstruction unit 13 may perform a reconstruction process designed to reconstruct only information other than faces, or in addition to the process of calculating a decoded image in which the subject can be recognized, an additional masking process may be performed, such as blacking out the facial information in the image so that it cannot be distinguished, to generate a decoded image in which human facial information has been excluded.

[0081] Furthermore, the removal of unnecessary information may be realized not by the image reconstruction unit 13 but by the optical modulation unit 11 or a combination of the optical modulation unit 11 and the image reconstruction unit 13. For example, the optical modulation unit 11 may not acquire information on spatial frequencies at which facial information can be resolved, and the image reconstruction unit 13 may generate, from the resulting encoded image DA, for example, silhouette information of a person that does not include facial information.

[0082] [4. Training Data Generation Device] It is possible to generate an image dataset used for machine learning of information extraction AI by having an operator assign labels indicating specific information to be extracted to a large number of encoded images actually generated by a lensless sensor. However, such an image dataset generation procedure increases the burden on the operator.

[0083] For this reason, in this embodiment, image data is generated by a training data generation device. Fig. 12 is a functional block diagram showing an example of the configuration of the training data generation device according to the embodiment. Fig. 13 is a flowchart showing an example of processing executed by the image data generation device according to the embodiment.

[0084] 12 , the training data generation device 200 includes a microcomputer having a CPU, ROM, RAM, etc., and various circuits. The training data generation device 200 includes a subject image generation unit 201, an optical element optical property calculation unit 202, and a pixel output calculation unit 203, which function when the CPU executes a program stored in the ROM using the RAM as a work area.

[0085] The subject image generation unit 201, the optical element optical property calculation unit 202, and the pixel output calculation unit 203 included in the training data generation device 200 may be configured in part or in whole using hardware such as an ASIC or FPGA.

[0086] The image information of the encoded image generated by the lensless sensor varies depending on disturbances and errors, such as the imaging environment of the subject, assembly errors of the optical modulation elements (masks), and differences in the optical characteristics of each mask.

[0087] Therefore, information extraction AI requires machine learning to enhance robustness against these disturbances and errors. Therefore, the training data generation device 200 generates, through simulation, an image dataset that serves as a learning material necessary to enhance robustness against disturbances and errors.

[0088] The training data generation device 200 simulates and reproduces an image captured by a lensless sensor by adding an effect that mimics the imaging process via an optical modulation element to an image captured by a normal sensor that is not a lensless sensor.

[0089] The training data generation device 200 generates a plurality of images captured by a lensless sensor through simulation. During the simulation process, the training data generation device 200 may also arbitrarily add the effects of errors and disturbances that may occur in an actual lensless sensor, and by utilizing the lensless simulation image dataset thus generated as a learning material for an information extraction AI, it is possible to achieve AI learning that is robust against disturbances and errors.

[0090] The training data generation device 200 is configured to perform different behavior for each generation of a lensless simulation image for phenomena that vary in behavior between individual devices and between imaging trials, such as noise and manufacturing errors of the lensless sensor. As a result, the training data generation device 200 realizes learning that is robust to the phenomenon in question by including the distribution of variation in the lensless simulation image dataset.

[0091] The object image generation unit 201 receives as input a reference image data set, such as an object image or an image captured by a normal camera. The object image generation unit 201 may also be configured to generate an object image based entirely on simulation using CG (Computer Graphics) or the like. The object image generation unit 201 may also be configured to generate an object image that mimics time behavior in order to reproduce phenomena involving time fluctuations, such as flicker or an object moving during the exposure period of an image sensor. The object image generation unit 201 outputs the input object image or the object image to be generated to the pixel output calculation unit 203.

[0092] The optical element optical characteristic calculation unit 202 is a functional block that simulates the optical characteristics of how an optical element spreads a point image of a subject scene onto an image plane. The optical element optical characteristic calculation unit 202 calculates image processing parameters that take into account differences in optical characteristics between individual optical modulation elements, such as assembly errors of the optical element with respect to the image sensor, variations in manufacturing shape, and deflection of the optical modulation element, as well as stray light incident from outside the angle of view.

[0093] The optical element optical characteristic calculation unit 202 may also be configured to calculate image processing parameters that take into account the influence on optical characteristics caused by defects or dust on the optical modulation element. The optical element optical characteristic calculation unit 202 outputs the calculated parameters to the pixel output calculation unit 203 and sets them.

[0094] The pixel output calculation unit 203 is a functional block that performs image processing on the image information of the subject input from the subject image generation unit 201, applying parameters set by the optical element optical property calculation unit 202, and imitates the type of image that a lensless sensor would output.

[0095] The pixel output calculation unit 203 may be configured to take into account variations such as observation noise of the image sensor (pixel unit 12) and differences in characteristics between individual image sensors (pixel units 12).The pixel output calculation unit 203 may be configured to take into account the influence of pixel oblique incidence characteristics on the imaging results.The pixel output calculation unit 203 may be configured to take into account the influence of pixel defects in the image sensor (pixel unit 12).

[0096] The pixel output calculation unit 203 generates a plurality of lensless simulation images that mimic the encoded images output from the lensless sensor, taking into account various disturbances and errors in this way, to create a lensless simulation image data set. At this time, the pixel output calculation unit 203 may create the lensless simulation image data set by attaching a label indicating the information to be extracted (the correct extraction result) to the created lensless simulation images.

[0097] In this way, the training data generation device 200 creates an image dataset of encoded images DA that simulates the imaging process using a lensless sensor by adding effects simulating the imaging process via an optical modulation element to images captured by a normal sensor. Furthermore, an image dataset may be created to achieve robust AI learning by intentionally adding the effects of errors and disturbances.

[0098] Therefore, the information extraction AI can perform machine learning with increased robustness against disturbances and errors by repeating the information extraction process to extract desired information from images in the image dataset created by the training data generation device 200.

[0099] When creating a lensless simulation image dataset to be used for machine learning by the image reconstruction unit 13 and the information extraction unit 33 shown in Figure 11, the training data generation device 200 generates a decoded image DB from the generated encoded image DA to create the lensless simulation image dataset.

[0100] When a reference image dataset is input, the training data generation device 200 executes the process shown in Fig. 13. Note that if a reference image dataset is not input, the training data generation device 200 can generate a reference image dataset itself, for example, using CG (Computer Graphics), and execute the process shown in Fig. 13.

[0101] 13 , when a reference image dataset is input, the training data generation device 200 determines whether or not an unreferenced image remains in the reference image dataset (step S101). If the training data generation device 200 determines that an unreferenced image remains (step S101, Yes), the training data generation device 200 acquires one unreferenced image from the reference image dataset (step S102).

[0102] Next, the training data generation device 200 inputs the acquired image to the object image generation unit 201 (step S103). The object image generation unit 201 outputs the input image to the pixel output calculation unit 203.

[0103] Thereafter, the learning data generating device 200 calculates the optical characteristics of the optical elements and sets them in the pixel output calculating unit 203 (step S104). Subsequently, the pixel output calculating unit 203 generates a lensless simulation image (step S105).

[0104] Thereafter, the pixel output calculation unit 203 stores the generated image in the lensless simulation image dataset (step S106), and proceeds to step S101. If the training data generation device 200 determines that no unreferenced images remain in the reference image dataset (step S101, No), it ends the process.

[0105] Note that the training data generation device 200 shown in Fig. 12 is an example. The training data sets used in machine learning of the information extraction AI may be created, for example, by the training data generation devices 200 and 300 shown in Figs. 14, 15, 19, and 21. Figs. 14, 15, 19, and 21 are functional block diagrams showing other configuration examples of the training data generation device according to the embodiment.

[0106] The training data generation device 200 shown in Fig. 14 differs from the training data generation device 200 shown in Fig. 12 in that the optical element optical property calculation unit 202 outputs the calculated parameters to the meta information conversion unit 204 in addition to the pixel output calculation unit 203. The other configurations are the same as those of the training data generation device 200 shown in Fig. 12.

[0107] 14 converts the subject position in the reference image into the subject position in the encoded image DA based on the parameters input from the optical element optical property calculation unit 202, and adds meta information indicating the conversion result to the lensless simulation image data set. As a result, meta information indicating the subject position in the encoded image DA is added to the lensless simulation image. Note that this meta information may include all of the desired information described above.

[0108] 15 includes an image reconstructing unit 501, a meta-information adding unit 502, and a meta-information converting unit 503. The image reconstructing unit 501 generates a reconstructed image (decoded image DB) from an encoded image DA of the encoded image dataset, and outputs the reconstructed image to the meta-information adding unit 502.

[0109] The image reconstruction unit 501 may output the generated reconstructed image to a lensless simulation image data set. The meta-information assignment unit 502 assigns meta-information indicating the subject position in the reconstructed image to the reconstructed image. The meta-information here may include all of the desired information described above.

[0110] The meta-information conversion unit 503 converts meta-information indicating the subject position in the reconstructed image into meta-information indicating the subject position in the coded image DA, and adds it to the lensless simulation image data set.

[0111] FIG. 16 is a flowchart showing an example of processing executed by the meta information conversion units 204 and 503 according to the embodiment.

[0112] 16, the meta-information conversion unit 204, 503 determines the effective area of ​​the optical characteristics of the optical element (step S201), and then calculates the center position, width, and height of the effective area of ​​the optical characteristics of the optical element (step S202).

[0113] Thereafter, the meta information conversion unit 204, 503 calculates the center position of the effective area of ​​the optical characteristics of the optical element and the shift amount of the image center (step S203), and then determines whether there is an unprocessed subject position of the subject T (step S204).

[0114] If the meta information conversion unit 204, 503 determines that there is an unprocessed subject position of subject T (step S204, Yes), it converts the subject position using the center position, width, and height of the effective area (step S205) and proceeds to step S204.

[0115] 17 and 18, an example of a meta information conversion method performed by the meta information conversion units 204 and 503 will be described. Figures 17 and 18 are explanatory diagrams showing an example of a meta information conversion method according to an embodiment.

[0116] For example, as shown in Figure 17, when the input image is a reference image DC of a reference image dataset, the meta information conversion unit 204, 503 uses the position MD2 of the effective area of ​​the optical characteristics of the optical element to convert the position MD1 of the subject T in the reference image DC to the position MD3 of the subject T in the encoded image DA.

[0117] Also, as shown in Figure 18, when the input image is an encoded image DA of an encoded image data set, the meta information conversion unit 204, 503 uses the position MD2 of the effective area of ​​the optical characteristics of the optical element to convert the position MD3 of the subject T in the encoded image DA to the position MD1 of the subject T in the reconstructed image DC.

[0118] At this time, the meta information conversion unit 204, 503 converts the position of the subject T using the following equations (1), (2), (3), and (4). Horizontal coordinate 1 of the subject position after conversion = horizontal coordinate 1 of the subject position - (width of the effective area of ​​the optical characteristics of the optical element / 2) - horizontal shift amount of the center position of the effective area of ​​the optical characteristics of the optical element (1) Horizontal coordinate 2 of the subject position after conversion = horizontal coordinate 2 of the subject position + (width of the effective area of ​​the optical characteristics of the optical element / 2) - horizontal shift amount of the center position of the effective area of ​​the optical characteristics of the optical element (2) Vertical coordinate 1 of the subject position after conversion = vertical coordinate 1 of the subject position - (height of the effective area of ​​the optical characteristics of the optical element / 2) - vertical shift amount of the center position of the effective area of ​​the optical characteristics of the optical element (3) Vertical coordinate 2 of the subject position after conversion = vertical coordinate 2 of the subject position + (height of the effective area of ​​the optical characteristics of the optical element / 2) - vertical shift amount of the center position of the effective area of ​​the optical characteristics of the optical element (4)

[0119] 16 , after the meta information conversion unit 204, 503 performs the conversion process in step S205, the process proceeds to step S204. If the meta information conversion unit 204, 503 determines that there is no subject position of the subject T that has not yet been processed (No in step S204), the process ends.

[0120] 19 includes a lensless camera 301, a normal camera 302, a general-purpose recognizer 303, and a pairing processing unit 304. The lensless camera 301 is the same as the imaging device 10 shown in FIG. 1, and any camera having a lensless image sensor can be used.

[0121] When creating a training dataset, the training data generation device 300 captures images of the same subject T using a lensless camera 301 and a normal camera 302. The lensless camera 301 captures an encoded image DA that is a problem from which the information extraction AI will extract desired information, and outputs the image to a pairing processing unit 304.

[0122] The training data generation device 300 can also obtain more accurate answer information for the lensless camera 301 by substantially or completely aligning the optical axes of the lensless camera 301 and the normal camera 302. For example, the optical axes of the lensless camera 301 and the normal camera 302 can be aligned using a beam splitter such as a prism or a half mirror.

[0123] The normal camera 302 captures an RGB (three primary colors) image of the subject and outputs it to the general-purpose recognizer 303. The general-purpose recognizer 303 recognizes desired information from the RGB image, which is the answer to the problem that causes the information extraction AI to extract the desired information. The general-purpose recognizer 303 then outputs the recognition result (answer) of the desired information to the pairing processing unit 304.

[0124] The pairing processing unit 304 pairs the coded image DA input from the lensless camera 301 with the recognition result input from the general-purpose recognizer 303 to create a learning dataset.

[0125] When the training data generation device 300 creates a training data set to be used for machine learning by the image reconstruction unit 13 and the information extraction unit 33 shown in FIG. 11, it generates a decoded image DB from the generated encoded image DA and pairs it with the recognition result to create a training data set.

[0126] Fig. 20 is a flowchart showing an example of processing executed by the pairing processing unit 304 shown in Fig. 19. As shown in Fig. 20, when the pairing processing unit 304 starts the pairing processing, first, it acquires an image of the normal camera 302 (step S301), and determines an area corresponding to the angle of view of the normal camera 302 from the image of the lensless camera 301 (step S302).

[0127] Thereafter, the pairing processing unit 304 crops a portion of the image of the lensless camera 301 that corresponds to the angle of view of the normal camera 302 (step S303), and resizes the cropped area of ​​the lensless camera 301 to the image size of the normal camera 302 (step S304).

[0128] Furthermore, when the pairing processing starts, the pairing processing unit 304 converts the subject position of the recognition result of the normal camera 302 into a subject position equivalent to that of the lensless camera 301 (step S305). Subsequently, the pairing processing unit 304 associates a correspondence relationship between the converted subject position equivalent to that of the lensless camera 301 and the subject type (step S306).

[0129] Thereafter, the pairing processing unit 304 pairs the coded image DA of the lensless camera 301 with the recognition result of the normal camera 302 (step S307), and ends the pairing processing.

[0130] The training data generation device 300 may also be configured as shown in Fig. 21. The training data generation device 300 shown in Fig. 21 differs in configuration from the training data generation device 300 shown in Fig. 19 in that the training data generation device 300 does not include the general-purpose recognizer 303 shown in Fig. 19.

[0131] 21 , a pairing processing unit 304 receives an input of a lensless encoded image (answer) from a lensless camera 301 and an input of an RGB image (answer) from a normal camera 302. In this case, the pairing processing unit 304 pairs the lensless encoded image (question) with the RGB image (answer) to create a training dataset.

[0132] Fig. 22 is a flowchart showing an example of processing executed by the pairing processing unit 304 shown in Fig. 21. As shown in Fig. 22, when the pairing processing unit 304 starts the pairing processing, it executes the same processing as steps S301 to S304 shown in Fig. 20.

[0133] Specifically, the pairing processing unit 304 first acquires an image from the normal camera 302 (step S301), and determines an area corresponding to the angle of view of the normal camera 302 from the image from the lensless camera 301 (step S302).

[0134] Thereafter, the pairing processing unit 304 crops a portion of the image of the lensless camera 301 that corresponds to the angle of view of the normal camera 302 (step S303), and resizes the cropped area of ​​the lensless camera 301 to the image size of the normal camera 302 (step S304).

[0135] Thereafter, the pairing processing unit 304 pairs the coded image DA of the lensless camera 301 with the image of the normal camera 302 (step S308), and ends the pairing process.

[0136] [5. Learning Device] The information extraction AI according to the embodiment can function as a learning device used for designing the lensless sensor 1 shown in FIG. 1, for example, by being configured to have the functions of the learning data generation devices 200 and 300 described with reference to FIGS. 12 to 22 .

[0137] 23 is an explanatory diagram of a learning device according to an embodiment. As shown in FIG. 23, a learning device 400 according to an embodiment includes a microcomputer having a CPU, ROM, RAM, etc., and various circuits.

[0138] The learning device 400 includes a setting unit 410 that functions by the CPU executing a program stored in the ROM using the RAM as a work area, an encoded image generation unit 403, and an information extraction AI 404. The setting unit 410 includes an optical element optical property setting unit 401 and an image sensor setting unit 402.

[0139] The optical element optical property setting unit 401, image sensor setting unit 402, encoded image generation unit 403, and information extraction AI 404 provided in the learning device 400 may be partially or entirely configured using hardware such as an ASIC or FPGA.

[0140] 12 , the optical element optical property setting unit 401 calculates image processing parameters for each input image that take into account differences in optical properties between individual optical modulation elements, such as assembly errors of the optical element with respect to the image sensor, variations in manufacturing shape, and deflection of the optical modulation element, as well as stray light incident from outside the angle of view, and sets these parameters in the encoded image generation unit 403. Furthermore, the optical element optical property setting unit 401 may also design the lensless sensor 1 based on the two-dimensional pattern of the optical modulation unit 11 and the distance between the mask and the image sensor, and calculate image processing parameters corresponding to these design values ​​and set these parameters in the encoded image generation unit 403.

[0141] The image sensor setting unit 402 calculates image processing parameters for each input image that take into account phenomena that cause variations in behavior between individual devices and each imaging attempt, such as noise, manufacturing errors in the lensless sensor, and errors in the mounting position, and sets these parameters in the encoded image generation unit 403.

[0142] The image sensor setting unit 402 may also design the lensless sensor 1 taking into account image sensor characteristics such as pixel pitch and the device structure of the lensless sensor such as the attachment position of optical elements, calculate image processing parameters corresponding to the design values, and set them in the encoded image generation unit 403. Furthermore, image processing parameters corresponding to the shooting conditions of the lensless sensor, such as exposure time and gain setting, for example, may be calculated and set in the encoded image generation unit 403.

[0143] 12, the encoded image generation unit 403 executes image processing according to set parameters to generate lensless simulation image data and outputs it to the information extraction AI 404. The information extraction AI extracts desired information from the lensless simulation image data, i.e., the encoded image DA, in the same way as the information extraction AI of the information extraction unit 30 described with reference to FIGS.

[0144] The learning device 400 then performs machine learning by repeatedly executing this process, and sequentially updates the parameters calculated by the optical element optical property setting unit 401 so as to improve the accuracy with which the information extraction AI 404 extracts the desired information. The learning device 400 also sequentially updates the parameters calculated by the image sensor setting unit 402 so as to improve the accuracy with which the information extraction AI 404 extracts the desired information.

[0145] The optical element optical characteristic setting unit 401 outputs the updated and optimized parameters as optical modulation element setting information. The image sensor setting unit 402 outputs the updated and optimized parameters as image sensor setting information.

[0146] As a result, for example, by using the optical modulation element setting information, the designer of the lensless sensor 1 can design the optical modulation unit 11 so as to improve the accuracy of extraction of desired information by the information extraction AI 404. Furthermore, by using the image sensor setting information, the designer can realize control and attachment position of the pixel unit 12 so as to improve the accuracy of extraction of desired information by the information extraction AI 404.

[0147] The image sensor setting unit 402 may be omitted from the learning device 400. The learning device 400 can generate optical modulation element setting information required for an optimal optical modulation unit 11 as long as it includes at least the optical element optical property setting unit 401, the encoded image generation unit 403, and the information extraction AI 404.

[0148] In addition, in the learning device 400, in order to improve the accuracy of extraction of desired information by the information extraction AI 404, the coefficients of the neural network of the information extraction AI 404 may also be updated at the same time as the parameters of the optical element optical property setting unit 401 and the image sensor setting unit 402.

[0149] The learning device 400 may also have a function for deploying an AI model appropriate for the lensless sensor depending on the type of optical element. Since the data format output by a typical RGB image sensor is an image, AI processing is possible by connecting various AI models downstream. On the other hand, a lensless sensor has the characteristic that the encoded image changes significantly depending on the optical element. Therefore, when selecting an AI model for a lensless sensor, it is necessary to consider the effect of optical modulation by each optical element and select an AI model that corresponds to this.

[0150] In contrast, one embodiment of the learning device 400 may provide a user with a user interface that enables the user to select an AI model that corresponds to the type of optical element used in the lensless sensor and deploy it on the lensless sensor. This allows the user of the lensless sensor to easily deploy the optimal AI model that corresponds to the optical element used.

[0151] The user may specify the type of optical element of the lensless sensor to be used via a user interface. Alternatively, instruction data that can identify the type of optical element installed may be provided to the lensless sensor, and the learning device may deploy an appropriate AI model by reading the instruction data from the connected lensless sensor camera.

[0152] Furthermore, when a user selects an AI model corresponding to an optical element type, the learning device 400 may present a user interface that allows the user to select from multiple candidates the type of information to be acquired from the lensless sensor. If multiple types of information can be extracted from the imaging results for the same optical element, candidate information extraction AIs corresponding to the selected optical element are presented to the user, and the user can select the desired information extraction AI from the presented information extraction AIs and deploy the AI ​​model. For example, information to be extracted from the lensless encoded image DA may include metadata such as image recognition and detection results, and image information related to a two-dimensional image and depth information of the subject T. By selecting the desired AI from these candidates, the AI ​​service user can easily deploy an AI that is linked to the type of optical element of the lensless sensor and can extract the type of information desired by the user.

[0153] The learning device 400 may also have a function that allows it to switch whether to input a reconstructed image or an encoded image to the AI ​​model depending on the type of AI model. For example, for an AI model that does not require consideration of privacy protection, such as object detection, inference is performed using reconstructed images, and an AI model for a general RGB sensor can be used as is. On the other hand, for an AI model that requires consideration of privacy protection, such as human recognition, inference is preferably performed using encoded images without undergoing reconstruction processing. For this reason, instruction data indicating whether to use encoded images or reconstructed images for inference is assigned to the AI ​​model deployed from the AI ​​learning service, and the model is deployed to the lensless sensor. The lensless sensor switches the signal processing method based on the instruction data, and then inputs appropriate data into the AI ​​model for inference.

[0154] In one embodiment of the learning device 400, a lensless sensor manufactured based on optimized design parameters may be provided to a user as a device. Alternatively, instruction data that enables determination of the design parameters on which the lensless sensor is based may be assigned to the lensless sensor, and the learning device may refer to the instruction data when deploying an AI model, and deploy an optimal AI model that corresponds to the design parameters of the lensless sensor.

[0155] Furthermore, in a lensless sensor, calculation processing required in AI, such as convolution, may be optically simulated, and the processing may be performed in the optical modulation unit 11. For this reason, constraints may be imposed when optimizing the two-dimensional pattern of the optical modulation unit so that the optical modulation unit 11 in the optical element optical property setting unit 401 of the learning device 400 exhibits such optical behavior.

[0156] Furthermore, the lensless sensor may optically realize more complex feature extraction operations by stacking optical modulation elements in multiple stages. Therefore, the optical modulation unit 11 in the optical element optical property setting unit 401 may assume such behavior. A lensless sensor designed in this way optically realizes the calculation processing required for the information extraction AI, thereby reducing the calculation load on the information extraction unit 30.

[0157] [6. Variations of the Pixel Unit: Wavelength Characteristics of the Pixel Unit] The pixel unit 12 is not limited to pixels of an image sensor equipped with a general RGB color filter. For example, the pixel unit 12 can be configured to receive light L incident via the optical modulation unit 11 using pixels of a multispectral sensor that can acquire a large amount of color information. In this case, the pixel unit 12 can acquire wavelength information of a subject with higher accuracy than an RGB sensor while having a lensless sensor configuration.

[0158] On the other hand, since the encoded image DA captured using this configuration has different image information from the image captured by the image sensor using a lens configuration, there is a risk that the desired information cannot be extracted if general image recognition AI or AI for this image sensor is applied.

[0159] Furthermore, since the color information acquired is different from that of an RGB sensor, the image information is also different from that of an image captured by a lensless sensor using an RGB sensor. Therefore, even if information extraction AI for a lensless sensor using an RGB sensor is applied, there is a risk that the desired information cannot be extracted.

[0160] Therefore, the pixel unit 12 according to one embodiment of the present invention is capable of extracting desired information from image information by including a dedicated information extraction AI in its configuration for a lensless sensor equipped with an image sensor other than an RGB sensor.

[0161] Furthermore, the training data generation device may also generate training data in a manner that takes into account the characteristics of the image sensor installed in the lensless sensor. For example, in a configuration using a multispectral sensor, the training data generation device 200 may reproduce a multispectral image output from the pixel unit 12 while taking into account the wavelength characteristics of the subject, the optical modulation element, and the image sensor. The training data set thus generated can be used as teaching material for the information extraction AI, thereby training an information extraction AI dedicated to the lensless sensor using the multispectral image sensor. The training data generation device 200 may reproduce disturbances and errors associated with the use of a multispectral sensor during the image generation process.

[0162] [7. Pixel Unit Variations: When SPAD (Single Photon Avalanche Diode) Pixels are Used] The pixel unit 12 may use a SPAD (Single Photon Avalanche Diode) pixel structure. SPAD pixels have a pixel structure that utilizes the avalanche multiplication effect, which multiplies electrons from a single incident photon, and are therefore characterized by their ability to easily detect even weak light. Therefore, by using SPAD pixels in a lensless sensor, it is possible to efficiently detect photons and acquire information about a subject even in an environment where only a small number of photons are received, such as a dark place, while still adopting a lensless sensor configuration.

[0163] On the other hand, because the imaging principle differs from that of a CMOS sensor, the encoded image DA captured by a lensless sensor equipped with SPAD pixels has different imaging characteristics, such as dynamic range and observation noise, compared to when a CMOS sensor is used. For this reason, when general image recognition AI is applied, it may not be possible to extract desired information, or the accuracy of information extraction may be impaired.

[0164] Therefore, the pixel section 12 according to one embodiment of the present invention is configured with a dedicated information extraction AI for a lensless sensor equipped with SPAD pixels, thereby enabling desired information to be extracted from image information.

[0165] Furthermore, in the training data generation device 200, a training data set generated by simulating the imaging process using SPAD pixels mounted in a lensless sensor can be used as a teaching material for the information extraction AI, thereby training an information extraction AI dedicated to lensless sensors using SPAD pixels. The training data generation device 200 may reproduce disturbances and errors associated with the use of SPAD pixels in the image generation process.

[0166] [8. Pixel Unit Variations: When an Event-Based Vision Sensor is Used] The pixel unit 12 may use pixels from an event-based vision sensor. An event-based vision sensor detects changes in brightness for each pixel and outputs only the changed data, combining it with the pixel's coordinates and time information. This allows for high speed and low latency. Therefore, using pixels from an event-based vision sensor in a lensless sensor enables imaging with features such as reduced output data volume during periods when there are no changes in the subject scene.

[0167] On the other hand, because event-based vision sensors have different imaging principles and output image information than CMOS image sensors, there is a risk that the desired information may not be extracted if general image recognition AI is applied to a lensless sensor equipped with an event-based vision sensor.

[0168] Furthermore, image recognition AI for general event-based vision sensors assumes imaging via a lens, and if this is applied to a lensless sensor equipped with an event-based vision sensor, there is a risk that the desired information will not be extracted.

[0169] Therefore, the pixel unit 12 according to one embodiment of the present invention is a lensless sensor having pixels of an event-based vision sensor, and by providing its configuration with a dedicated information extraction AI, it is possible to extract desired information from image information.

[0170] Furthermore, in the training data generation device 200, a training data set generated by simulating the imaging process using pixels of an event-based vision sensor mounted in a lensless sensor can be used as a teaching material for the information extraction AI, allowing the lensless sensor using pixels of the event-based vision sensor to be trained. In the training data generation device 200, disturbances and errors associated with mounting pixels of the event-based vision sensor may be reproduced in the image generation process.

[0171] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0172] Note that the present technology may also be configured as follows. (1) A non-transitory computer-readable storage medium that, when executed by an information processing system, causes the information processing system to perform a process, the process comprising: receiving an encoded image of a subject captured by a lensless image sensor; and extracting information from the encoded image using a trained machine learning model. (2) The non-transitory computer-readable storage medium according to (1), wherein the trained machine learning model is a deep neural network. (3) The non-transitory computer-readable storage medium according to (2), wherein the process further includes updating coefficients of the deep neural network. (4) The non-transitory computer-readable storage medium according to any one of (1) to (3), wherein the trained machine learning model is a convolutional neural network. (5) The non-transitory computer-readable storage medium according to any one of (1) to (3), wherein the trained machine learning model is a vision transformer. (6) The non-transitory computer-readable storage medium of any one of (1) to (5), wherein the information includes one or more of a position, a contour, and a type of an object in the encoded image. (7) The non-transitory computer-readable storage medium of any one of (1) to (6), wherein the encoded image is obtained by photoelectric conversion of light that has passed through a mask. (8) The non-transitory computer-readable storage medium of any one of (1) to (7), wherein the encoded image is an image of an object that is not recognizable to the naked eye. (9) The non-transitory computer-readable storage medium of any one of (1) to (8), wherein the processing further includes extracting information without outputting a decoded image. (10) The non-transitory computer-readable storage medium of any one of (1) to (9), wherein the processing further includes obtaining an image of the object, wherein the image of the object is input or generated from a reference image dataset.(11) The non-transitory computer-readable storage medium of (10), wherein a simulated object image to be used as the object image is generated using computer graphics when the object image is generated. (12) The non-transitory computer-readable storage medium of (11), wherein the object image is generated by imitating temporal behavior associated with temporal variations. (13) The non-transitory computer-readable storage medium of (10), wherein parameters for image processing are calculated based on optical characteristics of the lensless image sensor including at least one of one or more disturbances and one or more errors, and wherein image processing is performed by applying the calculated parameters to the object image, and wherein an image output by the lensless image sensor is simulated based on the image processing. (14) The non-transitory computer-readable storage medium of (13), wherein a plurality of lensless simulated images imitating encoded images output from the lensless image sensor are generated based on at least one of one or more disturbances and one or more errors, thereby creating a lensless simulated image dataset. (15) The non-transitory computer-readable storage medium according to (14), wherein a label indicating information to be extracted is assigned to each lensless simulation image created in the lensless simulation image dataset. (16) The non-transitory computer-readable storage medium according to (13), wherein at least one of the one or more disturbances and the one or more errors includes an environment in which the object is imaged, an assembly error of a mask, and a difference in optical properties between different masks. (17) An image processing method comprising receiving an encoded image of an object imaged by a lensless image sensor and extracting information from the encoded image using a trained machine learning model. (18) The image processing method according to (17), wherein information is extracted without outputting a decoded image. (19) The image processing method according to (17) or (18), wherein an object image is acquired, and the object image is input or generated from a reference image dataset.(20) The image processing method according to (19), which generates a simulated object image to be used as the object image using computer graphics when the object image is generated. (21) The image processing method according to (19) or (20), which calculates parameters for image processing based on optical characteristics of the lensless image sensor including at least one of one or more disturbances and one or more errors, applies the calculated parameters to the object image to perform image processing, and simulates an image output by the lensless image sensor based on the image processing. (22) The image processing method according to (21), which generates a plurality of lensless simulation images that imitate encoded images output from the lensless image sensor based on at least one of one or more disturbances and one or more errors, and creates a lensless simulation image dataset. (23) The image processing method according to (22), which assigns a label indicating information to be extracted to each lensless simulation image created in the lensless simulation image dataset. (24) An image processing system comprising an information extraction circuit that receives an image of a subject captured by a lensless image sensor, extracts information from the image using a trained machine learning model, and outputs the information. (25) The image processing system according to (24), wherein the received image is an encoded image. (26) The image processing system according to (24) or (25), comprising an image reconstruction circuit configured to receive the encoded image captured by the lensless image sensor, generate a decoded image from the encoded image, and output the decoded image to the information extraction circuit. (27) The image processing system according to (25) or (26), comprising a lensless image sensor configured to acquire image data, wherein the information extraction circuit is mounted on the lensless image sensor.(28) The image processing system according to (25) or (26), comprising: a lensless image sensor configured to acquire image data and output an encoded image; and an information processing device different from the lensless image sensor, wherein the information processing device includes the information extraction circuit. (29) The image processing system according to (25) or (26), comprising: a lensless image sensor configured to acquire image data and output an encoded image; and a cloud server configured to receive the encoded image, wherein the cloud server includes the information extraction circuit. (30) The image processing system described in (25) or (26), wherein the information extraction circuit includes a first information extraction circuit and a second information extraction circuit and extracts information in multiple stages; the image processing system has a lensless image sensor including the first information extraction circuit and an information processing device separate from the lensless image sensor; the first information extraction circuit receives an encoded image captured by the lensless image sensor, extracts a first portion of the information in a first stage, and outputs the first portion of the information to the second information extraction circuit; and the information processing device includes a second information extraction circuit, and the second information extraction circuit receives the first portion of the information and extracts a second portion of the information in a second stage.

[0173] REFERENCE SIGNS LIST 1 Lensless sensor 10 Imaging device 11 Optical modulation unit 12 Pixel unit 13 Image reconstruction unit 20 Signal processing device 30 Information extraction unit 31 First information extraction unit 32 Second information extraction unit 33 Information extraction unit 40 Data transfer unit 100 Information processing system 101 Lensless sensor 102 Lensless sensor 103 Information processing device 104 Cloud 105 Lensless sensor 106 Information processing device 110 Information processing system 200 Training data generation device 201 Subject image generation unit 202 Optical element optical property calculation unit 203 Pixel output calculation unit 300 Training data generation device 301 Lensless camera 302 Normal camera 303 General-purpose recognizer 304 Pairing processing unit 400 Training device 401 Optical element optical property setting unit 402 Image sensor setting unit 403 Encoded image generation unit 410 Setting unit DA Encoded image DB Decoded image

Claims

1. A non-transitory computer-readable storage medium that, when executed by an information processing system, causes the information processing system to perform a process, the process comprising: receiving an encoded image of a subject captured by a lensless image sensor; and extracting information from the encoded image by a trained machine learning model.

2. The non-transitory computer-readable storage medium of claim 1, wherein the trained machine learning model is a deep neural network.

3. The non-transitory computer-readable storage medium of claim 2, wherein the processing further comprises updating coefficients of the deep neural network.

4. The non-transitory computer-readable storage medium of claim 1, wherein the trained machine learning model is a convolutional neural network.

5. The non-transitory computer-readable storage medium of claim 1, wherein the trained machine learning model is a Vision Transformer.

6. The non-transitory computer-readable storage medium of claim 1, wherein the information includes one or more of a location, a contour, and a type of an object in the encoded image.

7. The non-transitory computer-readable storage medium of claim 1, wherein the encoded image is a photoelectric conversion of light passing through a mask.

8. The non-transitory computer-readable storage medium of claim 1, wherein the encoded image is an image of an object not perceptible to the naked eye.

9. The non-transitory computer-readable storage medium of claim 1, wherein the processing further comprises extracting information without outputting the decoded image.

10. The non-transitory computer-readable storage medium of claim 1, wherein the process further comprises obtaining an object image, the object image being input from or generated from a reference image dataset.

11. The non-transitory computer readable storage medium of claim 10, further comprising: generating a simulated object image for use as the object image using computer graphics when the object image is generated.

12. The non-transitory computer-readable storage medium of claim 11, wherein the object images are generated to mimic a temporal behavior associated with a temporal variation.

13. The non-transitory computer-readable storage medium of claim 10, further comprising: calculating parameters for image processing based on optical characteristics of the lensless image sensor including at least one of one or more disturbances and one or more errors; applying the calculated parameters to the subject image to perform image processing; and simulating an image output by the lensless image sensor based on the image processing.

14. The non-transitory computer readable storage medium of claim 13, further comprising: generating a plurality of lensless simulated images that mimic encoded images output from the lensless image sensor based on at least one of the one or more disturbances and the one or more errors; and creating a lensless simulated image data set.

15. The non-transitory computer readable storage medium of claim 14, further comprising: assigning a label to each lensless simulation image created in the lensless simulation image dataset that indicates information to be extracted.

16. The non-transitory computer-readable storage medium of claim 13, wherein at least one of the one or more disturbances and the one or more errors include an environment in which the object is imaged, an assembly error of the mask, and differences in optical properties in different masks.

17. An image processing method comprising: receiving an encoded image of a subject captured by a lensless image sensor; and extracting information from the encoded image using a trained machine learning model.

18. The image processing method of claim 17, further comprising extracting information without outputting the decoded image.

19. The image processing method of claim 17, further comprising acquiring an image of an object, the image of the object being input from or generated from a reference image data set.

20. The image processing method of claim 19, further comprising using computer graphics to generate a simulated object image for use as the object image when the object image is generated.

21. The image processing method of claim 19, further comprising: calculating parameters for image processing based on optical characteristics of the lensless image sensor including at least one of one or more disturbances and one or more errors; applying the calculated parameters to the subject image to perform image processing; and simulating an image output by the lensless image sensor based on the image processing.

22. The image processing method of claim 21, further comprising: generating a plurality of lensless simulation images that mimic encoded images output from the lensless image sensor based on at least one of the one or more disturbances and one or more errors; and creating a lensless simulation image data set.

23. The image processing method of claim 22, further comprising: assigning a label to each lensless simulation image created in the lensless simulation image data set that indicates information to be extracted.

24. An image processing system comprising: an information extraction circuit that receives an image of a subject captured by a lensless image sensor, extracts information from the image using a trained machine learning model, and outputs the information.

25. The image processing system of claim 24, wherein the received image is an encoded image.

26. The image processing system of claim 24, further comprising an image reconstruction circuit configured to receive an encoded image captured by the lensless image sensor, generate a decoded image from the encoded image, and output the decoded image to an information extraction circuit.

27. The image processing system of claim 25, further comprising a lensless image sensor configured to acquire image data, and wherein the information extraction circuitry is mounted on the lensless image sensor.

28. An image processing system according to claim 25, comprising: a lensless image sensor configured to acquire image data and output an encoded image; and an information processing device distinct from the lensless image sensor, said information processing device including said information extraction circuit.

29. The image processing system of claim 25, comprising: a lensless image sensor configured to acquire image data and output an encoded image; and a cloud server configured to receive the encoded image, the cloud server including the information extraction circuit.

30. The image processing system of claim 25, wherein the information extraction circuit includes a first information extraction circuit and a second information extraction circuit, and extracts information in multiple stages; the image processing system has a lensless image sensor including the first information extraction circuit, and an information processing device separate from the lensless image sensor, wherein the first information extraction circuit receives an encoded image captured by the lensless image sensor, extracts a first portion of the information in a first stage, and outputs the first portion of the information to a second information extraction circuit; and the information processing device includes a second information extraction circuit, and the second information extraction circuit receives the first portion of the information and extracts a second portion of the information in a second stage.

Images