IMAGE PROCESSING DEVICE, MACHINE LEARNING MODEL TRAINING METHOD, RECOGNITION DEVICE, AND IMAGE PROCESSING METHOD

Abstract

An image processing device comprising: a noise providing unit (12) for acquiring a captured image from an image capture device equipped with a mask having one or more openings formed therein; an MPH information acquiring unit (111) for acquiring opening mode information corresponding to a mode of the one or more openings; a noise providing unit (12) for providing the captured image with a noise determined by the opening mode information; and a transmission unit (13) for outputting the captured image provided with the noise.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a machine learning model training method, a classification device, and an image processing method. [Background technology]

[0002] In recent years, techniques for processing images to prevent unauthorized use of images have been researched and developed. For example, one example of processing an image is adding noise to the image. Patent Document 1 discloses a technique for adding noise to each pixel based on additional information indicating which of two noise processes, a first noise addition process or a second noise addition process, is to be performed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3919613 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technique described in Patent Document 1, there is a possibility that the processed image may be restored. For example, there is a possibility that noise added to the image may be removed.

[0005] Therefore, the present disclosure provides an image processing device, a method for training a machine learning model, a classification device, and an image processing method that can suppress the restoration of an image to which noise has been added. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present disclosure includes an image acquisition unit that acquires a first captured image from a first imaging device having a mask with one or more openings formed therein, an information acquisition unit that acquires opening mode information corresponding to the modes of the one or more openings, a noise addition unit that adds noise determined from the opening mode information to the first captured image, and an output unit that outputs the first captured image to which the noise has been added.

[0007] A method for training a machine learning model according to one embodiment of the present disclosure includes acquiring a dataset including images generated by adding noise determined from aperture mode information corresponding to the modes of the one or more openings to an image acquired from an imaging device having a mask with one or more openings formed therein, and training a machine learning model using the acquired dataset.

[0008] An identification device according to one embodiment of the present disclosure includes an image acquisition unit that acquires the first captured image to which the noise has been added from the image processing device, and an identification unit that identifies an object appearing in the first captured image to which the noise has been added by using a machine learning model trained with a dataset including an image generated by adding noise determined from the opening mode information to a second captured image acquired from a second imaging device equipped with the mask.

[0009] An image processing method according to one aspect of the present disclosure includes acquiring an image from an imaging device having a mask with one or more openings formed therein, acquiring opening mode information corresponding to the modes of the one or more openings, adding noise determined from the opening mode information to the image, and outputting the image with the noise added. Effect of the Invention

[0010] According to an image processing device and the like according to an aspect of the present disclosure, it is possible to suppress restoration of an image to which noise has been added. [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 is a block diagram showing a functional configuration of the information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a configuration of the sensing device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining various images according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the operation of the sensing device according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing the operation of the training device of the information processing system according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the operation of the identification device of the information processing system according to the first embodiment. [Figure 7] FIG. 7 is a block diagram illustrating a functional configuration of the sensing device according to the second embodiment. As shown in FIG. [Figure 8] FIG. 8 is a flowchart showing the operation of the sensing device according to the second embodiment. [Figure 9] FIG. 9 is a block diagram showing a functional configuration of a sensing device according to a modification of the second embodiment. In FIG. [Figure 10] FIG. 10 is a flowchart showing the operation of the sensing device according to the modification of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Background to this disclosure) Before describing the embodiments of the present disclosure, the background to the present disclosure will be described.

[0013] In recent years, cameras have been widely installed indoors and outdoors, making it easier to capture images that include privacy. For example, cameras are installed in drones and robots that operate in privacy-protected areas such as toilets and bathrooms in homes or elderly care facilities, or in convenience stores, making it easier to capture images that include privacy, such as people indoors, faces of passersby, and license plates of automobiles.

[0014] If such an image containing private information is leaked, the people and other objects in the image are exposed to the threat of privacy violation. Therefore, the importance of protecting the privacy of objects in images captured by cameras is increasing.

[0015] It is conceivable to reduce the threat of privacy violation by adding noise to a captured image (normal image) as in the technology of Patent Document 1. However, the technology of Patent Document 1 cannot protect privacy if the normal image itself is leaked.

[0016] In the present disclosure, from the viewpoint of protecting the privacy of an object captured in an image, for example, a multi-pinhole camera is used to capture an image of the object. Although details will be described later, an image captured by a multi-pinhole camera (multi-pinhole image: MPH image) is an image that includes parallax characteristics. In other words, an MPH image is a blurred image that includes blur. As a result, the acquired image is a blurred image, making it possible to protect the privacy of the object. Note that a multi-pinhole camera is a camera having a multi-pinhole mask (mask) in which multiple pinholes are formed.

[0017] However, even if the image is an MPH image, if the image is leaked, it may be restored to an image with the blur removed. For example, depending on the characteristics of the mask equipped on the camera, restoration may occur.

[0018] Therefore, the inventors of the present application conducted extensive research into image processing devices and the like that can prevent images, such as MPH images, that were already privacy-protected at the time they were acquired from being leaked, from being restored, and have devised the image processing devices and the like described below.

[0019] An image processing device according to one embodiment of the present disclosure includes an image acquisition unit that acquires a first captured image from a first imaging device having a mask with one or more openings formed therein, an information acquisition unit that acquires opening mode information corresponding to the modes of the one or more openings, a noise addition unit that adds noise determined from the opening mode information to the first captured image, and an output unit that outputs the first captured image to which the noise has been added.

[0020] As a result, noise corresponding to the state of the mask opening is added to the captured image. In the captured image to which such noise is added, the frequency components of the captured image and the frequency components of the noise are difficult to separate, compared to noise added regardless of the state of the mask opening. Therefore, according to the image processing device, when the image to which noise is added is leaked, it is possible to suppress restoration.

[0021] Also, for example, the noise imparting section may impart the noise having a frequency band wider than a predetermined band to the first captured image.

[0022] This widens the frequency band where the noise frequency components and the captured image frequency components overlap, making it more difficult to separate the captured image frequency components and the noise frequency components. Therefore, when an image with noise is leaked, it is more difficult to restore the image.

[0023] Also, for example, the image forming apparatus may further include a noise information determination section that selects the noise to be added by the noise adding section in accordance with the opening mode information.

[0024] This allows a noise to be added to the captured image by selecting from a plurality of noises according to the state of the opening, so that a noise more suitable for the captured image can be added. In other words, the frequency components of the noise are more difficult to remove.

[0025] Also, for example, the mask may be configured to be switchable between a first opening mode and a second opening mode different from the first opening mode, and may further include a switching unit that switches the opening mode of the mask from one of the first opening mode and the second opening mode to the other.

[0026] This allows the mask opening pattern to be switched, so that if an image with noise is leaked, it can be prevented from being restored, compared to when there is only one mask opening pattern.

[0027] Furthermore, for example, the noise may include at least one of salt-and-pepper noise, Laplace noise, noise that changes an output value of a partial area of ​​the first captured image to a constant value, white noise, and pink noise.

[0028] This makes it possible to prevent an image with added noise from being restored if it is leaked by simply adding at least one of salt-and-pepper noise, Laplace noise, noise that changes the output value of a portion of the captured image to a constant value, white noise, and pink noise to the captured image.

[0029] Furthermore, for example, when the frequency characteristic of the mask based on the opening mode corresponding to the opening mode information is broad across a first band and a second band that is higher than the first band, the noise information determination unit may select at least one of salt-and-pepper noise and a noise that changes an output value of a portion of the first captured image to a constant value as the noise to be added by the noise adding unit, and when the intensity of the frequency components of the first band is higher than the intensity of the frequency components of the second band, the noise information determination unit may select Laplace noise as the noise to be added by the noise adding unit.

[0030] This allows noise having the same frequency characteristics as the frequency characteristics of the mask to be added to the captured image, which further reduces the possibility of the image with the added noise being restored if the image is leaked.

[0031] Furthermore, for example, the opening mode information may include at least one of a PSF (Point Spread Function), a size and a shape of the opening, and information on the multiple openings in the mask.

[0032] This makes it possible to prevent restoration of an image with noise added when it is leaked, simply by obtaining at least one of the PSF, the size and shape of the opening, and information regarding the multiple openings in the mask.

[0033] Furthermore, for example, the first imaging device may be any one of a multi-pinhole camera, a lensless camera, and a coded aperture camera.

[0034] This makes it possible to add noise to images captured by a multi-pinhole camera, lensless camera, or coded aperture camera, thereby preventing images captured by the multi-pinhole camera, lensless camera, or coded aperture camera from being restored if they are leaked.

[0035] In addition, a method for training a machine learning model according to one embodiment of the present disclosure acquires a dataset including images generated by adding noise determined from opening mode information corresponding to the modes of one or more openings to an image acquired from an imaging device equipped with a mask having one or more openings, and trains a machine learning model using the acquired dataset.

[0036] This makes it possible to generate a trained model that can accurately identify objects even when the above-mentioned noise is present.

[0037] Furthermore, an identification device according to one embodiment of the present disclosure includes an image acquisition unit that acquires the first captured image to which the noise has been added from an image processing device described in any one of claims 1 to 8, and an identification unit that identifies an object appearing in the first captured image to which the noise has been added by using a machine learning model trained with a dataset including an image generated by adding noise determined from the opening mode information to a second captured image acquired from a second imaging device equipped with the mask.

[0038] This makes it possible to suppress a decrease in the classification performance of the classification device for a captured image containing noise. In other words, it is possible to realize a classification device that can accurately classify an object even in a captured image containing noise.

[0039] Moreover, an image processing method according to one aspect of the present disclosure includes acquiring an image from an imaging device having a mask with one or more openings formed therein, acquiring opening mode information corresponding to the modes of the one or more openings, adding noise determined from the opening mode information to the image, and outputting the image with the noise added.

[0040] This provides the same effects as the image processing device described above.

[0041] Furthermore, these comprehensive or specific aspects may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0042] Hereinafter, a specific example of an image processing device according to an aspect of the present disclosure will be described with reference to the drawings. Each of the embodiments shown here shows a specific example of the present disclosure. Therefore, the numerical values, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim will be described as optional components.

[0043] In addition, each figure is a schematic diagram and is not necessarily illustrated precisely. Therefore, for example, the scales in each figure do not necessarily match. In addition, in each figure, substantially the same configurations are given the same reference numerals, and duplicated explanations are omitted or simplified.

[0044] In addition, in this specification, terms indicating relationships between elements such as "identical" and numerical values ​​and numerical ranges are not expressions that only indicate the strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent (e.g., about 5%).

[0045] (Embodiment 1) Hereinafter, an information processing system including an image processing device according to the present embodiment will be described with reference to FIGS.

[0046] [1-1. Information processing system configuration] First, the configuration of an information processing system according to the present embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a block diagram showing the functional configuration of an information processing system 1 according to the present embodiment. Fig. 2 is a block diagram showing the configuration of a sensing device 10 according to the present embodiment. The information processing system 1 is a system for identifying an object appearing in an MPH image, for example, by using an image in which noise is further added to the MPH image.

[0047] 1, the information processing system 1 includes a sensing device 10, a classification device 20, and a training device 30. The sensing device 10 and the classification device 20, and the classification device 20 and the training device 30 are communicatively connected to each other. The sensing device 10 and the training device 30 may also be communicatively connected to each other.

[0048] The sensing device 10 obtains an image that can be prevented from being restored when an image with noise added thereto is leaked. In the present embodiment, the sensing device 10 generates an image in which a predetermined noise is added to an MPH image as an image that can be prevented from being restored.

[0049] The sensing device 10 has an MPH image capturing unit 11, a noise adding unit 12, and a transmission unit 13. Including at least the noise adding unit 12 and the transmission unit 13, an information processing device (image processing device) for performing image processing is realized.

[0050] The MPH image capturing unit 11 acquires an image with privacy protection. In this embodiment, the MPH image capturing unit 11 acquires an MPH image as an image with privacy protection. The MPH image capturing unit 11 has a multi-pinhole mask 15 in which a plurality of pinholes 15a are formed, and is capable of acquiring a plurality of images (pinhole images: PH images) with different viewpoints of an object (subject) in a single imaging operation by superimposing them. In the MPH image capturing unit 11, an image acquired by an imaging operation is an image with privacy protection already. In other words, the MPH image capturing unit 11 does not acquire a normal image with no privacy protection.

[0051] This makes it possible to prevent the privacy of the object from being violated by the leakage of the normal image itself, whose privacy is not protected. In the following, the multi-pinhole mask 15 is also referred to as the MPH mask 15. The MPH image capturing unit 11 is an example of an imaging device, the multi-pinhole mask 15 is an example of a mask, and the MPH image is an example of a captured image.

[0052] The normal image is an image captured by a normal imaging device that does not have the MPH mask 15, and is an image obtained by imaging light from an object collected by an optical system. An example of an optical system is a lens. When a person looks directly at an object in an environment in which the object exists, the person perceives the object in a state almost similar to that of a normal image. In other words, a person visually recognizes a normal image captured by a normal imaging device in the same way as in the state of real space. The normal imaging device is, for example, a camera having a lens.

[0053] 2, the MPH image capturing unit 11 is realized by, for example, a lens 14, an MPH mask 15, and an image sensor 16. The MPH image capturing unit 11 is, for example, a multi-pinhole camera. Note that the MPH image capturing unit 11 is not limited to being a multi-pinhole camera, and may be a lensless camera, a coded aperture camera, or the like.

[0054] The lens 14 focuses the incident light onto the image sensor 16. The lens 14 is realized by, for example, a convex lens.

[0055] The MPH mask 15 is a mask in which multiple pinholes 15a (multi-pinholes) are formed. The MPH mask 15 is disposed at a certain distance from the image sensor 16 on the object side. The MPH mask 15 is disposed, for example, between the lens 14 and the image sensor 16, but the position of the MPH mask 15 is not limited thereto. That is, light that has passed through the MPH mask 15 is incident on the image sensor 16. The pinhole 15a is an example of an opening.

[0056] The multiple pinholes 15a are arranged randomly or at equal intervals. The size and shape of the pinholes 15a, the position, number, and distance between adjacent pinholes 15a formed in the MPH mask 15 are an example of opening mode information corresponding to the mode of the opening (mask pattern). The opening mode information includes at least one of the size and shape of the pinhole 15a, the position, number, and distance between adjacent pinholes 15a formed in the MPH mask 15. At least one of the position, number, and distance between adjacent pinholes 15a is an example of information about the multiple pinholes 15a. In the following, the opening mode information is also referred to as MPH information.

[0057] The position of the pinhole 15a affects the position of the object projected onto the image sensor 16, and the size and number of the pinholes 15a affect the blur of the MPH image.

[0058] The number of the pinholes 15a is not particularly limited as long as it is, for example, two or more pinholes. The opening state may differ for each MPH image capturing unit 11, each protection target, each usage scene, etc. For each usage scene, information on the target to be concealed, such as concealing a face or concealing the entire background, is included.

[0059] The aperture mode information is not limited to information indicating the size, shape, etc. of the aperture as described above. The aperture mode information may include a PSF (Point Spread Function). The PSF indicates how an image of a point light source is blurred (deteriorated) by the optical system when the image is captured, and is a function that represents the intensity distribution when the image is blurred. It can also be said that the PSF is a function that represents how light rays spread when an ideal point light source passes through the optical system.

[0060] In this way, the aperture mode information may include information regarding the mode of the pinhole 15a, or may include information regarding the MPH image obtained depending on the aperture mode.

[0061] The image sensor 16 receives light that has passed through the MPH mask 15. It can also be said that the image sensor 16 acquires an image of the object (MPH image) through each pinhole 15a. The MPH image is an image acquired through each pinhole 15a. The acquired PH image differs depending on the position and size of each pinhole 15a. The image sensor 16 acquires a superimposed image (encoded image) of multiple PH images as an MPH image. The MPH image may be an image that cannot be visually recognized by humans, but by using computer processing, it is possible to acquire information contained in the image of the object to be imaged and the surrounding environment.

[0062] 1 again, the noise adding unit 12 adds noise to the MPH image obtained by the MPH image capturing unit 11. For example, the noise adding unit 12 adds noise determined from the opening mode information of the MPH mask 15 to the MPH image by image processing. For example, the noise adding unit 12 adds noise having the same frequency characteristics as the passing frequency characteristics of the MPH mask 15. It can also be said that the noise adding unit 12 adds noise according to the opening mode of the MPH mask 15.

[0063] The MPH image may have, for example, flat frequency characteristics (spatial frequency characteristics). Therefore, the noise adding unit 12 adds, for example, noise with flat frequency characteristics to the MPH image. Noise with flat frequency characteristics is noise (broad noise) whose frequency band (spatial frequency band) is wider than a predetermined band. It can also be said that noise with flat frequency characteristics is noise whose deviation in frequency characteristics is equal to or less than a threshold in a predetermined band. The predetermined band is a band based on the form of the opening of the MPH mask 15, and includes, for example, a low band and a high band that is higher than the low band. The predetermined band is a band that at least partially overlaps with the frequency band of the MPH image, and may be, for example, a band that includes all of the frequency band of the MPH image. Broad noise means, for example, noise whose intensity of frequency components in a predetermined band is within a predetermined range, or whose variation in intensity of frequency components in a predetermined band is within a predetermined range.

[0064] The noise added by the noise adding unit 12 may include at least one of salt-and-pepper noise, Laplace noise, noise in which the output value (pixel value) of a part of the MPH image is changed to a constant value (by cutout), white noise, and pink noise. The noise added by the noise adding unit 12 may also include at least one of scratch noise, which is a scratch-like noise like an old film, noise by random erasing that masks (blocks) a part of the MPH image, colored noise (for example, noise other than white noise), and frequency-biased noise in which the deviation of the frequency characteristics exceeds a threshold value.

[0065] In addition, an image to which sparse noise is added in the frequency space is easier to restore than an image to which broad noise is added in the frequency space. Therefore, the noise adding unit 12 does not add sparse noise in the frequency space to the MPH image. For example, the noise adding unit 12 does not add impulse noise in the frequency space to the MPH image.

[0066] 2, the noise adding unit 12 is realized by, for example, an ISP (Image Signal Processor) 17. Sparse noise is, for example, noise that has frequency components only in either the low frequency range or the high frequency range.

[0067] It should be noted that the noise adding unit 12 is not limited to adding noise by image processing. The noise added by the noise adding unit 12 may be, for example, noise caused by hot pixels and dead pixels, shot noise, etc. In this case, the noise adding unit 12 may be included in the MPH image capturing unit 11 (for example, the image sensor 16). It should be noted that a hot pixel is noise in which some pixel values ​​become white, and a dead pixel is noise in which some pixel values ​​become black.

[0068] Referring again to FIG. 1, the transmission unit 13 outputs the MPH image to which noise has been added by the noise adding unit 12 to the classification device 20. The transmission unit 13 outputs the MPH image to which noise has been added by wireless communication, but may also output the MPH image to which noise has been added by wired communication. The transmission unit 13 is an example of an output unit. In the following, the MPH image to which noise has been added is also referred to as a noise-added image.

[0069] As shown in FIG. 2, the transmission unit 13 is realized by a communication unit 18 having a communication interface such as an adapter for transmitting the noise-added image to the identification device 20.

[0070] The classification device 20 uses a trained learning model (trained model) to classify an object appearing in a noise-added image. For example, the classification device 20 may use the trained model to detect an object appearing in a noise-added image and output a detection result. The detection of an object in a noise-added image is an example of classification. The trained model is also referred to as a classifier.

[0071] The identification device 20 includes a receiving unit 21, an identification unit 22, and an output unit 23.

[0072] The receiving unit 21 communicates with the sensing device 10 and the training device 30. The receiving unit 21 receives a noise-added image from the sensing device 10, and receives a trained model from the training device 30. The receiving unit 21 is configured to include a communication interface such as an adapter for receiving various information from the sensing device 10 and the training device 30.

[0073] The identification unit 22 acquires information on objects (e.g., a target object and the surrounding environment of the target object) in the noise-added image using the trained model. The identification unit 22, for example, identifies objects in the noise-added image and acquires the positions of the objects in the noise-added image. That is, the information on the objects is the identification result of the trained model, and may include the presence or absence of an object and the position of the object. Furthermore, identifying an object may include, for example, detecting pixels in which an object is present.

[0074] The discrimination unit 22 inputs a noise-added image to a trained model trained by the training unit 33, and obtains an output from the trained model as a discrimination result. The trained model trained by the training unit 33 has improved discrimination performance for noise-added images. Therefore, by using the trained model, the discrimination unit 22 can suppress a decrease in discrimination performance for the noise-added image.

[0075] For example, when the identification device 20 is mounted on an automobile, examples of the object are a person, an automobile, a bicycle, or a traffic light. The identification device 20 may identify one predetermined type of object or a plurality of types of objects using a noise-added image. The identification device 20 may also identify objects by category, such as a moving object including a person, an automobile, or a bicycle.

[0076] The output unit 23 outputs the identification result of the identification unit 22. The output unit 23 may present the identification result to a user. The output unit 23 includes, for example, a display device or a sound output device.

[0077] The above-mentioned sensing device 10 and identification device 20 may be mounted on the same device. For example, the sensing device 10 and the identification device 20 may be mounted on a moving object such as a vehicle or a robot, or may be mounted on a fixed object such as a surveillance camera system.

[0078] The training device 30 generates a trained model that the identification device 20 uses to identify objects. The training device 30 has an MPH information acquisition unit 31, a noise information acquisition unit 32, a training unit 33, and a transmission unit 34. The training device 30 may also have a reception unit that receives input from a user. The reception unit is realized by a button, a touch panel, or the like, but may also be realized by a device that receives input by voice, etc.

[0079] The MPH information acquisition unit 31 acquires MPH information (opening mode information) corresponding to the opening mode of the MPH mask 15 of the sensing device 10. The MPH information acquisition unit 31 may acquire the opening mode information from the sensing device 10, for example, or may acquire the MPH information by input from a user.

[0080] The noise information acquisition unit 32 acquires noise information indicating the noise to be added to the MPH image by the noise addition unit 12 of the sensing device 10. The noise information acquisition unit 32 may acquire the noise information from the sensing device 10, for example, or may acquire the noise information by input from a user.

[0081] The training unit 33 trains the learning model that identifies objects from the noise-added image generated by the noise adding unit 12. The training unit 33 trains the learning model by machine learning using a data set generated using the MPH information acquired by the MPH information acquiring unit 31 and the noise information acquired by the noise information acquiring unit 32. It can also be said that the training unit 33 trains the learning model using the data set. The learning model is an example of a machine learning model that identifies objects based on an image, and is, for example, a machine learning model using a neural network such as Deep Learning, but may be another machine learning model. For example, the machine learning model may be a machine learning model using Random Forest, Genetic Programming, or the like.

[0082] The data set includes training images, which are images corresponding to the noise-added images, and correct answer information for the training images. The correct answer information is reference data in machine learning and is appropriately determined depending on the purpose of use of the classification device 20, and is, for example, the type of object and the position of the object on the image. The data set may be generated by the training device 30 or another device, for example.

[0083] An image equivalent to the noise-added image may be, for example, an image obtained by adding noise specified by the noise information acquired by the noise information acquisition unit 32 to an MPH image captured by an imaging device equipped with an MPH mask 15 having the same MPH information as the MPH information acquired by the MPH information acquisition unit 31. In addition, when the MPH information includes a PSF, an image equivalent to the noise-added image may be, for example, an image obtained by adding noise specified by the noise information acquired by the noise information acquisition unit 32 to an image generated by convolving the PSF with a normal image.

[0084] Furthermore, machine learning is realized by, for example, backpropagation (BP) in deep learning. Specifically, the training unit 33 inputs training images to an untrained learning model and obtains a recognition result output by the learning model. Then, the training unit 33 adjusts the learning model so that the recognition result becomes correct answer information. The training unit 33 repeats such adjustment for a plurality of different training images (for example, several thousand sets) and the corresponding correct answer information, thereby improving the recognition accuracy of the learning model.

[0085] The transmission unit 34 outputs the trained model generated by the training unit 33 to the classification device 20. The transmission unit 13 is configured to include a communication interface such as an adapter for transmitting the trained model to the classification device 20. Note that outputting the trained model means outputting information such as network parameters and a computation algorithm (machine learning algorithm) in the trained model. Note that the algorithm is not particularly limited and may be any existing algorithm.

[0086] Here, various images will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining various images according to the present embodiment.

[0087] FIG. 3 shows an MPH image captured by the MPH image capturing unit 11 of an object (person) indicated by a correct image (normal image), an image with salt-and-pepper noise and an image with Poisson noise, which are examples of noise-added images, and restored images obtained by restoring these images. Each image is restored, for example, by the SelfDeblur method (Ren D, Zhang K, Wang Q, et al. Neural blind deconvolution using deep priors. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020.), which is a neural network model that performs blur removal processing on an input image and outputs a blur-free image corresponding to the input image. The three restored images are performed by the same neural network model. The image with salt-and-pepper noise is an image in which salt-and-pepper noise is added to an MPH image, and the image with Poisson noise is an image in which Poisson noise is added to an MPH image.

[0088] As shown in Fig. 3, the correct image is an image in which it is possible to recognize that a person is drinking a beverage, but the MPH image, the salt-and-pepper noise-added image, and the Poisson noise-added image may be images that cannot be visually recognized by humans. Note that the MPH image, the salt-and-pepper noise-added image, and the Poisson noise-added image are images from which information contained in the image, such as the object and the surrounding environment, can be obtained by computer processing.

[0089] The restored image of the MPH image is restored to the extent that it is possible to recognize that a person is drinking a beverage. In this way, the MPH image may be restored to an extent that is visually recognizable by humans.

[0090] On the other hand, the restored images obtained by restoring the salt-and-pepper noise-added image and the Poisson noise-added image are images in which it is not possible to recognize that a person is drinking a beverage. In other words, the restored images are not restored to a level that can be visually recognized by humans. The salt-and-pepper noise-added image and the Poisson noise-added image are images to which broad noise with a frequency band wider than a predetermined band has been added. Therefore, the frequency components of the MPH image and the frequency components of the noise overlap in a predetermined band, making it difficult to remove only the frequency components of the noise from the noise-added image. In particular, when converting a signal by convolution as in a multi-pinhole camera, broad noise in frequency space can make image restoration difficult.

[0091] In this way, by adding broad noise whose frequency band is wider than a predetermined band to the MPH image, it is possible to realize an image that is difficult to restore if leaked.

[0092] When the noise added by the noise adding unit 12 is salt-and-pepper noise, that is, when the noise information includes information indicating that salt-and-pepper noise is to be added, the training device 30 trains the learning model using the salt-and-pepper noise-added image. When the noise added by the noise adding unit 12 is either salt-and-pepper noise or Poisson noise, that is, when the noise information includes information indicating that either salt-and-pepper noise or Poisson noise is to be added, the training device 30 trains the learning model using each of the salt-and-pepper noise-added image and the Poisson noise-added image.

[0093] [1-2. Operation of information processing system] Next, the operation of the information processing system 1 configured as above will be described with reference to Fig. 4 to Fig. 6. First, the operation of the sensing device 10 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the operation of the sensing device 10 according to the present embodiment.

[0094] 4, the MPH image capturing unit 11 of the sensing device 10 captures an image of an object to obtain an MPH image (S11). The MPH image capturing unit 11 outputs the obtained MPH image to the noise adding unit 12.

[0095] Next, when the noise adding unit 12 acquires the MPH image, it adds noise to the MPH image determined from the MPH information of the MPH mask 15 at the time the MPH image was captured (S12). When the state of the opening of the MPH mask 15 cannot be changed, the noise adding unit 12 reads out noise corresponding to the state of the opening from a storage unit (not shown) and adds the read out noise to the MPH image. In addition, the noise adding unit 12 adds noise to the entire MPH image, for example, but may add noise to only a partial area of ​​the MPH image. The noise to be added is stored in advance in the storage unit of the sensing device 10. The noise adding unit 12 functions as an image acquiring unit that acquires the MPH image.

[0096] The noise adding unit 12 may store the noise-added image obtained by adding noise to the MPH image in the storage unit. The noise adding unit 12 may not store the MPH image before noise addition, that is, the MPH image captured by the MPH image capturing unit 11, in the storage unit. The noise adding unit 12 may overwrite the MPH image with the noise-added image and store it, or may delete the MPH image if the MPH image is stored. This makes it possible to prevent the MPH image from leaking out of the sensing device 10.

[0097] Next, the transmission unit 13 transmits the MPH image (noise-added image) to which noise has been added by the noise adding unit 12 to the classification device 20 (S13). The transmission unit 13 transmits, for example, the salt-and-pepper noise-added image or the Poisson noise-added image shown in FIG. 3 to the classification device 20.

[0098] As described above, the sensing device 10 of the information processing system 1 generates a noise-added image by adding noise determined from the MPH information to the MPH image. As described in FIG. 3, such a noise-added image is more difficult to restore than an MPH image. Therefore, the sensing device 10 can suppress restoration of an image with noise added when the image is leaked.

[0099] Next, the operation of the training device 30 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the operation of the training device 30 according to this embodiment. The operation shown in Fig. 5 is executed, for example, before the operation shown in Fig. 4.

[0100] 5, first, the training device 30 acquires MPH information and noise information of the sensing device 10 (S21). Specifically, the MPH information acquisition unit 31 acquires the MPH information, and the noise information acquisition unit 32 acquires the noise information. When the mode of the pinhole 15a and the noise to be applied are exclusively associated with each other, the training device 30 may acquire at least one of the MPH information and the noise information from an external device or a user input.

[0101] In addition, when the opening state of the pinhole 15a in the MPH mask 15 is changeable, the MPH information includes information corresponding to each of the changeable opening states.

[0102] Next, the training unit 33 of the training device 30 acquires a training dataset (S22). The training unit 33 acquires a dataset corresponding to the MPH information and noise information acquired in step S21. The dataset may be generated by the training unit 33 based on the MPH information and noise information. Alternatively, a plurality of datasets may be stored in the training device 30, and the training unit 33 may read out a dataset corresponding to the MPH information and noise information acquired in step S21. Alternatively, the training unit 33 may acquire the dataset from an external device (for example, a device that manages various datasets).

[0103] If the opening state of the pinhole 15a is changeable, a data set corresponding to each of the changeable opening states may be acquired.

[0104] Next, the training unit 33 performs a learning process using the data set (S23). In step S23, the training unit 33 uses the data set to train the learning model by machine learning. The learning process is performed by, but is not limited to, a backpropagation method and may be performed by any known method.

[0105] Here, the noise added by the noise adding unit 12 is broad in frequency space, but is impulse-like noise in real space. Impulse-like noise in real space has little effect on training of the trained model, so that deterioration of the discrimination performance can be suppressed.

[0106] In addition, when the manner of the opening of the MPH mask 15 is changeable, the training unit 33 may perform the learning process of step S23 for one learning model using a plurality of data sets including training images with different noises added to each other, or may perform a learning process to generate a plurality of learning models corresponding to the plurality of data sets. This makes it possible to suppress a decrease in the object identification performance using the noise-added image even when the manner of the opening of the MPH mask 15 is changed, that is, when the noise added by the noise adding unit 12 is changed.

[0107] Next, the transmission unit 34 transmits the learned model trained by the training unit 33 to the classification device 20 (S24). The process of step S24 is performed, for example, before the sensing device 10 is shipped from a factory.

[0108] Next, the operation of the identification device 20 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the operation of the identification device 20 of the information processing system 1 according to the present embodiment.

[0109] As shown in Fig. 6, first, the receiving unit 21 of the classification device 20 receives the trained model transmitted from the training device 30 in step S24 shown in Fig. 5 (S31). The receiving unit 21 stores the received trained model in a storage unit (not shown). The receiving unit 21 functions as an acquisition unit.

[0110] Next, the receiving unit 21 of the identification device 20 receives the MPH image (noise-added image) to which noise has been added, transmitted from the sensing device 10 in step S13 shown in Fig. 4 (S32). The receiving unit 21 stores the received noise-added image in a storage unit (not shown).

[0111] Next, the discrimination unit 22 executes discrimination processing on the noise-added MPH image using the trained model (S33). The discrimination unit 22 inputs the noise-added MPH image into the trained model and obtains an output as a discrimination result. Since the trained model is a trained model trained using training images corresponding to the noise added by the noise adding unit 12, it is possible to perform discrimination processing on the noise-added image with high accuracy.

[0112] Next, the output unit 23 outputs the classification result (S34). The output unit 23 presents the classification result to the user, for example, by an image, a sound, or the like.

[0113] As described above, the classification device 20 of the information processing system 1 performs classification using a learned model trained by the training device 30 using MPH images with noise, thereby preventing a decrease in classification performance for MPH images with noise.

[0114] The information processing system 1 according to this embodiment can be a system that can simultaneously prevent an image with noise added thereto from being restored when the image is leaked, and prevent a deterioration in recognition performance due to the noise.

[0115] (Embodiment 2) [2-1. Information processing system configuration] The configuration of an information processing system 1 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a block diagram showing a functional configuration of a sensing device 110 according to the present embodiment. The information processing system 1 according to the present embodiment differs from the information processing system 1 according to the embodiment in that the information processing system 1 according to the present embodiment includes a sensing device 110 instead of the sensing device 10. Hereinafter, the sensing device 110 according to the present embodiment will be described, focusing on the differences from the sensing device 10 according to the embodiment. Furthermore, the same reference numerals as the sensing device 10 are used for configurations that are the same as or similar to the sensing device 10 according to the first embodiment, and descriptions thereof will be omitted or simplified.

[0116] As shown in FIG. 7, the sensing device 110 includes an MPH information acquisition unit 111 and a noise information determination unit 112 in addition to the components of the sensing device 10 according to the first embodiment.

[0117] The MPH information acquisition unit 111 acquires MPH information (opening mode information) corresponding to the opening modes of the pinholes 15a. The MPH information acquisition unit 111 is an example of an information acquisition unit.

[0118] The noise information determination unit 112 selects the noise to be added by the noise adding unit 12 according to the MPH information. The noise information determination unit 112 determines the noise to be added by the noise adding unit 12 according to, for example, the frequency characteristics (pass frequency characteristics) of the MPH mask 15. For example, the noise information determination unit 112 selects a noise having the same frequency characteristics as the frequency characteristics of the mask from among a plurality of noises, and adds the selected noise to the MPH image.

[0119] For example, when the frequency characteristic of the MPH mask 15 is flat, the noise information determination unit 112 determines salt-and-pepper noise and noise in which the output value (pixel value) of a part of the MPH image is changed to a constant value (due to cutout) as noise to be added to the MPH image. Also, when the frequency characteristic (pass frequency characteristic) of the MPH mask 15 is biased toward the low frequency range, the noise information determination unit 112 determines noise whose frequency characteristic is biased toward the low frequency range, such as Laplace noise, as noise to be added to the MPH image.

[0120] [2-2. Operation of sensing device] Next, the operation of the information processing system 1 configured as above will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the operation of the sensing device 110 according to this embodiment. Note that operations that are the same as or similar to those shown in Fig. 4 of the first embodiment are given the same reference numerals as in Fig. 4, and descriptions thereof will be omitted or simplified.

[0121] As shown in Fig. 8, the MPH information acquisition unit 111 of the sensing device 110 acquires MPH information of the MPH mask 15 possessed by the MPH image capturing unit 11 (S41). When the state of the opening of the MPH mask 15 is fixed (cannot be changed), the MPH information acquisition unit 111 may acquire the MPH information once, for example. The MPH information acquisition unit 111 outputs the acquired MPH information to the noise information determination unit 112. The MPH information acquisition unit 111 stores the acquired MPH information in a storage unit (not shown).

[0122] Next, the noise information determination unit 112 of the sensing device 110 selects a noise to be added by the noise adding unit 12 from among a plurality of noises based on the MPH information acquired by the MPH information acquisition unit 111, thereby determining the noise information to be output to the noise adding unit 12 (S42). When the frequency characteristics of the MPH mask 15 are flat in a frequency band including a low frequency band and a high frequency band higher than the low frequency band, the noise information determination unit 112 determines the noise information based on the MPH information so that a noise with a flatter frequency characteristic is added. An example of a noise with a flatter frequency characteristic is salt-and-pepper noise, or a noise in which the output value (pixel value) of a part of the MPH image is changed to a constant value (due to cutout). The low frequency band is an example of a first band, and the high frequency band is an example of a second band.

[0123] Furthermore, when the frequency characteristics of the MPH mask 15 include a large number of low-frequency components among the low and high frequencies, the noise information determination unit 112 determines the noise information based on the MPH information so that noise whose frequency characteristics are closer to the low frequency range is added. The noise whose frequency characteristics are closer to the low frequency range is, for example, Laplace noise.

[0124] In this way, when the frequency characteristics of the MPH mask 15 based on the opening configuration corresponding to the MPH information are broad across a first band and a second band that is higher than the first band, the noise information determination unit 112 selects at least one of salt-and-pepper noise and noise that changes the output value of a portion of the MPH image to a constant value as the noise to be added by the noise adding unit 12, and when the intensity of the frequency components in the first band is higher than the intensity of the frequency components in the second band, the noise information determination unit 112 selects Laplace noise as the noise to be added by the noise adding unit 12.

[0125] The frequency characteristic being broad across the first and second bands means, for example, that the difference between the intensity of the frequency components in the first band and the intensity of the frequency components in the second band is less than a predetermined value. The intensity of the frequency components in the first band is higher than the intensity of the frequency components in the second band means, for example, that the intensity of the frequency components in the first band is higher than the intensity of the frequency components in the second band by a predetermined value or more. For example, the above determination may be made by comparing the average value of the intensity of the frequency components in the first band with the average value of the intensity of the frequency components in the second band.

[0126] In step S42, the noise information determination unit 112 may determine, based on the MPH information, whether the frequency characteristics of the MPH mask 15 are flat in a frequency band including a low frequency band and a high frequency band higher than the low frequency band, and whether the frequency characteristics of the MPH mask 15 include many low frequency components out of the low frequency band and the high frequency band, and may determine the noise to be added based on the determination result. For example, if the positions and sizes of the pinholes 15a are random and the number of pinholes 15a (for example, the number in a predetermined area) is equal to or greater than a predetermined number, the noise information determination unit 112 may determine that the frequency characteristics of the MPH mask 15 are flat in a frequency band including a low frequency band and a high frequency band. Also, if the number of pinholes 15a (for example, the number in a predetermined area) is less than a predetermined number, the noise information determination unit 112 may determine that the MPH mask 15 includes many low frequency components. Also, the noise information determination unit 112 may have a table in which MPH information and mask information are associated with each other, and may select the noise to be added based on the table.

[0127] The noise information includes information for specifying the noise to be added by the noise adding unit 12. The noise information determination unit 112 outputs the determined noise information to the noise adding unit 12.

[0128] Next, the noise adding unit 12 adds noise to the MPH image based on the noise information (S43). The noise adding unit 12 reads out the noise specified by the noise information from the storage unit, and adds the read out noise to the MPH image.

[0129] As a result, noise according to the frequency characteristics of the MPH mask 15 is added to the MPH image, making it possible to more effectively generate a noise-added image that is difficult to restore.

[0130] (Modification of the second embodiment) The configuration of the information processing system 1 according to this modification will be described with reference to FIG. 9. FIG. 9 is a block diagram showing the functional configuration of a sensing device 110a according to this modification. The information processing system 1 according to this modification differs from the information processing system 1 according to the second embodiment in that the information processing system 1 according to this modification includes a sensing device 110a instead of the sensing device 110. Hereinafter, the sensing device 110a according to this modification will be described, focusing on the differences from the sensing device 110 according to the second embodiment. Furthermore, the same reference numerals as the sensing device 110 are used for configurations that are the same as or similar to the sensing device 110 according to the second embodiment, and descriptions thereof will be omitted or simplified.

[0131] As shown in FIG. 9, the sensing device 110a has an MPH information setting unit 111a instead of the MPH information acquisition unit 111 of the sensing device 110 according to the second embodiment. In addition, the MPH image capturing unit 11 according to this modification is configured to be able to dynamically change the state of the aperture (mask pattern). The MPH image capturing unit 11 is configured to be able to change, for example, between a first aperture state (first mask pattern) and a second aperture state (second mask pattern) different from the first aperture state. In other words, the MPH image capturing unit 11 is capable of switching the state of the aperture. The MPH mask 15 of the MPH image capturing unit 11 is realized by, for example, an SLM (Spatial Light Modulator) having a liquid crystal shutter or the like. In this case, the aperture is realized by a transparent portion of the liquid crystal shutter.

[0132] The MPH information setting unit 111a controls the state of the opening of the MPH mask 15 of the MPH image capturing unit 11. The MPH information setting unit 111a performs control to switch the state of the opening of the MPH mask 15, for example. The MPH information setting unit 111a changes the state of the opening of the MPH mask 15 over time, for example. For example, the MPH information setting unit 111a may switch the state of the opening at a predetermined time interval, may switch the state of the opening in response to an operation by a user, or may switch the state of the opening based on a preset schedule. The schedule includes information indicating the change over time in the state of the opening.

[0133] It can also be said that the MPH information setting unit 111a switches the mode of the openings of the MPH mask 15 from one of the first opening mode and the second opening mode different from the first opening mode to the other. The number of opening modes that the MPH information setting unit 111a can set is not particularly limited as long as it is two or more. The first opening mode and the second opening mode differ in at least one of the positions, number, and distance between adjacent openings, for example. The MPH information setting unit 111a functions as a switching unit.

[0134] The MPH information setting unit 111a acquires MPH information by switching the state of the opening. The MPH information setting unit 111a can also be said to be an MPH information acquisition unit.

[0135] The MPH image capturing unit 11 controls the MPH mask 15 (for example, an SLM) to realize the aperture state set by the MPH information setting unit 111a.

[0136] The noise information determination unit 112 selects noise according to the switched opening mode each time the MPH information setting unit 111a switches the opening mode. The noise information determination unit 112, for example, changes the noise to be added to the MPH image over time. The noise information determination unit 112 may have information on different noises, the number of which is equal to or greater than the number of opening modes that the MPH information setting unit 111a can set, and may set different noises for each of the multiple opening modes.

[0137] In addition, when the opening mode of the MPH mask 15 of the MPH image capturing unit 11 is switchable as in this modified example, for example, in the training device 30, training of the learning model is performed using a data set including training images corresponding to each of the switchable opening modes.

[0138] Next, the operation of the information processing system 1 configured as above will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the operation of the sensing device 110a according to this modification. Note that the same reference numerals as in Fig. 8 are used for operations that are the same as or similar to those shown in Fig. 8 of the second embodiment, and descriptions thereof will be omitted or simplified.

[0139] 10, the MPH information setting unit 111a of the sensing device 110a sets MPH information of the MPH mask 15 held by the MPH image capturing unit 11 (S51). The MPH information setting unit 111a sets the MPH information of the MPH mask 15 by selecting the opening mode currently used from a list including a plurality of opening modes for the MPH mask 15. The MPH information setting unit 111a outputs the MPH information to each of the MPH image capturing unit 11 and the noise information determination unit 112.

[0140] Note that, before step S51, the MPH information setting unit 111a may determine whether or not the MPH information needs to be switched, and if it is determined that the MPH information needs to be switched, execute the process of step S51, and if it is determined that the MPH information does not need to be switched, execute the process of step S11 and subsequent steps without executing the process of step S51 (without switching the MPH information). For example, the determination may be made based on the elapsed time since the MPH information was switched, or based on whether or not an input from the user has been received. Step S51 can also be said to be a process of updating the MPH information of the MPH mask 15 of the MPH image capturing unit 11.

[0141] Next, the MPH image capturing unit 11 captures an image of the object based on the MPH information set by the MPH information setting unit 111a to obtain an MPH image (S11). The MPH image capturing unit 11 switches the aperture state of the multi-pinhole mask 15 based on the MPH information set in step S51, and captures an image of the object using the switched aperture state.

[0142] The MPH image capturing unit 11 outputs the acquired MPH image to the noise adding unit 12. The MPH image capturing unit 11 may output the MPH image and MPH information indicating the state of the mouth when the MPH image was captured to the noise adding unit 12 in association with each other.

[0143] Next, the noise information determination unit 112 selects the noise to be added by the noise adding unit 12 based on the MPH information (S42). When the MPH information is set in step S51, the noise information determination unit 112 discards the previous MPH information in step S42, and selects noise according to the newly set MPH information using the newly set MPH information.

[0144] This allows the opening state of the MPH mask 15 to be switched, so that the sensing device 110a can better suppress the restoration of the leaked image compared to when the MPH mask 15 has only one opening state.

[0145] The process of step S42 may be performed before acquiring the MPH image. The process of step S42 may be performed, for example, between step S51 and step S11.

[0146] As a result, the noise added to the MPH image changes over time, so that the sensing device 110a can generate a noise-added image that is more difficult to restore. In addition, by training the learning model using images that include each of the noises added over time, it is possible to suppress a decrease in the discrimination performance in noise-added images in which different noises are added over time.

[0147] (Other embodiments) The present disclosure has been described above based on the embodiment and each modified example (hereinafter, also referred to as the embodiment, etc.), but the present disclosure is not limited to the above-mentioned embodiment, etc. As long as it does not deviate from the gist of the present disclosure, various modifications conceived by a person skilled in the art to the present embodiment, etc., and forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0148] For example, in the above embodiment and the like, the noise imparted by the noise imparting unit is described as being of one type, but the noise may be a combination of at least two of the noises exemplified in the above embodiment 1. The noise imparted image may be an MPH image to which two or more noises have been imparted.

[0149] Furthermore, the MPH image capturing unit in the above-mentioned embodiment and the like may be a lensless camera. The lensless camera has a transmission film on which a specific pattern such as a multi-pinhole or dot pattern is printed instead of a lens. The area in the transmission film that transmits light is an example of an opening. Furthermore, the transmission film is an example of a mask.

[0150] Furthermore, the MPH image capturing unit in the above-mentioned embodiments may be a coded aperture camera. A coded aperture camera is a camera using a coded aperture. In a camera using a coded aperture, a mask having a mask pattern (aperture shape) is arranged on the object side of the image sensor. The coded aperture has a function of blocking incident light by the mask pattern, and the PSF of the image is controlled by the mask pattern. The area in the mask that transmits light is an example of an aperture.

[0151] The shape of the openings formed in the mask in the above-mentioned embodiments is not particularly limited. The shape of the openings may be circular, doughnut-shaped, or may have a predetermined pattern. The number of openings formed in the mask may be one or more.

[0152] In addition, in the above-described embodiment and the like, an example has been described in which the object identification process using the trained model is performed in the identification device, but the object identification process may be performed in a sensing device. For example, the identification device may be built into the sensing device.

[0153] In the above embodiment, the training device trains the learned model to output an identification result of an object in a noise-added image, but may train the learned model to restore the noise-added image. The training device may train the learned model to input a noise-added image and output a correct answer image (normal image) corresponding to the noise-added image. In this case, the data set includes a training image that is an image corresponding to the MPH image to which noise has been added, and a correct answer image for the training image.

[0154] The order of the processes described in the above embodiment is merely an example. The order of the processes may be changed, the processes may be executed in parallel, or some of the processes may not be executed.

[0155] In addition, the division of functional blocks in the block diagram is one example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, some functions may be transferred to other functional blocks, and the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0156] Furthermore, each device included in the information processing system may be realized as a single device, or may be realized by multiple devices. When the information processing system is realized by multiple devices, the components included in the information processing system may be distributed in any manner among the multiple devices. Furthermore, the communication method between the multiple devices may be wired communication or wireless communication. Furthermore, the communication standard used for communication is not particularly limited.

[0157] Moreover, each of the components described in the above embodiments may be realized as software, or may be realized as an LSI, which is typically an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip to include some or all of them. Here, LSI is used, but it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration. Furthermore, the method of integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. After LSI manufacture, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connection or setting of the circuit cells inside the LSI may be used. Furthermore, if an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology or a different derived technology, it is natural that the integration of the components may be performed using that technology.

[0158] Furthermore, the technology of the present disclosure may be the above-mentioned program, or a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. Needless to say, the above-mentioned program can be distributed via a transmission medium such as the Internet. For example, the above-mentioned program and a digital signal consisting of the above-mentioned program may be transmitted via a telecommunication line, a wireless or wired communication line, a network such as the Internet, data broadcasting, or the like. The above-mentioned program and a digital signal consisting of the above-mentioned program may be executed by another independent computer system by being recorded on a recording medium and transferred, or by being transferred via a network, or the like.

[0159] In the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. [Industrial Applicability]

[0160] The present disclosure is widely applicable to devices that use images to identify objects. [Explanation of symbols]

[0161] 1. Information Processing Systems 10, 110, 110a Sensing device 11 MPH image capture unit 12 Noise Addition Section 13, 34 Transmitter 14 Lenses 15 Multi-Pinhole Mask 15a Pinhole 16 Image Sensor 17 ISP 18 Communications Department 20 Identification device 21 Receiving section 22 Identification section 23 Output section 30 Training equipment 31 MPH information acquisition section 32 Noise information acquisition section 33 Training Department 111 MPH information acquisition section 111a MPH information setting section 112 Noise information determination unit

Claims

1. an image acquisition unit that acquires a first captured image from a first imaging device having a mask with one or more openings formed therein; An information acquisition unit that acquires opening mode information corresponding to the one or more opening modes; a noise adding unit that adds noise determined from the opening mode information to the first captured image; an output unit that outputs the first captured image to which the noise has been added; Image processing device.

2. The noise adding unit adds the noise having a frequency band wider than a frequency band based on the mode of the one or more openings to the first captured image. The image processing device according to claim 1 .

3. a noise information determination unit that selects the noise to be added by the noise adding unit in accordance with the opening mode information.

3. The image processing device according to claim 1 or 2.

4. the mask is configured to be switchable between a first opening state and a second opening state different from the first opening state, The mask further includes a switching unit that switches the state of the opening of the mask from one of the first opening state and the second opening state to the other. The image processing device according to any one of claims 1 to 3.

5. The noise includes at least one of salt-and-pepper noise, Laplace noise, noise that changes an output value of a partial area of ​​the first captured image to a constant value, white noise, and pink noise. The image processing device according to any one of claims 1 to 4.

6. The noise information determination unit selects at least one of salt-and-pepper noise and a noise that changes an output value of a partial area of ​​the first captured image to a constant value as the noise to be added by the noise adding unit when a frequency characteristic of the mask based on the opening mode corresponding to the opening mode information is broad across a first band and a second band that is a band higher than the first band, and selects Laplace noise as the noise to be added by the noise adding unit when an intensity of a frequency component in the first band is higher than an intensity of a frequency component in the second band. The image processing device according to claim 3 .

7. The aperture mode information includes at least one of a PSF (Point Spread Function), a size and a shape of the aperture, and information on the plurality of apertures in the mask. The image processing device according to any one of claims 1 to 6.

8. The first imaging device is any one of a multi-pinhole camera, a lensless camera, and a coded aperture camera. The image processing device according to any one of claims 1 to 7.

9. A data set is obtained that includes an image generated by adding noise determined from aperture mode information corresponding to the modes of the one or more apertures to an image acquired from an imaging device equipped with a mask in which one or more apertures are formed; Training a machine learning model using the obtained dataset. How to train a machine learning model.

10. an image acquisition unit that acquires the first captured image to which the noise has been added from the image processing device according to any one of claims 1 to 8; and an identification unit that uses a machine learning model trained with a data set including an image generated by adding noise determined from the opening mode information to a second captured image acquired from a second imaging device equipped with the mask to identify an object appearing in the first captured image to which the noise has been added. Identification device.

11. Acquiring an image from an imaging device having a mask with one or more openings; Acquire opening mode information corresponding to the one or more opening modes; adding noise determined from the aperture mode information to the captured image; Outputting the captured image to which the noise has been added Image processing methods.

12. The noise imparting unit imparts the noise having a frequency band overlapping with at least a part of a frequency band of the first captured image to the first captured image. The image processing device according to claim 1 .

Images