Image processing device, training method of machine learning model, recognition device, and image processing method

Abstract

The image processing apparatus includes a noise-imposing unit (12) that acquires an image from an imaging device provided with a mask having one or more openings, an MPH information acquisition unit (111) that acquires opening shape information corresponding to one or more opening shapes, a noise-imposing unit (12) that imposes noise determined based on the opening shape information on the image, and a transmission unit (13) that outputs the image to which the noise is imposed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing apparatus, a training method of a machine learning model, an identification apparatus, and an image processing method. BACKGROUND

[0002] In recent years, techniques of processing an image to prevent unauthorized use of the image have been researched and developed. For example, as one example of processing an image, noise is imparted to the image. In Patent Literature 1, a technique of imparting noise to each pixel based on addition information indicating which of two noise addition processes, a first noise addition process and a second noise addition process, is performed is disclosed.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent No. 3919613 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, in the technique described in Patent Literature 1, the processed image can be restored. For example, it is possible to remove the noise imparted to the image.

[0008] Therefore, the present disclosure provides an image processing apparatus, a training method of a machine learning model, an identification apparatus, and an image processing method capable of suppressing a noise-imparted image from being restored.

[0009] MEANS FOR SOLVING PROBLEMS

[0010] An image processing apparatus of an aspect of the present disclosure includes: an image acquisition unit that acquires a first captured image from a first camera that has a mask formed with one or more openings; an information acquisition unit that acquires opening shape information corresponding to a shape of the one or more openings; a noise imparting unit that imparts noise determined in accordance with the opening shape information to the first captured image; and an output unit that outputs the first captured image to which the noise is imparted.

[0011] A training method of a machine learning model of an aspect of the present disclosure acquires a data set including an image generated by imparting noise to a captured image acquired by a camera that has a mask formed with one or more openings, the noise being determined in accordance with opening shape information corresponding to a shape of the one or more openings, and trains a machine learning model using the acquired data set.

[0012] The recognition device of one aspect of the present disclosure includes an image acquisition unit that acquires the first captured image to which the noise is added from the image processing device, and a recognition unit that recognizes an object appearing in the first captured image to which the noise is added using a machine learning model trained using a data set including an image generated by adding the noise determined based on the opening aspect information to a second captured image acquired from a second capturing device provided with the mask.

[0013] The image processing method of one aspect of the present disclosure acquires a captured image from a capturing device provided with a mask in which one or more openings are formed, acquires opening aspect information corresponding to an aspect of the one or more openings, adds noise determined based on the opening aspect information to the captured image, and outputs the captured image to which the noise is added.

[0014] Effects of Invention

[0015] The image processing device and the like according to one aspect of the present disclosure can suppress restoration of an image to which noise is added. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a block diagram showing a functional structure of an information processing system of Embodiment 1.

[0017] Figure 2 is a block diagram showing a structure of a sensing device of Embodiment 1.

[0018] Figure 3 is a diagram for explaining various images of Embodiment 1.

[0019] Figure 4 is a flowchart showing an operation in the sensing device of Embodiment 1.

[0020] Figure 5 is a flowchart showing an operation in a training device of the information processing system of Embodiment 1.

[0021] Figure 6 is a flowchart showing an operation in a recognition device of the information processing system of Embodiment 1.

[0022] Figure 7 is a block diagram showing a functional structure of a sensing device of Embodiment 2.

[0023] Figure 8 is a flowchart showing an operation in the sensing device of Embodiment 2.

[0024] Figure 9 is a block diagram showing a functional structure of a sensing device of a modified example of Embodiment 2.

[0025] Figure 10 is a flowchart showing the operation in the sensing device of the modification of Embodiment 2. DETAILED DESCRIPTION

[0026] (History of the achievement of the present disclosure)

[0027] Before explaining embodiments of the present disclosure and the like, the history of the achievement of the present disclosure will be explained.

[0028] In recent years, cameras are widely provided indoors and outdoors, and images containing privacy are easily taken. For example, cameras are provided in toilets and bathrooms and the like in homes or facilities for the elderly and the like, in unmanned aerial vehicles and robots and the like that move in privacy protection areas, or in stores of convenience stores and the like, and images containing privacy such as the appearance of a person indoors, the face of a passerby, and the license plate of a car are easily taken.

[0029] When such an image containing privacy is leaked, the object such as a person appearing in the image is threatened with invasion of privacy. Therefore, the importance of privacy protection for objects appearing in images taken by cameras is increasing.

[0030] As the technology of Patent Literature 1, it is considered to reduce the threat of invasion of privacy by imparting noise to a taken image (normal image). However, in the technology of Patent Literature 1, privacy cannot be protected in the case where the normal image itself is leaked.

[0031] In the present disclosure, from the viewpoint of privacy protection for objects appearing in images, for example, an object is imaged using a multi-pinhole camera. An image taken by the multi-pinhole camera (multi-pinhole image: MPH image) is an image containing a parallax characteristic, the details of which will be described later. In other words, the MPH image is a blurred image containing blur. Thereby, since the taken image becomes a blurred image, it is possible to protect the privacy of the object. Furthermore, the multi-pinhole camera is a camera having a multi-pinhole mask (mask) in which a multi-pinhole is formed.

[0032] However, even with the MPH image, when the image is leaked, it is possible to restore the image to an image in which the blur is removed. For example, it is possible to restore it in accordance with the characteristics of the mask possessed by the camera.

[0033] Therefore, the present inventors have made intensive studies on an image processing device and the like that can suppress restoration of an image in the case where an image in which privacy has been protected at the time point when an MPH image or the like is taken is leaked, and have created an image processing device and the like shown below.

[0034] An image processing apparatus of an aspect of the present disclosure includes: an image acquisition unit that acquires a first captured image from a first imaging device that includes a mask having one or more openings; an information acquisition unit that acquires opening shape information corresponding to the shape of the one or more openings; a noise addition unit that adds noise determined based on the opening shape information to the first captured image; and an output unit that outputs the first captured image to which the noise is added.

[0035] Thus, noise corresponding to the shape of the openings of the mask is added to the captured image. The captured image to which such noise is added is less likely to have frequency components of the noise and frequency components of the captured image separated than a captured image to which noise is added regardless of the shape of the openings of the mask. Therefore, according to the image processing apparatus, it is possible to suppress the image from being restored in a case where the image to which the noise is added is leaked out.

[0036] Further, for example, the noise addition unit can add the noise having a frequency band wider than a prescribed band to the first captured image.

[0037] Thus, since the frequency band in which the frequency components of the noise and the frequency components of the captured image overlap is widened, the frequency components of the noise and the frequency components of the captured image are more difficult to separate. Therefore, it is possible to further suppress the image from being restored in a case where the image to which the noise is added is leaked out.

[0038] Further, for example, the image processing apparatus can further include a noise information determination unit that selects the noise to be added by the noise addition unit based on the opening shape information.

[0039] Thus, the noise to be added is selected from among a plurality of noises based on the shape of the openings, and thus it is possible to add more appropriate noise to the captured image. That is, the frequency components of the noise are more difficult to be removed.

[0040] Further, for example, the mask can be configured to switch between a first opening shape and a second opening shape different from the first opening shape, and the image processing apparatus can further include a switching unit that switches the opening shape of the mask from one of the first opening shape and the second opening shape to the other.

[0041] Thus, since the opening shape of the mask is switched, it is possible to suppress the image from being restored in a case where the image to which the noise is added is leaked out, compared to a case where the opening shape of the mask is only one.

[0042] Further, for example, the noise can include at least one of salt and pepper noise, Laplacian noise, noise that changes the output value of a portion of the first captured image to a certain value, white noise, and pink noise.

[0043] Therefore, by simply applying at least one of salt-and-pepper noise, Laplacian noise, noise that changes the output value of a portion of the image to a certain value, white noise, and pink noise to the image, it is possible to suppress image restoration even when the image with noise has leaked out.

[0044] Furthermore, for example, if the frequency characteristics of the mask based on the opening shape corresponding to the opening shape information are widespread throughout the first band and the second band, which is the high band of the first band, the noise information determination unit selects at least one of salt-and-pepper noise and noise that changes the output value of a portion of the first camera image to a certain value as the noise assigned by the noise assignment unit. If the intensity of the frequency component in the first band is higher than the intensity of the frequency component in the second band, Laplace noise is selected as the noise assigned by the noise assignment unit.

[0045] Therefore, it is possible to impart noise with the same frequency characteristics as the mask to the photographic image. This further suppresses image restoration in the event that the noise-imparted image has leaked out.

[0046] Alternatively, for example, the opening morphology information may include at least one of the following: PSF (Point Spread Function), the size and shape of the opening, and information relating to the plurality of openings in the mask.

[0047] Therefore, by obtaining only one of the following information—the PSF, the size and shape of the opening, and at least one of the multiple openings in the mask—it is possible to suppress the image from being restored when the image with noise has leaked out.

[0048] Alternatively, for example, the first camera device may be one of a multi-hole camera, a lensless camera, and a coded aperture camera.

[0049] Therefore, noise can be added to images captured by multi-hole cameras, lensless cameras, or coded aperture cameras, thus preventing the images from being leaked and allowing image recovery.

[0050] Furthermore, one method for training a machine learning model disclosed herein involves acquiring a dataset containing images generated by adding noise to camera images, the camera images being acquired by a camera device having a mask forming one or more openings, the noise being determined based on opening shape information corresponding to the shape of the one or more openings, and using the acquired dataset to train a machine learning model.

[0051] Therefore, even with the aforementioned noise, a fully learned model that can accurately identify objects can still be generated.

[0052] Furthermore, one aspect of the recognition device disclosed herein includes: an image acquisition unit that acquires the first camera image with the noise applied from an image processing device described in any of the above aspects; and a recognition unit that uses a machine learning model to recognize an object reflected in the first camera image with the noise applied, the machine learning model being a machine learning model trained using a dataset containing an image generated by applying noise to a second camera image determined according to the opening shape information, the second camera image being acquired from a second camera device equipped with the mask.

[0053] Therefore, it is possible to suppress the reduction in recognition performance of recognition devices for noisy camera images. In other words, it is possible to achieve a recognition device that can accurately identify objects even with noisy camera images.

[0054] Furthermore, one aspect of the image processing method disclosed herein involves acquiring a camera image from a camera device having a mask having one or more openings, acquiring opening shape information corresponding to the shape of the one or more openings, assigning noise determined based on the opening shape information to the camera image, and outputting the camera image with the noise assigned.

[0055] Thus, it achieves the same effect as the image processing device described above.

[0056] Furthermore, these general or specific forms can be realized through systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer-readable CD-ROMs, or through any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0057] Hereinafter, specific examples of an image processing apparatus or the like according to one aspect of the present disclosure will be described with reference to the accompanying drawings. The embodiments shown herein are all specific examples of the present disclosure. Therefore, the numerical values, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure. Furthermore, constituent elements not described in the independent technical solutions among the constituent elements in the following embodiments will be described as arbitrary constituent elements.

[0058] Furthermore, these figures are schematic diagrams and not necessarily rigorous representations. Therefore, for example, the scales may not be consistent across different figures. Additionally, substantially identical structures are labeled with the same reference numerals across different figures, and repetitive descriptions are omitted or simplified.

[0059] Also in the present specification, the same or similar terms expressing the relationship between elements, and numerical values and numerical value ranges are not only expressions of strict meanings, but also mean expressions including substantially equivalent ranges, for example, a difference of several percent (for example, about 5%).

[0060] (Embodiment 1)

[0061] Hereinafter, referring to Figures 1-6 An information processing system provided with the image processing apparatus of the present embodiment is described.

[0062] [1-1. Structure of information processing system]

[0063] First, referring to Figure 1 and Figure 2 The structure of the information processing system of the present embodiment is described. Figure 1 is a block diagram showing the functional structure of the information processing system 1 of the present embodiment. Figure 2 is a block diagram showing the structure of the sensing apparatus 10 of the present embodiment. The information processing system 1 is, for example, a system for performing recognition of an object appearing in an MPH image using an image to which noise is further imparted to the MPH image.

[0064] As shown in Figure 1 , the information processing system 1 is provided with a sensing apparatus 10, a recognition apparatus 20, and a training apparatus 30. The sensing apparatus 10 and the recognition apparatus 20, and the recognition apparatus 20 and the training apparatus 30 are communicably connected, respectively. Also, the sensing apparatus 10 and the training apparatus 30 can be communicably connected.

[0065] The sensing apparatus 10 acquires an image that can suppress a restored image in a case where an image to which noise is imparted flows out. In the present embodiment, the sensing apparatus 10 generates an image to which a prescribed noise is imparted to an MPH image, as an image that can suppress a restored image.

[0066] The sensing apparatus 10 has an MPH image imaging section 11, a noise imparting section 12, and a transmission section 13. An information processing apparatus (image processing apparatus) for performing image processing, which at least includes the noise imparting section 12 and the transmission section 13, is implemented.

[0067] The MPH image camera unit 11 acquires privacy-protected images. In this embodiment, the MPH image camera unit 11 acquires MPH images as privacy-protected images. The MPH image camera unit 11 has a multi-pinhole mask 15 with multiple pinholes 15a formed thereon, which can overlap and acquire multiple images (pinhole images: PH images) with different viewpoints on the object (subject) in a single imaging operation. In the MPH image camera unit 11, the images acquired through the imaging operation are privacy-protected images. In other words, the MPH image camera unit 11 does not acquire ordinary images whose privacy is not protected.

[0068] Therefore, it is possible to suppress the invasion of an object's privacy due to the leakage of ordinary images that are not protected by privacy. Furthermore, the multi-pinhole mask 15 is also described as the MPH mask 15. In addition, the MPH image capturing unit 11 is an example of a capturing device, the multi-pinhole mask 15 is an example of a mask, and the MPH image is an example of a captured image.

[0069] Furthermore, a typical image is an image captured by a conventional imaging device without the MPH mask 15, obtained by imaging light from an object focused by an optical system. An example of such an optical system is a lens. When a person directly views the object in its presence, the person perceives the object in essentially the same way as in a typical image. In other words, the person visually recognizes a typical image captured by a conventional imaging device in the same way as in real space. A typical imaging device is, for example, a camera with a lens.

[0070] like Figure 2 As shown, the MPH image capture unit 11 is implemented, for example, by a lens 14, an MPH mask 15, and an image sensor 16. The MPH image capture unit 11 is, for example, a multi-hole camera. In addition, the MPH image capture unit 11 is not limited to a multi-hole camera, and may also be a lensless camera, a coded aperture camera, etc.

[0071] Lens 14 focuses the incident light onto image sensor 16. Lens 14 is implemented, for example, by a convex lens.

[0072] The MPH mask 15 is a mask with multiple pinholes 15a (multiple pinholes). The MPH mask 15 is disposed on the object side at a certain distance relative to the image sensor 16. The MPH mask 15 is disposed, for example, between the lens 14 and the image sensor 16, but its placement is not limited to this. That is, light passing through the MPH mask 15 is incident on the image sensor 16. The pinholes 15a are examples of openings.

[0073] In addition, the plurality of pinholes 15a are arranged at random or at equal intervals. The size and shape of the pinholes 15a, and the position and number of the pinholes 15a formed in the MPH mask 15 and the distance between adjacent pinholes 15a are an example of opening pattern information corresponding to the opening pattern (mask pattern). The opening pattern information includes at least one of the size and shape of the pinholes 15a, and the position, number, and distance between adjacent pinholes 15a of the plurality of pinholes 15a formed in the MPH mask 15. At least one of the position, number, and distance between adjacent pinholes 15a of the plurality of pinholes 15a is an example of information related to the plurality of pinholes 15a. In addition, hereinafter, the opening pattern information will also be described as MPH information.

[0074] The position of the pinholes 15a affects the position of the object projected on the image sensor 16, and the size and number of the pinholes 15a affect the blur of the MPH image.

[0075] In addition, the plurality of pinholes 15a are not particularly limited in number as long as there are two or more pinholes, for example. Furthermore, the opening pattern can differ depending on each MPH image capturing section 11, each protection target, each use scenario, and the like. Each use scenario includes information related to the object to be hidden, such as hiding the face, hiding the entire background, and the like.

[0076] Furthermore, the opening pattern information is not limited to information indicating the size, shape, and the like of the opening as described above. The opening pattern information can also include a PSF (Point Spread Function). The PSF indicates the blur method (degradation method) of the image of the optical system when a point light source is captured, and is a function indicating the intensity distribution at the time of blur. The PSF can also be said to be a function indicating how light spreads in the case where an ideal point light source passes through the optical system.

[0077] In this way, the opening pattern information can include information related to the pattern of the pinholes 15a, and can also include information related to the MPH image obtained depending on the opening pattern.

[0078] The image sensor 16 receives light that has passed through the MPH mask 15. The image sensor 16 can also be said to acquire an image (MPH image) of the object through each pinhole 15a. The MPH image is an image acquired via each pinhole 15a. Depending on the position and size of each pinhole 15a and the like, the acquired PH image differs. The image sensor 16 acquires an overlapping image (encoded image) of a plurality of PH images as the MPH image. The MPH image is an image that, although it can be an image that a person cannot visually recognize, if processed using a computer, it is possible to acquire information included in the image of the object to be captured and the surrounding environment and the like.

[0079] Referring again to Figure 1The noise imparting unit 12 imparts noise to the MPH image obtained by the MPH image capturing unit 11. For example, the noise imparting unit 12 imparts noise to the MPH image based on the opening shape information of the MPH mask 15 through image processing. The noise imparting unit 12 imparts noise having the same frequency characteristics as the transmission frequency characteristics of the MPH mask 15. For example, it can also be said that the noise imparting unit 12 imparts noise corresponding to the opening shape of the MPH mask 15.

[0080] The MPH image, for example, may have a flat frequency response (spatial frequency response). Therefore, the noise imposition unit 12, for example, impositions noise with a flat frequency response onto the MPH image. Flat frequency response noise is noise with a bandwidth (spatial bandwidth) wider than a specified band (broad noise). Flat frequency response noise can also be described as noise whose frequency response deviation is below a threshold within the specified band. The specified band is a band based on the opening shape of the MPH mask 15, for example, including a low band and a band higher than the low band, i.e., a high band. The specified band is a band that at least partially overlaps with the frequency band of the MPH image, for example, it could be a band including all frequency bands of the MPH image. Broad noise means, for example, noise where the intensity of frequency components in the specified band is within a specified range, or noise where the deviation of the intensity of frequency components in the specified band is within a specified range.

[0081] The noise imparted by the noise imparting unit 12 may include at least one of salt-and-pepper noise, Laplacian noise, truncated noise that changes the output value (pixel value) of a portion of the MPH image to a certain value, white noise, and pink noise. Furthermore, the noise imparted by the noise imparting unit 12 may include at least one of scratch noise, such as that found on old film, noise generated by random erasing of a portion of the MPH image, color noise (e.g., noise other than white noise), and frequency shift noise whose frequency characteristics deviate beyond a threshold.

[0082] Furthermore, an image with sparse noise in the frequency space is easier to restore than an image with broad noise in the frequency space. Therefore, the noise imposition unit 12 does not imposition sparse noise on the MPH image in the frequency space. For example, the noise imposition unit 12 does not imposition impulse noise in the frequency space on the MPH image.

[0083] like Figure 2 As shown, the noise imparting unit 12 is implemented, for example, by an ISP (Image Signal Processor) 17. Sparse noise refers to noise that has frequency components only in one of the low-band and high-band mentioned above.

[0084] Further, the noise imparting section 12 is not limited to imparting noise by image processing. The noise imparted by the noise imparting section 12 can be, for example, noise caused by a hot pixel and a dead pixel, shot noise, or the like. In this case, the noise imparting section 12 can be included in the MPH image capturing section 11 (for example, the image sensor 16). Further, the hot pixel is noise in which a part of pixel values becomes white, and the dead pixel is noise in which a part of pixel values becomes black.

[0085] Referring again to Figure 1 , the transmission section 13 outputs the MPH image to which noise is imparted by the noise imparting section 12 to the recognition device 20. The transmission section 13 outputs the MPH image to which noise is imparted by wireless communication, but can also output the MPH image to which noise is imparted by wired communication. The transmission section 13 is an example of an output section. Further, hereinafter, the MPH image to which noise is imparted is also referred to as a noise imparted image.

[0086] As shown in Figure 2 , the transmission section 13 is implemented by the communication section 18 having a communication interface such as an adapter for transmitting the noise imparted image to the recognition device 20.

[0087] The recognition device 20 performs recognition of the object represented in the noise imparted image using the trained learning model (learned model). The recognition device 20 can detect the object represented in the noise imparted image using the learned model, for example, and output the detection result. The detection of the object in the noise imparted image is an example of recognition. In addition, the learned model is also referred to as a recognizer.

[0088] The recognition device 20 has a reception section 21, a recognition section 22, and an output section 23.

[0089] The reception section 21 communicates with the sensing device 10 and the training device 30. The reception section 21 receives the noise imparted image from the sensing device 10 and receives the learned model from the training device 30. The reception section 21 is configured to include a communication interface such as an adapter for receiving various information from the sensing device 10 and the training device 30.

[0090] The recognition section 22 acquires information of the object (for example, the target object and the surrounding environment of the target object) in the noise imparted image using the learned model. The recognition section 22, for example, recognizes the object in the noise imparted image and acquires the position of the object in the noise imparted image. That is, the information of the object is the recognition result of the learned model, and can include the presence or absence of the object and the position of the object. Further, the recognition of the object can include, for example, detecting a pixel in which the object exists.

[0091] The recognition unit 22 inputs the noise-attached image to the learned model trained by the training unit 33, and acquires the output from the learned model as the recognition result. The recognition performance of the learned model trained by the training unit 33 with respect to the noise-attached image is improved. Therefore, the recognition unit 22 can suppress the decrease in the recognition performance with respect to the noise-attached image by using the learned model.

[0092] For example, in a case where the recognition device 20 is mounted on a car, examples of the object are a person, a car, a bicycle, or a signal. In addition, the recognition device 20 can recognize a predetermined one kind of object using the noise-attached image, or can recognize a plurality of kinds of objects. In addition, the recognition device 20 can recognize the object in a category unit including a mobile body such as a person, a car, or a bicycle.

[0093] The output unit 23 outputs the recognition result of the recognition unit 22. The output unit 23 can also prompt the recognition result to the user. The output unit 23 is configured, for example, by including a display device or a sound emitting device.

[0094] In addition, the sensing device 10 and the recognition device 20 described above can be mounted in the same device. For example, the sensing device 10 and the recognition device 20 can be mounted on a mobile body such as a vehicle and a robot, or can be mounted on a fixed object such as a monitoring camera system.

[0095] The training device 30 generates a learned model used in the recognition of the object by the recognition device 20. The training device 30 has an MPH information acquisition unit 31, a noise information acquisition unit 32, a training unit 33, and a transmission unit 34. In addition, the training device 30 can also have an acceptance unit that accepts an input from the user. The acceptance unit is realized by a button, a touch panel, or the like, but can also be realized by a device that accepts an input based on a voice or the like.

[0096] The MPH information acquisition unit 31 acquires MPH information (opening shape information) corresponding to the opening shape of the MPH mask 15 of the sensing device 10. The MPH information acquisition unit 31 can acquire the opening shape information from the sensing device 10, for example, or can acquire the MPH information according to an input from the user.

[0097] The noise information acquisition unit 32 acquires noise information indicating the noise attached to the MPH image by the noise attachment unit 12 of the sensing device 10. The noise information acquisition unit 32 can acquire the noise information from the sensing device 10, for example, or can acquire the noise information by an input from the user.

[0098] The training section 33 performs training of a learning model that performs recognition of an object with respect to a noise-imposed image generated by the noise-imposing section 12. The training section 33 performs training of the learning model by machine learning using a data set generated using the MPH information acquired by the MPH information acquisition section 31 and the noise information acquired by the noise information acquisition section 32. It can also be said that the training section 33 causes the learning model to be trained using the data set. The learning model is an example of a machine learning model that recognizes an object based on an image, and is, for example, a machine learning model using a neural network such as deep learning, but can also be another machine learning model. For example, the machine learning model can be a machine learning model using a random forest, genetic programming, or the like.

[0099] Further, the data set contains an image corresponding to the noise-imposed image, that is, a training image, and correct answer information with respect to the training image. The correct answer information is reference data in machine learning, and is appropriately decided in accordance with the use of the recognition device 20 and the like, and is, for example, the kind of the object and the position of the object on the image. Further, the data set can be generated by the training device 30, for example, or can be generated by another device.

[0100] The image corresponding to the noise-imposed image can be, for example, an image in which noise determined by the noise information acquired by the noise information acquisition section 32 is imposed on an MPH image imaged by an imaging device provided with an MPH mask 15 having the same MPH information as the MPH information acquired by the MPH information acquisition section 31. In addition, the image corresponding to the noise-imposed image can also be, for example, an image obtained by imposing noise determined by the noise information acquired by the noise information acquisition section 32 on an image generated by convolving a PSF on a normal image, in the case where the MPH information contains the PSF.

[0101] Further, machine learning is implemented, for example, by an error backpropagation method (BP: Back Propagation) such as deep learning. Specifically, the training section 33 inputs the training image to the learning model that has not been trained, and acquires a recognition result output by the learning model. Then, the training section 33 adjusts the learning model so that the recognition result becomes the correct answer information. The training section 33 improves the recognition accuracy of the learning model by repeating such adjustment with respect to a plurality of (for example, several thousand sets) different training images and the correct answer information corresponding to the training images.

[0102] The transmission unit 34 outputs the learned model generated by the training unit 33 to the recognition device 20. The transmission unit 13 is configured to include a communication interface such as an adapter for transmitting the learned model to the recognition device 20. Further, outputting the learned model means outputting information such as network parameters in the learned model, an algorithm (machine learning algorithm) of the operation, and the like. In addition, the algorithm is not particularly limited and can be any algorithm that exists.

[0103] Here, reference will be made to Figure 3 Various images will be described. Figure 3 is a diagram for explaining various images of the present embodiment.

[0104] In Figure 3 , an MPH image of an object (person) shown in a correct image (normal image) imaged by the MPH image imaging unit 11, and a salt-and-pepper noise-imposed image and a Poisson noise-imposed image as examples of noise-imposed images, and a restored image obtained by restoring these images are shown. The restoration of each image is, for example, an image obtained by performing a restoration process by the SelfDeblur method (Ren D, Zhang K, Wang Q, et al., Neural blind deconvolution using deep priors. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020.), which is a neural network model that performs blur removal processing on an input image and outputs a deblurred image corresponding to the input image. The three restored images are executed by the same neural network model. The salt-and-pepper noise-imposed image is an image to which salt-and-pepper noise has been imposed on the MPH image, and the Poisson noise-imposed image is an image to which Poisson noise has been imposed on the MPH image.

[0105] As Figure 3 shown, the correct image is an image in which a person can be recognized as drinking a beverage, but the MPH image, the salt-and-pepper noise-imposed image, and the Poisson noise-imposed image can be images that a person cannot visually recognize. In addition, the MPH image, the salt-and-pepper noise-imposed image, and the Poisson noise-imposed image are images in which information contained in an image of an object and a surrounding environment, and the like can be obtained if processed using a computer.

[0106] The restored image obtained by restoring the MPH image is restored to an extent in which a person can be recognized as drinking a beverage. Therefore, the MPH image can be an image that is restored to an extent in which a person can visually recognize.

[0107] On the other hand, the image to which the salt-and-pepper noise is imparted and the restored image to which the Poisson noise is imparted are images in which it is not possible to identify that the person is drinking the beverage. That is, the restored image is not an image that has been restored to a degree that a person can visually identify. The image to which the salt-and-pepper noise is imparted and the image to which the Poisson noise is imparted are images to which wide noise having a wider frequency band than the prescribed band is imparted. Therefore, the frequency component of the MPH image and the frequency component of the noise overlap in the prescribed band, and it is difficult to remove only the frequency component of the noise from the noise-imparted image. In particular, in a case where the signal is converted by convolution as in a multi-pinhole camera, wide noise in the frequency space can make it difficult to restore the image.

[0108] Thus, by imparting wide noise having a wider frequency band than the prescribed band to the MPH image, it is possible to realize an image that is difficult to restore in the case of outflow.

[0109] Further, in a case where the noise imparted by the noise imparting section 12 is salt-and-pepper noise, that is, in a case where the noise information is information indicating that salt-and-pepper noise is imparted, the training device 30 performs training of the learning model using the salt-and-pepper noise-imparted image. Further, in a case where the noise imparted by the noise imparting section 12 is one of salt-and-pepper noise and Poisson noise, that is, in a case where the noise information is information indicating that one of salt-and-pepper noise and Poisson noise is imparted, the training device 30 performs training of the learning model using the salt-and-pepper noise-imparted image and the Poisson noise-imparted image, respectively.

[0110] [1-2. Action of information processing system]

[0111] Next, the action of the information processing system 1 configured as described above will be described with reference to Figures 4-6 to the flowchart of the action in the sensing device 10 of the present embodiment. Figure 4 The action of the sensing device 10 will be described. Figure 4 is a flowchart indicating the action in the sensing device 10 of the present embodiment.

[0112] As shown in Figure 4 , the MPH image capturing section 11 of the sensing device 10 acquires an MPH image by capturing an object (S11). The MPH image capturing section 11 outputs the acquired MPH image to the noise imparting section 12.

[0113] Next, when the MPH image is acquired, the noise-imposing section 12 imposes noise decided in accordance with the MPH information of the MPH mask 15 at the time of imaging the MPH image to the MPH image (S12). In a case where the opening form of the MPH mask 15 is not changeable, the noise-imposing section 12 reads out noise corresponding to the opening form from a storage section (not shown) and imposes the read-out noise to the MPH image. Further, the noise-imposing section 12, for example, imposes noise to the entire MPH image, but can impose noise to only a part of the MPH image. The imposed noise is stored in advance in the storage section of the sensing device 10. The noise-imposing section 12 functions as an image acquisition section that acquires the MPH image.

[0114] The noise-imposing section 12 can store the noise-imposed image in which noise is imposed to the MPH image in the storage section. In addition, the noise-imposing section 12 can not store the MPH image before the noise is imposed, that is, the MPH image imaged by the MPH image imaging section 11, in the storage section. The noise-imposing section 12 can save the noise-imposed image by overwriting the MPH image, and in a case where the MPH image is saved, the MPH image can be deleted. Thereby, the MPH image can be suppressed from flowing out of the sensing device 10.

[0115] Next, the transmission section 13 transmits the MPH image to which noise is imposed by the noise-imposing section 12 (noise-imposed image) to the recognition device 20 (S13). The transmission section 13, for example, transmits the salt and pepper noise-imposed image or the Poisson noise-imposed image shown in Figure 3 to the recognition device 20.

[0116] As described above, the sensing device 10 of the information processing system 1 generates a noise-imposed image by imposing noise decided in accordance with MPH information to an MPH image. As described above, the noise-imposed image is difficult to restore the image compared to the MPH image. Therefore, the sensing device 10 can suppress the image from being restored in a case where the image to which noise is imposed flows out. Figure 3

[0117] Next, the action of the training device 30 will be described with reference to Figure 5 Figure 5 is a flowchart showing the action in the training device 30 of the present embodiment. Figure 5 Figure 4 is executed before the action shown in

[0118] As described above, the sensing device 10 of the information processing system 1 generates a noise-imposed image by imposing noise decided in accordance with MPH information to an MPH image. As described above, the noise-imposed image is difficult to restore the image compared to the MPH image. Therefore, the sensing device 10 can suppress the image from being restored in a case where the image to which noise is imposed flows out. Figure 5 ​​​As shown, first, the training device 30 acquires the MPH information and the noise information of the sensing device 10 (S21). Specifically, the MPH information acquisition section 31 acquires the MPH information, and the noise information acquisition section 32 acquires the noise information. Further, in a case where the form of the pinhole 15a and the noise to be imparted exclusively establish a correspondence, the training device 30 acquires at least one of the MPH information and the noise information through an input from an external device or a user.

[0119] Further, in a case where the opening form of the pinhole 15a in the MPH mask 15 is changeable, the MPH information includes information corresponding to the changeable opening forms respectively.

[0120] Next, the training section 33 of the training device 30 acquires a data set for training (S22). The training section 33 acquires a data set corresponding to the MPH information and the noise information acquired in step S21. The data set can be generated by the training section 33 based on the MPH information and the noise information. Further, it can also be that a plurality of data sets are stored in the training device 30, and the training section 33 reads out a data set corresponding to the MPH information and the noise information acquired in step S21. Further, the training section 33 can acquire the data set from an external device (for example, a device that manages various data sets).

[0121] In addition, in a case where the opening form of the pinhole 15a is changeable, a data set corresponding to the changeable opening forms respectively can also be acquired.

[0122] Next, the training section 33 performs a learning process using the data set (S23). In step S23, the training section 33 performs training of a learning model through machine learning using the data set. The learning process is performed by an error backpropagation method or the like, but is not limited thereto, and can be performed by any known method.

[0123] Here, although the noise imparted by the noise imparting section 12 is wide in the frequency space, it is impulse noise in the actual space. The influence of the impulse noise in the actual space on the training of the completed learning model is small, and thus it is possible to suppress a decrease in recognition performance.

[0124] Further, in a case where the opening form of the MPH mask 15 is changeable, the training section 33 can perform the learning process of step S23 on one learning model using a plurality of data sets including training images different from each other in the noise to be imparted, or can perform a learning process of generating a plurality of learning models corresponding to the plurality of data sets. Thereby, in a case where the opening form of the MPH mask 15 is changed, that is, even in a case where the noise imparted by the noise imparting section 12 is changed, it is possible to suppress a decrease in recognition performance of an object using the noise imparted image.

[0125] Next, the transmission unit 34 transmits the learned model trained by the training unit 33 to the recognition device 20 (S24). The processing of step S24 is performed, for example, before the sensing device 10 is shipped.

[0126] Next, referring to Figure 6 The operation of the recognition device 20 will be described. Figure 6 is a flowchart showing the operation in the recognition device 20 of the information processing system 1 of the present embodiment.

[0127] As shown in Figure 6 First, the reception unit 21 of the recognition device 20 receives the learned model transmitted from the training device 30 in step S24 shown in Figure 5 The reception unit 21 stores the received learned model in a storage unit (not shown). The reception unit 21 functions as an acquisition unit.

[0128] Next, the reception unit 21 of the recognition device 20 receives the noise-imposed MPH image (noise-imposed image) transmitted from the sensing device 10 in step S13 shown in Figure 4 The reception unit 21 stores the received noise-imposed image in a storage unit (not shown).

[0129] Next, the recognition unit 22 performs recognition processing on the noise-imposed MPH image using the learned model (S33). The recognition unit 22 acquires the output obtained by inputting the noise-imposed MPH image to the learned model as a recognition result. Since the learned model is a learned model trained using the training image corresponding to the noise imposed by the noise-imposing unit 12, the recognition processing on the noise-imposed image can be performed with high accuracy.

[0130] Next, the output unit 23 outputs the recognition result (S34). The output unit 23 prompts the user of the recognition result, for example, by an image, a sound, or the like.

[0131] As described above, since the recognition device 20 of the information processing system 1 performs recognition using the learned model trained by the training device 30 using the noise-imposed MPH image, the decrease in recognition performance on the noise-imposed MPH image is suppressed.

[0132] The information processing system 1 of the present embodiment can be a system capable of balancing the restoration of the image in the case where the image to which noise is imposed leaks out and the suppression of the decrease in recognition performance caused by the noise.

[0133] (Embodiment 2)

[0134] [2-1. Structure of Information Processing System]

[0135] Referring toFigure 7 The structure of the information processing system 1 in this embodiment is explained. Figure 7 This is a block diagram illustrating the functional structure of the sensing device 110 in this embodiment. The information processing system 1 of this embodiment differs from the information processing system 1 of the other embodiment in that it includes a sensing device 110 instead of a sensing device 10. Hereinafter, the sensing device 110 of this embodiment will be described focusing on its differences from the sensing device 10 of the other embodiment. Furthermore, structures identical or similar to the sensing device 10 of Embodiment 1 will be labeled with the same reference numerals as those for the sensing device 10, and descriptions will be omitted or simplified.

[0136] like Figure 7 As shown, in addition to the sensing device 10 of Embodiment 1, the sensing device 110 also has an MPH information acquisition unit 111 and a noise information determination unit 112.

[0137] MPH information acquisition unit 111 acquires MPH information (opening shape information) corresponding to the opening shape of the plurality of pinholes 15a. MPH information acquisition unit 111 is an example of an information acquisition unit.

[0138] The noise information determination unit 112 selects the noise applied by the noise application unit 12 based on the MPH information. For example, the noise information determination unit 112 determines the noise applied by the noise application unit 12 based on the frequency characteristics of the MPH mask 15 (via frequency characteristics). For example, the noise information determination unit 112 selects noise with the same frequency characteristics as the mask from a plurality of noises and applies the selected noise to the MPH image.

[0139] For example, when the frequency response of the MPH mask 15 is flat, the noise information determination unit 112 determines that noise (based on truncation) that changes the output value (pixel value) of a portion of the MPH image to a certain value is applied to the MPH image. Furthermore, for example, when the frequency response (via frequency response) of the MPH mask 15 is close to the low band, the noise information determination unit 112 determines that noise with a frequency response close to the low band, such as Laplacian noise, is applied to the MPH image.

[0140] [2-2. Operation of the sensing device]

[0141] Next, we will refer to Figure 8 Explain the operation of the information processing system 1 constructed as described above. Figure 8 This is a flowchart illustrating the operation of the sensing device 110 in this embodiment. Furthermore, regarding the implementation of Embodiment 1... Figure 4 Actions that are the same as or similar to the actions shown, and are attached to Figure 4 The same reference numerals are used in the accompanying drawings, and descriptions are omitted or simplified.

[0142] As Figure 8 shown, the MPH information acquisition section 111 of the sensing device 110 acquires MPH information of the MPH mask 15 included in the MPH image capturing section 11 (S41). In a case where the opening form of the MPH mask 15 is fixed (unchangeable), the MPH information acquisition section 111 can acquire, for example, MPH information once. The MPH information acquisition section 111 outputs the acquired MPH information to the noise information decision section 112. The MPH information acquisition section 111 stores the acquired MPH information in a storage section (not shown).

[0143] Next, the noise information decision section 112 of the sensing device 110 selects the noise to be imparted by the noise imparting section 12 from a plurality of noises based on the MPH information acquired by the MPH information acquisition section 111, thereby deciding the noise information to be output to the noise imparting section 12 (S42). In a case where the frequency characteristic of the MPH mask 15 is flat in a frequency band including a low band and a high band higher than the low band, the noise information decision section 112 decides the noise information based on the MPH information so as to impart a noise having a more flat frequency characteristic. The noise having a more flat frequency characteristic is, for example, salt and pepper noise, or a noise (based on truncation) that changes the output value (pixel value) of a part of the region of the MPH image to a certain value. The low band is an example of the first band, and the high band is an example of the second band.

[0144] Further, in a case where the frequency characteristic of the MPH mask 15 includes a large amount of frequency components of the low band and the high band, the noise information decision section 112 decides the noise information based on the MPH information so as to impart a noise having a frequency characteristic closer to the low band. The noise having a frequency characteristic closer to the low band is, for example, Laplacian noise.

[0145] In this way, in a case where the frequency characteristic of the MPH mask 15 based on the opening form corresponding to the MPH information is wide-ranging over the first band and the second band which is a high band of the first band, the noise information decision section 112 selects at least one of salt and pepper noise and a noise that changes the output value of a part of the region of the MPH image to a certain value as the noise to be imparted by the noise imparting section 12, and selects Laplacian noise as the noise to be imparted by the noise imparting section 12 in a case where the intensity of the frequency component of the first band is higher than the intensity of the frequency component of the second band.

[0146] Further, the frequency characteristic being wide-ranging over the first band and the second band means, for example, that the difference between the intensity of the frequency component of the first band and the intensity of the frequency component of the second band is less than a predetermined value. In addition, the intensity of the frequency component of the first band being higher than the intensity of the frequency component of the second band means, for example, that the intensity of the frequency component of the first band is higher than the intensity of the frequency component of the second band by more than a predetermined value. For example, the above-described determination can be made by comparing the average of the intensity of the frequency component of the first band and the average of the intensity of the frequency component of the second band.

[0147] In step S42, the noise information decision section 112 can determine, based on the MPH information, whether the frequency characteristic of the MPH mask 15 is flat in a frequency band including the low band and a high band higher than the low band, and whether the frequency characteristic of the MPH mask 15 contains a frequency component of a large amount of the low band among the low band and the high band, and decide the noise to be imparted based on the determination result. For example, when the positions and sizes of the pinholes 15a are random and the number of the pinholes 15a (e.g., the number in a prescribed region) is equal to or more than a prescribed number, the noise information decision section 112 can determine that the frequency characteristic of the MPH mask 15 is flat in the frequency band including the low band and the high band. Also, for example, when the number of the pinholes 15a (e.g., the number in a prescribed region) is less than the prescribed number, the noise information decision section 112 can determine that the frequency component of a large amount of the low band is contained. Further, the noise information decision section 112 has a table in which the MPH information and the mask information are established in correspondence, and can select the noise to be imparted based on the table.

[0148] Further, the noise information contains information that determines the noise to be imparted by the noise imparting section 12. The noise information decision section 112 outputs the decided noise information to the noise imparting section 12.

[0149] Next, the noise imparting section 12 imparts noise to the MPH image based on the noise information (S43). The noise imparting section 12 reads out the noise determined by the noise information from the storage section, and imparts the read noise to the MPH image.

[0150] Thus, since the noise corresponding to the frequency characteristic of the MPH mask 15 is imparted to the MPH image, it is possible to more effectively generate a noise imparted image that is difficult to be restored.

[0151] (Modified example of Embodiment 2)

[0152] The configuration of the information processing system 1 of the present modified example will be described with reference to Figure 9 The configuration of the information processing system 1 of the present modified example will be described with reference to Figure 9 is a block diagram showing the functional configuration of the sensing device 110a of the present modified example. The information processing system 1 of the present modified example differs from the information processing system 1 of Embodiment 2 in that the sensing device 110a is provided instead of the sensing device 110. Hereinafter, the sensing device 110a of the present modified example will be described focusing on the difference from the sensing device 110 of Embodiment 2. Further, the same or similar configuration as the sensing device 110 of Embodiment 2 is attached with the same reference numeral as the sensing device 110, and the description thereof is omitted or simplified.

[0153] As Figure 9As shown, the sensing device 110a has an MPH information setting section 111a instead of the MPH information acquisition section 111 of the sensing device 110 of Embodiment 2. In addition, the MPH image capturing section 11 of the present modified example is configured so that the opening pattern (mask pattern) can be dynamically changed. The MPH image capturing section 11 is configured so that it can be changed to, for example, a first opening pattern (first mask pattern) and a second opening pattern (second mask pattern) different from the first opening pattern. That is, the MPH image capturing section 11 can switch the opening pattern. The MPH mask 15 of the MPH image capturing section 11 is realized by, for example, an SLM (Spatial Light Modulator) having a liquid crystal shutter or the like. In this case, the opening is realized by the transmission portion of the liquid crystal shutter.

[0154] The MPH information setting section 111a controls the opening pattern of the MPH mask 15 of the MPH image capturing section 11. The MPH information setting section 111a, for example, performs control to switch the opening pattern of the MPH mask 15. The MPH information setting section 111a, for example, changes the opening pattern of the MPH mask 15 with time. For example, the MPH information setting section 111a can switch the opening pattern at a prescribed time interval, can switch the opening pattern by operation from a user, and can switch the opening pattern based on a pre-set time table. The time table contains information indicating the time change of the opening pattern.

[0155] It can also be said that the MPH information setting section 111a switches the opening pattern of the MPH mask 15 from one of the first opening pattern and the second opening pattern different from the first opening pattern to the other. In addition, the number of opening patterns that the MPH information setting section 111a can set is not particularly limited as long as it is two or more. At least one of, for example, the positions, the number, and the distance between adjacent openings of the openings of the first opening pattern and the second opening pattern is different. The MPH information setting section 111a functions as a switching section.

[0156] The MPH information setting section 111a acquires MPH information by switching the opening pattern. The MPH information setting section 111a can also be said to be an MPH information acquisition section.

[0157] The MPH image capturing section 11 controls the MPH mask 15 (for example, an SLM) in order to realize the opening pattern set by the MPH information setting section 111a.

[0158] Every time the MPH information setting section 111a switches the opening form, the noise information determining section 112 selects the noise corresponding to the switched opening form. The noise information determining section 112, for example, changes the noise to be imparted to the MPH image with time. The noise information determining section 112, for example, can have information on mutually different noises equal to or more than the number of opening forms that the MPH information setting section 111a can set, and sets mutually different noises in each of the plurality of opening forms.

[0159] Further, in a case where the opening form of the MPH mask 15 of the MPH image capturing section 11 is switchable as in this modification example, for example, in the training device 30, training of the learning model is performed using a data set containing training images corresponding to the switchable opening forms respectively.

[0160] Next, the operation of the information processing system 1 configured as described above will be described with reference to Figure 10 Figure 10 is a flowchart showing the operation in the sensing device 110a of the present modification example. Further, the same or similar operations as those shown in Figure 8 will be described. The same reference numerals as those in Figure 8 will be attached to the same or similar operations, and the description will be omitted or simplified.

[0161] As shown in Figure 10 , the MPH information setting section 111a of the sensing device 110a sets the MPH information of the MPH mask 15 that the MPH image capturing section 11 has (S51). The MPH information setting section 111a sets the MPH information of the MPH mask 15 by selecting the opening form to be used at the current time point from a list containing a plurality of opening forms, from the opening form of the MPH mask 15. The MPH information setting section 111a outputs the MPH information to the MPH image capturing section 11 and the noise information determining section 112 respectively.

[0162] Further, it can also be that the MPH information setting section 111a determines whether or not the MPH information needs to be switched before step S51, and in a case where it is determined that the MPH information needs to be switched, performs the processing of step S51, and in a case where it is determined that the MPH information does not need to be switched, does not perform the processing of step S51 (does not switch the MPH information) and performs the processing after step S11. For example, this determination can be made based on the elapsed time from when the MPH information was switched, or can be made based on whether or not an input from the user has been accepted. It can also be said that step S51 is processing to update the MPH information of the MPH mask 15 that the MPH image capturing section 11 has.

[0163] ​Next, the MPH image capturing section 11 captures an MPH image by photographing the object based on the MPH information set by the MPH information setting section 111a (S11). The MPH image capturing section 11 switches the opening form of the multi-pinhole mask 15 based on the MPH information set in step S51, and captures the object using the switched opening form.

[0164] The MPH image capturing section 11 outputs the captured MPH image to the noise imparting section 12. The MPH image capturing section 11 can also output the MPH image to the noise imparting section 12 in association with the MPH information indicating the opening form at the time of capturing the MPH image.

[0165] Next, the noise information deciding section 112 selects the noise imparted by the noise imparting section 12 based on the MPH information (S42). In the case where the MPH information is set in step S51, the noise information deciding section 112 discards the MPH information of the previous time in step S42, and selects the noise corresponding to the newly set MPH information using the newly set MPH information.

[0166] Thus, the opening form of the MPH mask 15 is switched, and therefore, the sensing device 110a can further suppress the restoration of the outflow image compared to the case where the MPH mask 15 has only one opening form.

[0167] In addition, the processing of step S42 can also be performed before the MPH image is captured. The processing of step S42 can also be performed, for example, between step S51 and step S11.

[0168] Thus, since the noise imparted to the MPH image changes over time, the sensing device 110a can generate a noise imparted image that is more difficult to restore. Furthermore, by training the learning model using images each containing noise imparted over time, it is possible to suppress a decrease in recognition performance in noise imparted images in which the noise imparted over time differs.

[0169] (Other Embodiments)

[0170] The present disclosure has been described above based on the embodiments and the various modifications (hereinafter, also referred to as the embodiments, etc.). However, the present disclosure is not limited to the above-described embodiments, etc. As long as the gist of the present disclosure is not deviated from, a mode obtained by applying various modifications thought by those skilled in the art to the present embodiments, etc., and a mode constructed by combining the constituent elements in different embodiments can also be included in the scope of one or more modes of the present disclosure.

[0171] For example, in the above embodiment and the like, it is described that the noise imparted by the noise imparting section is an example of one kind of noise, but the noise can be a combination of at least two of the noises exemplified in Embodiment 1 described above. The noise imparted image can also be an MPH image to which two or more kinds of noise are imparted.

[0172] In addition, the MPH image imaging section in the above embodiment and the like can also be a lensless camera. The lensless camera has a transmission film on which a specific pattern such as a multi-pin hole, a dot pattern, or the like is printed, in place of a lens. In the transmission film, the region that transmits light is an example of an opening. In addition, the transmission film is an example of a mask.

[0173] In addition, the MPH image imaging section in the above embodiment and the like can also be a coded opening camera. The coded opening camera is a camera that uses a coded aperture. In the camera that uses the coded aperture, a mask having a mask pattern (aperture shape) is disposed on the object side of the image sensor. The coded aperture has a function of blocking incident light by the mask pattern, and controls the PSF of the image by the mask pattern. The region that transmits light in the mask is an example of an opening.

[0174] In addition, the shape of the opening formed on the mask in the above embodiment and the like is not particularly limited. The shape of the opening can be circular, can be annular, or can be a prescribed pattern. In addition, the number of openings formed on the mask can be one or more.

[0175] Further, in the above embodiment and the like, an example is described in which the object recognition processing using the learned model is performed in the recognition device, but the object recognition processing can also be performed in the sensing device. For example, the recognition device can also be built into the sensing device.

[0176] Further, in the above embodiment and the like, the training device has trained the learned model to output the recognition result of the object in the noise imparted image, but can also be trained, for example, to restore the noise imparted image. The training device can train the learning model, for example, to input the noise imparted image, and output a correct image (normal image) corresponding to the noise imparted image. In this case, the data set contains an image corresponding to the MPH image to which noise is imparted, that is, a training image, and a correct image for the training image.

[0177] Further, the order of the plurality of processes described in the above embodiment and the like is an example. The order of the plurality of processes can be changed, and the plurality of processes can also be executed in parallel. Further, a part of the plurality of processes can also not be executed.

[0178] Furthermore, the division of the functional blocks in the block diagrams is one example, and a plurality of functional blocks can be implemented as one functional block, or one functional block can be divided into a plurality of functional blocks, or a part of the functions can be transferred to another functional block. In addition, a plurality of functional blocks having similar functions can be processed by a single hardware or software in parallel or time-division.

[0179] Furthermore, each device possessed by the information processing system can be implemented as a single device, or can be implemented by a plurality of devices. In the case where the information processing system is implemented by a plurality of devices, the constituent elements possessed by the information processing system can be allocated to the plurality of devices in any manner. In addition, the communication method between the plurality of devices can be wired communication, or can be wireless communication. In addition, the communication standard used for the communication is not particularly limited.

[0180] In addition, each constituent element described in the above-described embodiments and the like can be implemented as software, and typically can be implemented as an integrated circuit, that is, an LSI. These can be individually subjected to single-chip implementation, or a part or all thereof can be subjected to single-chip implementation. Here, the LSI is assumed, but depending on the degree of integration, it can be also referred to as an IC, a system LSI, a super LSI, or a ultra LSI. In addition, the method of integration is not limited to the LSI, and a dedicated circuit or a general-purpose processor can be used. After LSI manufacturing, a FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor where the connection or the configuration of circuit cells inside the LSI can be reconfigured can be used. Furthermore, if a technology that replaces LSI appears as a result of technological development, it is naturally also possible to apply the technology to the integration of a constituent element.

[0181] Furthermore, the technology of the present disclosure can be the above-described program, or a non-transitory computer-readable recording medium that records the above-described program. In addition, the above-described program can naturally be distributed via a transmission medium such as the Internet. For example, the above-described program and a digital signal configured by the above-described program can be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like. In addition, the above-described program and a digital signal configured by the above-described program can be recorded in a recording medium and transferred, or transferred via a network or the like, and executed by another independent computer system.

[0182] In addition, in the above-described embodiments and the like, each constituent element can be configured by a dedicated hardware, or implemented by executing a software program suitable for each constituent element. Each constituent element can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0183] Industrial applicability

[0184] The present disclosure can be widely applied to an apparatus that uses images to recognize objects.

[0185] BRIEF DESCRIPTION OF DRAWINGS

[0186] 1 Information processing system

[0187] 10, 110, 110a Sensing device

[0188] 11 MPH image imaging section

[0189] 12 Noise imparting section

[0190] 13, 34 Transmitting section

[0191] 14 Lens

[0192] 15 Multi-pinhole mask

[0193] 15a Pinhole

[0194] 16 Image sensor

[0195] 17 ISP

[0196] 18 Communication section

[0197] 20 Recognition apparatus

[0198] 21 Receiving section

[0199] 22 Recognition section

[0200] 23 Output section

[0201] 30 Training apparatus

[0202] 31 MPH information obtaining section

[0203] 32 Noise information obtaining section

[0204] 33 Training section

[0205] 111 MPH information obtaining section

[0206] 111a MPH information setting section

[0207] 112 Noise information determining section

Claims

1. An image processing apparatus, comprising, Possessing: An image acquisition unit that acquires a first captured image from a first image pickup device that possesses a mask in which one or more openings are formed; An information acquisition unit that acquires opening shape information corresponding to a shape of the one or more openings; A noise imparting unit that imparts, to the first captured image, (i) noise having a frequency characteristic determined in accordance with a frequency characteristic of the mask, or (ii) noise having a frequency band determined in accordance with a band of the mask corresponding to the frequency characteristic of the mask, wherein the frequency characteristic of the mask is a frequency characteristic based on the shape of the one or more openings corresponding to the opening shape information; And An output unit that outputs the first captured image to which the noise is imparted.

2. The image processing apparatus according to claim 1, wherein The noise imparting unit imparts, to the first captured image, the noise having a frequency band wider than a prescribed band.

3. The image processing apparatus according to claim 1 or 2, wherein Further possessing a noise information determination unit that selects, from among a plurality of noises, noise having the same frequency characteristic as the frequency characteristic of the mask, as the noise imparted by the noise imparting unit, based on the frequency characteristic of the mask corresponding to the opening shape information.

4. The image processing apparatus according to claim 1 or 2, wherein The mask is configured to be capable of switching between a first opening shape and a second opening shape different from the first opening shape, Further possessing a switching unit that switches the opening shape of the mask from one of the first opening shape and the second opening shape to the other.

5. The image processing apparatus according to claim 1 or 2, wherein The noise includes at least one of salt and pepper noise, Laplacian noise, noise that changes an output value of a partial region of the first captured image to a certain value, white noise, and pink noise.

6. The image processing apparatus according to claim 3, wherein In a case where the frequency characteristic of the mask based on the opening shape corresponding to the opening shape information indicates a wideband characteristic including a first band and a second band higher than the first band, the noise information determination unit selects at least one of salt and pepper noise and noise that changes an output value of a partial region of the first captured image to a certain value, as the noise imparted by the noise imparting unit, and in a case where an intensity of a frequency component of the first band is higher than an intensity of a frequency component of the second band, selects Laplacian noise as the noise imparted by the noise imparting unit.

7. The image processing apparatus according to claim 1 or 2, wherein The opening shape information includes at least one of a point spread function (PSF), a size of the opening, a shape, and information related to a plurality of the openings in the mask.

8. The image processing apparatus according to claim 1 or 2, wherein The first image pickup device is one of a multi-pinhole camera, a lensless camera, and an encoded aperture camera.

9. A training method of a machine learning model, wherein acquire a data set including an image generated by imparting noise to a captured image acquired by an imaging device provided with a mask formed with one or more openings, the noise (i) having a frequency characteristic decided in accordance with a frequency characteristic of the mask or (ii) having a frequency band decided in accordance with a band of the mask corresponding to the frequency characteristic of the mask, wherein the frequency characteristic of the mask is decided based on opening shape information corresponding to a shape of the one or more openings, train a machine learning model using the acquired data set.

10. An identification device, wherein, provided with: an image acquisition section that acquires the first captured image to which the noise is imparted from the image processing device according to any one of claims 1 to 8; and an identification section that identifies an object appearing in the first captured image to which the noise is imparted using a machine learning model that is trained using a data set including an image generated by imparting noise decided in accordance with the opening shape information to a second captured image acquired from a second imaging device provided with the mask.

11. An image processing method in which, a captured image is acquired from an imaging device provided with a mask formed with one or more openings, opening shape information corresponding to a shape of the one or more openings is acquired, noise decided in accordance with a frequency characteristic of the mask is imparted to the captured image, the noise (i) having a frequency characteristic decided in accordance with the frequency characteristic of the mask or (ii) having a frequency band decided in accordance with a band of the mask corresponding to the frequency characteristic of the mask, wherein the frequency characteristic of the mask is a frequency characteristic based on a shape of the one or more openings corresponding to the opening shape information, and the captured image to which the noise is imparted is output.

Images