OBJECT CLASSIFICATION METHOD, VEHICLE CONTROL METHOD, INFORMATION DISPLAY METHOD, AND OBJECT CLASSIFICATION DEVICE
The object recognition method using a filter array with diverse wavelength filters and machine learning improves accuracy and efficiency by encoding images for precise object identification.
Patent Information
- Application Number
- JP2023147063
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-10-15
- Filing Date
- 2023-09-11
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-09-24
AI Technical Summary
Conventional object recognition using RGB images struggles with accuracy, particularly in distinguishing between real objects and their signs or posters, and hyperspectral imaging faces challenges with low spatial resolution and high computational demands.
An object recognition method utilizing a filter array with multiple filters having different wavelength dependencies, enabling accurate object recognition by capturing encoded image data without reconstructing wavelength-specific images, and employing machine learning for classification.
Enhances object recognition accuracy by maintaining high spatial resolution and reducing computational resources, allowing for efficient processing and improved distinction between objects.
Smart Images

Figure 0007591709000001 
Figure 0007591709000002 
Figure 0007591709000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an object classification method, a vehicle control method, an information display method, and an object classification device. [Background technology]
[0002] In object recognition using machine learning, monochrome or RGB images are generally used as training data. Meanwhile, attempts to use multispectral images, which contain more wavelength information than RGB images, for object recognition are also being considered.
[0003] Patent Document 1 discloses a spectral camera in which multiple filters that pass light in different wavelength ranges are spatially arranged in a mosaic pattern as a sensor for acquiring a multispectral image. Patent Document 2 discloses a method for learning images of immune cells for multiple image channels using a convolutional neural network in order to improve the recognition accuracy of immune cells in an image. Patent Document 3 discloses a machine learning method in which multispectral images or hyperspectral images are used as training data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Special Publication No. 2015-501432 [Patent Document 2] International Publication No. 2015 / 177268 [Patent Document 3] US Patent Application Publication No. 2017 / 0076438 Summary of the Invention [Problem to be solved by the invention]
[0005] The present disclosure provides a novel object recognition method that enables highly accurate object recognition from encoded image data. [Means for solving the problem]
[0006] An object recognition method according to an embodiment of the present disclosure includes acquiring image data of an image including feature information indicating features of an object, and recognizing the object included in the image based on the feature information. The image data is acquired by capturing the image using a first imaging device including an image sensor and a filter array arranged in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having different wavelength dependencies of optical transmittance, each of the two or more filters having a maximum optical transmittance in a plurality of wavelength ranges. Effect of the Invention
[0007] According to the present disclosure, highly accurate object recognition becomes possible. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an object recognition device according to an exemplary embodiment of the present disclosure. [Figure 2A] FIG. 2A is a schematic diagram illustrating an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of a spatial distribution of the transmittance of light in each of a plurality of wavelength ranges included in the target wavelength range. [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of the region A1 in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of the region A2 in the filter array shown in FIG. 2A. [Figure 3A] FIG. 3A is a diagram illustrating an example of a two-dimensional distribution of a filter array. [Figure 3B] FIG. 3B is a diagram illustrating another example of a two-dimensional distribution of a filter array. [Figure 4A]FIG. 4A is a flowchart illustrating an example of an object recognition method using the object recognition device in an exemplary embodiment. [Figure 4B] FIG. 4B is a flowchart illustrating an example of a process for generating a classification model. [Figure 4C] FIG. 4C is a schematic diagram illustrating an example of multiple training data sets in an exemplary embodiment. [Figure 4D] FIG. 4D is a diagram illustrating an example in which an object recognition result is fed back to a classification model. [Figure 4E] FIG. 4E is a flowchart showing another example of an object recognition method using the object recognition device in an exemplary embodiment. [Figure 5A] FIG. 5A is a diagram illustrating a function of displaying a recommended area for object recognition to assist in imaging. [Figure 5B] FIG. 5B is a diagram illustrating the magnification of an object by an optical system having a zoom function. [Figure 5C] FIG. 5C is a diagram illustrating a modified example of the filter array. [Figure 6A] FIG. 6A is a diagram illustrating an example of application of an object recognition device in an exemplary embodiment. [Figure 6B] FIG. 6B is a diagram illustrating another application example of the object recognition device according to the exemplary embodiment. [Figure 6C] FIG. 6C is a diagram illustrating another application example of the object recognition device according to the exemplary embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of vehicle control using the object recognition device in the exemplary embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an object recognition device according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.
[0010] Conventional object recognition using RGB images has limitations in its recognition capabilities. For example, it may not be possible to distinguish between an actual object and a signboard or poster. This is generally due to the small difference between the amount of each of the R, G, and B components of the light reflected from the actual object and the amount of each of the R, G, and B components of the light reflected from the signboard or poster. In order to distinguish between an actual object and a signboard or poster, for example, it is conceivable to use multi-wavelength spectral data. This may make it possible to detect minute differences in spectral data due to differences in the materials of the objects.
[0011] In a conventional hyperspectral camera, as disclosed in Patent Document 1, for example, a plurality of wavelength filters with different transmission wavelength ranges are arranged two-dimensionally. When one frame of an image is acquired in a single shot, such as in video shooting, there is a trade-off between the number of wavelength ranges and the spatial resolution. That is, when many filters with different transmission wavelength ranges are spatially distributed and arranged to acquire a multi-wavelength image, the spatial resolution of the image acquired for each wavelength range is low. Therefore, even if a hyperspectral image is used for object recognition in the hope of improving the object recognition accuracy, in reality, the recognition accuracy may decrease due to the low spatial resolution.
[0012] It is also possible to improve both the wavelength resolution and the resolution by increasing the number of pixels of the image sensor. In this case, a large amount of three-dimensional data is handled by adding multi-wavelength data to two-dimensional spatial data. When applying machine learning to such large data, a lot of time or resources are spent on preprocessing, learning, communication, and data storage.
[0013] Based on the above considerations, the present inventor has come up with the object recognition method described in the following items.
[0014] [Item 1] The object recognition method according to the first aspect includes acquiring image data of an image including feature information indicating features of an object, and recognizing the object included in the image based on the feature information. The image data is acquired by capturing the image using a first imaging device including an image sensor and a filter array arranged in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having different wavelength dependencies of optical transmittance, the optical transmittance of each of the two or more filters having a maximum value in a plurality of wavelength ranges.
[0015] [Item 2] In the object recognition method related to the first item, recognizing the object is performed by applying a classification model trained by a machine learning algorithm to the image data, and the classification model may be trained in advance using a plurality of first training data sets each including training image data and label data that identifies the object included in the training image represented by the training image data.
[0016] [Item 3] In the object recognition method related to the second item, the multiple learning image data included in the multiple first training data sets may include learning image data generated by a second imaging device different from the first imaging device.
[0017] [Item 4] In the object recognition method according to the third aspect, the second imaging device may include a filter array having the same characteristics as the filter array in the first imaging device.
[0018] [Item 5] The object recognition method according to any of the second to fourth items may further include, after the object is recognized, further training the classification model using a second training data set including the image data and second label data that identifies the object.
[0019] [Item 6] In an object recognition method relating to any one of the second to fifth items, the position of the object in the training image in a plurality of training image data included in the plurality of first training data sets may be different from one another in the plurality of training image data.
[0020] [Item 7] In the object recognition method relating to any one of items 2 to 6, the training image data may be acquired by capturing an image of the object in a state in which the object occupies a predetermined range or more within the training image.
[0021] [Item 8] In the object recognition method relating to any of items 1 to 7, acquiring the image data is performed using an imaging device including a display, and the object recognition method may further include causing the display to display an auxiliary display to inform a user of the area in the image in which the object should be located or the range in which the object should occupy before the image data is acquired.
[0022] [Item 9] In the object recognition method according to any one of items 1 to 8, the multiple filters may have different wavelength dependencies of light transmittance, and the light transmittance of each of the multiple filters may have a maximum value in multiple wavelength ranges.
[0023] [Item 10] A vehicle control method according to a tenth item is a vehicle control method using the object recognition method according to any one of the first to ninth items, wherein the first imaging device is attached to a vehicle, and includes controlling operation of the vehicle based on a result of recognizing the object.
[0024] [Item 11] An information display method according to an eleventh item is an information display method using an object recognition method according to any one of the first to ninth items, and includes obtaining data indicating at least one selected from the group consisting of a name of the object and a description of the object from a database based on a result of recognizing the object, and displaying the at least one selected from the group consisting of the name of the object and the description of the object on a display.
[0025] [Item 12] The object recognition method according to the twelfth aspect includes acquiring image data of an image including feature information indicating features of an object, and recognizing the object included in the image based on the feature information. The image data is acquired by repeating an operation of capturing the image with some of the light sources emitting light by a first imaging device including an image sensor and a light source array including a plurality of light sources emitting light in different wavelength ranges, while changing a combination of light sources included in the portion of the plurality of light sources.
[0026] [Item 13] In the object recognition method related to the twelfth item, recognizing the object is performed by applying a classification model trained by a machine learning algorithm to the image data, and the classification model may have been trained in advance using a plurality of first training data sets each including training image data and label data that identifies the object included in the training image represented by the training image data.
[0027] [Item 14] In the object recognition method related to the thirteenth item, the multiple learning image data included in the multiple first training data sets may include learning image data generated by a second imaging device different from the first imaging device.
[0028] [Item 15] In the object recognition method according to the fourteenth aspect, the second imaging device may include a light source array having characteristics equivalent to those of the light source array in the first imaging device.
[0029] [Item 16] The object recognition method according to any one of items 13 to 15 may further include, after the object is recognized, further training the classification model with a second training data set including the image data and second label data that identifies the object.
[0030] [Item 17] In an object recognition method according to any one of items 13 to 16, the positions of the object in the training images in a plurality of training image data included in the plurality of first training data sets may be different from each other in the plurality of training image data.
[0031] [Item 18] In the object recognition method relating to any of items 13 to 17, the training image data may be acquired by capturing an image of the object occupying a predetermined range or more within the training image.
[0032] [Item 19] In an object recognition method relating to any of items 12 to 18, acquiring the image data is performed using an imaging device including a display, and the object recognition method may further include causing the display to display an auxiliary display to inform a user of the area in the image in which the object should be located or the range in which the object should occupy before the image data is acquired.
[0033] [Item 20] A vehicle control method according to the 20th item is a vehicle control method using the object recognition method according to any one of the 12th to 19th items, wherein the first imaging device is attached to a vehicle, and includes controlling operation of the vehicle based on the result of recognizing the object.
[0034] [Item 21] An information display method according to the 21st item is an information display method using an object recognition method according to any one of the 12th to 19th items, and includes obtaining data indicating at least one selected from the group consisting of a name of the object and a description of the object from a database based on a result of recognizing the object, and displaying the at least one selected from the group consisting of the name of the object and the description of the object on a display.
[0035] [Item 22] The object recognition device according to the 22nd item comprises an image sensor that generates image data of an image including feature information indicating features of an object, a filter array arranged in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having different wavelength dependencies of optical transmittance, each of the two or more filters having a maximum optical transmittance in a plurality of wavelength ranges, and a signal processing circuit that recognizes the object included in the image based on the feature information.
[0036] [Item 23] The object recognition device according to the 23rd item comprises an image sensor that generates an image signal of an image including an object, a light source array including a plurality of light sources that emit light in different wavelength ranges, a control circuit that controls the image sensor and the plurality of light sources, the control circuit repeating an operation of causing the image sensor to capture an image with a portion of the plurality of light sources emitting light a plurality of times while changing the combination of light sources included in the portion of the plurality of light sources, and a signal processing circuit that recognizes the object included in the image based on feature information indicating features of the object included in image data composed of the image signals generated by the image sensor for each of the plurality of captures.
[0037] [Item 24] An object recognition device according to a twenty-fourth aspect includes a memory and a signal processing circuit. The signal processing circuit receives two-dimensional image data of an image including a plurality of pixels, the two-dimensional image data being multi / hyperspectral image data in which information of a plurality of wavelength ranges is multiplexed into data of each of the plurality of pixels and the luminance distribution of each of the plurality of pixels is encoded, and recognizes an object included in a scene represented by the two-dimensional image data based on feature information included in the two-dimensional image data.
[0038] [Item 25] In the object recognition device according to the 24th item, the feature information may be extracted from the two-dimensional image data without reconstructing images of each of the multiple wavelength bands based on the two-dimensional image data.
[0039] [Item 26] The object recognition device according to a twenty-fourth aspect may further include an imaging device that acquires the two-dimensional image data.
[0040] [Item 27] In the object recognition device according to the twenty-sixth item, the two-dimensional image data may be acquired by capturing an image of the object in a state in which the object occupies a predetermined range or more in an imaging area of the imaging device.
[0041] [Item 28] The object recognition device according to the 27th item may further include a display that displays an auxiliary display to inform a user of the area in which the object should be located or the range which the object should occupy in an image captured by the imaging device before the two-dimensional image data is acquired by the imaging device.
[0042] [Item 29] In the object recognition device related to the 26th item, the imaging device may include an image sensor and a filter array arranged in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having different wavelength dependencies of optical transmittance, and the optical transmittance of each of the two or more filters having a maximum value in a plurality of wavelength ranges.
[0043] [Item 30] In the object recognition device according to the twenty-ninth item, the plurality of filters may include a plurality of subsets that are periodically arranged.
[0044] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, and arrangement positions of the components shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components.
[0045] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, although it is called an LSI or an IC, the name may change depending on the degree of integration, and it may be called a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of an LSI, or a reconfigurable logic device, which can reconfigure the connection relationship inside the LSI or set up the circuit partition inside the LSI, may also be used for the same purpose.
[0046] Furthermore, all or part of the functions or operations of the circuit, unit, device, member, or part can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.
[0047] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[0048] (Embodiment 1) FIG. 1 is a diagram illustrating an example of an object recognition device 300 in an exemplary embodiment 1 of the present disclosure. FIG. 1 illustrates a situation in which a mushroom is photographed as an example. The photographed object 70 may be any object. The object recognition device 300 in the embodiment 1 includes an imaging device 150, a signal processing circuit 200, a display 400, and a memory 500. The imaging device 150 includes an optical system 40, a filter array 100C, and an image sensor 60. The object recognition device 300 may be a computer such as a smartphone or a tablet computer. A camera mounted on these computers may function as the imaging device 150.
[0049] The filter array 100C is disposed in the optical path of light incident on the image sensor 60. In this embodiment, the filter array 100C is disposed at a position facing the image sensor 60. The filter array 100C may be disposed at another position. The image of the light from the object 70 is encoded by the filter array 100C. Here, "encoding" means modulating the image by attenuating the light incident on the filter array 100C at an attenuation rate that depends on the wavelength and position of the light. The image data generated based on the image modulated in this way is called "encoded image data". The configuration and encoding of the filter array 100C will be described in detail later.
[0050] The image sensor 60 may be a monochrome type imaging element having a plurality of photodetection cells, which are a plurality of pixels arranged two-dimensionally on an imaging surface. The image sensor 60 may be, for example, a charge-coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, an infrared array sensor, a terahertz array sensor, or a millimeter wave array sensor. The photodetection cells include, for example, photodiodes. The image sensor 60 does not necessarily have to be a monochrome type imaging element. For example, a color type imaging element having an R / G / B, R / G / B / IR, or R / G / B / W filter may be used. The image sensor 60 may have detection sensitivity not only in the visible wavelength range, but also in the wavelength ranges of X-rays, ultraviolet, near infrared, mid-infrared, far infrared, microwaves, and radio waves.
[0051] The image sensor 60 is disposed in the optical path of the light that has passed through the filter array 100C. The image sensor 60 receives the light that has passed through the filter array 100C and generates an image signal. Each photodetection cell in the image sensor 60 outputs a photoelectric conversion signal according to the amount of light received. The image signal is generated by a plurality of photoelectric conversion signals output from a plurality of photodetection cells. FIG. 1 shows a schematic diagram of an example of a captured image 120 that is configured by the image signal, i.e., encoded image data.
[0052] The optical system 40 includes at least one lens. In the example shown in FIG. 1, the optical system 40 is depicted as one lens, but may be configured by a combination of multiple lenses. The optical system 40 may have a zoom function as described later. The optical system 40 forms an image of light from the object 70 on the filter array 100C.
[0053] The signal processing circuit 200 is a circuit that processes an image signal output from the image sensor 60. The signal processing circuit 200 can be realized by, for example, a combination of a central processing unit (CPU), an image processing arithmetic processor (GPU), and a computer program. Such a computer program is stored in a recording medium such as a memory, and a processor such as the CPU or GPU executes the program to perform the recognition process described below. The signal processing circuit 200 may be a digital signal processor (DSP) or a programmable logic device (PLD) such as a field programmable gate array (FPGA). The signal processing circuit 200 may be included in a server computer connected to an imaging device 150 or a device such as a smartphone via a network such as the Internet.
[0054] The signal processing circuit 200 recognizes the object 70 from the encoded image data. For example, a model trained by a known machine learning algorithm may be used to recognize the object 70. Details of the object recognition method will be described later.
[0055] The display 400 displays information associated with the recognized object 70. The display 400 may be, for example, the display of a smartphone or a tablet computer. The display 400 may also be a display connected to a personal computer or the like, or a display built into a laptop computer.
[0056] Next, the configuration and encoding of filter array 100C will be described in detail.
[0057] FIG. 2A is a diagram showing a schematic example of a filter array 100C. The filter array 100C has a plurality of regions arranged two-dimensionally. In this specification, the regions may be referred to as "cells." A filter having an individually set spectral transmittance is arranged in each region. Here, "spectral transmittance" means a light transmittance having wavelength dependency. The spectral transmittance is expressed by a function T(λ) where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take a value between 0 and 1. Thus, the filter array 100C includes a plurality of filters arranged two-dimensionally along a plane intersecting the optical path.
[0058] In the example shown in FIG. 2A, the filter array 100C has 48 rectangular regions arranged in 6 rows and 8 columns. In practical applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels of a general imaging element such as an image sensor. The number of pixels may be, for example, hundreds of thousands to tens of millions. In one example, the filter array 100C may be disposed directly above the imaging element such that each region corresponds to one pixel of the imaging element. Each region faces, for example, one or more pixels of the imaging element.
[0059] FIG. 2B is a diagram showing an example of the spatial distribution of the light transmittance of each of multiple wavelength bands W1, W2, ..., Wi included in the target wavelength band. In the example shown in FIG. 2B, the difference in shading of each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.
[0060] 2C and 2D are diagrams showing examples of the spectral transmittance of the region A1 and the region A2 included in the multiple regions of the filter array 100C shown in FIG. 2A, respectively. The spectral transmittance of the region A1 and the spectral transmittance of the region A2 are different from each other. In this way, the spectral transmittance of the filter array 100C differs depending on the region. However, it is not necessary that the spectral transmittance of all the regions is different. The spectral transmittance of at least two of the multiple regions in the filter array 100C is different from each other. That is, the filter array 100C includes two or more filters having different spectral transmittances from each other. The spectral transmittance of each of the two or more filters has a maximum value in multiple wavelength ranges and a minimum value in multiple other wavelength ranges.
[0061] Here, the meaning of "maximum value" and "minimum value" in this disclosure will be explained. When the spectral transmittance of the filter of interest is normalized so that the maximum value is 1 and the minimum value is 0, a value that exceeds 0.5 and has a difference of 0.2 or more with an adjacent minimum value is defined as a "maximum value" in this disclosure. Similarly, when the above normalization is performed, a value that is less than 0.5 and has a difference of 0.2 or more with an adjacent maximum value is defined as a "minimum value" in this disclosure. All of the spectral transmittances of the multiple filters in the filter array 100C may be different from each other. In this case, the spectral transmittance of each filter may have a maximum value in multiple wavelength ranges and a minimum value in multiple other wavelength ranges. In an example, the number of patterns of the spectral transmittance of the multiple filters included in the filter array 100C may be the same as or greater than the number i of wavelength ranges included in the target wavelength range. Typically, the filter array 100C may be designed so that the spectral transmittances of more than half of the filters are different.
[0062] The filter array 100C modulates the incident light into light having multiple discrete intensity peaks for each wavelength, and outputs the multiple wavelengths of light in a superimposed manner, whereby the image of the light passing through the filter array 100C is encoded.
[0063] The resolution in the wavelength direction of the spectral transmittance of each region can be set to approximately the bandwidth of a desired wavelength range. In other words, the width of the range that has a value equal to or greater than the average value between the minimum value closest to the maximum value and the maximum value in a wavelength range that includes one maximum value in the curve of the spectral transmittance can be set to approximately the bandwidth of the desired wavelength range. In this case, if the spectral transmittance is resolved into frequency components by, for example, Fourier transform, the value of the frequency component corresponding to that wavelength range becomes relatively large.
[0064] The filter array 100C is typically divided into a number of cells corresponding to a number of regions divided into a grid pattern, as shown in FIG. 2A. These cells have different spectral transmittances. The wavelength distribution and spatial distribution of the light transmittance of each region of the filter array 100C may be, for example, a random distribution or a quasi-random distribution.
[0065] The concept of random distribution and quasi-random distribution is as follows. First, each region in the filter array 100C can be considered as a vector element having a value, for example, from 0 to 1, depending on the light transmittance. Here, when the transmittance is 0, the value of the vector element is 0, and when the transmittance is 1, the value of the vector element is 1. In other words, a set of regions arranged in a row or column direction can be considered as a multidimensional vector having a value from 0 to 1. Therefore, it can be said that the filter array 100C has a plurality of multidimensional vectors in the column or row direction. In this case, the random distribution means that any two multidimensional vectors are independent, that is, not parallel. Moreover, the quasi-random distribution means that a configuration in which some multidimensional vectors are not independent from each other is included. Therefore, in the random distribution and the quasi-random distribution, a vector whose elements are the values of the transmittance of light in the first wavelength range in each region belonging to a set of regions arranged in one row or column included in a plurality of regions, and a vector whose elements are the values of the transmittance of light in the first wavelength range in each region belonging to a set of regions arranged in another row or column are independent of each other. Similarly, for a second wavelength range different from the first wavelength range, a vector whose elements are the values of the transmittance of light in the second wavelength range in each region belonging to a set of regions arranged in one row or column contained in the multiple regions, and a vector whose elements are the values of the transmittance of light in the second wavelength range in each region belonging to a set of regions arranged in another row or column are independent of each other.
[0066] When the filter array 100C is disposed near or directly above the image sensor 60, the cell pitch, which is the distance between the multiple regions in the filter array 100C, may be approximately equal to the pixel pitch of the image sensor 60. In this way, the resolution of the encoded light image emitted from the filter array 100C approximately matches the pixel resolution. When the filter array 100C is disposed away from the image sensor 60, the cell pitch may be made finer depending on the distance.
[0067] In the examples shown in Figures 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessary to use a grayscale transmittance distribution. For example, a binary-scale transmittance distribution in which the transmittance of each region can take a value of either approximately 0 or approximately 1 may be adopted. In a binary-scale transmittance distribution, each region transmits most of the light in at least two of the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.
[0068] A part of all the cells, for example, half of the cells, may be replaced with a transparent region. Such a transparent region transmits light from all wavelength ranges W1 to Wi included in the target wavelength range with the same high transmittance. The high transmittance is, for example, 0.8 or more. In such a configuration, the multiple transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the multiple regions in the filter array 100C, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately. In the example shown in FIG. 2A, the two arrangement directions are the horizontal direction and the vertical direction. By extracting the components transmitted through the transparent regions arranged in a checkerboard pattern, monochrome images can be simultaneously acquired with one camera.
[0069] The filter array 100C may be composed of at least one selected from the group consisting of a multilayer film, an organic material, a diffraction grating structure, and a microstructure containing a metal. In the case of a multilayer film, for example, a dielectric multilayer film or a multilayer film containing a metal film is used. At this time, at least one of the thickness, material, and stacking order of the multilayer film can be designed to be different in each cell. This allows different spectral characteristics to be realized in each cell. In addition, the multilayer film can realize spectral characteristics with a sharp rise or fall. In the case of using an organic material, different spectral characteristics can be realized in each cell by using different pigments or dyes, or by stacking different materials. In the case of a diffraction grating structure, different spectral characteristics can be realized in each cell by providing a diffraction structure with a different diffraction pitch or depth. In the case of a microstructure containing a metal, different spectral characteristics can be realized by spectroscopic analysis due to the plasmon effect.
[0070] The filter array 100C is disposed near or directly above the image sensor 60. Here, "near" means that the image of the light from the optical system 40 is formed on the surface of the filter array 100C with a certain degree of clarity. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 100C and the image sensor 60 may be integrated. The filter array 100C is a mask having a spatial distribution of light transmittance. The filter array 100C modulates the intensity of the incident light and passes it through.
[0071] 3A and 3B are diagrams illustrating an example of a two-dimensional distribution of the filter array 100C.
[0072] As shown in FIG. 3A, the filter array 100C may be configured by a binary mask. The black portion represents light blocking, and the white portion represents light transmission. 100% of the light passing through the white portion is transmitted, and 100% of the light passing through the black portion is blocked. The two-dimensional distribution of the transmittance of the mask may be a random distribution or a quasi-random distribution. The two-dimensional distribution of the transmittance of the mask does not necessarily have to be completely random. This is because the encoding by the filter array 100C is performed to distinguish each image of each wavelength. In addition, the ratio of the black portion to the white portion does not have to be 1:1. For example, the white portion:black portion may be 1:9. As shown in FIG. 3B, the filter array 100C may be a mask having a grayscale transmittance distribution.
[0073] 3A and 3B, the filter array 100C has a different spatial distribution of transmittance for each of the wavelength bands W1, W2, ..., Wi. The spatial distributions of the transmittance for each of the wavelength bands do not match even when translated.
[0074] The image sensor 60 may be a monochrome type imaging element having two-dimensional pixels. However, the image sensor 60 does not necessarily have to be composed of a monochrome type imaging element. The image sensor 60 may be a color type imaging element having, for example, R / G / B, R / G / B / IR, or R / G / B / W filters. The color type imaging element can increase the amount of information related to wavelengths. This makes it possible to complement the characteristics of the filter array 100C and facilitates filter design.
[0075] Next, a process of acquiring image data showing the captured image 120 by the object recognition device 300 of this embodiment will be described. The image of light from the object 70 is formed by the optical system 40 and encoded by the filter array 100C installed just before the image sensor 60. As a result, images having different encoded information for each wavelength range are superimposed on each other and formed as a multiple image on the image sensor 60. This results in the captured image 120. At this time, since a spectroscopic element such as a prism is not used, no spatial shift of the image occurs. This makes it possible to maintain high spatial resolution even in the case of multiple images. As a result, it becomes possible to improve the accuracy of object recognition.
[0076] The wavelength range may be limited by installing a bandpass filter in a part of the object recognition device 300. When the wavelength range of the object 70 is known to some extent, the identification range can also be limited by limiting the wavelength range. As a result, high object recognition accuracy can be achieved.
[0077] Next, an object recognition method using the object recognition device 300 in this embodiment will be described.
[0078] 4A is a flowchart showing an example of an object recognition method using the object recognition device 300 in this embodiment. This object recognition method is executed by the signal processing circuit 200. The signal processing circuit 200 executes a computer program stored in the memory 500 to perform the processes of steps S101 to S104 shown in FIG. 4A.
[0079] First, a user captures an image of an object 70 using the imaging device 150 included in the object recognition device 300. As a result, an encoded captured image 120 is obtained.
[0080] In step S101, the signal processing circuit 200 acquires image data generated by the imaging device 150. The image data represents the captured image 120 that has been encoded.
[0081] In step S102, the signal processing circuit 200 performs preprocessing on the acquired image data. The preprocessing is performed to improve the recognition accuracy. The preprocessing may include, for example, area extraction, smoothing processing for noise removal, feature extraction, and other processing. The preprocessing may be omitted if unnecessary.
[0082] In step S103, the signal processing circuit 200 applies the trained classification model to the image data to identify the object 70 included in the scene represented by the preprocessed image data. The classification model is trained in advance, for example, by a known machine learning algorithm. The classification model will be described in detail later.
[0083] In step S104, the signal processing circuit 200 outputs information associated with the object 70. The signal processing circuit 200 outputs information such as the name and / or detailed information of the object 70 to the display 400. The display 400 displays an image showing the information. The information is not limited to an image and may be presented by, for example, sound.
[0084] Next, the classification model used in the object recognition method will be described.
[0085] FIG. 4B is a flowchart illustrating an example of a process for generating a classification model.
[0086] In step S201, the signal processing circuit 200 collects a plurality of training data sets. Each of the plurality of training data sets includes learning image data and label data. The label data is information for identifying an object 70 included in a scene represented by the learning image data. The learning image data is image data encoded in a manner similar to that of the image data described above. The plurality of learning image data included in the plurality of training data sets may include learning image data generated by the imaging device 150 in this embodiment or another imaging device. Details of the plurality of training data sets will be described later.
[0087] In step S202, the signal processing circuit 200 performs pre-processing on the learning image data included in each training data. The pre-processing is as described above.
[0088] In step S203, the signal processing circuit 200 generates a classification model from a plurality of training data sets by machine learning. For machine learning, algorithms such as deep learning, support vector machine, decision tree, genetic programming, or Bayesian network may be used. When deep learning is used, algorithms such as convolutional neural network (CNN) or recurrent neural network (RNN) may be used.
[0089] In this embodiment, by utilizing a model trained by machine learning, information about objects in a scene can be obtained directly from the encoded image data. In order to do the same thing with conventional technology, many calculations were required. For example, it was necessary to reconstruct image data of each wavelength range from the encoded image data using a method such as compressed sensing, and to identify the object from the image data. In contrast, in this embodiment, it is not necessary to reconstruct image data of each wavelength range from the encoded image data. Therefore, it is possible to save time or computational resources spent on the reconstruction process.
[0090] FIG. 4C is a diagram showing a schematic example of a plurality of training data sets in this embodiment. In the example shown in FIG. 4C, each training data set includes coded image data showing one or more mushrooms and label data showing whether the mushrooms are edible or poisonous. In this way, for each training data set, coded image data and label data showing a correct label correspond one-to-one. The correct label may be, for example, information showing the name, characteristics, sensory evaluation such as "delicious" or "bad", or judgment such as "good" or "bad". In general, the more training data sets there are, the higher the accuracy of learning can be. Here, the position of the object 70 in the image in the plurality of learning image data included in the plurality of training data sets may differ depending on the learning image data. The coding information differs for each pixel. Therefore, the more learning image data in which the position of the object 70 in the image differs, the higher the accuracy of object recognition by the classification model can be.
[0091] In the object recognition device 300 of this embodiment, the classification model is incorporated in the signal processing circuit 200 before the user uses it. As another method, the encoded image data representing the captured image 120 may be transmitted to a classification system prepared separately outside via a network or cloud. The classification system is capable of high-speed processing using, for example, a supercomputer. As a result, even if the processing speed of the terminal on the user side is weak, as long as it can be connected to the network, the recognition result of the object 70 can be provided to the user at high speed.
[0092] The image data acquired in step S101 in FIG. 4A and the learning image data acquired in step S201 in FIG. 4B may be encoded by, for example, a filter array having equivalent characteristics. In this case, the recognition accuracy of the object 70 can be improved. Here, the filter array having equivalent characteristics does not need to have the same characteristics exactly, and some filters may have different spectral transmission characteristics. For example, the characteristics of several percent to several tens of percent of the filters may be different. When the learning image data is generated by another imaging device, the other imaging device may include a filter array having equivalent characteristics to the filter array 100C included in the imaging device 150.
[0093] The recognition results of the object 70 may be fed back to the classification model, so that the classification model can be further trained.
[0094] 4D is a diagram showing an example of feeding back the recognition result of the object 70 to the classification model. In the example shown in FIG. 4D, the learned classification model is applied to the preprocessed encoded image data, and the classification result is output. The result is then added to a dataset, and further machine learning is performed using the dataset. This allows the model to be further trained, and the prediction accuracy can be improved.
[0095] FIG. 4E is a flowchart showing in more detail the operation when the recognition result is fed back to the classification model.
[0096] Steps S301 to S304 shown in Fig. 4E are the same as steps S101 to S104 shown in Fig. 4A, respectively. Then, steps S305 to S307 are executed.
[0097] In step S305, the signal processing circuit 200 generates a new training data set including the image data acquired in step S301 and label data indicating the object 70 recognized in step S303.
[0098] In step S306, the signal processing circuit 200 further trains the classification model with new training data sets. This training process is similar to the training processes shown in steps S202 and S203 in FIG. 4B.
[0099] In step S307, the signal processing circuit 200 judges whether or not to continue recognizing the object 70. If the judgment is Yes, the signal processing circuit 200 executes the process of step S301 again. If the judgment is No, the signal processing circuit 200 ends the recognition of the object 70.
[0100] In this way, by feeding back the recognition result of the object 70 to the classification model, the recognition accuracy of the classification model can be improved. Furthermore, it becomes possible to create a classification model suited to the user.
[0101] If a classification system is provided separately, the user may transmit a dataset including the recognition result of the object 70 to the classification system via a network for feedback. The dataset may include data indicating the captured image 120 generated by imaging, or data obtained by preprocessing the captured image 120, and label data indicating the recognition result by the classification model or a correct label based on the user's knowledge. The user who transmits the dataset for feedback may be given an incentive, such as a reward or points, by the provider of the classification system. An access permission for the captured image 120 taken by the user or an authentication of whether to automatically transmit the captured image 120 may be displayed on the display 400, for example, by a screen pop-up, before transmission.
[0102] The filter array 100C can multiplex multiple wavelength information per pixel, instead of one wavelength information per pixel. The captured image 120 includes multiplexed two-dimensional information. The two-dimensional information is, for example, randomly encoded spectral information for space and wavelength. When a fixed pattern is used as the filter array 100C, the encoding pattern is learned by machine learning. As a result, although the input data is two-dimensional, the information is substantially three-dimensional (i.e., two-dimensional position and one-dimensional wavelength) for object recognition.
[0103] Since the image data in this embodiment is data in which wavelength information is multiplexed, it is possible to increase the spatial resolution per wavelength compared to conventional hyperspectral images that sacrifice spatial resolution. Furthermore, the object recognition device 300 in this embodiment is capable of acquiring one frame of image data in a single shot. This allows object recognition that is resistant to moving objects or camera shake compared to conventional hyperspectral imaging methods that use a scanning method with high resolution.
[0104] Conventional hyperspectral image capture has a problem of low detection sensitivity per wavelength. For example, when decomposing into 40 wavelengths, the amount of light per pixel is reduced to 1 / 40 compared to when no decomposition is performed. In contrast, in the method of the present embodiment, as illustrated in Figs. 3A and 3B, for example, about 50% of the amount of incident light is detected by the image sensor 60. This increases the amount of detected light per pixel compared to conventional hyperspectral images. As a result, the signal-to-noise ratio of the image is increased.
[0105] Next, examples of other functions of the imaging device implementing the object recognition method according to this embodiment will be described.
[0106] FIG. 5A is a diagram showing a function of displaying a recommended area for object recognition to assist in imaging by a camera. When an object 70 is imaged on the image sensor 60 as being extremely small or extremely large, a difference occurs between the image of the imaged object 70 and the image of the training data set recognized during learning, and the recognition accuracy decreases. The filter array 100C, for example, contains different wavelength information for each pixel. Therefore, if the object 70 is detected only in a part of the imaging area of the image sensor 60, a bias occurs in the wavelength information. In order to prevent a bias in the wavelength information, the object 70 can be photographed as widely as possible in the imaging area of the image sensor 60. Also, if the image of the object 70 is photographed in a state where it protrudes from the imaging area of the image sensor 60, a lack of information on the spatial resolution of the object 70 occurs. Therefore, the recommended area for object recognition is slightly inside the imaging area of the image sensor 60. In the example shown in FIG. 5A, an auxiliary display 400a indicating the recommended area for object recognition is displayed on the display 400. In FIG. 5A, the entire area of the display 400 corresponds to the imaging area of the image sensor 60. For example, an area of 60% to 98% of the width or height of the imaging area may be displayed on the display 400 as a recommended area for object recognition. The recommended area for object recognition may be an area of 70% to 95% of the width or height of the shooting area, or an area of 80% to 90%. In this manner, before the image data is acquired by the imaging device 150, the auxiliary display 400a may be displayed on the display 400. The auxiliary display 400a informs the user of the area in which the object 70 should be located in the scene to be captured or the range that the object 70 should occupy. Similarly, each of the multiple learning image data included in the multiple training data sets may be acquired by capturing an image of the object 70 in a state in which the object 70 occupies a predetermined range or more in the image.
[0107] Fig. 5B is a diagram showing a schematic view of an object 70 being enlarged by an optical system having a zoom function. In the example shown in the left part of Fig. 5B, the object 70 before enlargement is displayed on the display 400, and in the example shown in the right part of Fig. 5B, the object 70 after enlargement is displayed on the display 400. In this way, the optical system 40 having a zoom function can form a wide image of the object 70 on the image sensor 60.
[0108] FIG. 5C is a diagram showing a schematic diagram of a modified example of the filter array 100C. In the example shown in FIG. 5C, a group of regions AA consisting of a collection of a plurality of regions (A1, A2, ...) is periodically arranged. The plurality of regions have different spectral characteristics from each other. Periodic means that the group of regions AA is repeated two or more times in the vertical direction and / or horizontal direction while maintaining the spectral characteristics. The filter array 100C shown in FIG. 5C can prevent spatial bias of wavelength information. Furthermore, in learning object recognition, learning may be performed not by the entire filter array 100C shown in FIG. 5C but by only the group of regions AA, which is a subset of the periodic structure. This can shorten the learning time. By periodically arranging filters having the same spectral characteristics in space, object recognition becomes possible even when an object is captured in a portion of the imaging area instead of the entire area.
[0109] The image encoded by the filter array 100C may include, for example, randomly multiplexed wavelength information. This makes the image difficult for the user to see. Therefore, the object recognition device 300 may be provided with a separate normal camera for displaying to the user. That is, the object recognition device 300 may be provided with a binocular configuration of the imaging device 150 and a normal camera. This allows the user to see a non-encoded visible monochrome image on the display 400. As a result, the user can easily grasp the positional relationship between the object 70 and the imaging area of the image sensor 60.
[0110] The object recognition device 300 may have a function of extracting the contour of the object 70 in the image. By extracting the contour, it is possible to remove unnecessary background around the object 70. The image data from which the unnecessary background has been removed may be used as learning image data. In this case, it is possible to further improve the recognition accuracy. The object recognition device 300 may have a function of displaying the contour recognition result on the display 400 and allowing the user to fine-tune the contour.
[0111] 6A to 6C are diagrams that diagrammatically show application examples of the object recognition device 300 according to this embodiment.
[0112] Part (a) of FIG. 6A shows an example of application to discrimination of plant species. Part (b) of FIG. 6A shows an example of application to displaying the names of foods. Part (c) of FIG. 6A shows an example of application to analysis of mineral resources. Part (d) of FIG. 6A shows an example of application to identification of insect species. In addition, the object recognition device 300 in this embodiment is effective for applications such as security authentication and unlocking such as face recognition, or person detection. In the case of a normal monochrome image or RGB image, there is a possibility that the human eye will misrecognize an object at first glance. In contrast, by adding multi-wavelength information as in this embodiment, it is possible to improve the accuracy of object recognition.
[0113] FIG. 6B shows an example in which detailed information of the object 70 is displayed on a smartphone that implements the object recognition method of the present embodiment. In this example, the object recognition device 300 is mounted on a smartphone. By simply holding the smartphone over the object 70, it is possible to identify what the object 70 is, and based on the result, the name of the object 70 and its explanatory information can be collected and displayed from a database via a network. In this way, a mobile information device such as a smartphone can be used as an "image search encyclopedia." In the "image search encyclopedia," if complete identification is difficult, multiple candidates may be presented in order of likelihood. In this way, based on the recognition result of the object 70, data indicating the name and explanatory information of the object 70 may be obtained from a database, and the name and / or explanatory information may be displayed on the display 400.
[0114] FIG. 6C shows an example in which a smartphone recognizes a plurality of objects present in a city. The smartphone is equipped with an object recognition device 300. When an object 70 is specified, such as an object to be inspected on a production line, the inspection device acquires only information of a specific wavelength corresponding to the object 70. On the other hand, in a situation in which the target of the object 70 is not specified, such as when used in a city, it is effective to acquire multi-wavelength information as with the object recognition device 300 in this embodiment. The object recognition device 300 may be disposed on the display 400 side of the smartphone or on the opposite side of the display 400 depending on the use example.
[0115] In addition, the object recognition method in this embodiment can be applied to a wide range of fields where recognition by artificial intelligence (AI) can be performed, such as map applications, automatic driving, or car navigation. As described above, the object recognition device can also be mounted on portable devices such as smartphones, tablets, or head-mounted display devices. If it can be photographed by a camera, a living body such as a person, a face, or an animal can also be the object 70.
[0116] The captured image 120 represented by the image data input to the signal processing circuit 200 is a multiplexed encoded image. For this reason, it is difficult to determine what is in the captured image 120 at first glance. However, the captured image 120 contains feature information that is information that indicates the features of the object 70. Therefore, the AI can recognize the object 70 directly from the captured image 120. This eliminates the need for arithmetic processing for reconstructing an image, which takes a relatively large amount of time.
[0117] (Embodiment 2) The object recognition device 300 according to the second embodiment is applied to a sensing device for autonomous driving. In the following, detailed description of the same contents as those in the first embodiment will be omitted, and differences from the first embodiment will be mainly described.
[0118] 7 is a diagram illustrating an example of vehicle control using the object recognition device 300 in this embodiment. The object recognition device 300 mounted on the vehicle can sense the environment outside the vehicle and recognize one or more objects 70 around the vehicle that are within the field of view of the object recognition device 300. The objects 70 around the vehicle can include, for example, oncoming vehicles, parallel vehicles, parked vehicles, pedestrians, bicycles, roads, lanes, white lines, sidewalks, curbs, gutters, signs, traffic lights, utility poles, stores, plants, obstacles, or fallen objects.
[0119] The object recognition device 300 includes an imaging device similar to that in the first embodiment. The imaging device generates image data of a moving image at a predetermined frame rate. The image data indicates a captured image 120 in which light from an object 70 around the vehicle passes through the filter array 100C and is multiplex-coded. The signal processing circuit 200 acquires the image data, extracts one or more objects 70 in the field of view from the image data, estimates what each of the extracted objects 70 is, and labels each object 70. Based on the recognition result of the object 70, the signal processing circuit 200 can, for example, understand the surrounding environment, determine danger, or display a target driving trajectory 420. Data such as the surrounding environment, danger information, and the target driving trajectory 420 can be used to control on-board equipment such as the steering or transmission of the vehicle body. This can enable automatic driving. The recognition result such as the object recognition label or the traveling route may be displayed on a display 400 installed in the vehicle as shown in FIG. 7 so that the driver can understand it. In this manner, the vehicle control method in this embodiment includes controlling the operation of the vehicle to which the imaging device 150 is attached, based on the recognition result of the object 70.
[0120] In conventional object recognition using RGB or monochrome images, it is difficult to distinguish between a photograph and the real thing. For this reason, for example, a photograph of a signboard or poster may be mistakenly recognized as the real thing. However, in the object recognition device 300, by using multi-wavelength information, the difference in the spectral distribution between the paint of the signboard and the real car can be taken into consideration. This makes it possible to improve the recognition accuracy. Furthermore, in the object recognition device 300, two-dimensional data on which multi-wavelength information is superimposed is acquired. This results in a smaller amount of data than conventional three-dimensional hyperspectral data. As a result, the time required for reading and transferring data, and the processing time for machine learning can be shortened.
[0121] In addition to the misrecognition of a photograph and a real object, an object may accidentally appear as something different in a camera image. In the example shown in FIG. 7, a roadside tree may appear to have a human shape depending on its growth rate or the viewing angle. For this reason, in conventional object recognition based on shape, the roadside tree shown in FIG. 7 may be misrecognized as a person. In this case, in an autonomous driving environment, the vehicle may be instructed to slow down or brake suddenly due to the misrecognition that a person has jumped out. As a result, an accident may be induced. For example, on a highway, the vehicle must not suddenly stop due to misrecognition. Even in such an environment, the object recognition device 300 can improve the recognition accuracy compared to conventional object recognition by utilizing multi-wavelength information.
[0122] The object recognition device 300 can be used in combination with various sensors such as a millimeter wave radar, a laser range finder (Lidar), or a GPS. This can further improve the recognition accuracy. For example, by linking it to pre-recorded road map information, the accuracy of generating the target travel trajectory can be improved.
[0123] (Embodiment 3) In the third embodiment, unlike the first embodiment, encoded image data is acquired by using a plurality of light sources with different emission wavelength ranges instead of the filter array 100C. In the following, detailed description of the same contents as in the first embodiment will be omitted, and the differences from the first embodiment will be mainly described.
[0124] 8 is a diagram illustrating an example of an object recognition device 300 in this embodiment. The object recognition device 300 in this embodiment includes an imaging device 150, a signal processing circuit 200, a display 400, and a memory 500. The imaging device 150 includes an optical system 40, an image sensor 60, a light source array 100L, and a control circuit 250.
[0125] The light source array 100L includes a plurality of light sources each emitting light in a different wavelength range. The control circuit 250 controls the image sensor 60 and the plurality of light sources included in the light source array 100L. The control circuit 250 repeats the operation of making the image sensor 60 capture an image with some or all of the plurality of light sources emitting light a plurality of times while changing the combination of the light sources to be emitted. As a result, light with different spectral characteristics is emitted from the light source array 100L for each capture. The combination of the light sources to be emitted does not include exactly the same combination. However, among the plurality of combinations, some light sources may overlap in two or more combinations. Therefore, the captured images 120G1, 120G2, 120G3, . . . , 120Gm obtained in each capture at the capture times T1, T2, T3, . . . , Tm have different intensity distributions. In this embodiment, the image data input to the signal processing circuit 200 is a collection of image signals generated by the image sensor 60 in the imaging device 150 for each capture of multiple times.
[0126] The control circuit 250 may not only change each light source to a binary state of on or off, but also adjust the light amount of each light source. Even when such adjustment is performed, multiple image signals having different wavelength information can be obtained. Each light source may be, for example, an LED, an LD, a laser, a fluorescent lamp, a mercury lamp, a halogen lamp, a metal halide lamp, or a xenon lamp, but is not limited thereto. In addition, when emitting light in a wavelength range of the terahertz order, an ultrafast fiber laser such as a femtosecond laser may be used as the light source.
[0127] The signal processing circuit 200 performs learning and classification of the object 70 using all or any of the captured images 120G1, 120G2, 120G3, . . . , 120Gm included in the image data.
[0128] The control circuit 250 may emit light having a spatially random intensity distribution, for example, to the light source array 100L, instead of light having a spatially uniform illuminance distribution. The light emitted from the multiple light sources may have a two-dimensional illuminance distribution that differs for each wavelength. As shown in FIG. 8, the image of the light emitted from the light source array 100L toward the object 70 and passing through the optical system 40 is formed on the image sensor 60. In this case, the light incident on each pixel or each set of pixels of the image sensor 60 has spectral characteristics including different spectral peaks, similar to the example shown in FIG. 2. This enables object recognition in a single shot, similar to the first embodiment.
[0129] As in the first embodiment, the multiple learning image data included in the multiple training data sets include learning image data generated by the imaging device 150 or another imaging device. When the learning image data is generated by another imaging device, the other imaging device may be equipped with a light source array having the same characteristics as the light source array 100L included in the imaging device 150. When the image data to be recognized and each learning image data are encoded by a light source array having the same characteristics, high recognition accuracy of the object 70 can be obtained.
[0130] The object recognition method of the present disclosure includes acquiring image data in which each pixel is multiplexed with multiple wavelength information, and recognizing an object included in a scene represented by the image data by applying a classification model learned by a machine learning algorithm to the image data in which the multiple wavelength information is multiplexed. The object recognition method of the present disclosure also includes reinforcing classification model learning using the image data in which the multiple wavelength information is multiplexed. The means for acquiring image data in which each pixel is multiplexed with multiple wavelength information is not limited to the imaging device described in the above embodiment.
[0131] The present disclosure also includes programs and methods that define the operations performed by the signal processing circuit 200. [Industrial Applicability]
[0132] The object recognition device disclosed herein can be used in measuring equipment that identifies objects with high accuracy during measurement. The object recognition device can also be applied to, for example, plant / food / organism type identification, route guidance / navigation, mineral exploration, bio / medical / beauty sensing, food foreign body / pesticide residue inspection systems, remote sensing systems, and in-vehicle sensing systems such as autonomous driving. [Explanation of symbols]
[0133] 40 Optical system 60 Image Sensor 70 Object 100C Filter Array 100L Light Source Array 120 Captured images 200 Signal Processing Circuit 250 Control circuit 300 Object recognition device 400 Display 400a auxiliary display 420 Target run trajectory 500 Memory
Claims
1. acquiring image data of an image including feature information indicative of a feature of an object; classifying the object included in the image based on the feature information; Including, The image data is An image sensor; a filter array disposed in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having optical transmittances with different wavelength dependencies, the optical transmittances of the two or more filters each having a maximum value in a plurality of wavelength ranges; The image is acquired by capturing the image using a first imaging device comprising: The feature information is extracted from the image data without reconstructing images of each of the plurality of wavelength ranges based on the image data. Object classification method.
2. classifying the object by applying a classification model trained by a machine learning algorithm to the image data; The classification model is trained in advance using a plurality of first training data sets each including training image data and label data for identifying the object included in the training image represented by the training image data. The method of claim 1 .
3. The plurality of learning image data included in the plurality of first training data sets include learning image data generated by a second imaging device different from the first imaging device. The method of claim 2 .
4. the second imaging device includes a filter array having the same characteristics as the filter array in the first imaging device; The method of claim 3 .
5. After the object is classified, the classification model is further trained with a second training data set including the image data and second label data that identifies the object. The object classification method according to any one of claims 2 to 4.
6. The object classification method according to claim 2 , wherein the positions of the object in the training images in the plurality of training image data included in the plurality of first training data sets are different from each other in the plurality of training image data.
7. The learning image data is acquired by capturing an image of the object in a state where the object occupies a predetermined range or more in the learning image. The object classification method according to any one of claims 2 to 4.
8. acquiring the image data using an imaging device including a display; and displaying, before the image data is acquired, an auxiliary display on the display to inform a user of an area in the image in which the object should be located or a range in which the object should occupy. The method for classifying objects according to any one of claims 1 to 4.
9. The plurality of filters have mutually different wavelength dependencies of light transmittance, The light transmittance of each of the plurality of filters has a maximum value in a plurality of wavelength ranges. The method for classifying objects according to any one of claims 1 to 4.
10. A vehicle control method using the object classification method according to any one of claims 1 to 4, comprising: The first imaging device is mounted on a vehicle, controlling an operation of the vehicle based on a result of classifying the object. A vehicle control method.
11. An information display method using the object classification method according to any one of claims 1 to 4, comprising the steps of: obtaining data indicating at least one selected from the group consisting of a name of the object and a description of the object from a database based on a result of classifying the object; displaying at least one selected from the group consisting of a name of the object and a description of the object on a display; Including, Information display method.
12. an image sensor for generating image data including feature information indicating features of an object; a filter array disposed in an optical path of light incident on the image sensor, the filter array including a plurality of light-transmitting filters arranged two-dimensionally along a plane intersecting the optical path, the plurality of filters including two or more filters having optical transmittances with wavelength dependencies different from each other, and the optical transmittances of the two or more filters each having a maximum value in a plurality of wavelength ranges; a signal processing circuit for classifying the object included in the image based on the feature information; Equipped with The feature information is extracted from the image data without reconstructing images of each of the plurality of wavelength ranges based on the image data. Object classification device.
Citation Information
Patent Citations
A spectral camera equipped with a mosaic filter for each pixel.
JP2015501432A
Imaging device and spectroscopy system
JP2016156801A
Data processor, data processing method, program, and electronic equipment
JP2018096834A
Systems and methods for analyzing remote sensing imagery
US20170076438A1
Multi-point spectral system
US20170163901A1
