Object classification method, information display method, and object classification device

Abstract

The invention relates to an object classification method, an information display method and an object classification device. An object classification method according to the present disclosure includes: acquiring image data of an image of an object; and a step of classifying the object included in the image, the image data being data of a multiplexed image in which a plurality of pieces of wavelength information are multiplexed in one pixel, and the classification of the object being performed by inputting the data of the multiplexed image into a classification model trained by a machine learning algorithm.

Description

[0001] This application is a divisional of an invention patent application with an application date of September 24, 2019, application number 201980031643.3, and invention name “Object recognition method, vehicle control method, information display method and object recognition device”. Technical Field

[0002] The present disclosure relates to an object recognition method, a vehicle control method, an information display method, and an object recognition device. Background Art

[0003] In object recognition using machine learning, generally, monochrome images or RGB images are used as learning data. On the other hand, attempts to perform object recognition using multispectral images that contain information of more wavelengths than RGB images have also been studied.

[0004] Patent document 1 discloses a spectral camera that spatially arranges multiple filters that allow light of different wavelength regions to pass in a mosaic shape as a sensor for obtaining multispectral images. Patent document 2 discloses a method for learning images of immune cells using a convolutional neural network for multiple image channels in order to improve the recognition accuracy of immune cells in an image. Patent document 3 discloses a method for machine learning that uses multispectral images or super-spectral images as training data.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application No. 2015-501432

[0008] Patent Document 2: International Publication No. 2015 / 177268

[0009] Patent Document 3: U.S. Patent Application Publication No. 2017 / 0076438 Summary of the invention

[0010] Problems to be solved by the invention

[0011] The present disclosure provides a new object recognition method capable of performing high-precision object recognition based on encoded image data.

[0012] Means for solving problems

[0013] An object classification method according to one embodiment of the present invention includes: a step of obtaining image data of an image of an object; and a step of classifying the object contained in the image, wherein the image data is data of a multiplexed image in which information of multiple wavelengths is multiplexed in one pixel, and the object is classified by inputting the data of the multiplexed image into a classification model trained by a machine learning algorithm.

[0014] Effects of the Invention

[0015] According to the present disclosure, it is possible to perform highly accurate object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a diagram schematically showing an example of an object recognition device in an exemplary embodiment of the present disclosure.

[0017] Figure 2A It is a diagram schematically showing an example of a filter array.

[0018] Figure 2B This is a diagram showing an example of spatial distribution of light transmittance in each of a plurality of wavelength regions included in the target wavelength region.

[0019] Figure 2C Yes means Figure 2A A diagram showing an example of the spectral transmittance of area A1 in the filter array shown.

[0020] Figure 2D Yes means Figure 2A A diagram showing an example of the spectral transmittance of area A2 in the filter array shown.

[0021] Figure 3A This is a diagram schematically showing an example of a two-dimensional distribution of a filter array.

[0022] Figure 3B This is a diagram schematically showing another example of the two-dimensional distribution of the filter array.

[0023] Figure 4A This is a flowchart showing an example of an object recognition method using the object recognition device in the exemplary embodiment.

[0024] Figure 4B This is a flowchart showing an example of a classification model generation process.

[0025] Figure 4C FIG. 1 is a diagram schematically showing examples of multiple training data sets in an exemplary embodiment.

[0026] Figure 4D This is a diagram schematically showing an example of feeding back the object recognition result to the classification model.

[0027] Figure 4E This is a flowchart showing another example of the object recognition method using the object recognition device in the exemplary embodiment.

[0028] Figure 5A This is a diagram schematically showing a function of displaying a recommended area for object recognition to assist in photographing.

[0029] Figure 5B It is a diagram schematically showing an enlarged view of an object of an optical system having a zoom function.

[0030] Figure 5C A diagram schematically showing a modified example of a filter array.

[0031] Fig. 6A It is a diagram schematically showing an application example of the object recognition device in the exemplary embodiment.

[0032] Figure 6B It is a diagram schematically showing another application example of the object recognition device in the exemplary embodiment.

[0033] Figure 6C It is a diagram schematically showing another application example of the object recognition device in the exemplary embodiment.

[0034] Figure 7 FIG. 1 is a diagram schematically showing an example of vehicle control using the object recognition device in the exemplary embodiment.

[0035] Figure 8 FIG. 1 is a diagram schematically showing an example of an object recognition device in an exemplary embodiment. DETAILED DESCRIPTION

[0036] Before describing the embodiments of the present disclosure, the knowledge that is the basis of the present disclosure will be described.

[0037] In object recognition using existing RGB images, the recognition capability is limited. For example, there is a situation where it is impossible to distinguish between a real object and its billboard or poster. This is because the difference between the amount of each component of R, G, and B of the light reflected from the real object and the amount of each component of R, G, and B of the light reflected from its billboard or poster is usually small. In order to distinguish between a real object and its billboard or poster, for example, it is considered to use multi-wavelength spectral data. In this way, it is possible to detect subtle differences in spectral data caused by differences in the materials of the object.

[0038] In an existing hyperspectral camera, for example, as disclosed in Patent Document 1, a plurality of wavelength filters having different transmission wavelength regions are arranged two-dimensionally. When a single shot is used to obtain an image of one frame, such as in dynamic image shooting, the number of wavelength regions and the spatial resolution become a trade-off. That is, if a large number of filters having different transmission wavelength regions are spatially dispersed in order to obtain a multi-wavelength image, the spatial resolution of the image obtained in each wavelength region becomes low. Therefore, even if the hyperspectral image is used for object recognition in the expectation of improving the recognition accuracy of the object, there is a possibility that the recognition accuracy will be reduced due to the low spatial resolution in reality.

[0039] It is also considered to improve both wavelength resolution and resolution by increasing the number of pixels of the image sensor. In this case, a large volume of three-dimensional data is obtained by adding multi-wavelength data to the two-dimensional data of the space. When machine learning is applied to such large-scale data, pre-processing, learning, communication, and data storage take a lot of time or resources.

[0040] Based on the above studies, the present inventors have conceived of the object recognition method described in the following items.

[0041] [Project 1]

[0042] The object recognition method according to item 1 includes: a step of obtaining image data of an image including feature information indicating features of an object; and a step of recognizing the object included in the image based on the feature information. The image data is obtained by capturing the image with a first camera, the first camera having an image sensor and a filter array, the filter array being arranged in an optical path of light incident on the image sensor, and including a plurality of light-transmitting filters arranged two-dimensionally along a surface intersecting the optical path, the plurality of filters including two or more filters having mutually different wavelength dependencies of light transmittance, and the light transmittance of each of the two or more filters having a local maximum value in a plurality of wavelength regions.

[0043] [Project 2]

[0044] Alternatively, in the object recognition method involved in the first item, recognition of the object is performed by applying a classification model trained by a machine learning algorithm to the image data, the classification model being pre-trained by a plurality of first training data sets, the plurality of first training data sets respectively including learning image data and label data, the label data identifying the object contained in the learning image shown by the learning image data.

[0045] [Item 3]

[0046] In the object recognition method according to the second item, the plurality of learning image data included in the plurality of first training data sets may include learning image data generated by a second imaging device different from the first imaging device.

[0047] [Item 4]

[0048] In the object recognition method according to item 3, the second imaging device may include a filter array having characteristics equivalent to those of the filter array in the first imaging device.

[0049] [Item 5]

[0050] Alternatively, the object recognition method involved in any one of items 2 to 4 also includes: after the object is recognized, the classification model is further trained through a second training data set, and the second training data set includes the image data and second label data for identifying the object.

[0051] [Item 6]

[0052] Alternatively, in the object recognition method according to any one of items 2 to 5, positions of the objects in the multiple learning image data included in the multiple first training data sets within the learning images are different from each other in the multiple learning image data.

[0053] [Item 7]

[0054] In the object recognition method according to any one of items 2 to 6, the learning image data may be acquired by capturing the object in a state where the object occupies a predetermined area or more in the learning image.

[0055] [Item 8]

[0056] Alternatively, in the object recognition method involved in any one of items 1 to 7, the image data is obtained using a photographing device including a display, and the object recognition method further includes: before obtaining the image data, causing the display to display an auxiliary display for notifying a user of the area where the object in the image should be located or the range that the object should occupy.

[0057] [Item 9]

[0058] In the object recognition method according to any one of items 1 to 8, the wavelength dependencies of the light transmittances of the plurality of filters may be different from each other, and the light transmittance of each of the plurality of filters may have local maxima in a plurality of wavelength regions.

[0059] [Item 10]

[0060] The vehicle control method of item 10 is a vehicle control method using the object recognition method of any one of items 1 to 9, wherein the first camera is installed on a vehicle, and the vehicle control method includes controlling the movement of the vehicle based on a result of recognizing the object.

[0061] [Item 11]

[0062] The information display method of Item 11 is an information display method using the object recognition method of any one of Items 1 to 9, comprising: a step of obtaining data representing at least one selected from the group consisting of the name of the object and the description of the object from a database based on the result of recognizing the object; and a step of displaying the at least one selected from the group consisting of the name of the object and the description of the object on a display.

[0063] [Item 12]

[0064] The object recognition method according to item 12 comprises: a step of obtaining image data of an image including feature information indicating features of an object; and a step of recognizing the object included in the image based on the feature information. The image data is obtained by repeatedly capturing the image multiple times while changing a combination of light sources included in a part of a plurality of light sources by a first imaging device, the first imaging device comprising an image sensor and a light source array including the plurality of light sources emitting light in different wavelength regions.

[0065] [Item 13]

[0066] Alternatively, in the object recognition method involved in Item 12, recognition of the object is performed by applying a classification model trained by a machine learning algorithm to the image data, the classification model being pre-trained by a plurality of first training data sets, the plurality of first training data sets respectively including learning image data and label data, the label data identifying the object contained in the learning image shown by the learning image data.

[0067] [Item 14]

[0068] In the object recognition method according to item 13, the plurality of learning image data included in the plurality of first training data sets may include learning image data generated by a second imaging device different from the first imaging device.

[0069] [Item 15]

[0070] In the object recognition method according to item 14, the second imaging device may include a light source array having the same characteristics as those of the light source array in the first imaging device.

[0071] [Item 16]

[0072] Alternatively, the object recognition method involved in any one of items 13 to 15 also includes: after the object is recognized, the classification model is further trained through a second training data set, and the second training data set includes the image data and second label data for identifying the object.

[0073] [Item 17]

[0074] Alternatively, in the object recognition method according to any one of items 13 to 16, positions of the objects in the multiple learning image data included in the multiple first training data sets within the learning images are different from each other in the multiple learning image data.

[0075] [Item 18]

[0076] In the object recognition method according to any one of items 13 to 17, the learning image data may be acquired by capturing the object in a state where the object occupies a predetermined range or more in the learning image.

[0077] [Item 19]

[0078] Alternatively, in the object recognition method involved in any one of items 12 to 18, the image data is obtained using a photographing device including a display, and the object recognition method further includes: before obtaining the image data, causing the display to display an auxiliary display for notifying a user of the area where the object in the image should be located or the range that the object should occupy.

[0079] [Item 20]

[0080] The vehicle control method involved in Item 20 is a vehicle control method using the object recognition method involved in any one of Items 12 to 19, wherein the first camera is installed on the vehicle, and the vehicle control method includes a step of controlling the movement of the vehicle based on the result of recognizing the object.

[0081] [Item 21]

[0082] The information display method involved in Item 21 is an information display method using the object recognition method involved in any one of Items 12 to 19, comprising: a step of obtaining data representing at least one selected from the group consisting of the name of the object and the description of the object from a database based on the result of recognizing the object; and a step of displaying the at least one selected from the group consisting of the name of the object and the description of the object on a display.

[0083] [Item 22]

[0084] The object recognition device involved in Item 22 comprises: an image sensor that generates image data of an image including characteristic information representing characteristics of an object; a filter array that is arranged in the optical path of light incident on the image sensor, including a plurality of light-transmitting filters arranged in two dimensions along a surface intersecting the optical path, the plurality of filters including two or more filters having wavelength dependence of light transmittance that is different from each other, and the light transmittance of each of the two or more filters has a maximum value in a plurality of wavelength regions; and a signal processing circuit that recognizes the object contained in the image based on the characteristic information.

[0085] [Item 23]

[0086] The object recognition device involved in Item 23 comprises: an image sensor that generates an image signal containing an image of an object; a light source array that includes a plurality of light sources that emit light in different wavelength regions; a control circuit that controls the image sensor and the plurality of light sources, the control circuit changing the combination of light sources included in a part of the plurality of light sources while repeatedly causing the image sensor to photograph while causing the part of the plurality of light sources to emit light; and a signal processing circuit that recognizes the object contained in the image based on feature information, the feature information being information representing features of the object contained in image data consisting of the image signal generated by the image sensor in each of the plurality of photographing.

[0087] [Item 24]

[0088] The object recognition device according to item 24 comprises a memory and a signal processing circuit. The signal processing circuit receives two-dimensional image data and recognizes an object contained in a scene shown by the two-dimensional image data based on feature information contained in the two-dimensional image data, wherein the two-dimensional image data is two-dimensional image data of an image containing a plurality of pixels, and is multi-spectral / hyper-spectral image data obtained by multiplexing information of a plurality of wavelength regions on data of each of the plurality of pixels and encoding brightness distribution of each of the plurality of pixels.

[0089] [Item 25]

[0090] In the object recognition device according to item 24, the feature information may be extracted from the two-dimensional image data without reconstructing the images of the respective plurality of wavelength regions based on the two-dimensional image data.

[0091] [Item 26]

[0092] The object recognition device according to Item 24 may further include a photographing device for acquiring the two-dimensional image data.

[0093] [Item 27]

[0094] In the object recognition device according to item 26, the two-dimensional image data may be acquired by capturing the object in a state where the object occupies a predetermined range or more in a capturing area of ​​the capturing device.

[0095] [Item 28]

[0096] Alternatively, the object recognition device involved in Item 27 may also further include a display, which displays an auxiliary display for notifying a user of the area where the object should be located or the range that the object should occupy in the image captured by the camera before the two-dimensional image data is acquired by the camera.

[0097] [Item 29]

[0098] Alternatively, in the object recognition device involved in Item 26, the shooting device includes an image sensor and a filter array, the filter array is arranged in the optical path of light incident on the image sensor, and includes a plurality of light-transmitting filters arranged in two dimensions along a surface intersecting the optical path, the plurality of filters include two or more filters having wavelength dependence of light transmittance different from each other, and the light transmittance of each of the two or more filters has a maximum value in a plurality of wavelength regions.

[0099] [Item 30]

[0100] In the object recognition device according to Item 29, the plurality of filters may include a plurality of periodically arranged partial sets.

[0101] The embodiments described below all represent general or specific examples. The numerical values, shapes, materials, constituent elements, and the configuration positions of the constituent elements shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, the constituent elements in the following embodiments that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements.

[0102] In the present disclosure, all or part of a circuit, unit, device, component or part, or all or part of a functional block of a block diagram may also be executed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC) or an LSI (large-scale integration). LSI or IC may be integrated into one chip or may be composed of a combination of multiple chips. For example, functional blocks other than storage elements may also be integrated into one chip. Here, it is referred to as LSI or IC, but the name varies depending on the degree of integration, and may also be referred to as system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). A field programmable gate array (FPGA) that is programmed after the manufacture of the LSI, or a reconfigurable logic device that can reconstruct the bonding relationship within the LSI or set the circuit division within the LSI can also be used for the same purpose.

[0103] Furthermore, all or part of the functions or operations of a circuit, unit, device, component or part can be executed by software processing. In this case, the software is recorded in one or more non-temporary recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processing device (processor), the functions determined by the software are executed by the processing device (processor) and peripheral devices. The system or device may also have one or more non-temporary recording media with software recorded, a processing device (processor) and necessary hardware devices, such as interfaces.

[0104] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0105] (Implementation Method 1)

[0106] Figure 1 Schematically shows an example of an object recognition device 300 in the first exemplary embodiment of the present disclosure. As an example, Figure 1Indicates the state of photographing mushrooms. The photographed object 70 may be any object. The object recognition device 300 in Embodiment 1 includes a photographing device 150, a signal processing circuit 200, a display 400, and a memory 500. The photographing device 150 includes an optical system 40, a filter array 100C, and an image sensor 60. The object recognition device 300 may be, for example, a computer such as a smartphone or a tablet computer. The camera mounted on these computers may also function as the photographing device 150.

[0107] The filter array 100C is arranged in the optical path of the light incident on the image sensor 60. In the present embodiment, the filter array 100C is arranged at a position opposite to the image sensor 60. The filter array 100C may also be arranged at other positions. The image of the light from the object 70 is encoded by the filter array 100C. Here, "encoding" means modulating the image by attenuating the light incident on the filter array 100C at an attenuation rate that depends on the wavelength and position of the light. The image data generated based on the modulated image is referred to as "encoded image data". The structure of the filter array 100C and the details of the encoding will be described later.

[0108] The image sensor 60 may be a monochrome imaging element having a plurality of pixels, i.e., a plurality of light detection units, arranged two-dimensionally on the imaging surface. The image sensor 60 may be, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, an infrared array sensor, a terahertz array sensor, or a millimeter wave array sensor. The light detection unit may include, for example, a photodiode. The image sensor 60 is not necessarily a monochrome imaging element. For example, a color imaging element having a filter of R / G / B, R / G / B / IR, or R / G / B / W may also be used. The image sensor 60 is not limited to the visible wavelength range, and may also have detection sensitivity within the wavelength range of X-rays, ultraviolet rays, near infrared, mid-infrared, far infrared, and microwaves / radio waves.

[0109] The image sensor 60 is arranged in the optical path of the light that has passed through the filter array 100C. The image sensor 60 receives the light that has passed through the filter array 100C and generates an image signal. Each light detection unit in the image sensor 60 outputs a photoelectric conversion signal corresponding to the amount of light received. The image signal is generated by the multiple photoelectric conversion signals output from the multiple light detection units. Figure 1 An example of a captured image 120 composed of the image signal, that is, encoded image data is schematically shown.

[0110] The optical system 40 includes at least one lens. Figure 1 In the example shown, the optical system 40 is depicted as a single lens, but may be composed of a combination of a plurality of lenses. The optical system 40 may also have a zoom function as described later. The optical system 40 forms an image of light from an object 70 on the filter array 100C.

[0111] The signal processing circuit 200 is a circuit for processing an image signal output from the image sensor 60. The signal processing circuit 200 can be implemented, for example, by a combination of a central processing unit (CPU) and a processing unit for image processing (GPU) and a computer program. Such a computer program is stored in a recording medium such as a memory, and a processor such as a CPU or a GPU executes the program, thereby being able to perform the recognition processing described later. The signal processing circuit 200 can also be a programmable logic device (PLD) such as a digital signal processor (DSP) or a field programmable gate array (FPGA). The signal processing circuit 200 can also have a server computer connected to a device such as a camera 150 or a smart phone via a network such as the Internet.

[0112] The signal processing circuit 200 recognizes the object 70 based on the encoded image data. For example, a model trained by a known machine learning algorithm can be used to recognize the object 70. The details of the object recognition method will be described later.

[0113] The display 400 displays information associated with the recognized object 70. The display 400 may be, for example, a display of a smartphone or a tablet computer. The display 400 may also be a display connected to a personal computer or the like, or a display built into a notebook computer.

[0114] Next, the details of the structure and encoding of the filter array 100C will be described.

[0115] Figure 2A : is a diagram schematically showing an example of a filter array 100C. The filter array 100C has a plurality of regions arranged in two dimensions. In this specification, the region is sometimes referred to as a "unit". A filter having a separately set spectral transmittance is arranged in each region. Here, "spectral transmittance" refers to a light transmittance that has wavelength dependence. The spectral transmittance is represented by a function T(λ), with the wavelength of the incident light being λ. The spectral transmittance T(λ) can take a value greater than 0 and less than 1. In this way, the filter array 100C includes a plurality of filters arranged in two dimensions along a surface that intersects the optical path.

[0116] exist Figure 2AIn the example shown, the filter array 100C has 48 rectangular areas arranged in 6 rows and 8 columns. In actual use, more areas can be set. The number can be, for example, the same as the number of pixels of a general imaging element such as an image sensor. The number of pixels is, for example, hundreds of thousands to tens of millions. In one example, the filter array 100C is arranged directly above the imaging element, and each area can be arranged in a manner corresponding to one pixel of the imaging element. Each area is, for example, opposite to one or more pixels of the imaging element.

[0117] Figure 2B : is a diagram showing an example of the spatial distribution of the transmittance of light in each of the plurality of wavelength regions W1, W2, ..., Wi included in the target wavelength region. Figure 2B In the example shown, the difference in lightness and darkness of each area indicates the difference in transmittance. The lighter the area, the higher the transmittance, and the darker the area, the lower the transmittance. Figure 2B As shown, the spatial distribution of light transmittance differs depending on the wavelength region.

[0118] Figure 2C as well as Figure 2D They are respectively Figure 2A FIG. 1 is a diagram showing an example of spectral transmittance of region A1 and region A2 included in a plurality of regions of the filter array 100C shown. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 100C is different depending on the region. However, it is not necessarily necessary for the spectral transmittance of all regions to be different. The spectral transmittance of at least two of the plurality of regions in the filter array 100C is different from each other. That is, the filter array 100C includes more than two filters having different spectral transmittances. The spectral transmittance of each of the two or more filters has a maximum value in a plurality of wavelength regions and a minimum value in other plurality of wavelength regions.

[0119] Here, the meaning of "maximum" and "minimum" in the present disclosure is explained. When the maximum value of the spectral transmittance of the filter of interest is regularized to 1 and the minimum value is regularized to 0, a value that exceeds 0.5 and has a difference of 0.2 or more from the adjacent minimum value is defined as a "maximum" in the present disclosure. Similarly, when performing the above regularization, a value that is less than 0.5 and has a difference of 0.2 or more from the adjacent maximum value is defined as a "minimum" in the present disclosure. All spectral transmittances of multiple filters in the filter array 100C may also be different from each other. In this case, the spectral transmittance of each filter may have a maximum value in multiple wavelength regions and a minimum value in other multiple wavelength regions. In one example, the number of modes of spectral transmittance of multiple filters included in the filter array 100C may be the same as or greater than the number i of wavelength regions included in the object wavelength region. Typically, the filter array 100C can be designed so that the spectral transmittances of more than half of the filters are different.

[0120] The filter array 100C modulates the incident light into a plurality of discrete intensity peaks for each wavelength for each region, and superimposes and outputs the multi-wavelength lights. Thus, the image of the light passing through the filter array 100C is encoded.

[0121] The resolution of the spectral transmittance in the wavelength direction of each region can be set to the bandwidth of the desired wavelength region. In other words, the width of the range of values ​​above the minimum value closest to the maximum value and the average value of the maximum value in the wavelength range containing a maximum value in the spectral transmittance curve can be set to the bandwidth of the desired wavelength region. In this case, if the spectral transmittance is decomposed into frequency components by, for example, Fourier transform, the value of the frequency component corresponding to the wavelength region becomes relatively large.

[0122] Typically, such as Figure 2A As shown, the filter array 100C is divided into a plurality of units corresponding to a plurality of regions divided into a grid shape. These units have different spectral transmittances. The wavelength distribution and spatial distribution of the light transmittance of each region of the filter array 100C can be, for example, random distribution or quasi-random distribution.

[0123] The method of considering random distribution and quasi-random distribution is as follows. First, each area in the filter array 100C can be considered as a vector element having a value of, for example, 0 to 1 according to the light transmittance. Here, when the transmittance is 0, the value of the vector element is 0, and when the transmittance is 1, the value of the vector element is 1. In other words, a set of areas arranged in a row direction or a column direction can be regarded as a multidimensional vector having values ​​from 0 to 1. Therefore, the filter array 100C can be said to have multiple multidimensional vectors in the column direction or the row direction. At this time, random distribution means that any two multidimensional vectors are independent, that is, not parallel. In addition, quasi-random distribution refers to a structure that is not independent between a part of the multidimensional vectors. Therefore, in random distribution and quasi-random distribution, the vector with the transmittance value of the light in the first wavelength region in each area of ​​the set of areas arranged in one row or column contained in multiple areas as an element and the vector with the transmittance value of the light in the first wavelength region in each area of ​​the set of areas arranged in other rows or columns as an element are independent of each other. Similarly, for the second wavelength region different from the first wavelength region, a vector having as elements the transmittance values ​​of light in the second wavelength region in each region of a set of regions arranged in one row or column included in a plurality of regions and a vector having as elements the transmittance values ​​of light in the second wavelength region in each region of a set of regions arranged in another row or column are independent of each other.

[0124] When the filter array 100C is arranged near or just above the image sensor 60, the mutual spacing of the plurality of regions in the filter array 100C, i.e., the cell pitch, may be substantially consistent with the pixel pitch of the image sensor 60. In this way, the resolution of the image of the encoded light emitted from the filter array 100C is substantially consistent with the resolution of the pixel. When the filter array 100C is arranged separately from the image sensor 60, the cell pitch may be made thinner according to the distance.

[0125] exist FIG. 2A to FIG. 2D In the example shown, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value greater than 0 and less than 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary-scale transmittance distribution can be adopted in which the transmittance of each region can take any value between approximately 0 and approximately 1. In the binary-scale transmittance distribution, each region transmits most of the light in at least two wavelength regions among the multiple wavelength regions included in the target wavelength region, and does not transmit most of the light in the remaining wavelength regions. Here, "most of" means approximately 80% or more.

[0126] It is also possible to replace a portion of all the units, such as half of the units, with transparent areas. Such transparent areas allow light in wavelength area Wi to be transmitted from all wavelength areas W1 included in the target wavelength area with the same degree of high transmittance. This high transmittance is, for example, greater than 0.8. In such a structure, multiple transparent areas can be arranged in a checkerboard pattern, for example. That is, in the two arrangement directions of the multiple areas in the filter array 100C, areas with different light transmittances depending on the wavelength and transparent areas can be arranged alternately. Figure 2A In the example shown, the two arrangement directions are horizontal and vertical. By extracting the components that pass through the transparent areas arranged in a checkerboard pattern, a single camera can simultaneously acquire a monochrome image.

[0127] The filter array 100C can be composed of at least one selected from the group consisting of a multilayer film, an organic material, a diffraction grating structure, and a fine structure containing metal. In the case of a multilayer film, for example, a multilayer film containing a dielectric multilayer film or a metal film is used. At this time, in each unit, at least one of the thickness, material, and stacking order of the multilayer film can be designed to be different. Thereby, different spectral characteristics can be achieved in each unit. In addition, a spectral characteristic with a sharp rise or fall can be achieved by a multilayer film. In the case of using an organic material, in each unit, different spectral characteristics can be achieved by different pigments or dyes, or by stacking different kinds of materials. In the case of a diffraction grating structure, different spectral characteristics can be achieved by setting a diffraction structure with different diffraction spacing or depth in each unit. In the case of a fine structure containing metal, different spectral characteristics can be achieved by spectroscopy based on the plasma effect.

[0128] The filter array 100C is arranged near or directly above the image sensor 60. Here, "near" means that the image of the light from the optical system 40 is close to being formed on the surface of the filter array 100C in a state of being clear to a certain extent. "Directly above" means that the two are close with almost no gap. The filter array 100C and the image sensor 60 can be integrated. The filter array 100C is a mask with a spatial distribution of light transmittance. The filter array 100C modulates the intensity of the incident light and allows it to pass.

[0129] Figure 3A as well as Figure 3B 100C is a diagram schematically showing an example of two-dimensional distribution of the filter array 100C.

[0130] like Figure 3AAs shown, the filter array 100C can also be composed of a binary mask. The black part represents shading, and the white part represents transmission. The light passing through the white part is 100% transmitted, and the light passing through the black part is 100% blocked. The two-dimensional distribution of the transmittance of the mask can be a random distribution or a quasi-random distribution. The two-dimensional distribution of the transmittance of the mask does not have to be completely random. This is because the encoding of the filter array 100C is performed to distinguish images of each wavelength. In addition, the ratio of the black part to the white part does not need to be 1:1. For example, it can also be white part: black part = 1:9. As Figure 3B As shown, the filter array 100C may also be a mask having a grayscale transmittance distribution.

[0131] like Figure 3A as well as Figure 3B As shown, the filter array 100C has a different spatial distribution of transmittance for each wavelength region W1, W2, ..., Wi. The spatial distribution of transmittance in each wavelength region does not coincide even when the wavelength region is moved in parallel.

[0132] The image sensor 60 may be a monochrome type imaging element having two-dimensional pixels. However, the image sensor 60 does not necessarily have to be composed of a monochrome type imaging element. For example, the image sensor 60 may use a color type imaging element having filters of R / G / B, R / G / B / IR, or R / G / B / W. The color type imaging element can increase the amount of information related to the wavelength. Thus, the characteristics of the filter array 100C can be supplemented, and the filter design becomes easy.

[0133] Next, the process of obtaining image data representing the captured image 120 by the object recognition device 300 of the present embodiment is described. The image of the light from the object 70 is formed by the optical system 40 and encoded by the filter array 100C arranged just before the image sensor 60. As a result, images with different encoded information in each wavelength region overlap with each other and are imaged as multiple images on the image sensor 60. Thus, the captured image 120 is obtained. At this time, since a spectroscopic element such as a prism is not used, spatial displacement of the image does not occur. Therefore, even multiple images can maintain a high spatial resolution. As a result, the accuracy of object recognition can be improved.

[0134] The wavelength region can also be limited by providing a bandpass filter in a part of the object recognition device 300. When the wavelength range of the object 70 is known to some extent, the recognition range can be limited by limiting the wavelength region. As a result, high recognition accuracy of the object can be achieved.

[0135] Next, an object recognition method using the object recognition device 300 in this embodiment will be described.

[0136] Figure 4A1 is a flowchart showing an example of an object recognition method using the object recognition device 300 in this embodiment. The object recognition method is executed by the signal processing circuit 200. The signal processing circuit 200 executes a computer program stored in the memory 500 to execute Figure 4A The processing of steps S101 to S104 is shown.

[0137] First, the user photographs the object 70 using the photographing device 150 included in the object recognition device 300. This obtains the coded photographed image 120.

[0138] In step S101, the signal processing circuit 200 acquires image data generated by the imaging device 150. The image data represents the captured image 120 after being encoded.

[0139] In step S102, the signal processing circuit 200 performs pre-processing on the acquired image data. Pre-processing is performed to improve recognition accuracy. Pre-processing may include, for example, region extraction, smoothing processing for noise removal, and feature extraction. Pre-processing may be omitted if not necessary.

[0140] In step S103, the signal processing circuit 200 applies the trained classification model to the image data to determine the objects 70 included in the scene represented by the pre-processed image data. The classification model is pre-trained, for example, by a well-known machine learning algorithm. The details of the classification model will be described later.

[0141] In step S104, the signal processing circuit 200 outputs information associated with the object 70. The signal processing circuit 200 outputs information such as the name and / or detailed information of the object 70 to the display 400. The display 400 displays an image representing the information. The information is not limited to images, and may be presented by sound, for example.

[0142] Next, the classification model used in the object recognition method is described.

[0143] Figure 4B This is a flowchart showing an example of a classification model generation process.

[0144] In step S201, the signal processing circuit 200 collects multiple training data sets. The multiple training data sets respectively include learning image data and label data. The label data is information for identifying the object 70 included in the scene represented by the learning image data. The learning image data is image data encoded using the same method as the above-mentioned image data. The multiple learning image data included in the multiple training data sets may include learning image data generated by the shooting device 150 or other shooting devices in this embodiment. Details about the multiple training data sets are described later.

[0145] In step S202, the signal processing circuit 200 performs pre-processing on the learning image data included in each training data. The pre-processing is as described above.

[0146] In step S203, the signal processing circuit 200 generates a classification model through machine learning based on multiple training data sets. Machine learning can use algorithms such as deep learning, support vector machine, decision tree, genetic programming, or Bayesian network. In the case of deep learning, for example, algorithms such as convolutional neural network (CNN) or recurrent neural network (RNN) can be used.

[0147] In this embodiment, by using a model trained by machine learning, information related to objects in the scene can be directly obtained from the coded image data. In order to perform the same situation with the prior art, a large amount of calculations are required. For example, it is necessary to reconstruct image data of each wavelength region from the coded image data by methods such as compressed sensing, and determine the object based on these image data. In contrast, in this embodiment, it is not necessary to reconstruct image data of each wavelength region based on the coded image data. Therefore, the time or computing resources spent on the reconstruction process can be saved.

[0148] Figure 4C FIG. 2 is a diagram schematically showing examples of multiple training data sets in this embodiment. Figure 4C In the example shown, each training data set includes encoded image data representing one or more mushrooms and label data indicating whether the mushroom is an edible mushroom or a poisonous mushroom. In this way, for each training data set, the encoded image data and the label data representing the correct label correspond 1:1. The correct label can be, for example, the name of the object 70, its characteristics, sensory evaluation such as "delicious" or "unpalatable", or information indicating judgments such as "good" or "bad". Generally speaking, the more multiple training data sets there are, the higher the accuracy of learning can be. Here, the position of the object 70 in the multiple learning image data contained in the multiple training data sets can also be different according to the learning image data. The encoded information is different for each pixel. Therefore, the more learning image data with different positions of the object 70 in the image, the higher the accuracy of object recognition based on the classification model can be.

[0149] In the object recognition device 300 of this embodiment, the classification model is assembled into the signal processing circuit 200 before the user uses it. As another method, the encoded image data representing the captured image 120 can also be sent to a classification system prepared externally via a network or a cloud. In this classification system, for example, high-speed processing based on a supercomputer can be performed. As a result, even if the processing speed of the terminal on the user side is fragile, as long as it can be connected to the network, the recognition result of the object 70 can be provided to the user at a high speed.

[0150] exist Figure 4A The image data obtained in step S101 and Figure 4B The learning image data obtained in step S201 in can be encoded by, for example, a filter array having equivalent characteristics. In this case, the recognition accuracy of the object 70 can be improved. Here, the filter array having equivalent characteristics does not need to strictly have equivalent characteristics, and the spectral transmission characteristics may be different in some filters. For example, the characteristics of the filters may differ from a few percent to tens of percent of the whole. In the case where the learning image data is generated by other shooting devices, the other shooting devices may have a filter array having equivalent characteristics to the filter array 100C included in the shooting device 150.

[0151] The recognition result of the object 70 may also be fed back to the classification model, thereby enabling the classification model to be further trained.

[0152] Figure 4D Schematically shows an example of feeding back the recognition result of the object 70 to the classification model. Figure 4D In the example shown, the trained classification model is applied to the pre-processed coded image data, and the classification result is output. Then, the result is added to the data set, and the data set is used to further perform machine learning. In this way, the model can be further trained to improve the prediction accuracy.

[0153] Figure 4E This is a flowchart showing in more detail the operation when the recognition result is fed back to the classification model.

[0154] Figure 4E Steps S301 to S304 are respectively Figure 4A The steps S101 to S104 are the same. Afterwards, steps S305 to S307 are performed.

[0155] In step S305 , the signal processing circuit 200 generates a new training data set including the image data acquired in step S301 and label data indicating the object 70 recognized in step S303 .

[0156] In step S306, the signal processing circuit 200 further trains the classification model using multiple new training data sets. Figure 4B The learning processes shown in step S202 and step S203 are the same.

[0157] In step S307 , the signal processing circuit 200 determines whether to continue recognizing the object 70 . If the determination is “yes”, the signal processing circuit 200 executes the process of step S301 again. If the determination is “no”, the signal processing circuit 200 ends recognizing the object 70 .

[0158] In this way, the recognition accuracy of the classification model can be improved by feeding back the recognition result of the object 70 to the classification model. In addition, a classification model suitable for the user can be created.

[0159] In the case where a classification system is provided separately, the user may also send a data set including the recognition result of the object 70 to the classification system via a network for feedback. The data set may include data representing the captured image 120 generated by shooting, or data pre-processed thereon, and label data representing the recognition result based on the classification model or the correct label based on the user's opinion. The provider of the classification system may also give rewards such as remuneration or points to the user who sent the data set for feedback. The access permission of the captured image 120 taken by the user, or the authentication of whether it can be automatically sent, may also be displayed on the display 400 before sending, for example, by a pop-up screen.

[0160] The filter array 100C can multiplex multiple wavelength information in one pixel instead of one wavelength information. The captured image 120 contains multiplexed two-dimensional information. The two-dimensional information is about space and wavelength, for example, spectral information randomly encoded. In the case of using a fixed pattern as the filter array 100C, the encoding pattern is learned by machine learning. Thus, although it is two-dimensional input data, in fact, three-dimensional (i.e., two-dimensional position and one-dimensional wavelength) information is used for object recognition.

[0161] The image data in this embodiment is data in which wavelength information is multiplexed, so the spatial resolution per wavelength can be improved compared to the conventional hyperspectral image that sacrifices spatial resolution. Furthermore, the object recognition device 300 in this embodiment can obtain image data of one frame using a single lens. Thus, compared to the conventional hyperspectral imaging method of a high-resolution scanning method, it is possible to recognize moving objects or objects with strong hand shaking.

[0162] In the conventional hyperspectral image capturing, there is a problem of low detection sensitivity per wavelength. For example, when the image is decomposed into 40 wavelengths, the light amount per pixel is reduced to 1 / 40 compared to the case where the image is not decomposed. In contrast, in the method of this embodiment, Figure 3A as well as Figure 3BAs illustrated, the image sensor 60 detects, for example, about 50% of the incident light. This increases the amount of light detected per pixel compared to conventional hyperspectral images. As a result, the SN ratio of the image increases.

[0163] Next, examples of other functions of the imaging device in which the object recognition method according to the present embodiment is implemented will be described.

[0164] Figure 5A This is a diagram schematically showing a function of displaying a recommended area for object recognition to assist in photographing by a camera. If the object 70 is imaged extremely small or extremely large on the image sensor 60, there will be a difference between the imaged image of the object 70 and the image of the training data set recognized during learning, and the recognition accuracy will decrease. The filter array 100C, for example, contains different wavelength information for each pixel. Therefore, if the object 70 is detected only by a part of the photographing area of ​​the image sensor 60, the wavelength information will be biased. In order to prevent the deviation of the wavelength information, the object 70 can be photographed as widely as possible in the photographing area of ​​the image sensor 60. In addition, if the image of the object 70 is photographed in a state where it protrudes from the photographing area of ​​the image sensor 60, the information on the spatial resolution of the object 70 will be lacking. Therefore, the recommended area for object recognition is slightly inside the photographing area of ​​the image sensor 60. Figure 5A In the example shown, an auxiliary display 400a showing a recommended area for object recognition is displayed on the display 400. Figure 5A In the figure, the entire area of ​​the display 400 corresponds to the shooting area of ​​the image sensor 60. For example, an area from 60% to 98% of the horizontal width or vertical width of the shooting area can be displayed on the display 400 as a recommended area for object recognition. The recommended area for object recognition can also be an area from 70% to 95% or an area from 80% to 90% of the horizontal width or vertical width of the shooting area. In this way, before acquiring image data through the shooting device 150, the auxiliary display 400a can also be displayed on the display 400. The auxiliary display 400a will inform the user of the area where the object 70 should be located in the captured scene or the range that the object 70 should occupy. Similarly, each of the multiple learning image data contained in the multiple training data sets can be obtained by capturing the object 70 in a state where it occupies more than a specified range in the image.

[0165] Figure 5B Schematically shows an object 70 magnified by an optical system having a zoom function. Figure 5B In the example shown in the left part of FIG. 1 , the object 70 before enlargement is displayed on the display 400. Figure 5BIn the example shown in the right part of FIG. 4 , the magnified object 70 is displayed on the display 400. In this way, the object 70 can be imaged widely on the image sensor 60 by the optical system 40 having a zoom function.

[0166] Figure 5C 1 is a diagram schematically showing a modified example of the filter array 100C. Figure 5C In the example shown, the region group AA composed of a set of multiple regions (A1, A2, ...) is periodically arranged. The multiple regions have different spectral characteristics. The so-called periodicity means that the region group AA is repeated more than twice in the longitudinal and / or transverse directions while maintaining the spectral characteristics. Figure 5C The filter array 100C shown can prevent spatial deviation of wavelength information. Figure 5C The entire filter array 100C shown in the figure is not used for learning, but only for learning the area group AA which is a partial set of periodic structures. In this way, the learning time can be shortened. By periodically configuring filters with the same spectral characteristics in space, object recognition can be performed even when the object is not captured in the entire shooting area but in a part of it.

[0167] The image encoded by the filter array 100C may include, for example, randomly multiplexed wavelength information. Therefore, the image is difficult for the user to view. Therefore, the object recognition device 300 may also be provided with a conventional camera for displaying to the user. That is, the object recognition device 300 may also be provided with a binocular structure of the camera 150 and a conventional camera. Thus, the user can display a visible monochrome image that is not encoded on the display 400. As a result, the user can easily grasp the positional relationship between the object 70 and the shooting area of ​​the image sensor 60.

[0168] The object recognition device 300 may also have a function of extracting the contour of the object 70 in the image. By extracting the contour, the unnecessary background around the object 70 can be removed. The image data from which the unnecessary background has been removed may also be used as learning image data. In this case, the recognition accuracy can be further improved. The object recognition device 300 may also have a function of displaying the recognition result of the contour on the display 400 so that the user can make fine adjustments to the contour.

[0169] FIG. 6A to FIG. 6C FIG. 3 is a diagram schematically showing an application example of the object recognition device 300 in the present embodiment.

[0170] Fig. 6A Part (a) shows an application example of distinguishing the types of plants. Fig. 6A Part (b) shows an application example of displaying the name of food. Fig. 6APart (c) shows an application example of analysis of mineral resources. Fig. 6A Part (d) shows an application example of determining the type of insect. In addition, the object recognition device 300 in this embodiment is effective for security authentication / lock release such as facial authentication, or human detection. In the case of a normal monochrome image or RGB image, it is possible that the human eye will misidentify an object at a glance. In contrast, by applying multi-wavelength information as in this embodiment, the recognition accuracy of the object can be improved.

[0171] Figure 6B An example of displaying detailed information of an object 70 on a smartphone installed with the object recognition method in the present embodiment is shown. In this example, an object recognition device 300 is mounted on a smartphone. Simply by holding the smartphone on the object 70, it is possible to determine what the object 70 is, and based on the result, the name of the object 70 and its description information are collected from a database via a network and displayed. In this way, a portable information device such as a smartphone can be used as an "image retrieval encyclopedia". In the "image retrieval encyclopedia", when it is difficult to fully recognize, multiple candidates can also be prompted in order of probability from high to low. In this way, based on the recognition result of the object 70, data representing the name and description information of the object 70 can be obtained from the database, and the name and / or description information can be displayed on the display 400.

[0172] Figure 6C An example of recognizing a plurality of objects existing on a street by a smartphone is shown. The smartphone is equipped with an object recognition device 300. When the object 70 is determined, such as an inspection object on a production line, the inspection device only obtains information of a specific wavelength corresponding to the object 70. On the other hand, in a situation where the target of the object 70 is not determined, such as use on the street, it is effective to obtain multi-wavelength information as the object recognition device 300 in this embodiment. The object recognition device 300 can be configured on the display 400 side of the smartphone according to the use example, or it can be configured on the surface opposite to the display 400.

[0173] In addition, the object recognition method in this embodiment can also be applied to a wide range of fields that can perform recognition based on artificial intelligence (AI), such as map applications, autonomous driving, or car navigation. As described above, the object recognition device can also be installed in portable devices such as smartphones, tablet computers, or head-mounted display devices. If it can be photographed with a camera, biological bodies such as people, faces, or animals can also become objects 70.

[0174] The captured image 120 represented by the image data input to the signal processing circuit 200 is a multi-coded image. Therefore, it is difficult to determine what is captured in the captured image 120 at first glance. However, the captured image 120 contains information representing the characteristics of the object 70, that is, feature information. Therefore, the AI ​​can directly recognize the object 70 from the captured image 120. As a result, the computational processing of image reconstruction, which takes a long time, is not required.

[0175] (Implementation Method 2)

[0176] The object recognition device 300 of the second embodiment is applied to a sensor device for autonomous driving. Detailed description of the same contents as those of the first embodiment will be omitted, and the description will focus on the differences from the first embodiment.

[0177] Figure 7 : is a diagram schematically showing an example of vehicle control using the object recognition device 300 in the present embodiment. The object recognition device 300 mounted on the vehicle can sense the environment outside the vehicle and recognize one or more objects 70 around the vehicle that enter the field of view of the object recognition device 300. The objects 70 around the vehicle may include, for example, oncoming vehicles, parallel vehicles, parked vehicles, pedestrians, bicycles, roads, lanes, white lines, sidewalks, curbs, grooves, signs, signals, telephone poles, shops, trees, obstacles, or fallen objects.

[0178] The object recognition device 300 has the same shooting device as the object recognition device in Embodiment 1. The shooting device generates image data of a dynamic image at a predetermined frame rate. The image data represents a captured image 120 after light from an object 70 around the vehicle passes through a filter array 100C and is multiplexed and encoded. The signal processing circuit 200 obtains the image data, extracts one or more objects 70 within the field of view from the image data, estimates what each extracted object 70 is, and labels each object 70. Based on the recognition result of the object 70, the signal processing circuit 200 can, for example, understand the surrounding environment, judge danger, or display the target driving trajectory 420. Data such as the surrounding environment, danger information, and the target driving trajectory 420 can be used to control vehicle-mounted equipment such as the steering or transmission of the vehicle body. In this way, automatic driving can be performed. The recognition result of the object recognition label or the travel path, etc., is as follows: Figure 7 As shown, it can also be displayed on a display 400 installed in the vehicle so that the driver can understand it. In this way, the vehicle control method in this embodiment includes a step of controlling the action of the vehicle equipped with the camera 150 based on the recognition result of the object 70.

[0179] In object recognition using existing RGB or monochrome images, it is difficult to distinguish between photos and real objects. Therefore, for example, there is a situation where photos and real objects on signs or posters are mistakenly recognized. However, in the object recognition device 300, by using multi-wavelength information, the difference in spectral distribution between the paint of the sign and the real car can be considered. Thus, the recognition accuracy can be improved. Furthermore, in the object recognition device 300, two-dimensional data with superimposed multi-wavelength information is obtained. Thus, compared with the previous three-dimensional hyperspectral data, the amount of data is small. As a result, the time required for reading and transmitting data and the processing time of machine learning can be shortened.

[0180] In addition to misidentification between photos and real objects, objects in camera images sometimes appear to be different objects by accident. Figure 7 In the example shown, the street trees appear to be in the shape of people depending on their growth or viewing angle. Figure 7 The street tree shown is mistakenly identified as a person. In this case, in an autonomous driving environment, the sudden appearance of a person can be mistakenly identified to instruct the vehicle to slow down or brake suddenly. As a result, an accident may occur. For example, on a highway, the vehicle should not stop suddenly due to a mistaken identification. In such an environment, the object recognition device 300 can improve the recognition accuracy compared with the previous object recognition by utilizing multi-wavelength information.

[0181] The object recognition device 300 can be used in combination with various sensors such as millimeter wave radar, laser rangefinder (Lidar) or GPS. In this way, the recognition accuracy can be further improved. For example, by linking with the information of the pre-recorded road map, the generation accuracy of the target's driving trajectory can be improved.

[0182] (Implementation 3)

[0183] In Embodiment 3, different from Embodiment 1, coded image data is obtained by using a plurality of light sources having different emission wavelength regions instead of filter array 100C. Detailed description of the same contents as Embodiment 1 is omitted below, and the description will focus on the differences from Embodiment 1.

[0184] Figure 8 Schematically shows an example of an object recognition device 300 in this embodiment. The object recognition device 300 in this embodiment includes an imaging device 150, a signal processing circuit 200, a display 400, and a memory 500. The imaging device 150 includes an optical system 40, an image sensor 60, a light source array 100L, and a control circuit 250.

[0185] The light source array 100L includes a plurality of light sources that each emits light in a different wavelength region. The control circuit 250 controls the image sensor 60 and the plurality of light sources included in the light source array 100L. The control circuit 250 changes the combination of the light sources that emit light, and repeats the action of making the image sensor 60 shoot multiple times in a state where a part or all of the plurality of light sources emit light. Thus, each time shooting, light with different spectral characteristics is emitted from the light source array 100L. The combination of light sources that emit light does not include exactly the same combination. However, a part of the light sources may be repeated in more than two of the plurality of combinations. Therefore, the captured images 120G1, 120G2, 120G3, ..., 120Gm obtained in each shooting at shooting time T1, T2, T3, ..., Tm have different intensity distributions. In the present embodiment, the image data input to the signal processing circuit 200 is a collection of image signals generated each time the image sensor 60 in the shooting device 150 shoots multiple times.

[0186] Alternatively, the control circuit 250 not only makes each light source perform a binary change of lighting or extinguishing, but also adjusts the light amount of each light source. When such an adjustment is performed, multiple image signals with different wavelength information can also be obtained. Each light source can be, for example, an LED, an LD, a laser, a fluorescent lamp, a mercury lamp, a halogen lamp, a metal halide lamp, or a xenon lamp, but is not limited thereto. In addition, in the case of emitting light in a terahertz-level wavelength region, the light source can use an ultra-high-speed fiber laser such as a femtosecond laser.

[0187] The signal processing circuit 200 performs learning and classification of the object 70 using all or any of the captured images 120G1 , 120G2 , 120G3 , . . . , 120Gm included in the image data.

[0188] The control circuit 250 is not limited to light with uniform spatial illuminance distribution, and may, for example, emit light with random spatial intensity distribution to the light source array 100L. The light emitted from the plurality of light sources may also have a two-dimensional illuminance distribution that is different for each wavelength. Figure 8 As shown in FIG. 2 , an image of light emitted from the light source array 100L toward the object 70 and passing through the optical system 40 is formed on the image sensor 60. In this case, the light incident on each pixel or each plurality of pixels of the image sensor 60 has a spectral characteristic including a plurality of different spectral peaks, similarly to the example shown in FIG. 2 . Thus, similarly to the first embodiment, object recognition can be performed with a single lens.

[0189] As in Embodiment 1, the plurality of learning image data included in the plurality of training data sets include learning image data generated by the imaging device 150 or other imaging devices. When the learning image data is generated by other imaging devices, the other imaging devices may include a light source array having the same characteristics as the light source array 100L included in the imaging device 150. When the image data of the recognition object and each learning image data are encoded by the light source array having the same characteristics, a high recognition accuracy of the object 70 can be obtained.

[0190] The object recognition method in the present disclosure includes: the step of obtaining image data in which multiple wavelength information is multiplexed in each pixel; and the step of applying a classification model trained by a machine learning algorithm to the image data in which multiple wavelength information is multiplexed, thereby identifying objects contained in the scene represented by the image data. In addition, the object recognition method in the present disclosure includes the step of strengthening the classification model training using the image data in which multiple wavelength information is multiplexed. The means for obtaining image data in which multiple wavelength information is multiplexed in each pixel is not limited to the shooting device described in the above-mentioned embodiment.

[0191] The present disclosure also includes programs and methods that specify actions performed by the signal processing circuit 200 .

[0192] Industrial Applicability

[0193] The object recognition device disclosed in the present invention can be used in measuring equipment that can accurately recognize an object during measurement. The object recognition device can also be applied to, for example, plant / food / organism category recognition, road guidance / navigation, mineral exploration, sensing for organisms / medical / cosmetics, food foreign matter / pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems such as autonomous driving.

[0194] Description of Reference Numerals

[0195] 40 Optical System

[0196] 60 Image Sensor

[0197] 70 objects

[0198] 100C filter array

[0199] 100L light source array

[0200] 120 images captured

[0201] 200 signal processing circuit

[0202] 250 control circuit

[0203] 300 Object Recognition Device

[0204] 400 Display

[0205] 400a auxiliary display

[0206] 420 target driving trajectory

[0207] 500 Memory

Claims

1. A method for classifying objects, wherein: include: The step of obtaining image data of an image of an object; as well as a step of classifying said objects contained in said image, The image data is data of a multiplexed image in which information of a plurality of wavelengths is multiplexed in one pixel. Classifying the object is performed by inputting the data of the multiplexed image into a classification model trained by a machine learning algorithm.

2. The object classification method according to claim 1, wherein: The classification model is pre-trained using a plurality of first training data sets, each of which includes learning image data and label data, wherein the label data identifies the object included in the learning image indicated by the learning image data.

3. The object classification method according to claim 2, wherein: The plurality of learning image data included in the plurality of first training data sets include learning image data generated by a second imaging device different from a first imaging device used to acquire the image data.

4. The object classification method according to claim 3, wherein: The second imaging device includes a filter array having the same characteristics as those of the filter array in the first imaging device.

5. The object classification method according to any one of claims 2 to 4, wherein: Also includes: After the object is classified, the classification model is further trained using a second training data set, wherein the second training data set includes the image data and second label data for identifying the object.

6. The object classification method according to any one of claims 2 to 4, wherein: The positions of the objects in the learning images in the plurality of learning image data included in the plurality of first training data sets are different from one another in the plurality of learning image data.

7. The object classification method according to any one of claims 2 to 4, wherein: The learning image data is acquired by capturing the object in a state where the object occupies a predetermined area or more in the learning image.

8. The object classification method according to any one of claims 1 to 4, wherein: The image data is obtained by using a camera including a display. The object classification method further includes a step of causing the display to display an auxiliary display for notifying a user of an area in which the object should be located or a range that the object should occupy in the image before acquiring the image data.

9. An information display method using the object classification method according to any one of claims 1 to 4, wherein: Include: A step of acquiring, from a database, data representing at least one selected from the group consisting of a name of the object and a description of the object based on a result of classifying the object; as well as The step of displaying the at least one selected from the group consisting of the name of the object and the description of the object on a display.

10. An object classification device, wherein: have: An image sensor that acquires image data of an image of an object; and Information processing circuits, The information processing circuit, classifying the object contained in the image, The image data is data of a multiplexed image in which information of a plurality of wavelengths is multiplexed in one pixel. The classification of the object is performed by inputting the data of the multiplexed image into a classification model trained by a machine learning algorithm.

Images