Specific device, specific method, and program

Abstract

To provide a specific device capable of improving the accuracy of image recognition using infrared images. [Solution] The identification device 10 according to this disclosure comprises an image acquisition unit 11 that acquires an infrared image, a region detection unit 12 that detects an image region containing a specific object from the infrared image, a material identification unit 14 that identifies the material corresponding to a pixel based on the spectral reflectance corresponding to the pixel of the specific object included in the image region, and a type identification unit 15 that identifies the type of the specific object based on the material corresponding to the pixel.

Description

Technical Field

[0004] , ,

[0001] The present disclosure relates to a specific device, a specific method, and a program.

Background Art

[0002] There is known a technique of performing imaging using a surveillance camera provided around a road or the like and performing image recognition on a person or a vehicle included in the captured image.

[0003] As a related technique, Patent Document 1 discloses a system for performing human body detection based on a visible light image and a far infrared image. In the system disclosed in Patent Document 1, human body detection is performed at night or in a backlight environment by a method using thermal detection or background difference method. The system acquires a visible light image and a far infrared image, extracts an image of an animal from the far infrared image, recognizes the contour shape of the extracted animal image or the coordinate values of the contour shape, and specifies an image part corresponding to the contour shape or the coordinate values in the visible light image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In environments such as at night or in backlight, image recognition using an image of a far infrared camera is effective. On the other hand, in image recognition using an image of a far infrared camera, there is a problem that there are many misrecognitions such as a person riding a motorcycle being recognized as a pedestrian. The technique disclosed in Patent Document 1 does not sufficiently consider such problems.

[0006] An object of the present disclosure is to provide a specific device, a specific method, and a program capable of improving the accuracy of image recognition using an infrared image in view of the above-described problems. [Means for solving the problem]

[0007] The specified device relating to this disclosure is An image acquisition unit that acquires infrared images, A region detection unit that detects an image region containing a specific object from the aforementioned infrared image, A material identification unit identifies the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, The system includes a type identification unit that identifies the type of the specified object based on the material corresponding to the pixel.

[0008] The method of identification relating to this disclosure is: Image acquisition step to acquire infrared image, A region detection step for detecting an image region containing a specific object from the aforementioned infrared image, A material identification step to identify the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, The method includes a type identification step of identifying the type of the specified object based on the material corresponding to the pixel.

[0009] Programs related to this disclosure Image acquisition step to acquire infrared image, A region detection step for detecting an image region containing a specific object from the aforementioned infrared image, A material identification step to identify the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, The computer is instructed to perform a type identification step, which involves identifying the type of the specific object based on the material corresponding to the pixel, and a type identification step, which involves having the computer perform these steps. [Effects of the Invention]

[0010] The identification device, identification method, and program described herein can improve the accuracy of image recognition using infrared images.

Brief Description of the Drawings

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of a specific system. [Figure 2] FIG. 2 is a diagram showing an example of a far-infrared image with a recognition frame. [Figure 3] FIG. 3 is a table showing an example of the spectral reflectance for each material. [Figure 4] FIG. 4 is a diagram showing an example of a material corresponding to an image area including a person on a scooter. [Figure 5] FIG. 5 is a diagram showing an example of a material corresponding to an image area including a person on a large motorcycle. [Figure 6] FIG. 6 is a diagram showing an example of a material corresponding to an image area including a person on a bicycle. [Figure 7] FIG. 7 is a diagram showing an example of a material corresponding to an image area including a person. [Figure 8] FIG. 8 is a diagram showing an example of a table indicating the relationship between the material corresponding to the pixels included in the recognition frame and the type of a specific object. [Figure 9] FIG. 9 is a flowchart showing the processing performed by a specific device. [Figure 10] FIG. 10 is a diagram showing an example of a table indicating the relationship between the occupancy rate of pixels for each material included in the recognition frame and the type of a specific object. [Figure 11] FIG. 11 is a flowchart showing the processing performed by a specific device. [Figure 12] FIG. 12 is a diagram showing an example of a table indicating the relationship between the arrangement and occupancy rate of pixels for each material included in the recognition frame and the type of a specific object. [Figure 13] FIG. 13 is a flowchart showing the processing performed by a specific device.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For the sake of clarity of explanation, duplicate explanations are omitted as necessary.

[0013] <Embodiment 1> (Specific System 1) Referring to FIG. 1, Embodiment 1 will be described. FIG. 1 is a block diagram showing the configuration of a specific system 1 according to this embodiment. The specific system 1 includes a specific device 10, an infrared camera 20, and a hyperspectral camera 30. In Embodiment 1, the case where the specific device 10 is installed in a multifunctional utility pole used as road infrastructure such as a smart pole (registered trademark) will be exemplified. The multifunctional utility pole of Embodiment 1 is installed on, for example, a street and includes an antenna and communication equipment for providing a wireless communication function, and an infrared camera 20 and a hyperspectral camera 30 for photographing people and vehicles passing on or around the road. The specific device 10 of Embodiment 1 may be in a form fixed at a predetermined location such as a multifunctional utility pole, or may be mounted on a moving body or flying body such as a vehicle or a drone.

[0014] The specific system 1 is an information processing system that recognizes a specific target included in an infrared image that is a target for specifying a type, and specifies the type of the recognized specific target. For example, the specific system 1 uses an infrared image taken by an infrared camera to specify the type of a specific target existing on or around the road. Not limited to this, the specific system 1 may be used in various environments where the type of a specific target can be specified.

[0015] The specific target is an object to be the target of image recognition. For example, the specific target is a person or a vehicle existing within the shooting range. The specific target may be a single person or a person in a vehicle. The vehicle is, for example, a motorcycle, a bicycle, or a kick scooter.

[0016] The type of specified object indicates a category for classifying the specified object. The type of specified object may also be information indicating the mode of transportation of a person riding in a vehicle. For example, the type of specified object may be a motorcycle, bicycle, or person (pedestrian). However, it is not limited to these, and information on various modes of transportation may be included as the type of specified object. For example, a kick scooter may be included as the type of specified object.

[0017] Furthermore, the types of specified objects may be further subdivided according to the characteristics of the materials included in the specified object. For example, the types of specified objects may be subdivided according to the size of the vehicle. For example, if the type of specified object is a motorcycle, then depending on the size of the motorcycle, small motorcycles, medium motorcycles, or large motorcycles may be used as types of specified objects. Alternatively, based on the type of motorcycle, scooters may be used as types of specified objects.

[0018] The specific device 10 acquires data from the far-infrared camera 20 and the hyperspectral camera 30, and performs the processing described herein based on the acquired data. The specific device 10 is connected to the far-infrared camera 20 and the hyperspectral camera 30 by a network (not shown). The network is a wired or wireless communication line. The network is, for example, the Internet, but the form of communication is not limited thereto.

[0019] The number of specific devices 10 provided in specific system 1 is arbitrary. Furthermore, specific devices 10 may include multiple far-infrared cameras 20 and hyperspectral cameras 30. The various components of specific system 1 will be described in detail below.

[0020] (Far-infrared camera 20) The far-infrared camera 20 is an example of an infrared camera that photographs a subject and acquires an infrared image. In this disclosure, the far-infrared camera 20 photographs the environment and acquires a far-infrared image that includes a specific object. In the following description, a far-infrared camera is used as an example of an infrared camera, but the specific system 1 may include cameras other than a far-infrared camera. For example, the specific system 1 may include a near-infrared camera or a mid-infrared camera in place of, or in addition to, the far-infrared camera.

[0021] The far-infrared camera 20 is installed in an environment where a specific target exists, at a position where the specific target can be photographed. For example, the far-infrared camera 20 is installed on or near a road. The far-infrared camera 20 photographs the environment in a predetermined direction. The far-infrared camera 20 acquires a far-infrared image by taking photographs and transmits the acquired far-infrared image to the identification device 10. The far-infrared image may be a still image or a moving image. The far-infrared camera 20 may transmit the far-infrared image to the identification device 10 at predetermined time intervals.

[0022] (Hyperspectral camera 30) The hyperspectral camera 30 is an example of a multi-wavelength spectroscopic camera that photographs a subject and acquires a spectral image. A multi-wavelength spectroscopic camera detects light at multiple different wavelengths and acquires spectral information of a subject at multiple wavelengths. In the following explanation, a hyperspectral camera will be used as an example of a multi-wavelength spectroscopic camera, but the specific system 1 may also include cameras other than a hyperspectral camera. For example, the specific system 1 may include a multispectral camera or an ultraspectral camera in place of, or in addition to, the hyperspectral camera.

[0023] The hyperspectral camera 30, like the far-infrared camera 20, is positioned in an environment where a specific target exists and capable of photographing the specific target. The hyperspectral camera 30 photographs the environment in a predetermined direction and acquires a spectral image. The spectral image includes spectral information corresponding to each pixel. The hyperspectral camera 30 transmits the acquired spectral image to the identification device 10. The hyperspectral camera 30 may transmit spectral images to the identification device 10 at predetermined time intervals.

[0024] (Specific device 10) The specific device 10 acquires data from the far-infrared camera 20 and the hyperspectral camera 30 and performs the processing described herein. For example, if the specific system 1 is a smart pole, the specific device 10 is a control device installed as a smart pole, or a server device that processes information acquired from the smart pole via communication. If the specific system 1 is a vehicle safety driving support system, the specific device 10 is a control device mounted on the vehicle, or a server device that processes information acquired from the vehicle via communication. The specific device 10 comprises an image acquisition unit 11, a region detection unit 12, a spectral reflectance calculation unit 13, a material identification unit 14, a type identification unit 15, and a storage unit 19.

[0025] The specific device 10 includes a processor and memory, although these are not shown in the diagram. The storage unit 19, which is a storage device, stores a computer program on which the processing described herein is implemented. The storage unit 19 includes, for example, a non-volatile storage device such as a hard disk or flash memory and a memory such as RAM (Random Access Memory), i.e., a volatile storage device. The processor can load the computer program from the storage unit 19 into the memory and execute the computer program. As a result, the processor realizes the functions of the image acquisition unit 11, the region detection unit 12, the spectral reflectance calculation unit 13, the material identification unit 14, and the type identification unit 15.

[0026] Alternatively, the image acquisition unit 11, region detection unit 12, spectral reflectance calculation unit 13, material identification unit 14, and type identification unit 15 may each be implemented with dedicated hardware. Furthermore, some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the above-mentioned circuits, etc., and programs. In addition, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc., can be used as the processor.

[0027] Furthermore, if some or all of the components of the specific device 10 are realized by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be realized in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. Also, the functions of the specific device 10 may be provided in SaaS (Software as a Service) format.

[0028] The image acquisition unit 11 acquires an infrared image including a specific target. Specifically, the image acquisition unit 11 acquires a far-infrared image including the specific target from the far-infrared camera 20. The image acquisition unit 11 outputs the acquired far-infrared image to the region detection unit 12. The image acquisition unit 11 also acquires a spectral image from the hyperspectral camera 30. The image acquisition unit 11 outputs the spectral image to the spectral reflectance calculation unit 13.

[0029] In this example, the image acquisition unit 11 acquires images from both the far-infrared camera 20 and the hyperspectral camera 30, but it is not limited to this. The specific system 1 may also include a far-infrared image acquisition unit that acquires far-infrared images from the far-infrared camera 20 and a spectral image acquisition unit that acquires spectral images from the hyperspectral camera 30, separately.

[0030] The region detection unit 12 detects an image region containing a specific object from the far-infrared image acquired by the image acquisition unit 11. The region detection unit 12 recognizes the specific object in the far-infrared image using a known object recognition method that uses a learning model trained on images of the specific object, such as YOLO (YOU Only Look Once), and assigns a rectangular recognition frame to the image region containing the specific object. In other words, the image region enclosed by the recognition frame is the image region containing the specific object. Note that the shape of the recognition frame is not limited to a rectangle. Furthermore, hereafter, when simply referred to as "recognition frame," it refers to the image region containing the specific object.

[0031] The region detection unit 12, when it recognizes a specific object in a far-infrared image using a learning model, determines whether there are multiple candidate types that are candidates for the type of the recognized specific object. Here, "when there are multiple candidate types that are candidates for the type of the recognized specific object" refers to the case where, when identifying the type of the specific object using multiple learning models, there are multiple recognition results that are equal to or higher than a predetermined recognition score.

[0032] For example, suppose the region detection unit 12 identifies the type of specific object using a person learning model and a bicycle learning model that is different from the person learning model. The person learning model is a learning model for recognizing people who are pedestrians. The bicycle learning model is a learning model for recognizing people who are riding bicycles. Each learning model outputs a recognition score according to the similarity between the specific object recognized by each learning model and the specific object. The recognition score indicates a high degree of similarity as it approaches 1, and a low degree of similarity as it approaches 0.

[0033] For example, the region detection unit 12 obtains a person recognition score SC1 as a recognition result using a person learning model, and a bicycle recognition score SC2 as a recognition result using a bicycle learning model. The region detection unit 12 compares a predetermined threshold with the recognition score and extracts candidate types of specific targets based on the comparison result.

[0034] For example, suppose a first threshold and a second threshold lower than the first threshold are set as thresholds. For example, the first threshold is 0.8 and the second threshold is 0.7. The first and second thresholds can be set arbitrarily. The region detection unit 12 extracts types for which a recognition score of the second threshold or higher has been obtained as candidate types.

[0035] For example, the region detection unit 12 extracts "person" as a candidate type if the person recognition score SC1 of the specific target included in the recognition frame R1 is equal to or greater than the second threshold. Similarly, the region detection unit 12 extracts "person riding a bicycle" as a candidate type if the bicycle recognition score SC2 of the specific target included in the recognition frame R1 is equal to or greater than the second threshold. If only one candidate type is extracted, it does not fall under the category of "there are multiple candidate types that are candidates for the type of the recognized specific target."

[0036] On the other hand, if multiple candidate types are extracted with recognition scores of 2 or higher, this falls under the category of "cases where there are multiple candidate types that are candidates for the type of specific object recognized." For example, this applies when the person recognition score SC1 is 1 or higher, and the bicycle recognition score SC2 is below 1 and 2 or higher. However, this does not apply when the person recognition score SC1 is 1 or higher, and the bicycle recognition score SC2 is below 2.

[0037] Figure 2 shows an example of a far-infrared image 100 with a recognition frame R1. The far-infrared image 100 is an image of a person P1 riding a scooter V1. The far-infrared image 100 includes a recognition frame R1 enclosed by a thick solid line. The recognition frame R1 is the image region of the far-infrared image 100 that includes the scooter V1 and the person P1 riding the scooter V1. For example, the recognition frame R1 may indicate the region within the bounding rectangle of the combined image region of the scooter V1 and the person P1 riding the scooter V1.

[0038] The far-infrared image 100 also includes a type display area A that indicates the type of the specific object. The type display area A is an area that displays the type of the specific object identified as a result of image recognition in the specific system 1. In Figure 2, the word "bike" indicating the type of the specific object is displayed in the type display area A.

[0039] Here, we will specifically explain the issues addressed in this disclosure. When using far-infrared cameras to perform image recognition on people or vehicles passing on a road, it is necessary to display the type of the specific object in a display unit that does not show the type, as shown in type display area A in Figure 2. The display unit is, for example, a display installed in a designated monitoring center.

[0040] For example, System 1 displays the type of object being identified near the recognition frame R1, such as "motorcycle" if a person on a motorcycle is recognized, "bicycle" if a person on a bicycle is recognized, and "person" if a pedestrian is recognized. By displaying information in this way, monitors stationed at the monitoring center can intuitively grasp the type of object being identified, thereby preventing accidents and other incidents.

[0041] Thus, when monitoring the environment using far-infrared cameras, it is common practice to recognize specific objects within the environment using a predetermined learning model. The learning model is pre-configured to recognize the type of specific object included in the recognition frame. Examples of learning models include dictionary files published on designated websites or dictionary files created using deep learning.

[0042] For example, a method is known to individually recognize motorcycles, bicycles, and people using motorcycle learning models, bicycle learning models, and person learning models, respectively, for recognizing motorcycles, bicycles, and people. In such a method, the type of object is identified depending on whether the recognition results from multiple learning models match or not.

[0043] For example, suppose a far-infrared image contains a person riding a motorcycle. In this case, if the recognition result from the motorcycle learning model matches the recognition result from the person learning model, the type of object is identified as a motorcycle. If these recognition results do not match, the type of object is identified as a person. The same applies to the bicycle learning model. Therefore, methods using such learning models may not correctly recognize motorcycles, bicycles, or people. Consequently, there is a problem in that objects that are actually motorcycles or bicycles are often mistakenly identified as people.

[0044] In contrast, the identification system 1 according to this embodiment solves the above problem by improving the accuracy of identifying motorcycles, bicycles, and people through sensor fusion of a far-infrared camera 20 and a hyperspectral camera 30, as described below. For example, the identification device 10 can identify various materials (e.g., aluminum, iron, rubber, plastic, cloth, or glass) using spectral information acquired from the hyperspectral camera 30, and accurately identify the type of object using the identification results. Details regarding the identification of the type of object will be described later.

[0045] Returning to Figure 1, the spectral reflectance calculation unit 13 calculates the spectral reflectance of a specific target corresponding to a pixel included in the recognition frame. In the example in Figure 2, the spectral reflectance calculation unit 13 calculates the spectral reflectance of the scooter V1 and the person P1 corresponding to pixels included in the recognition frame R1.

[0046] Specifically, the spectral reflectance calculation unit 13 acquires a spectral image 100S (not shown) corresponding to the far-infrared image 100, and calculates the spectral reflectance corresponding to the pixels of a specific target included in the image region of the recognition frame R1 of the far-infrared image 100 based on the spectral information of the acquired spectral image 100S.

[0047] For example, the spectral reflectance calculation unit 13 integrates the information contained in the far-infrared image 100 and the information contained in the spectral image 100S by performing sensor fusion processing. This allows the spectral reflectance calculation unit 13 to associate pixels contained in the far-infrared image 100 with pixels contained in the spectral image 100S. The spectral reflectance calculation unit 13 also calculates the spectral reflectance corresponding to each pixel contained in the recognition frame R1. This allows the material identification unit 14 to identify the material for each pixel.

[0048] In calculating spectral reflectance, the unit of pixels to be calculated is arbitrary. The spectral reflectance calculation unit 13 only needs to calculate the spectral reflectance using one or more pixels as the unit. For example, the spectral reflectance calculation unit 13 may calculate the spectral reflectance for each individual pixel, or it may calculate the spectral reflectance by calculating the average value of the spectral reflectance of each individual pixel for multiple pixels.

[0049] For example, the spectral reflectance calculation unit 13 may calculate the spectral reflectance corresponding to multiple pixels included in a predetermined line or multiple pixels included in a predetermined block in the recognition frame R1. By calculating the spectral reflectance of each individual pixel, the material type identification unit 15, described later, can easily calculate the proportion of pixels corresponding to a specific material within the recognition frame R1.

[0050] Although this explanation describes an example in which the spectral reflectance calculation unit 13 calculates the spectral reflectance, it is not limited to this. The spectral reflectance calculation unit 13 may also acquire the spectral reflectance calculated by a device other than the specified device 10 from that device.

[0051] Referring to Figure 3, the relationship between spectral reflectance and material will be explained. Figure 3 is Table T1, which shows an example of spectral reflectance for each material. As shown in Figure 3, the spectral reflectance at each wavelength differs for each material. Therefore, the spectral reflectance corresponding to a pixel differs depending on the material corresponding to the pixel included in the recognition frame. For example, the spectral reflectance of latex and aluminum differ significantly; at a wavelength of 900 nm, the spectral reflectance of latex is 0.85, while that of aluminum is 0.28.

[0052] Furthermore, when the spectral reflectance shown in Table T1 is graphed, graphs of spectral reflectance waveforms that differ for each material can be obtained. For example, if the spectral reflectance waveform shown in Table T1 is used as a reference waveform, and the spectral reflectance waveform corresponding to a pixel included in the recognition frame is used as a comparison waveform, the material corresponding to that pixel can be identified by comparing the reference waveform and the comparison waveform and determining whether they are similar or not. In addition, by identifying the material, the material corresponding to that pixel (for example, a vehicle part) can be identified.

[0053] Although Table T1 shows spectral reflectances for each 100 nm wavelength, the spectral reflectance calculation unit 13 may calculate the spectral reflectance at any wavelength.

[0054] Here, referring to Figures 4 to 7, the relationship between the material shown in Figure 3 and the specific target will be explained for each type of specific target. Figures 4 to 7 are diagrams illustrating examples of materials for each type of specific target. Note that in Figures 4 to 7, only the recognition frames R1 to R4 containing the specific target are shown in the far-infrared image, and the areas other than the recognition frames R1 to R4 and the type display area are omitted from the illustration.

[0055] Figures 4 and 5 show examples of material corresponding to an image region containing a person riding a motorcycle. Figure 4 shows an example of material corresponding to an image region (recognition frame R1) containing a person P1 riding a scooter V1. The scooter V1 comprises a frame 101a1, a frame 101a2, an exterior 101b, and tires 101c. Person P1 is wearing clothing 101d.

[0056] Figure 5 shows an example of material corresponding to an image region (recognition frame R2) that includes a person P2 riding a large motorcycle V2. The large motorcycle V2 comprises a frame 102a, exterior 102b, and tires 102c. Person P2 is wearing clothing 102d.

[0057] Examples of materials used for motorcycle tires include natural rubber and synthetic rubber. Natural rubber is, for example, latex. Synthetic rubber is, for example, nitrile. Examples of materials used for motorcycle frames include, for example, steel or aluminum alloy. Examples of materials used for motorcycle bodywork include, for example, ABS (Acrylonitrile Butadiene Styrene) resin, carbon fiber, or FRP (Fiber Reinforced Plastic).

[0058] Figure 6 shows an example of materials corresponding to an image region (recognition frame R3) that includes a person P3 riding a bicycle V3. The bicycle V3 comprises a frame 103a, exterior 103b, and tires 103c. Person P3 is wearing clothing 103d. Examples of tire materials used for the bicycle are the same as those for the scooter V1 and large motorcycle V2 described above. Examples of frame materials for the bicycle include chrome molybdenum steel, aluminum, titanium, carbon, or stainless steel.

[0059] Figure 7 shows an example of material corresponding to the image region (recognition frame R4) that includes person P4. Since person P4 is a pedestrian, recognition frame R4 does not include means of transportation such as a motorcycle. Therefore, recognition frame R4 does not include parts such as frames or exteriors. Also, person P4 is wearing clothing 104d.

[0060] As shown in Figures 4 to 7, the material corresponding to the pixels included in recognition frames R1 to R4 differs depending on the type of specific object. Therefore, the spectral reflectance calculation unit 13 calculates different spectral reflectances for the pixels included in recognition frames R1 to R4. For example, since recognition frame R4 shown in Figure 7 includes only the person P4 who is a pedestrian, the spectral reflectance is calculated only for the material, such as cloth, unlike recognition frames R1 to R3 shown in Figures 4 to 6.

[0061] Returning to Figure 1, the material identification unit 14 identifies the material corresponding to a pixel based on the spectral reflectance of the pixel corresponding to the specific object included in the image area of ​​the recognition frame. Specifically, the material identification unit 14 identifies the material corresponding to one or more pixels included in the recognition frame based on the spectral reflectance of the pixel. The material corresponding to a pixel included in the recognition frame could be, for example, a part of a vehicle or clothing worn by a person.

[0062] For example, the material identification unit 14 identifies the material using Table T1 shown in Figure 3. The material identification unit 14 sets the spectral reflectance waveform of each material shown in Table T1 as a reference waveform and sets the spectral reflectance waveform corresponding to the pixel included in the recognition frame as a comparison waveform. The material identification unit 14 compares the reference waveform and the comparison waveform. For example, the material identification unit 14 determines whether the reference waveform corresponding to material X and the comparison waveform corresponding to pixel Y included in the recognition frame R1 are similar to a predetermined level or higher. If it is determined that they are similar to a predetermined level or higher, the material identification unit 14 identifies the material corresponding to pixel Y as material X. The material identification unit 14 is not limited to this and may use any method to compare the reference waveform and the comparison waveform. For example, the material identification unit 14 may use the correlation coefficient to perform the comparison. Alternatively, the material identification unit 14 may compare the spectral reflectance at each wavelength of the reference waveform and the comparison waveform and perform the comparison using the number of wavelengths in which the difference in spectral reflectance is within a predetermined range.

[0063] The type identification unit 15 identifies the type of object to be identified based on the material corresponding to the pixels included in the recognition frame, which has been identified by the material identification unit 14.

[0064] Figure 8 shows an example of Table T2, which illustrates the relationship between the material corresponding to the pixels included in the recognition frame and the type of identified object. In Table T2, examples of materials include tires, frames, exteriors, and clothing. In Table T2, the score for whether or not the pixels corresponding to each material are included in the recognition frame is represented by "1" or "0". The score is 1 if the pixels corresponding to each material are included in the recognition frame, and 0 if they are not.

[0065] The score may be expressed in a format other than a numerical value. For example, the score may be expressed as "yes" and "no". Also, the score may be expressed in stages according to the number of pixels in the recognition frame of each material. For example, the score may be expressed in multiple stages such as 0 to 5, or it may be expressed as "○", "△", and "×".

[0066] The type identification unit 15 obtains material information corresponding to the pixels included in the recognition frame from the material identification unit 14 and identifies the type of the identified object by referring to Table T2. For example, suppose the recognition frame contains pixels corresponding to tires, frame, exterior, and clothing. In this case, the scores for tires, frame, exterior, and clothing are all 1. Based on Table T2, the type identification unit 15 identifies the identified object as a motorcycle or bicycle. Also, if the scores for tires, frame, and exterior are 0 and the score for clothing is 1, the type identification unit 15 identifies the type of the identified object as a person.

[0067] Returning to Figure 1, the memory unit 19 is a storage device that stores programs for realizing each function of the specific device 10. The memory unit 19 also stores the learning models 191 described above. The learning models 191 include, for example, a motorcycle learning model, a bicycle learning model, and a person learning model.

[0068] The configurations of the specific system 1 have been described above. Note that the configuration of the specific system 1 described above is merely an example and can be modified as appropriate. For example, in Figure 1, the far-infrared camera 20 and the hyperspectral camera 30 are shown outside the specific device 10, but this is not the only configuration. The specific device 10 may include at least one of the far-infrared camera 20 and the hyperspectral camera 30.

[0069] (Processing by specific device 10) Next, with reference to Figure 9, the process performed by the specific device 10 will be explained. Figure 9 is a flowchart showing the process performed by the specific device 10.

[0070] First, the image acquisition unit 11 acquires a far-infrared image from the far-infrared camera 20 (S11). The region detection unit 12 detects an image region containing a specific object from the acquired far-infrared image and assigns a recognition frame to it (S12).

[0071] Next, the region detection unit 12 extracts candidate types that are candidates for the type of the specified object and determines whether there are multiple candidate types (S13). For example, the region detection unit 12 extracts candidate types using the motorcycle learning model, bicycle learning model, and person learning model included in the learning model 191 stored in the memory unit 19. For example, the region detection unit 12 extracts types whose recognition score is equal to or higher than the second threshold as candidate types.

[0072] If it is determined that there is one or fewer candidate types (NO in S13), the region detection unit 12 uses those candidate types to identify the type of the target to be identified (S22). If it is determined that there are multiple candidate types (YES in S13), the process proceeds to step S14.

[0073] First, the image acquisition unit 11 acquires a spectral image from the hyperspectral camera 30 (S14). Next, the spectral reflectance calculation unit 13 calculates the spectral reflectance of a specific target corresponding to a pixel included in the recognition frame (S15). The spectral reflectance calculation unit 13 may associate pixels included in the far-infrared image with pixels included in the spectral image by performing sensor fusion processing. Subsequently, the material identification unit 14 identifies the material corresponding to the pixel based on the spectral reflectance (S16).

[0074] In the processing from step S17 onward, the type identification unit 15 identifies the type of the object to be identified based on the material corresponding to the pixels included in the recognition frame. First, the type identification unit 15 determines whether or not pixels corresponding to tires, frames, exteriors, and clothing are included in the recognition frame (S17). If it is determined that all pixels corresponding to tires, frames, exteriors, and clothing are included in the recognition frame (YES in S17), the type identification unit 15 identifies the type of the object to be identified as a motorcycle or a bicycle (S18).

[0075] If the system determines that at least one of the pixels corresponding to the tires, frame, exterior, and clothing is not included in the recognition frame (NO in S17), the type identification unit 15 determines whether or not the pixels corresponding to the clothing are included in the recognition frame (S19). If the system determines that the pixels corresponding to the clothing are included in the recognition frame (YES in S19), the type identification unit 15 identifies the type of the identified object as a person (S20). If the system determines that the pixels corresponding to the clothing are not included in the recognition frame (NO in S19), the type identification unit 15 identifies the type of the identified object as something other than a person (S21). An example of something being identified as something other than a person is when an object in the environment (e.g., a utility pole) is misrecognized.

[0076] The specified system 1 may have a display control unit (not shown) display the type of the specified target on the display unit. The display unit may be the display unit of the specified device 10, a display unit provided in a predetermined monitoring center, or a display unit of a terminal device used by a predetermined monitor.

[0077] For example, the display control unit causes the display unit to display an image such as the far-infrared image 100 shown in Figure 2. The display control unit may display an image in which the far-infrared image 100 and the type display area A are superimposed, as shown in the image. This allows a user who views the far-infrared image 100 to intuitively understand the type of a specific object. The display control unit may adjust the display mode of the far-infrared image 100 according to the type of the specific object. For example, the display control unit may display the type display area A in different colors depending on the type of the specific object.

[0078] As described above, the identification system 1 according to this embodiment identifies the material corresponding to a pixel based on the spectral reflectance of the target corresponding to the pixel included in the recognition frame, and then identifies the type of target based on the identified material. As a result, the identification system 1 can improve the accuracy of image recognition using infrared images.

[0079] This allows users who monitor the environment, such as roads, using far-infrared images recorded by surveillance cameras, to easily identify the type of specific object even in far-infrared images taken at night or in backlit conditions.

[0080] For example, while the related technologies mentioned above can recognize pedestrians, they do not take into account the recognition of people riding motorcycles or bicycles. In contrast, the identification system 1 according to this embodiment uses spectral images in addition to far-infrared images to identify the type of target, so it can accurately distinguish between people riding bicycles or motorcycles and pedestrians. As a result, the identification system 1 can perform human body detection in various states.

[0081] <Embodiment 2> Next, Embodiment 2 will be described. In Embodiment 1, the type of the target object was identified by determining whether or not the pixels corresponding to each material were included in the recognition frame. Embodiment 2 differs from Embodiment 1 in that it identifies the type of the target object using the proportion of the pixels corresponding to each material within the recognition frame. Below, we will mainly describe the differences from Embodiment 1, and explain any points that overlap with Embodiment 1 as appropriate.

[0082] The specified system 1a (not shown) in this embodiment includes a specified device 10a (not shown) instead of the specified device 10 described above. The block diagram showing the configuration of the specified system 1a can be explained by replacing the specified system 1 and specified device 10 shown in Figure 1 with the specified system 1a and specified device 10a, respectively. Therefore, the illustration of the specified system 1a is omitted. The function of the type-specific unit 15 in the specified device 10a differs from that of Embodiment 1.

[0083] In the identification device 10a, the type identification unit 15 identifies the type of the identified object based on the proportion of pixels corresponding to the identified material within the recognition frame. For example, the type identification unit 15 determines the number of pixels for each material included in the recognition frame. The type identification unit 15 may simply count the number of pixels for each material from the recognition frame and calculate the proportion of pixels corresponding to the identified material within the recognition frame using the total number of pixels included in the recognition frame, or it may calculate the proportion of pixels corresponding to the identified material within the recognition frame using the number of pixels remaining after subtracting the number of pixels corresponding to asphalt from the total number of pixels included in the recognition frame.

[0084] The type identification unit 15 uses the total number of pixels included in the recognition frame and the number of pixels for each material to calculate the proportion of pixels corresponding to the identified material within the recognition frame. In other words, the type identification unit 15 calculates the occupancy rate of pixels corresponding to the identified material within the recognition frame. For example, the type identification unit 15 calculates the occupancy rate of the pixel within the recognition frame as the degree of occupancy. The type identification unit 15 may also calculate an occupancy level indicating the degree of occupancy instead of the occupancy rate.

[0085] Figure 10 shows an example of Table T3, which illustrates the relationship between the pixel occupancy rate for each material included in the recognition frame and the type of specific object. In Table T3, the percentage of pixel occupancy in the recognition frame is shown, along with a score corresponding to that occupancy rate, expressed as a numerical value from 0 to 2. This score corresponds to a higher occupancy rate. Here, both the percentage of occupancy rate and the score are shown as an example in Table T3, but Table T3 may include either one or the other.

[0086] For example, suppose the following conditions are met: the pixel occupancy rate for the tires is 2.5% or more, the pixel occupancy rate for the frames is 1.5% or more, the pixel occupancy rate for the exterior is 10% or more, and the pixel occupancy rate for clothing is 10% or more. In this case, the type identification unit 15 identifies the type of the identified object as a scooter.

[0087] In this way, the type identification unit 15 identifies the type of the target object using the occupancy rate of the recognition frame of the pixels corresponding to each material. This allows the type identification unit 15 to identify the type of the target object in more detail.

[0088] (Processing by specific device 10a) Next, with reference to Figure 11, the processing performed by the specific device 10a will be explained. Figure 11 is a flowchart showing the processing performed by the specific device 10a. Steps S11 to S16 and S22 are the same as the processing of the specific device 10 explained using Figure 9, so their explanation will be omitted, and the processing from step S31 onwards will be explained.

[0089] First, the type identification unit 15 calculates the occupancy rate of the pixels corresponding to the identified material within the recognition frame (S31). For example, the type identification unit 15 determines the number of pixels for each material included in the recognition frame. The type identification unit 15 calculates the occupancy rate of the pixels corresponding to the identified material within the recognition frame for each material by dividing the number of pixels for each material by the total number of pixels included in the recognition frame.

[0090] First, the type identification unit 15 determines whether the occupancy rate of pixels corresponding to the tires, frame, and exterior is above a threshold (S32). If it determines that the occupancy rate of pixels corresponding to the tires, frame, and exterior is all above the threshold (YES in S32), the type identification unit 15 identifies the type of the identified object as a motorcycle (S33).

[0091] If the type identification unit 15 determines that at least one of the pixel occupancy rates corresponding to the tires, frame, and exterior is below a threshold (NO in S32), it determines whether the pixel occupancy rates corresponding to the tires, frame, and clothing are above a threshold (S34). If the type identification unit 15 determines that the pixel occupancy rates corresponding to the tires, frame, and clothing are all above a threshold (YES in S34), it identifies the type of the identified object as a bicycle (S35). If the type identification unit 15 determines that at least one of the pixel occupancy rates corresponding to the tires, frame, and clothing is below a threshold (NO in S34), it identifies the type of the identified object as a person (S36).

[0092] As described above, the identification system 1a according to this embodiment identifies the type of object based on the proportion of pixels corresponding to the identified material within the recognition frame. This enables the identification system 1a to perform image recognition with greater accuracy.

[0093] <Embodiment 3> Next, Embodiment 3 will be described. In Embodiment 2, the type of the identified object was identified based on the proportion of pixels corresponding to the identified material within the recognition frame. Embodiment 3 differs from Embodiments 1 and 2 in that the type of the identified object is identified based on the arrangement of pixels corresponding to the identified material within the recognition frame. Below, the differences from Embodiments 1 and 2 will be mainly described, and explanations of points that overlap with them will be omitted as appropriate.

[0094] The specified system 1b (not shown) in this embodiment includes a specified device 10b (not shown) instead of the specified device 10 described above. The block diagram showing the configuration of the specified system 1b can be explained by replacing the specified system 1 and specified device 10 shown in Figure 1 with the specified system 1b and specified device 10b, respectively. Therefore, the illustration of the specified system 1b is omitted. The function of the type-specific unit 15 in the specified device 10b differs from that of embodiments 1 and 2.

[0095] In the identification device 10b, the type identification unit 15 identifies the type of the identified object based on the arrangement of pixels corresponding to the identified material in the recognition frame. Sensor fusion processing in the spectral reflectance calculation unit 13 associates pixels included in the far-infrared image with pixels included in the spectral image. Therefore, the type identification unit 15 uses the correspondence between the addresses of pixels included in the far-infrared image and the addresses of pixels included in the spectral image to identify the arrangement of pixels corresponding to the identified material in the recognition frame.

[0096] Figure 12 shows an example of Table T4, which illustrates the relationship between the pixel arrangement and occupancy rate for each material included in the recognition frame and the type of specific object. In Table T4, the pixel arrangement and occupancy rate for each material included in the recognition frame are shown as percentages, along with a score corresponding to the occupancy rate, expressed as a number from 0 to 2. This score is expressed as a number corresponding to a higher occupancy rate. Here, both the percentage and score of occupancy rate are shown as an example in Table T4, but Table T4 may include either one or the other.

[0097] Note that in Table T4, the recognition frame is simply referred to as "region." Also, in the columns for tires, frame, and exterior, the pixel occupancy rate is shown in parentheses. This occupancy rate corresponds to the occupancy rate in Table T3 shown in Figure 10. In the column for clothing, the pixel occupancy rate in the upper or lower half of the recognition frame is used to indicate the arrangement of pixels corresponding to the clothing.

[0098] For example, suppose the following conditions are met: the pixel corresponding to the tire is located in the center of the lower half of the recognition frame, the pixel corresponding to the frame is located in the lower 3 / 4 of the recognition frame, the pixel corresponding to the exterior is located in the lower 3 / 4 of the recognition frame, and the pixels corresponding to the clothing are located in the upper and lower halves of the recognition frame. Furthermore, for the pixels corresponding to the clothing, the occupancy rate in the upper half of the recognition frame is 8% or more, and the occupancy rate in the lower half of the recognition frame is 2.9% or more. In this case, the type identification unit 15 identifies the type of the identified object as a scooter.

[0099] In this way, the type identification unit 15 identifies the type of the target based on the arrangement of pixels in the recognition frame. This allows the type identification unit 15 to identify the type of the target in more detail.

[0100] (Processing by specific device 10b) Next, with reference to Figure 13, the processing performed by the specific device 10b will be explained. Figure 13 is a flowchart showing the processing performed by the specific device 10b. Steps S11 to S16 and S22 are the same as the processing of the specific device 10 explained using Figure 9, so their explanation will be omitted, and the processing from step S41 onwards will be explained.

[0101] First, the type identification unit 15 identifies the arrangement of pixels in the recognition frame corresponding to the identified material (S41). Next, the type identification unit 15 determines whether the arrangement of pixels corresponding to the tires, frame, and exterior corresponds to a motorcycle (S42). If it determines that the arrangement of pixels corresponding to the tires, frame, and exterior corresponds to a motorcycle (YES in S42), the type identification unit 15 identifies the type of the identified object as a motorcycle (S43).

[0102] If the arrangement of pixels corresponding to the tires, frame, and exterior is determined not to correspond to a motorcycle (NO in S42), the type identification unit 15 determines whether the arrangement of pixels corresponding to the tires, frame, and clothing corresponds to a bicycle (S44).

[0103] If the arrangement of pixels corresponding to the tires, frame, and clothing determines that it corresponds to a bicycle (YES in S44), the type identification unit 15 identifies the type of the identified object as a bicycle (S45). If the arrangement of pixels corresponding to the tires, frame, and clothing determines that it does not correspond to a bicycle (NO in S44), the type identification unit 15 identifies the type of the identified object as a person (S46).

[0104] As described above, the identification system 1b according to this embodiment identifies the type of identified object based on the arrangement of pixels corresponding to the identified material within the recognition frame. This enables the identification system 1b to perform image recognition with greater accuracy.

[0105] <Example Hardware Configuration> Each of the functional components of the specified device 10, specified device 10a, and specified device 10b described above may be implemented by hardware that realizes each functional component (e.g., hardwired electronic circuits, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). For example, the present disclosure can also be implemented by having a CPU execute a computer program.

[0106] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in various types of non-transitory computer-readable medium or tangible storage medium. Examples, but not limited to, include RAM (Random-Access Memory), ROM (Read-Only Memory), flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may also be transmitted over various types of transient computer-readable medium or communication medium. Examples, but not limited to, include transient computer-readable medium or communication medium, including electrically, optically, acoustically, or otherwise propagating signals.

[0107] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the embodiments described above can be combined in any way. For example, embodiments 1 to 3 can be combined in any way. [Explanation of Symbols]

[0108] 1, 1a, 1b Specific Systems 10, 10a, 10b specific equipment 11 Image acquisition unit 12 Region detection unit 13 Spectral reflectance calculation section 14. Material Identification Section 15. Specific Category 19 Memory section 20 Far-infrared cameras 30 Hyperspectral cameras 100 Far-infrared images 101a1, 101a2, 102a, 103a frames 101b, 102b, 103b Exterior 101c, 102c, 103c tires 101d, 102d, 103d, 104d Clothes 191 Learning Models A Type display area P1~P4 People R1~R4 Recognition Frame Table showing an example of spectral reflectance for each T1 material. Table showing the relationship between the material corresponding to the T2 pixel and the type of specific object. Table showing the relationship between the pixel occupancy rate for each T3 material and the type of specific object. Table showing the relationship between the pixel arrangement and occupancy rate for each T4 material and the type of specific object. V1 Scooter V2 Large Motorcycle V3 Bicycle

Claims

1. An image acquisition unit that acquires infrared images, A region detection unit that detects an image region containing a specific object from the aforementioned infrared image, A material identification unit identifies the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, The system includes a type identification unit that identifies the type of the specified object based on the material corresponding to the pixel. Specific equipment.

2. The type identification unit identifies the type of the identified object based on the proportion of pixels in the image area corresponding to the identified material. The specific device according to claim 1.

3. The type identification unit identifies the type of the identified object based on the arrangement of pixels corresponding to the identified material in the image area. The specific device according to claim 1 or 2.

4. Image acquisition step to acquire infrared image, A region detection step for detecting an image region containing a specific object from the aforementioned infrared image, A material identification step to identify the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, A type identification step includes identifying the type of the specified object based on the material corresponding to the pixel. Specific method.

5. Image acquisition step to acquire infrared image, A region detection step for detecting an image region containing a specific object from the aforementioned infrared image, A material identification step to identify the material corresponding to the pixel based on the spectral reflectance corresponding to the pixel of the specified target included in the image region, The computer is instructed to perform a type identification step, which involves identifying the type of the specified object based on the material corresponding to the pixel. program.

Images