Information processing apparatus, information processing system, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing system, an information processing method, and an information processing program capable of improving detection accuracy of an object from a visible image including an infrared light component by simple processing.SOLUTION: An information processing device 10 includes an acquisition unit 12 that acquires a visible image obtained by imaging an illumination region including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit by an imaging element including both a captured pixel of infrared light and a captured pixel of visible light, and a processing unit 14 that generates at least one of information on presence or absence of a target in the illumination region or information on a distance to a target present in the illumination region on the basis of information on a high-intensity pixel having a predetermined intensity or more among pixels included in the visible image.SELECTED DRAWING: Figure 5A

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, an image processing method is known that suppresses the occurrence of false colors by performing saturation processing when visible light pixels become saturated pixels in a sensor that includes visible light pixels and near-infrared light pixels (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-145167 Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting an object from a visible image containing pixels that detect a combination of visible light and infrared light, removing the infrared light component does not necessarily contribute to improving the object detection accuracy. Furthermore, the processing load for removing the infrared light component increases. Therefore, it is desirable to improve the object detection accuracy from a visible image containing infrared light components using simple processing.

[0005] An object of the present disclosure is to provide an information processing device, an information processing system, an information processing method, and an information processing program that can improve the accuracy of detecting an object from a visible image that includes an infrared light component through simple processing. [Means for solving the problem]

[0006] An information processing device according to an embodiment of the present disclosure includes an acquisition unit and a processing unit. The acquisition unit acquires a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, captured by an image sensor having both infrared light imaging pixels and visible light imaging pixels. The processing unit generates at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image.

[0007] An information processing system according to an embodiment of the present disclosure includes the information processing device and an imaging device having the imaging element.

[0008] An information processing method according to an embodiment of the present disclosure includes an information processing device acquiring a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an image sensor having both infrared light imaging pixels and visible light imaging pixels. The information processing method includes the information processing device generating at least one of information regarding the presence or absence of an object in the illumination area or information regarding a distance to an object present in the illumination area, based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image.

[0009] An information processing program according to an embodiment of the present disclosure causes an information processing device to acquire a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, by an image sensor having both infrared light imaging pixels and visible light imaging pixels. The information processing program causes the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image. [Effects of the Invention]

[0010] According to an information processing device, an information processing system, an information processing method, and an information processing program according to an embodiment of the present disclosure, the accuracy of detecting an object from a visible image containing an infrared light component can be improved through simple processing. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a schematic configuration example of a detection system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating generation of an infrared image and a visible image from a captured image. [Figure 3] 10 is a graph showing an example of sensitivity characteristics of an imaging device. [Figure 4A] 10 is a graph showing an example of the detected intensity of infrared light and each of RGB colors. [Figure 4B] 10 is a graph showing an example of the detection intensity of infrared light and each of the RGB colors when the detection intensity of each of the RGB colors exceeds the upper detection limit. [Figure 5A] FIG. 2 is a diagram illustrating an example of an illumination area in front of a vehicle. [Figure 5B] 5B is a diagram showing an example of a visible image of an object present in the illumination area of ​​FIG. 5A. FIG. [Figure 6A] FIG. 5B is a diagram showing a case where the object is closer to the vehicle than in FIG. 5A. [Figure 6B] 6B is a diagram showing an example of a visible image of an object present in the illumination area of ​​FIG. 6A. FIG. [Figure 7A] FIG. 6B shows a case where the object is closer to the vehicle than in FIG. 6A. [Figure 7B] 7B is a diagram showing an example of a visible image of an object present in the illumination area of ​​FIG. 7A. FIG. [Figure 8] FIG. 1 is a block diagram showing an example of a learning flow for generating a recognition model. [Figure 9] 10 is an example of a table that associates a visible image of an object with a correct value of the distance to the object and a degree of danger. [Figure 10]FIG. 10 is a block diagram showing an example of a processing flow for estimating information about the distance to an object using a recognition model. [Figure 11] 10 is a flowchart illustrating an example of the procedure of an information processing method for detecting an object present in an illumination area based on high-intensity pixels of a visible image. [Figure 12] 10 is a flowchart illustrating an example of a procedure of an information processing method including a procedure of calculating a distance to an object based on high-intensity pixels of a visible image. DETAILED DESCRIPTION OF THE INVENTION

[0012] An image sensor that captures visible light is also sensitive to infrared light, but by using a filter that attenuates infrared light, it is possible to capture an image of an object using visible light so that the infrared light component is not detected. On the other hand, an image sensor that has both pixels that detect visible light and pixels that detect infrared light cannot use a filter that attenuates infrared light to capture infrared light, and the pixels that detect visible light detect both visible light and infrared light components. As a result, an image captured by the pixels that detect visible light is affected by the infrared light component.

[0013] When a pixel that detects visible light detects both a visible light component and an infrared light component, the detection intensity of the pixel that detects visible light may become saturated. In the present disclosure, the pixel whose detection intensity is saturated is used to generate information about the distance to an object captured in an image by the pixel that detects visible light.

[0014] (Configuration example of information processing system 1) As shown in FIG. 1, an information processing system 1 according to an embodiment of the present disclosure includes an information processing device 10, an imaging device 20, and an illumination device 30. In the present disclosure, the information processing system 1 is assumed to be mounted on a vehicle 70 (see FIG. 5A, etc.). The information processing system 1 may also be mounted on other moving bodies. The information processing system 1 may also be mounted on various devices, not limited to moving bodies.

[0015] <Information processing device 10> The information processing device 10 includes an acquisition unit 12, a processing unit 14, and an output unit 16.

[0016] The acquisition unit 12 acquires image data from the photographing device 20. The acquisition unit 12 may acquire various other data or information. The acquisition unit 12 may include a communication interface for wired or wireless communication with the photographing device 20 or other devices. The communication interface may be configured to be capable of communication using a communication method based on various communication standards. The communication interface may be configured based on known communication technology.

[0017] The acquisition unit 12 may include an input device that accepts input from a user. The input device may include, for example, a keyboard or physical keys, or a pointing device such as a touch panel, a touch sensor, or a mouse. The input device is not limited to these examples and may include various other devices. The acquisition unit 12 may be configured to be able to communicate with an external input device.

[0018] The processing unit 14 processes the image data. The processing unit 14 may include at least one processor to provide control and processing power for performing various functions. Each function of the processing unit 14 may be implemented by one processor or several processors. The combined functions of the processing unit 14 may be implemented by one processor. The processor may be implemented as a single integrated circuit (IC). The processor may be implemented as multiple communicatively connected integrated circuits or discrete circuits. The processor may be implemented based on various other known technologies.

[0019] The processor may include a general-purpose processor that loads a specific program to execute a specific function, or a dedicated processor specialized for a specific process. The general-purpose processor may include, for example, a central processing unit (CPU) or a digital signal processor (DSP). The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The processing unit 14 may include either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0020] The information processing device 10 may include a memory unit 18. The memory unit 18 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The memory unit 18 stores various information. The memory unit 18 stores programs to be executed by a processor or the like that functions as the processing unit 14. The memory unit 18 may be configured as a non-transitory readable medium. The memory unit 18 may function as a work memory for the processing unit 14. At least a part of the memory unit 18 may be configured integrally with the processing unit 14.

[0021] The output unit 16 outputs the processing result of the image data by the processing unit 14. The output unit 16 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or an inorganic EL display.

[0022] The output unit 16 may include an audio output device such as a speaker, etc. The output unit 16 is not limited to these examples and may include various other devices.

[0023] A part of the information processing device 10 may be installed externally rather than mounted on the vehicle 70 (see FIG. 5A, etc.). The part of the information processing device 10 that is installed externally may be communicably connected to the part that is mounted on the vehicle 70, etc. For example, at least a part of the functions of the processing unit 14 of the information processing device 10 may be realized by a cloud service.

[0024] <Photographing device 20> The image capturing device 20 includes an image sensor including both pixels that capture infrared light and pixels that capture visible light. The image capturing device 20 generates a captured image including both pixels that represent the detected intensity of infrared light and pixels that represent the detected intensity of visible light. The pixels that capture infrared light detect light in a wavelength range of, for example, 780 nm to 1000 nm. The infrared light may be light in the wavelength range of near-infrared light. The pixels that capture visible light detect light in a wavelength range of, for example, 380 nm to 780 nm. The wavelength ranges of infrared light and visible light are not limited to the values ​​described above and may be changed as appropriate when implementing the invention.

[0025] As illustrated in FIG. 2, the captured image includes pixels that capture infrared light represented by I and pixels that capture visible light in the three colors of red, green, and blue represented by R, G, and B. The image capturing device 20 performs demosaicing on the captured image. Demosaicing is a process that separates pixels that are mixed in the captured image to generate multiple images. By performing demosaicing on the captured image, the image capturing device 20 generates an infrared image made up of pixels that capture infrared light represented by I and a visible image made up of pixels that capture visible light in the three colors of red, green, and blue represented by R, G, and B.

[0026] Here, as illustrated in FIG. 3, the wavelength characteristics of sensitivity in a pixel capturing infrared light have a peak in the infrared band. On the other hand, the wavelength characteristics of sensitivity in a pixel capturing each of R, G, and B visible light have peaks in both the R, G, and B wavelengths included in the visible band and the infrared band. In other words, a pixel capturing visible light detects not only visible light components but also infrared light components. Therefore, the detection intensity corresponding to each pixel of the visible image is the detection intensity of a combined component of visible light components and infrared light components.

[0027] The imaging element may be, for example, a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS) sensor.

[0028] In the present disclosure, the image capturing device 20 is configured as an on-board camera of a vehicle 70 (see FIG. 5A, etc.). The image capturing device 20 is not limited to an on-board camera. The number of image capturing devices 20 is not limited to one, but may be two or more.

[0029] <Lighting device 30> The lighting device 30 includes an infrared light irradiating unit that irradiates infrared light and a visible light irradiating unit that irradiates visible light. An area illuminated by at least one of the infrared light and the visible light irradiated from the lighting device 30 is also referred to as an illumination area. A portion of the illumination area illuminated by the infrared light irradiated from the infrared light irradiating unit is also referred to as an infrared light illumination area. A portion of the illumination area illuminated by the visible light irradiated from the visible light irradiating unit is also referred to as a visible light illumination area.

[0030] The illumination device 30 is disposed so that a portion of the infrared light illumination region and a portion of the visible light illumination region overlap each other. In other words, the illumination region includes a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit. The portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit includes a portion illuminated by the infrared light irradiation unit but not the visible light irradiation unit, and a portion illuminated by the visible light irradiation unit but not the infrared light irradiation unit.

[0031] In the present disclosure, the lighting device 30 is configured as a headlight of a vehicle 70 (see FIG. 5A, etc.). The lighting device 30 is not limited to a headlight. The number of lighting devices 30 is not limited to one, but may be two or more. When the lighting device 30 is a headlight of the vehicle 70, the lighting device 30 may be installed at two locations on the left and right sides of the front of the vehicle 70.

[0032] (Example of operation of information processing system 1) The acquisition unit 12 of the information processing device 10 acquires an infrared image and a visible image from the imaging device 20. As described above, each pixel of the visible image represents the detection intensity of a component that is a combination of a visible light component and an infrared light component. As illustrated in Fig. 4A, the detection intensity at a pixel corresponding to each of the colors red, green, and blue represented by R, G, and B in the visible image is a value obtained by adding the detection intensity of the infrared light component represented by IR to the detection intensity of the red, green, and blue components when an object is imaged using only visible light.

[0033] In the information processing system 1 according to the present disclosure, the photographing device 20 photographs an illumination area with an image sensor to generate a visible image. The illumination area includes a portion illuminated by both the infrared light irradiating unit and the visible light irradiating unit of the illumination device 30, and a portion illuminated by only either the infrared light irradiating unit or the visible light irradiating unit. Therefore, in the visible image, pixels photographed in a portion illuminated by both the infrared light irradiating unit and the visible light irradiating unit correspond to pixels that detect both the infrared light component and the visible light component. Furthermore, in the visible image, pixels photographed in a portion illuminated by only either the infrared light irradiating unit or the visible light irradiating unit correspond to pixels that detect only either the infrared light component or the visible light component.

[0034] Here, it is assumed that the object has uniform reflectance for infrared light and visible light. When the object is located in an illumination area across a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit and a portion illuminated by only either the infrared light irradiation unit or the visible light irradiation unit, in a visible image of the object, the detection intensity of pixels capturing the portion of the object illuminated by both the infrared light irradiation unit and the visible light irradiation unit is higher than the detection intensity of pixels capturing the portion of the object illuminated by only either the infrared light irradiation unit or the visible light irradiation unit. Based on the detection intensity of each pixel in the visible image, the processing unit 14 of the information processing device 10 can classify pixels capturing the portion of the object illuminated by both the infrared light irradiation unit and the visible light irradiation unit into pixels capturing the portion of the object illuminated by only either the infrared light irradiation unit or the visible light irradiation unit, and pixels capturing the portion of the object illuminated by only either the infrared light irradiation unit or the visible light irradiation unit.

[0035] For example, the processing unit 14 may set a predetermined intensity as a threshold for the detection intensity, and classify pixels of the visible image whose detection intensity is equal to or greater than the predetermined intensity as pixels representing a portion of the object illuminated by both the infrared light irradiator and the visible light irradiator. The processing unit 14 may set the predetermined intensity based on the intensities of the infrared light and visible light irradiated by the lighting device 30 and the detection sensitivity of the image capture device 20. Pixels of the visible image whose detection intensity is equal to or greater than the predetermined intensity are also referred to as high-intensity pixels.

[0036] When a pixel capturing an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit is adjacent to a pixel capturing an image of a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, the detection intensity between the adjacent pixels in the visible image changes rapidly. Specifically, the difference in detection intensity between the adjacent pixels may be the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The processing unit 14 may detect a portion in the visible image where the absolute value of the difference in detection intensity between adjacent pixels is equal to or greater than a difference threshold as a portion where the detection intensity changes rapidly. The difference threshold may be set based on the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The difference threshold may be set to the smaller value of the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The difference threshold may be set to the intensity value of the infrared light irradiated by the infrared light irradiating unit itself, or may be set to a value obtained by multiplying the intensity value of the infrared light irradiated by the infrared light irradiating unit by a positive coefficient less than 1. The difference threshold may be set to the intensity value of the visible light irradiated by the visible light irradiating unit itself, or may be set to a value obtained by multiplying the intensity value of the visible light irradiated by the visible light irradiating unit by a positive coefficient less than 1.

[0037] In a portion where the detection intensity changes suddenly, a pixel with a higher detection intensity corresponds to a pixel capturing an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit. The processing unit 14 may set a value between the higher and lower detection intensities of adjacent pixels across the portion where the detection intensity changes suddenly as a predetermined intensity, thereby classifying pixels whose detection intensity is equal to or greater than a predetermined intensity, i.e., high-intensity pixels, as pixels capturing an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit. The processing unit 14 may classify pixels whose detection intensity increases suddenly as pixels capturing an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit.

[0038] The intensity of light incident from an area illuminated by both the infrared light irradiation unit and the visible light irradiation unit may exceed the upper detection limit, as illustrated in FIG. 4B. In this case, the detection intensity of pixels where the light intensity exceeds the upper detection limit saturates at the upper detection limit. In a visible image, pixels where the detection intensity of each RGB color is saturated are displayed in white. Furthermore, pixels where the detection intensity of each RGB color is close to the upper detection limit are displayed in gray close to white. For example, when the imaging device 20 captures a visible image of a person in the illumination area, the color of pixels that represent the person's face or clothing in an area illuminated by both the infrared light irradiation unit and the visible light irradiation unit appears almost white, regardless of their actual color. Pixels in a visible image where the detection intensity saturates at the upper detection limit and appears white, or pixels where the detection intensity approaches the upper detection limit and appears almost white, are also referred to as overexposed pixels. Overexposed pixels may correspond to pixels captured in an area illuminated by both the infrared light irradiation unit and the visible light irradiation unit, i.e., high-intensity pixels.

[0039] For example, as shown in a side view of a vehicle 70 in FIG. 5A , when the vehicle 70 is equipped with the lighting device 30 as a headlight that illuminates the road ahead, an illumination area is generated in front of the vehicle 70 by emitting infrared light from the infrared light emitting unit of the lighting device 30 and visible light from the visible light emitting unit. Of the illumination area, the portion illuminated by infrared light emitting from the infrared light emitting unit, i.e., the infrared light illumination area, is the area represented by R_IR between the two dashed lines. The portion illuminated by visible light emitting from the visible light emitting unit, i.e., the visible light illumination area, is the area represented by R_VL between the two dashed lines. Furthermore, the portion illuminated by both infrared light emitting from the infrared light emitting unit and visible light emitting from the visible light emitting unit, i.e., the overlapping portion of the infrared light illumination area and the visible light illumination area, is represented by R_OL.

[0040] A pedestrian 80 as an object is present in the illumination area shown in FIG. 5A. The pedestrian 80 present in the illumination area is dazzled when illuminated with visible light from the lighting device 30 serving as the headlights of the vehicle 70. Therefore, the visible light irradiating unit of the lighting device 30 is configured to irradiate visible light toward the ground in a so-called low beam direction so that the head of the pedestrian 80 present in the illumination area does not enter the visible light illumination area (R_VL). The low beam of the headlights of the vehicle 70 can also be referred to as a passing headlight. On the other hand, in order for the vehicle 70 to travel safely, it is necessary for the pedestrian 80 to be photographed by the imaging device 20. The pedestrian 80 is not dazzled when illuminated with infrared light. Therefore, the infrared light irradiating unit of the lighting device 30 is configured to irradiate infrared light toward the air in a so-called high beam direction so that most of the pedestrian 80 present in the illumination area falls within the infrared light illumination area (R_IR). The high beams in the headlights of the vehicle 70 can also be referred to as driving headlights.

[0041] In the present disclosure, the infrared light irradiating unit of the lighting device 30 may be configured to irradiate infrared light toward the ground in a so-called low beam direction so that the heads of pedestrians 80 present in the lighting area do not enter the infrared light illumination area (R_IR). The visible light irradiating unit of the lighting device 30 may be configured to irradiate infrared light toward the air in a so-called high beam direction so that most of the pedestrians 80 present in the lighting area enter the visible light illumination area (R_VL).

[0042] In the present disclosure, the infrared light irradiation unit and the visible light irradiation unit of the lighting device 30 may be configured to irradiate infrared light and visible light such that at least a portion of the irradiation ranges of the infrared light and the visible light do not overlap, and such that at least a portion of the irradiation ranges of the infrared light and the visible light overlap.

[0043] 5A, the boundary between the infrared light illumination region (R_IR) and the visible light illumination region (R_VL) is represented by a dashed line. The boundary of R_IR may be determined based on the central axis of the irradiation direction of infrared light from the infrared light irradiator and the spread angle from the central axis. The boundary of R_VL may be determined based on the central axis of the irradiation direction of visible light from the visible light irradiator and the spread angle from the central axis.

[0044] The boundary line of R_IR may be a line connecting positions where the intensity of infrared light decreases to a value obtained by multiplying the maximum intensity of infrared light in R_IR by an infrared light boundary threshold. Furthermore, the boundary line of R_VL may be a line connecting positions where the intensity of visible light decreases to a value obtained by multiplying the maximum intensity of visible light in R_VL by a visible light boundary threshold. The infrared light boundary threshold and the visible light boundary threshold may be set to different values ​​or may be set to the same value. The infrared light boundary threshold and the visible light boundary threshold may be set so that the difference between the detected intensity of a pixel capturing an overlapping portion (R_OL) between the infrared light illumination area and the visible light illumination area and the detected intensity of a pixel capturing R_IR or R_VL adjacent to the outside of R_OL is equal to or greater than a difference threshold. Furthermore, the boundary line of R_IR may be a line defining a predetermined first solid angle range relative to the illumination direction of infrared light. The boundary line of R_VL may be a line defining a predetermined second solid angle range relative to the illumination direction of visible light. The first solid angle range and the second solid angle range may be the same or different, and either the first solid angle range or the second solid angle range may be larger.

[0045] In FIG. 5A, the feet of the pedestrian 80 are located within R_OL. On the other hand, the part above the feet of the pedestrian 80 is located within R_IR but outside R_OL. Here, FIG. 5B shows a visible image of the pedestrian 80 in FIG. 5A captured by the imaging device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, pixels capturing the feet of the pedestrian 80 capture a part located within R_OL, which is a part illuminated by both the infrared light emitted from the infrared light irradiation unit and the visible light emitted from the visible light irradiation unit. Therefore, the pixels capturing the feet of the pedestrian 80 are high-intensity pixels 82. On the other hand, pixels capturing the part above the feet of the pedestrian 80 capture a part located outside R_OL, and are not high-intensity pixels 82.

[0046] By detecting that the visible image in FIG. 5B contains high-intensity pixels 82, the processing unit 14 can detect the presence of an object within the portion of the illumination area represented by R_OL. In FIG. 5A, the portion represented by R_OL is a finite range from the illumination device 30 to the point where the portion intersects with the ground and disappears. Therefore, the processing unit 14 can detect the presence of an object within the finite range corresponding to R_OL. The distance from the illumination device 30 to the point where the portion represented by R_OL intersects with the ground and disappears is determined based on the position of the illumination device 30, the direction in which infrared light is emitted from the infrared light emitter of the illumination device 30, and the direction in which visible light is emitted from the visible light emitter, i.e., a predetermined distance. By detecting that the visible image in FIG. 5B contains high-intensity pixels 82, the processing unit 14 can detect the presence of an object within a predetermined distance from the front end of the vehicle 70. In other words, the processing unit 14 can generate information indicating the presence of an object within a predetermined distance as information regarding the distance to the object.

[0047] As illustrated in Fig. 6A, when the distance from vehicle 70 to pedestrian 80 is shorter than the distance to pedestrian 80 in Fig. 5A, the part of pedestrian 80 around the knees is located within R_OL. On the other hand, the part of pedestrian 80 above the knees is located within R_IR but outside R_OL. Furthermore, the part of pedestrian 80 below the knees is located within R_VL but outside R_OL.

[0048] Here, FIG. 6B shows a visible image of the pedestrian 80 in FIG. 6A captured by the image capture device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, pixels capturing the area around the knees of the pedestrian 80 capture an area located within the R_OL, which is an area illuminated by both the infrared light emitted from the infrared light emitter and the visible light emitted from the visible light emitter. Therefore, the pixels capturing the area around the knees of the pedestrian 80 are high-intensity pixels 82. On the other hand, pixels capturing the area above or below the knees of the pedestrian 80 capture an area located outside the R_OL, and are not high-intensity pixels 82.

[0049] 7A, when the distance from vehicle 70 to pedestrian 80 is shorter than the distance to pedestrian 80 in FIG. 6A, the part of pedestrian 80 around the waist is located within R_OL. On the other hand, the part of pedestrian 80 above the waist is located within R_IR but outside R_OL. Furthermore, the part of pedestrian 80 below the waist is located within R_VL but outside R_OL.

[0050] Here, FIG. 7B shows a visible image of the pedestrian 80 in FIG. 7A captured by the image capture device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, pixels capturing the area around the waist of the pedestrian 80 capture a portion located within the R_OL, which is a portion illuminated by both the infrared light emitted from the infrared light emitter and the visible light emitted from the visible light emitter. Therefore, the pixels capturing the area around the waist of the pedestrian 80 are high-intensity pixels 82. On the other hand, pixels capturing the area above or below the waist of the pedestrian 80 capture an area located outside the R_OL, and are not high-intensity pixels 82.

[0051] 5B, 6B, and 7B, the R_OL range approaches the ground as it moves away from the lighting device 30, and therefore the height of the high-intensity pixels 82 from the ground changes depending on the distance from the front end of the vehicle 70, i.e., the lighting device 30, to the pedestrian 80. When the relationship between the height of the high-intensity pixels 82 from the ground in a visible image capturing the pedestrian 80 and the distance to the pedestrian 80 is known, the processing unit 14 can calculate the distance to the pedestrian 80 by detecting the height of the high-intensity pixels 82 from the ground in the visible image and applying the detected height to the known relationship. In other words, the processing unit 14 can calculate the distance to an object based on the height of the high-intensity pixels 82 from the ground in a visible image capturing the object.

[0052] The illumination device 30 is configured so that the overlapping portion (R_OL) of the infrared light illumination region (R_IR) and the visible light illumination region (R_VL) disappears within a finite range. The illumination device 30 may be configured so that the R_OL disappears when it intersects with the ground as shown in FIG. 5A . Alternatively, the illumination device 30 may be configured so that the R_OL disappears in midair without intersecting with the ground. In this case, in a side view of the illumination region, the boundary line of the R_IR that overlaps with the R_VL and the boundary line of the R_VL that overlaps with the R_IR intersect at a position a finite distance from the illumination device 30, causing the R_OL to disappear in midair.

[0053] When the illumination device 30 is configured so that the R_OL disappears in the air, the greater the distance to an object located within the R_OL, the smaller the area of ​​high-intensity pixels in the visible image of the object. If the relationship between the area of ​​high-intensity pixels in the visible image of the object and the distance to the object is known, the processing unit 14 can calculate the distance to the object by calculating the area of ​​the high-intensity pixels from the visible image and applying the known relationship. In other words, the processing unit 14 can calculate the distance to the object based on the area of ​​high-intensity pixels in the visible image of the object.

[0054] The processing unit 14 of the present disclosure may calculate the distance to the object based on the distance of the high-intensity pixel 82 in the visible image of the object from a predetermined reference. In the above description, the predetermined reference is the ground. However, the predetermined reference may be, for example, the height of the image capture device 20 or the lighting device 30 from the ground.

[0055] <Recognition model> The processing unit 14 may use a recognition model to recognize objects appearing in the infrared image and the visible image. The recognition model may be configured to output a result of recognizing an object appearing in the infrared image and the visible image when the infrared image and the visible image are input. The recognition model may also be configured to output a result of recognizing the presence of an object in the overlapping portion (R_OL) of the infrared light illumination region and the visible light illumination region when a visible image including high-intensity pixels of the object is input.

[0056] The recognition model may be configured to calculate the distance to a specific object. The specific object for which the distance is to be calculated is also referred to as a recognized object. The recognized object may include, for example, a pedestrian 80 located ahead of the vehicle 70 in the direction of travel. The recognition model may be configured to output a calculated distance from the vehicle 70 or the lighting device 30 mounted on the vehicle 70 to a recognized object present on the R_OL when a visible image including high-intensity pixels of the recognized object is input. The recognition model may be configured to output a hazard level of the recognized object present on the R_OL to the vehicle 70 when a visible image including high-intensity pixels of the recognized object is input. The hazard level of the recognized object represents the likelihood of a dangerous situation occurring, such as the recognized object approaching or colliding with the vehicle 70. In the present disclosure, the recognized object may be, in addition to a pedestrian 80, a standing (stationary) person, a jogger, a wheelchair, a bicycle, a motorcycle, a car, an animal such as a dog or a cat, a rock, or any other suitable object.

[0057] The recognition model may be configured to recognize the presence of a recognition target in the R_OL when it is determined that a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to output a distance to the recognition target in the R_OL or a level of danger of the target when it is determined that a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to determine whether an object captured in the visible image is a recognition target based on both the infrared image and the visible image, and to determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to determine whether an object captured in the visible image is a recognition target based only on the visible image, and to determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target.

[0058] The recognition model may include an input unit and an output unit. The input unit is a unit for inputting infrared images and visible images. The output unit is a unit for outputting the recognition result of the object.

[0059] The recognition model may be a rule-based model that specifies rules for extracting features of objects appearing in infrared and visible images. The recognition model may be a trained model generated by performing machine learning using infrared and visible images as training data. The machine learning method may include, for example, a method using Haar-like features or Hog features, or their derivative methods such as joint Haar-like, joint HoG, or Shapelet.

[0060] The recognition model may be generated by deep learning, which may be, for example, a convolutional neural network (CNN) model such as Renet or DenseNet, or a transformer model such as ViT.

[0061] When the recognition model is configured to output a result of recognizing the presence of an object in R_OL, it may be generated by performing learning using training data including visible images containing high-intensity pixels of the object.When the recognition model is configured to output a result of calculating the distance to the recognition object, it may be generated by performing learning using training data that associates visible images containing high-intensity pixels of the recognition object with correct values ​​of the distance to the recognition object.

[0062] As described above, the processing unit 14 generates information about the distance to the target object based on the high-intensity pixels in the visible image. The recognition model may be configured to output a result of calculating the distance to the target object based on the position or area of ​​the high-intensity pixels. In this case, training data for generating the recognition model may be teacher data in which the distance to the target object in the visible image is used as ground truth data and associated with the visible image.

[0063] The processing unit 14 may generate the recognition model by performing learning within the processing unit 14. The processing unit 14 may also acquire the recognition model from an external device.

[0064] The relationship between each image and the learning of the recognition model in the generation of the above-mentioned recognition model will be explained with reference to the block diagram shown in Fig. 8. The correct answer may be associated with the visible image by measuring the distance to the recognition target using another means such as a distance measuring device. The recognition model is generated by performing learning using an infrared image, a visible image, and data in which the correct answer is associated with the visible image as learning data.

[0065] The correspondence between the visible image and the correct answer data may be represented by a table as shown in Fig. 9. In the table of Fig. 9, a correct answer value of the distance from the vehicle 70 to the pedestrian 80 is associated with a visible image that captures the pedestrian 80 as a recognition object. The recognition object may be an object that satisfies the condition that the correct answer value of the distance from the vehicle 70 or the lighting device 30 to the recognition object is associated with a visible image that captures the recognition object.

[0066] The second row of the table shows the correspondence between a visible image when the pedestrian 80 is not in the overlapping area (R_OL) between the infrared light illumination area and the visible light illumination area, i.e., a visible image that does not include the high-intensity pixels 82 of the pedestrian 80, and the correct value of the distance from the vehicle 70 to the pedestrian 80. In this example, the range of R_OL intersects with the ground and disappears at a position 40 m away from the vehicle 70. Therefore, when the pedestrian 80 is not in R_OL, the correct value of the distance from the vehicle 70 to the pedestrian 80 is 40 m or more.

[0067] The third to fifth rows of the table represent correspondence between a visible image when a pedestrian 80 is present in the R_OL, i.e., a visible image including high-intensity pixels 82 of the pedestrian 80, and the correct value of the distance from the vehicle 70 to the pedestrian 80. The third row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 5A and 5B. The fourth row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 6A and 6B. The fifth row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 7A and 7B. The correct value of the distance from the vehicle 70 to the pedestrian 80 is 35 m in the third row corresponding to FIG. 5A, 25 m in the fourth row corresponding to FIG. 6A, and 15 m in the fifth row corresponding to FIG. 7A.

[0068] 9, a visible image showing a pedestrian 80, which is a recognition object, is associated with a risk level indicating the likelihood that the pedestrian 80 will approach or collide with the vehicle 70. In the second row of the table, a risk level of "none" is associated with a visible image that does not include high-intensity pixels 82 of the pedestrian 80. In the third to fifth rows of the table, a risk level of "low," "medium," or "high" is associated with a visible image that includes high-intensity pixels 82 of the pedestrian 80, depending on the distance to the pedestrian 80. The recognition object may be an object that satisfies the condition that the correct value of the risk level of the recognition object vis-à-vis the vehicle 70 is associated with a visible image showing the recognition object.

[0069] The expression of the risk level is not limited to the expressions exemplified in the table of Fig. 9. For example, in the second row of the table, a visible image that does not include high-intensity pixels 82 of a pedestrian 80 may be associated with a risk level of "very low," which indicates that the risk level is not zero but is very low, instead of "none." Furthermore, the risk level may be a numerical value that indicates the probability that the pedestrian 80 will approach or collide with the vehicle 70.

[0070] The processing unit 14 recognizes an object from an infrared image and a visible image using the recognition model generated as described above. The recognition process of the processing unit 14 will be described with reference to the block diagram shown in Fig. 10. The processing unit 14 acquires an infrared image and a visible image using the acquisition unit 12. The processing unit 14 inputs the infrared image and the visible image into the recognition model and acquires the recognition result of the object output from the recognition model.

[0071] The processing unit 14 may generate danger information related to the object based on the recognition result of the object. The danger information related to the object may include information on the presence of the object, information on the distance to the recognized object, or the danger level of the recognized object. The processing unit 14 may output the recognition result of the object or the danger information related to the object via the output unit 16 and notify the outside. If the information processing device 10 is installed in a vehicle 70, the recipients of the danger information may include the driver of the vehicle 70, etc.

[0072] <Example of information processing procedure> The processing unit 14 of the information processing device 10 may recognize the presence of an object based on a visible image including high-intensity pixels by executing an information processing method including the steps of the flowchart illustrated in Fig. 11. The information processing method may be realized as an information processing program executed by a processor included in the information processing device 10. The information processing program may be stored in a non-transitory computer-readable medium.

[0073] The processing unit 14 acquires an infrared image and a visible image from the imaging device 20 using the acquisition unit 12 (step S1). The processing unit 14 determines whether there are high-intensity pixels in the visible image (step S2). If there are no high-intensity pixels in the visible image (step S2: NO), the processing unit 14 ends the execution of the flowchart in FIG. 11.

[0074] If there is a high-intensity pixel in the visible image (step S2: YES), the processing unit 14 recognizes the presence of an object in the overlapping portion (R_OL) of the infrared light illumination area and the visible light illumination area, generates danger information about the object, and notifies the outside via the output unit 16 (step S3). After executing the procedure of step S3, the processing unit 14 ends the execution of the flowchart in FIG.

[0075] In the determination procedure of step S2, the processing unit 14 may determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a part of a recognition target object such as a pedestrian 80 or the like.

[0076] The processing unit 14 of the information processing device 10 may calculate the distance to the recognition target object based on a visible image including high-intensity pixels by executing an information processing method including the steps of the flowchart illustrated in Fig. 12. The information processing method may be realized as an information processing program executed by a processor included in the information processing device 10. The information processing program may be stored in a non-transitory computer-readable medium.

[0077] The processing unit 14 acquires an infrared image and a visible image from the imaging device 20 using the acquisition unit 12 (step S11). The processing unit 14 determines whether there are high-intensity pixels in the visible image (step S12). If there are no high-intensity pixels in the visible image (step S12: NO), the processing unit 14 ends the execution of the flowchart in FIG. 12.

[0078] If there is a high-intensity pixel in the visible image (step S12: YES), the processing unit 14 determines whether the high-intensity pixel is a pixel that captures at least a part of the recognition object, such as a pedestrian 80 (step S13). If the high-intensity pixel is a pixel that captures at least a part of the recognition object (step S13: YES), the processing unit 14 calculates the distance to the recognition object (step S14). If the high-intensity pixel is not a pixel that captures at least a part of the recognition object (step S13: NO), the processing unit 14 proceeds to step S15.

[0079] The processing unit 14 generates danger information and notifies the outside via the output unit 16 (step S15). If the processing unit 14 determines in the procedure of step S13 that the high-intensity pixel is a pixel that captures at least a part of the recognition target object, it may generate danger information about the recognition target object. If the processing unit 14 determines in the procedure of step S13 that the high-intensity pixel is not a pixel that captures at least a part of the recognition target object, it may generate danger information about an object that is not the recognition target object. After executing the procedure of step S15, the processing unit 14 ends execution of the flowchart of FIG. 12.

[0080] (summary) As described above, the information processing system 1, information processing device 10, and information processing method according to the present disclosure generate a visible image capturing an illumination area including an overlapping portion between an infrared light illumination area and a visible light illumination area. Then, without removing the infrared light component from the visible image, the presence of an object in the overlapping portion between the infrared light illumination area and the visible light illumination area is recognized based on information about pixels that detect both the infrared light component and the visible light component. In other words, it is determined whether an object present in the illumination area is located in the overlapping portion between the infrared light illumination area and the visible light illumination area.

[0081] In a comparative example, when both infrared light and visible light are irradiated onto the entire illumination area, it is difficult to identify the position of an object within the illumination area without removing the infrared light component from the visible image. In contrast to the comparative example, according to the present disclosure, it is easy to identify whether an object present within the illumination area is located in the overlapping portion of the infrared light illumination area and the visible light illumination area.

[0082] Furthermore, according to the present disclosure, the distance from the illumination device 30 to the object is calculated based on the position or area of ​​high-intensity pixels included in the visible image. On the other hand, in the above-mentioned comparative example, the visible image does not include high-intensity pixels that change depending on the distance to the object, so the distance cannot be calculated based on the high-intensity pixels. Therefore, according to the present disclosure, the position of the object can be easily identified.

[0083] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0084] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to be logically inconsistent, and multiple components can be combined into one or divided. It should be understood that these modifications are also included in the scope of the present disclosure.

[0085] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0086] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0087] The vehicle 70 according to the present disclosure may include, for example, automobiles, industrial vehicles, rail vehicles, utility vehicles, or fixed-wing aircraft traveling on runways. The automobile may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. The industrial vehicle may include, for example, agricultural or construction industrial vehicles. The industrial vehicle may include, for example, forklifts or golf carts. The agricultural industrial vehicle may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. The construction industrial vehicle may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, or road rollers. The vehicle 70 may include a human-powered vehicle. The classification of the vehicle 70 is not limited to the above examples. For example, the automobile may include an industrial vehicle capable of traveling on roads. The same type of vehicle 70 may be included in multiple classifications.

[0088] In addition to the vehicle 70, the moving body according to the present disclosure may also include an aircraft, a ship, or the like.

[0089] The above has described an embodiment of an information processing method using information processing system 1, but embodiments of the present disclosure can also be embodied as a method or program for implementing the device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0090] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0091] In one embodiment, (1) an information processing device includes an acquisition unit that acquires a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an image sensor that has both infrared light shooting pixels and visible light shooting pixels; and a processing unit that generates at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

[0092] (2) In the information processing device described in (1) above, the processing unit may calculate a distance to the object based on the positions of the high-intensity pixels.

[0093] (3) In the information processing device described in (1) or (2) above, the processing unit may calculate the distance to the object based on an area of ​​the high-intensity pixels in the visible image.

[0094] (4) In the information processing device described in any one of (1) to (3) above, the processing unit may detect a portion in the visible image where the absolute value of the difference in intensity between adjacent pixels is equal to or greater than a difference threshold, and set the predetermined intensity to a value between the higher intensity and the lower intensity of the adjacent pixels in the detected portion.

[0095] (5) In the information processing device described in any one of (1) to (4) above, the processing unit may input the visible image to a recognition model generated by performing learning using learning data that associates the high-intensity pixels with information regarding the distance to the object, and obtain information regarding the distance to the object output from the recognition model.

[0096] (6) In the information processing device described in any one of (1) to (5) above, the processing unit may input the visible image to a recognition model generated by performing learning using learning data that associates the high-intensity pixels with the risk level of the object, and obtain the risk level of the object output from the recognition model.

[0097] In one embodiment, (7) an information processing system includes the information processing device according to any one of (1) to (6) above, and an imaging device having the imaging element.

[0098] (8) The information processing system described in (7) above may further include an illumination device having the infrared light irradiating unit and the visible light irradiating unit.

[0099] (9) In the information processing system described in (8) above, the lighting device may be mounted on a moving body, and the infrared light irradiating unit may be configured to irradiate infrared light in the direction of the moving body's high beam, and the visible light irradiating unit may be configured to irradiate visible light in the direction of the moving body's low beam.

[0100] In one embodiment, (10) an information processing method includes an information processing device acquiring a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, using an image sensor having both infrared light shooting pixels and visible light shooting pixels; and the information processing device generating at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information of high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

[0101] In one embodiment, (11) an information processing program causes an information processing device to acquire a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, by an image sensor having both infrared light shooting pixels and visible light shooting pixels, and causes the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image. [Explanation of symbols]

[0102] 1. Information Processing Systems 10 information processing device (12: acquisition unit, 14: processing unit, 16: output unit, 18: storage unit) 20 Imaging equipment 30 Lighting equipment 70 vehicles 80 Pedestrians 82 high intensity pixels

Claims

1. an acquisition unit that acquires a visible image of an illumination area including a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, the visible image being captured by an image sensor that has both infrared light imaging pixels and visible light imaging pixels; a processing unit that generates at least one of information regarding the presence or absence of an object in the illumination area and information regarding the distance to the object present in the illumination area based on information regarding high-intensity pixels with an intensity equal to or greater than a predetermined intensity among pixels included in the visible image; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the processing unit calculates a distance to the object based on the positions of the high-intensity pixels.

3. The information processing device according to claim 1 , wherein the processing unit calculates the distance to the object based on an area of ​​the high-intensity pixels in the visible image.

4. 2. The information processing device according to claim 1, wherein the processing unit detects a portion in the visible image where an absolute value of a difference in intensity between adjacent pixels is equal to or greater than a difference threshold, and sets the predetermined intensity to a value between a higher intensity and a lower intensity of adjacent pixels in the detected portion.

5. 2. The information processing device according to claim 1, wherein the processing unit inputs the visible image into a recognition model generated by performing learning using learning data that associates the high-intensity pixels with information regarding the distance to the object, and acquires information regarding the distance to the object output from the recognition model.

6. The information processing device according to claim 1, wherein the processing unit inputs the visible image into a recognition model generated by performing learning using learning data that associates the high-intensity pixels with the risk level of the object, and obtains the risk level of the object output from the recognition model.

7. An information processing system comprising: the information processing device according to claim 1; and an imaging device having the imaging element.

8. The information processing system according to claim 7 , further comprising an illumination device having the infrared light irradiating section and the visible light irradiating section.

9. 9. The information processing system according to claim 8, wherein the lighting device is mounted on a moving body, and the infrared light irradiating unit is configured to irradiate infrared light in a direction of a high beam of the moving body, and the visible light irradiating unit is configured to irradiate visible light in a direction of a low beam of the moving body.

10. an information processing device acquires a visible image by capturing an illumination area including a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an imaging element having both infrared light imaging pixels and visible light imaging pixels; the information processing device generates at least one of information regarding the presence or absence of an object in the illumination area and information regarding the distance to the object present in the illumination area based on information regarding high intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image; An information processing method, including:

11. an information processing device is caused to acquire a visible image of an illumination area including a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, by using an image sensor having both infrared light imaging pixels and visible light imaging pixels; causing the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area and information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image; Information processing program.

Images