Information processing device, information processing system, information processing method, and information processing program

Abstract

This information processing device is provided with: an acquisition unit for acquiring a visible image in which an illumination region including a section illuminated by both an infrared light irradiation unit and a visible light irradiation unit and a section illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit is captured by an imaging element including both an infrared light imaging pixel and a visible light imaging pixel; and a processing unit for generating at least one of information relating to the presence or absence of an object in the illumination region and information relating to the distance to an object present in the illumination region on the basis of information of a high-intensity pixel having a prescribed intensity or higher among the pixels included in the visible image.

Description

Information processing device, information processing system, information processing method, and information processing program Cross-reference to related applications

[0001] This application claims priority from Japanese Patent Application No. 2024-116185 (filed July 19, 2024), the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program.

[0003] As described in Patent Document 1, an image processing method is known that suppresses the occurrence of false colors by performing saturation processing when visible light pixels become saturated pixels in a sensor that includes visible light pixels and near-infrared light pixels.

[0004] Japanese Patent Application Laid-Open No. 2021-145167

[0005] An information processing device according to an embodiment of the present disclosure includes an acquisition unit and a processing unit. The acquisition unit acquires a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, captured by an image sensor having both infrared light imaging pixels and visible light imaging pixels. The processing unit generates at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image.

[0006] An information processing system according to an embodiment of the present disclosure includes the information processing device and an imaging device having the imaging element.

[0007] An information processing method according to an embodiment of the present disclosure includes an information processing device acquiring a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an image sensor having both infrared light imaging pixels and visible light imaging pixels. The information processing method includes the information processing device generating at least one of information regarding the presence or absence of an object in the illumination area or information regarding a distance to an object present in the illumination area, based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image.

[0008] An information processing program according to an embodiment of the present disclosure causes an information processing device to acquire a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, by an image sensor having both infrared light imaging pixels and visible light imaging pixels. The information processing program causes the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information regarding high-intensity pixels having an intensity equal to or greater than a predetermined intensity among pixels included in the visible image.

[0009] 6A . FIG. 7A is a block diagram showing an example of a schematic configuration of a detection system according to an embodiment. FIG. 7B is a diagram illustrating generation of an infrared image and a visible image from a captured image. FIG. 7C is a graph showing an example of the sensitivity characteristics of an imaging device. FIG. 7D is a graph showing an example of the detection intensity of infrared light and each of the RGB colors. FIG. 7E is a graph showing an example of the detection intensity of infrared light and each of the RGB colors when the detection intensity of each of the RGB colors exceeds an upper detection limit. FIG. 7F is a diagram showing an example of an illumination area ahead of a vehicle. FIG. 7G is a diagram showing an example of a visible image of an object present in the illumination area of ​​FIG. 5A. FIG. 7H is a diagram showing a case where the object is closer to the vehicle than in FIG. 5A. FIG. 7H is a diagram showing an example of a visible image of an object present in the illumination area of ​​FIG. 7A. FIG. 7H is a diagram showing a case where the object is even closer to the vehicle than in FIG. 7A. FIG. 7H is a block diagram showing an example of a learning flow for generating a recognition model. FIG. 7I is an example of a table that associates a visible image of an object with a correct value of the distance to the object and a risk level. FIG. 7I is a block diagram showing an example of a processing flow for estimating information related to the distance to the object using a recognition model. FIG. 7I is a flowchart showing an example of the procedure of an information processing method for detecting an object present in an illumination area based on high-intensity pixels of a visible image. 10 is a flowchart illustrating an example of a procedure of an information processing method including a procedure of calculating a distance to an object based on high-intensity pixels of a visible image.

[0010] When detecting an object from a visible image including pixels that detect a combined component of visible light and infrared light, removing the infrared light component does not necessarily contribute to improving the accuracy of object detection. Furthermore, the processing load of removing the infrared light component increases. It is desired to improve the accuracy of object detection from a visible image including an infrared light component using simple processing. According to an information processing device, an information processing system, an information processing method, and an information processing program according to an embodiment of the present disclosure, the accuracy of object detection from a visible image including an infrared light component can be improved using simple processing.

[0011] An image sensor that captures visible light is also sensitive to infrared light, but by using a filter that attenuates infrared light, it is possible to capture an image of an object using visible light so that the infrared light component is not detected. On the other hand, an image sensor that has both pixels that detect visible light and pixels that detect infrared light cannot use a filter that attenuates infrared light to capture infrared light, and the pixels that detect visible light detect both visible light and infrared light components. As a result, an image captured by the pixels that detect visible light is affected by the infrared light component.

[0012] When a pixel that detects visible light detects both a visible light component and an infrared light component, the detection intensity of the pixel that detects visible light may become saturated. In the present disclosure, the pixel whose detection intensity is saturated is used to generate information about the distance to an object captured in an image by the pixel that detects visible light.

[0013] (Configuration example of information processing system 1) As shown in Fig. 1 , an information processing system 1 according to an embodiment of the present disclosure includes an information processing device 10, an imaging device 20, and an illumination device 30. In the present disclosure, the information processing system 1 is assumed to be mounted on a vehicle 70 (see Fig. 5A etc.). The information processing system 1 may also be mounted on other moving bodies. The information processing system 1 may also be mounted on various devices, not limited to moving bodies.

[0014] <Information Processing Device 10 > The information processing device 10 includes an acquisition unit 12 , a processing unit 14 , and an output unit 16 .

[0015] The acquisition unit 12 acquires image data from the image capture device 20. The acquisition unit 12 may acquire various other data or information. The acquisition unit 12 may include a communication interface for wired or wireless communication with the image capture device 20 or other devices. The communication interface may be configured to be capable of communication using a communication method based on various communication standards. The communication interface may be configured based on known communication technology.

[0016] The acquisition unit 12 may include an input device that accepts input from a user. The input device may include, for example, a keyboard or physical keys, or a pointing device such as a touch panel, a touch sensor, or a mouse. The input device is not limited to these examples and may include various other devices. The acquisition unit 12 may be configured to be able to communicate with an external input device.

[0017] The processing unit 14 processes the image data. The processing unit 14 may include at least one processor to provide control and processing power for performing various functions. Each function of the processing unit 14 may be implemented by one processor or several processors. The combined functions of the processing unit 14 may be implemented by one processor. The processor may be implemented as a single integrated circuit (IC). The processor may be implemented as multiple communicatively connected integrated circuits or discrete circuits. The processor may be implemented based on various other known technologies.

[0018] The processor may include a general-purpose processor that loads a specific program to execute a specific function, or a dedicated processor specialized for a specific process. The general-purpose processor may include, for example, a central processing unit (CPU) or a digital signal processor (DSP). The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The processing unit 14 may include either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0019] The information processing device 10 may include a storage unit 18. The storage unit 18 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit 18 stores various types of information. The storage unit 18 stores programs to be executed by a processor or the like that functions as the processing unit 14. The storage unit 18 may be configured as a non-transitory readable medium. The storage unit 18 may function as a work memory for the processing unit 14. At least a portion of the storage unit 18 may be configured integrally with the processing unit 14.

[0020] The output unit 16 outputs the processing result of the image data by the processing unit 14. The output unit 16 may include a display device such as a display. The display may include various types of displays such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or an inorganic EL display.

[0021] The output unit 16 may include an audio output device such as a speaker, etc. The output unit 16 is not limited to these examples and may include various other devices.

[0022] A part of the information processing device 10 may be installed externally rather than mounted on the vehicle 70 (see FIG. 5A etc.). The part of the information processing device 10 that is installed externally may be communicably connected to the part that is mounted on the vehicle 70 etc. For example, at least a part of the functions of the processing unit 14 of the information processing device 10 may be realized by a cloud service.

[0023] <Photographing device 20> The photographing device 20 has an imaging element that includes both pixels that capture infrared light and pixels that capture visible light. The photographing device 20 generates a captured image that includes both pixels that represent the detected intensity of infrared light and pixels that represent the detected intensity of visible light. The pixels that capture infrared light detect light in a wavelength range of, for example, 780 nm to 1000 nm. The infrared light may be light in the wavelength range of near-infrared light. The pixels that capture visible light detect light in a wavelength range of, for example, 380 nm to 780 nm. The wavelength ranges of infrared light and visible light are not limited to the values ​​described above and may be changed as appropriate when implementing the invention.

[0024] As illustrated in Fig. 2 , the captured image includes pixels that capture infrared light represented by I and pixels that capture visible light in the three colors of red, green, and blue represented by R, G, and B. The image capturing device 20 performs demosaicing on the captured image. Demosaicing is a process that separates pixels that are mixed in the captured image to generate multiple images. By performing demosaicing on the captured image, the image capturing device 20 generates an infrared image made up of pixels that capture infrared light represented by I and a visible image made up of pixels that capture visible light in the three colors of red, green, and blue represented by R, G, and B.

[0025] As shown in Fig. 3, the wavelength characteristics of sensitivity in a pixel capturing infrared light have a peak in the infrared band. On the other hand, the wavelength characteristics of sensitivity in a pixel capturing each of the R, G, and B visible light have peaks in both the R, G, and B wavelengths included in the visible band and in the infrared band. In other words, a pixel capturing visible light detects not only visible light components but also infrared light components. Therefore, the detection intensity corresponding to each pixel of the visible image is the detection intensity of a component that combines visible light components and infrared light components.

[0026] The imaging element may be, for example, a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS) sensor.

[0027] In the present disclosure, the image capturing device 20 is configured as an in-vehicle camera of a vehicle 70 (see FIG. 5A etc.). The image capturing device 20 is not limited to an in-vehicle camera. The number of image capturing devices 20 is not limited to one, and may be two or more.

[0028] <Illumination device 30> The illumination device 30 includes an infrared light irradiation unit that irradiates infrared light and a visible light irradiation unit that irradiates visible light. An area illuminated by at least one of the infrared light and the visible light irradiated from the illumination device 30 is also referred to as an illumination area. A portion of the illumination area that is illuminated by the infrared light irradiated from the infrared light irradiation unit is also referred to as an infrared light illumination area. A portion of the illumination area that is illuminated by the visible light irradiated from the visible light irradiation unit is also referred to as a visible light illumination area.

[0029] The illumination device 30 is arranged so that a portion of the infrared light illumination region and a portion of the visible light illumination region overlap each other. In other words, the illumination region includes a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit. The portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit includes a portion illuminated by the infrared light irradiation unit but not the visible light irradiation unit, and a portion illuminated by the visible light irradiation unit but not the infrared light irradiation unit.

[0030] In the present disclosure, the lighting device 30 is configured as a headlight of a vehicle 70 (see FIG. 5A , etc.). The lighting device 30 is not limited to a headlight. The number of lighting devices 30 is not limited to one, but may be two or more. When the lighting device 30 is a headlight of the vehicle 70, the lighting device 30 may be installed in two locations, one on the left and one on the front side of the vehicle 70.

[0031] (Example of operation of information processing system 1) The acquisition unit 12 of the information processing device 10 acquires an infrared image and a visible image from the imaging device 20. As described above, each pixel of the visible image represents the detection intensity of a component that is a combination of a visible light component and an infrared light component. As illustrated in Fig. 4A , the detection intensity at a pixel corresponding to each color of red, green, and blue represented by R, G, and B in the visible image is a value obtained by adding the detection intensity of the infrared light component represented by IR to the detection intensity of the red, green, and blue components when an image of an object is captured using only visible light.

[0032] In the information processing system 1 according to the present disclosure, the photographing device 20 photographs an illumination area with an image sensor to generate a visible image. The illumination area includes a portion illuminated by both the infrared light irradiating unit and the visible light irradiating unit of the illumination device 30, and a portion illuminated by only either the infrared light irradiating unit or the visible light irradiating unit. Therefore, in the visible image, pixels photographed in a portion illuminated by both the infrared light irradiating unit and the visible light irradiating unit correspond to pixels that detect both the infrared light component and the visible light component. Furthermore, in the visible image, pixels photographed in a portion illuminated by only either the infrared light irradiating unit or the visible light irradiating unit correspond to pixels that detect only either the infrared light component or the visible light component.

[0033] Here, it is assumed that the object has uniform reflectance for infrared light and visible light. When the object is located in an illumination area across a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, in a visible image of the object, the detection intensity of pixels capturing the portion of the object illuminated by both the infrared light irradiation unit and the visible light irradiation unit is higher than the detection intensity of pixels capturing the portion of the object illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit. Based on the detection intensity of each pixel in the visible image, the processing unit 14 of the information processing device 10 can classify pixels capturing the portion of the object illuminated by both the infrared light irradiation unit and the visible light irradiation unit into pixels capturing the portion of the object illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, and pixels capturing the portion of the object illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit.

[0034] For example, the processing unit 14 may set a predetermined intensity as a threshold for the detection intensity, and classify pixels of the visible image whose detection intensity is equal to or greater than the predetermined intensity as pixels representing an image of a portion of the object illuminated by both the infrared light irradiator and the visible light irradiator. The processing unit 14 may set the predetermined intensity based on the intensities of the infrared light and visible light irradiated by the lighting device 30 and the detection sensitivity of the image capture device 20. Pixels of the visible image whose detection intensity is equal to or greater than the predetermined intensity are also referred to as high-intensity pixels.

[0035] When a pixel capturing an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit is adjacent to a pixel capturing an image of a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, the detection intensity between the adjacent pixels in the visible image changes rapidly. Specifically, the difference in detection intensity between the adjacent pixels may be the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The processing unit 14 may detect a portion in the visible image where the absolute value of the difference in detection intensity between adjacent pixels is equal to or greater than a difference threshold as a portion where the detection intensity changes rapidly. The difference threshold may be set based on the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The difference threshold may be set to the smaller value of the intensity of infrared light emitted by the infrared light irradiation unit or the intensity of visible light emitted by the visible light irradiation unit. The difference threshold may be set to the intensity value of the infrared light irradiated by the infrared light irradiating unit itself, or may be set to a value obtained by multiplying the intensity value of the infrared light irradiated by the infrared light irradiating unit by a positive coefficient less than 1. The difference threshold may be set to the intensity value of the visible light irradiated by the visible light irradiating unit itself, or may be set to a value obtained by multiplying the intensity value of the visible light irradiated by the visible light irradiating unit by a positive coefficient less than 1.

[0036] In a portion where the detection intensity changes suddenly, a pixel with a higher detection intensity corresponds to a pixel that captures an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit. The processing unit 14 may set a value between the higher and lower detection intensities of adjacent pixels across the portion where the detection intensity changes suddenly as a predetermined intensity, thereby classifying pixels whose detection intensity is equal to or greater than a predetermined intensity, i.e., high-intensity pixels, as pixels that capture an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit. The processing unit 14 may classify pixels whose detection intensity increases suddenly as pixels that capture an image of a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit.

[0037] The intensity of light incident from a portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit may exceed the upper detection limit, as illustrated in FIG. 4B . In this case, the detection intensity of pixels where the light intensity exceeds the upper detection limit saturates at the upper detection limit. In the visible image, pixels where the detection intensity of each RGB color is saturated are displayed in white. Furthermore, pixels where the detection intensity of each RGB color is close to the upper detection limit are displayed in gray close to white. For example, when the imaging device 20 captures a visible image of a person present in the illumination area, the color of pixels depicting the person's face or clothing in the portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit appears almost white, regardless of their actual color. Pixels in the visible image where the detection intensity saturates at the upper detection limit and appears white, or pixels where the detection intensity approaches the upper detection limit and appears almost white, are also referred to as blown-out highlight pixels. Blown-out highlight pixels may correspond to pixels in the portion illuminated by both the infrared light irradiation unit and the visible light irradiation unit, i.e., high-intensity pixels.

[0038] For example, as shown in a side view of vehicle 70 in FIG. 5A , when vehicle 70 is equipped with lighting device 30 as a headlight that illuminates the road ahead, infrared light is emitted from the infrared light emitting unit of lighting device 30, and visible light is emitted from the visible light emitting unit, thereby generating an illumination area in front of vehicle 70. Of the illumination area, the portion illuminated by infrared light emitted from the infrared light emitting unit, i.e., the infrared light illumination area, is the area represented by R_IR between the two dashed lines. The portion illuminated by visible light emitted from the visible light emitting unit, i.e., the visible light illumination area, is the area represented by R_VL between the two dashed lines. Furthermore, the portion illuminated by both infrared light emitted from the infrared light emitting unit and visible light emitted from the visible light emitting unit, i.e., the overlapping portion of the infrared light illumination area and the visible light illumination area, is represented by R_OL.

[0039] A pedestrian 80 as an object is present within the illumination area shown in FIG. 5A . The pedestrian 80 present within the illumination area is dazzled when illuminated with visible light from the illumination device 30 serving as the headlight of the vehicle 70. Therefore, the visible light irradiation unit of the illumination device 30 is configured to irradiate visible light toward the ground in a so-called low beam direction so that the head of the pedestrian 80 present within the illumination area does not enter the visible light illumination area (R_VL). The low beam of the headlight of the vehicle 70 can also be referred to as a passing headlight. On the other hand, in order for the vehicle 70 to travel safely, it is necessary for the pedestrian 80 to be photographed by the imaging device 20. The pedestrian 80 is not dazzled when illuminated with infrared light. Therefore, the infrared light irradiation unit of the illumination device 30 is configured to irradiate infrared light toward the air in a so-called high beam direction so that most of the pedestrian 80 present within the illumination area falls within the infrared light illumination area (R_IR). The high beam of the headlight of the vehicle 70 can also be referred to as a driving headlight.

[0040] In the present disclosure, the infrared light irradiation unit of the lighting device 30 may be configured to irradiate infrared light toward the ground in a so-called low beam direction so that the heads of pedestrians 80 present in the illumination area do not enter the infrared light illumination area (R_IR). The visible light irradiation unit of the lighting device 30 may be configured to irradiate infrared light toward the air in a so-called high beam direction so that most of the pedestrians 80 present in the illumination area enter the visible light illumination area (R_VL).

[0041] In the present disclosure, the infrared light irradiation unit and the visible light irradiation unit of the lighting device 30 may be configured to irradiate infrared light and visible light such that at least a portion of the irradiation ranges of the infrared light and the visible light do not overlap, and such that at least a portion of the irradiation ranges of the infrared light and the visible light overlap.

[0042] 5A , the boundary between the infrared light illumination region (R_IR) and the visible light illumination region (R_VL) is represented by a dashed line. The boundary of R_IR may be determined based on the central axis of the irradiation direction of infrared light from the infrared light irradiator and the spread angle from the central axis. The boundary of R_VL may be determined based on the central axis of the irradiation direction of visible light from the visible light irradiator and the spread angle from the central axis.

[0043] The boundary line of R_IR may be a line connecting positions where the intensity of infrared light decreases to a value obtained by multiplying the maximum intensity of infrared light in R_IR by an infrared light boundary threshold. Furthermore, the boundary line of R_VL may be a line connecting positions where the intensity of visible light decreases to a value obtained by multiplying the maximum intensity of visible light in R_VL by a visible light boundary threshold. The infrared light boundary threshold and the visible light boundary threshold may be set to different values ​​or may be set to the same value. The infrared light boundary threshold and the visible light boundary threshold may be set so that the difference between the detected intensity of a pixel capturing an overlapping portion (R_OL) between the infrared light illumination region and the visible light illumination region and the detected intensity of a pixel capturing an R_IR or R_VL adjacent to the outside of R_OL is equal to or greater than a difference threshold. Furthermore, the boundary line of R_IR may be a line defining a predetermined first solid angle range relative to the illumination direction of infrared light. The boundary line of R_VL may be a line defining a predetermined second solid angle range relative to the illumination direction of visible light. The first solid angle range and the second solid angle range may be the same or different. Either the first solid angle range or the second solid angle range may be larger.

[0044] In FIG. 5A , the feet of the pedestrian 80 are located within the R_OL. On the other hand, the part above the feet of the pedestrian 80 is located within the R_IR but outside the R_OL. Here, FIG. 5B shows a visible image of the pedestrian 80 in FIG. 5A captured by the imaging device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, the pixels capturing the feet of the pedestrian 80 are located in the R_OL, and are illuminated by both the infrared light emitted from the infrared light irradiation unit and the visible light emitted from the visible light irradiation unit. Therefore, the pixels capturing the feet of the pedestrian 80 are high-intensity pixels 82. On the other hand, the pixels capturing the part above the feet of the pedestrian 80 are located outside the R_OL, and are not high-intensity pixels 82.

[0045] By detecting that the visible image of FIG. 5B contains high-intensity pixels 82, the processing unit 14 can detect the presence of an object within the portion of the illumination area represented by R_OL. In FIG. 5A , the portion represented by R_OL is a finite range from the illumination device 30 to the point where the portion intersects with the ground and disappears. Therefore, the processing unit 14 can detect the presence of an object within the finite range corresponding to R_OL. The distance from the illumination device 30 to the point where the portion represented by R_OL intersects with the ground and disappears is determined based on the position of the illumination device 30, the direction in which infrared light is emitted from the infrared light emitter of the illumination device 30, and the direction in which visible light is emitted from the visible light emitter, i.e., a predetermined distance. By detecting that the visible image of FIG. 5B contains high-intensity pixels 82, the processing unit 14 can detect the presence of an object within a predetermined distance from the front end of the vehicle 70. In other words, the processing unit 14 can generate information indicating the presence of an object within a predetermined distance as information regarding the distance to the object.

[0046] 6A , when the distance from vehicle 70 to pedestrian 80 is shorter than the distance to pedestrian 80 in FIG. 5A , the portion of pedestrian 80 near the knees is located within R_OL. On the other hand, the portion of pedestrian 80 above the knees is located within R_IR but outside R_OL. Furthermore, the portion of pedestrian 80 below the knees is located within R_VL but outside R_OL.

[0047] Here, Figure 6B shows a visible image of the pedestrian 80 in Figure 6A captured by the imaging device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, pixels capturing the area around the knees of the pedestrian 80 capture an area located within the R_OL, which is an area illuminated by both the infrared light emitted from the infrared light irradiation unit and the visible light emitted from the visible light irradiation unit. Therefore, the pixels capturing the area around the knees of the pedestrian 80 are high-intensity pixels 82. On the other hand, pixels capturing the area above or below the knees of the pedestrian 80 capture an area located outside the R_OL, and are not high-intensity pixels 82.

[0048] As illustrated in Fig. 7A , when the distance from vehicle 70 to pedestrian 80 is even shorter than the distance to pedestrian 80 in Fig. 6A , the portion of pedestrian 80 around the waist is located within R_OL. On the other hand, the portion of pedestrian 80 above the waist is located within R_IR but outside R_OL. Furthermore, the portion of pedestrian 80 below the waist is located within R_VL but outside R_OL.

[0049] 7B shows a visible image of the pedestrian 80 in FIG. 7A captured by the image capture device 20 mounted on the vehicle 70. In the visible image of the pedestrian 80, pixels capturing the area around the waist of the pedestrian 80 capture a portion located within the R_OL, which is a portion illuminated by both the infrared light emitted from the infrared light emitter and the visible light emitted from the visible light emitter. Therefore, the pixels capturing the area around the waist of the pedestrian 80 are high-intensity pixels 82. On the other hand, pixels capturing the area above or below the waist of the pedestrian 80 capture a portion located outside the R_OL, and are not high-intensity pixels 82.

[0050] 5B , 6B , and 7B , the R_OL range approaches the ground as it moves away from the lighting device 30, and therefore the height of the high-intensity pixels 82 from the ground changes depending on the distance from the front end of the vehicle 70, i.e., the lighting device 30, to the pedestrian 80. When the relationship between the height of the high-intensity pixels 82 from the ground in a visible image capturing the pedestrian 80 and the distance to the pedestrian 80 is known, the processing unit 14 can calculate the distance to the pedestrian 80 by detecting the height of the high-intensity pixels 82 from the ground in the visible image and applying the detected height to the known relationship. In other words, the processing unit 14 can calculate the distance to an object based on the height of the high-intensity pixels 82 from the ground in a visible image capturing the object.

[0051] The lighting device 30 is configured so that the overlapping portion (R_OL) of the infrared light illumination region (R_IR) and the visible light illumination region (R_VL) disappears within a finite range. The lighting device 30 may be configured so that the R_OL disappears when it intersects with the ground, as shown in FIG. 5A . Alternatively, the lighting device 30 may be configured so that the R_OL disappears in midair without intersecting with the ground. In this case, in a side view of the illumination region, the boundary line of the R_IR that overlaps with the R_VL and the boundary line of the R_VL that overlaps with the R_IR intersect at a position a finite distance from the lighting device 30, causing the R_OL to disappear in midair.

[0052] When the lighting device 30 is configured so that the R_OL disappears in mid-air, the greater the distance to an object located within the R_OL, the smaller the area of ​​high-intensity pixels in the visible image of the object. If the relationship between the area of ​​high-intensity pixels in the visible image of the object and the distance to the object is known, the processing unit 14 can calculate the distance to the object by calculating the area of ​​the high-intensity pixels from the visible image and applying the known relationship. In other words, the processing unit 14 can calculate the distance to the object based on the area of ​​high-intensity pixels in the visible image of the object.

[0053] The processing unit 14 of the present disclosure may calculate the distance to the object based on the distance of the high-intensity pixel 82 in the visible image of the object from a predetermined reference. In the above description, the predetermined reference is the ground. However, the predetermined reference may be, for example, the height of the image capture device 20 or the lighting device 30 from the ground.

[0054] <Recognition Model> The processing unit 14 may use a recognition model to recognize objects appearing in an infrared image and a visible image. The recognition model may be configured to output a result of recognizing an object appearing in the infrared image and the visible image when the infrared image and the visible image are input. Furthermore, the recognition model may be configured to output a result of recognizing the presence of an object in an overlapping portion (R_OL) of the infrared light illumination region and the visible light illumination region when a visible image including high-intensity pixels of the object is input.

[0055] The recognition model may be configured to calculate the distance to a specific object. The specific object for which the distance is calculated is also referred to as a recognized object. The recognized object may include, for example, a pedestrian 80 located ahead of the vehicle 70 in the direction of travel. The recognition model may be configured to output a calculated distance from the vehicle 70 or the lighting device 30 mounted on the vehicle 70 to a recognized object present on the R_OL when a visible image including high-intensity pixels of the recognized object is input. The recognition model may be configured to output a hazard level of the recognized object present on the R_OL to the vehicle 70 when a visible image including high-intensity pixels of the recognized object is input. The hazard level of the recognized object represents the likelihood of a dangerous situation occurring, such as the recognized object approaching or colliding with the vehicle 70. In the present disclosure, the recognized object may be, in addition to a pedestrian 80, a standing (stationary) person, a jogger, a wheelchair, a bicycle, a motorcycle, a car, an animal such as a dog or a cat, a rock, or any other suitable object.

[0056] The recognition model may be configured to recognize the presence of a recognition target in the R_OL when it is determined that a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to output a distance to the recognition target in the R_OL or a level of danger of the target when it is determined that a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to determine whether an object captured in the visible image is a recognition target based on both the infrared image and the visible image, and to determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target. The recognition model may be configured to determine whether an object captured in the visible image is a recognition target based only on the visible image, and to determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a portion of the recognition target.

[0057] The recognition model may include an input unit and an output unit. The input unit is a unit for inputting infrared images and visible images. The output unit is a unit for outputting the recognition result of the object.

[0058] The recognition model may be a rule-based model that specifies rules for extracting features of objects appearing in infrared images and visible images. The recognition model may be a trained model generated by performing machine learning using infrared images and visible images as training data. The machine learning method may include, for example, a method using Haar-like features or Hog features, or derivative methods thereof such as Joint Haar-like, Joint HoG, or Shapelet.

[0059] The recognition model may be generated by deep learning, which may be, for example, a convolutional neural network (CNN) model such as Renet or DenseNet, or a transformer model such as ViT.

[0060] When the recognition model is configured to output a result of recognizing the presence of an object in R_OL, it may be generated by performing learning using training data including visible images containing high-intensity pixels of the object.When the recognition model is configured to output a result of calculating the distance to the recognized object, it may be generated by performing learning using training data that associates visible images containing high-intensity pixels of the recognized object with correct values ​​of the distance to the recognized object.

[0061] As described above, the processing unit 14 generates information about the distance to the recognition target object based on the high-intensity pixels in the visible image. The recognition model may be configured to output a result of calculating the distance to the recognition target object based on the position or area of ​​the high-intensity pixels. In this case, training data for generating the recognition model may be teacher data in which the distance to the recognition target object in the visible image is used as ground truth data and associated with the visible image.

[0062] The processing unit 14 may generate the recognition model by performing learning within the processing unit 14. The processing unit 14 may also acquire the recognition model from an external device.

[0063] The relationship between each image and the learning of the recognition model in the generation of the above-mentioned recognition model will be explained with reference to the block diagram shown in Fig. 8. The association of the correct answer may be performed by associating the result of measuring the distance to the recognition target by another means such as a distance measuring device with the visible image. The recognition model is generated by performing learning using the infrared image, the visible image, and data in which the correct answer is associated with the visible image as learning data.

[0064] The correspondence between the visible image and the correct answer data may be represented by a table as exemplified in Fig. 9. In the table of Fig. 9, a correct answer value of the distance from the vehicle 70 to the pedestrian 80 is associated with a visible image that captures the pedestrian 80 as a recognition object. The recognition object may be an object that satisfies the condition that the correct answer value of the distance from the vehicle 70 or the lighting device 30 to the recognition object is associated with a visible image that captures the recognition object.

[0065] The second row of the table shows the correspondence between a visible image when the pedestrian 80 is not in the overlapping portion (R_OL) of the infrared light illumination region and the visible light illumination region, i.e., a visible image that does not include the high-intensity pixels 82 of the pedestrian 80, and the correct value of the distance from the vehicle 70 to the pedestrian 80. In this example, the R_OL range intersects with the ground and disappears at a position 40 m away from the vehicle 70. Therefore, when the pedestrian 80 is not in the R_OL, the correct value of the distance from the vehicle 70 to the pedestrian 80 is 40 m or more.

[0066] The third to fifth rows of the table represent correspondence between a visible image when the pedestrian 80 is present within the R_OL, i.e., a visible image including high-intensity pixels 82 of the pedestrian 80, and the correct value of the distance from the vehicle 70 to the pedestrian 80. The third row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 5A and 5B. The fourth row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 6A and 6B. The fifth row of the table corresponds to the visible image of the pedestrian 80 illustrated in FIGS. 7A and 7B. The correct value of the distance from the vehicle 70 to the pedestrian 80 is 35 m in the third row corresponding to FIG. 5A, 25 m in the fourth row corresponding to FIG. 6A, and 15 m in the fifth row corresponding to FIG. 7A.

[0067] 9, a visible image showing a pedestrian 80, which is a recognition object, is associated with a risk level indicating the likelihood that the pedestrian 80 will approach or collide with the vehicle 70. In the second row of the table, a risk level of "none" is associated with a visible image that does not include high-intensity pixels 82 of the pedestrian 80. In the third to fifth rows of the table, a risk level of "low," "medium," or "high" is associated with a visible image that includes high-intensity pixels 82 of the pedestrian 80, depending on the distance to the pedestrian 80. The recognition object may be an object that satisfies the condition that the correct value of the risk level of the recognition object vis-à-vis the vehicle 70 is associated with a visible image showing the recognition object.

[0068] The expression of the risk level is not limited to the expressions exemplified in the table of Fig. 9. For example, in the second row of the table, a visible image that does not include high-intensity pixels 82 of the pedestrian 80 may be associated with a risk level of "very low," which indicates that the risk level is not zero but is very low, instead of "none." Furthermore, the risk level may be a numerical value that indicates the probability that the pedestrian 80 will approach or collide with the vehicle 70.

[0069] The processing unit 14 recognizes an object from an infrared image and a visible image using the recognition model generated as described above. The recognition process of the processing unit 14 will be described with reference to the block diagram shown in Fig. 10. The processing unit 14 acquires an infrared image and a visible image using the acquisition unit 12. The processing unit 14 inputs the infrared image and the visible image into the recognition model and acquires the recognition result of the object output from the recognition model.

[0070] The processing unit 14 may generate danger information related to the object based on the recognition result of the object. The danger information related to the object may include information on the presence of the object, information on the distance to the recognized object, or the danger level of the recognized object. The processing unit 14 may output the recognition result of the object or the danger information related to the object via the output unit 16 and notify the outside. If the information processing device 10 is installed in a vehicle 70, the recipients of the danger information may include the driver of the vehicle 70, etc.

[0071] <Example of Procedure of Information Processing Method> The processing unit 14 of the information processing device 10 may recognize the presence of an object based on a visible image including high-intensity pixels by executing an information processing method including the procedures of the flowchart illustrated in Fig. 11. The information processing method may be realized as an information processing program executed by a processor included in the information processing device 10. The information processing program may be stored in a non-transitory computer-readable medium.

[0072] The processing unit 14 acquires an infrared image and a visible image from the image capturing device 20 using the acquisition unit 12 (step S1). The processing unit 14 determines whether there are high-intensity pixels in the visible image (step S2). If there are no high-intensity pixels in the visible image (step S2: NO), the processing unit 14 ends the execution of the flowchart in FIG. 11 .

[0073] If there is a high-intensity pixel in the visible image (step S2: YES), the processing unit 14 recognizes the presence of an object in the overlapping portion (R_OL) of the infrared light illumination area and the visible light illumination area, generates danger information about the object, and notifies the outside via the output unit 16 (step S3). After executing the procedure of step S3, the processing unit 14 ends execution of the flowchart in FIG.

[0074] In the determination procedure of step S2, the processing unit 14 may determine whether a high-intensity pixel included in the visible image is a pixel that captures at least a part of a recognition target object such as a pedestrian 80.

[0075] The processing unit 14 of the information processing device 10 may calculate the distance to the recognition target object based on a visible image including high-intensity pixels by executing an information processing method including the steps of the flowchart illustrated in Fig. 12. The information processing method may be realized as an information processing program executed by a processor included in the information processing device 10. The information processing program may be stored in a non-transitory computer-readable medium.

[0076] The processing unit 14 acquires an infrared image and a visible image from the image capturing device 20 using the acquisition unit 12 (step S11). The processing unit 14 determines whether there are high-intensity pixels in the visible image (step S12). If there are no high-intensity pixels in the visible image (step S12: NO), the processing unit 14 ends the execution of the flowchart in FIG. 12 .

[0077] If there is a high-intensity pixel in the visible image (step S12: YES), the processing unit 14 determines whether the high-intensity pixel is a pixel that captures at least a part of the recognition target object, such as a pedestrian 80 (step S13). If the high-intensity pixel is a pixel that captures at least a part of the recognition target object (step S13: YES), the processing unit 14 calculates the distance to the recognition target object (step S14). If the high-intensity pixel is not a pixel that captures at least a part of the recognition target object (step S13: NO), the processing unit 14 proceeds to step S15.

[0078] The processing unit 14 generates danger information and notifies the outside via the output unit 16 (step S15). If the processing unit 14 determines in the procedure of step S13 that the high-intensity pixel is a pixel that captures at least a part of the recognition target object, it may generate danger information related to the recognition target object. If the processing unit 14 determines in the procedure of step S13 that the high-intensity pixel is not a pixel that captures at least a part of the recognition target object, it may generate danger information related to an object that is not the recognition target object. After executing the procedure of step S15, the processing unit 14 ends execution of the flowchart of FIG. 12.

[0079] (Summary) As described above, according to the information processing system 1, information processing device 10, and information processing method of the present disclosure, a visible image is generated by capturing an illumination area including an overlapping portion between an infrared light illumination area and a visible light illumination area. Then, without removing the infrared light component from the visible image, the presence of an object in the overlapping portion between the infrared light illumination area and the visible light illumination area is recognized based on information about pixels that detect both the infrared light component and the visible light component. In other words, it is determined whether an object present in the illumination area is located in the overlapping portion between the infrared light illumination area and the visible light illumination area.

[0080] In a comparative example, when both infrared light and visible light are irradiated onto the entire illumination area, it is difficult to identify the position of an object within the illumination area without removing the infrared light component from the visible image. In contrast to the comparative example, according to the present disclosure, it is easy to identify whether an object present within the illumination area is located in the overlapping portion of the infrared light illumination area and the visible light illumination area.

[0081] Furthermore, according to the present disclosure, the distance from the illumination device 30 to the object is calculated based on the position or area of ​​high-intensity pixels included in the visible image. On the other hand, in the above-described comparative example, the visible image does not include high-intensity pixels that change depending on the distance to the object, so the distance cannot be calculated based on the high-intensity pixels. Therefore, according to the present disclosure, the position of the object can be easily identified.

[0082] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0083] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to be logically inconsistent, and multiple components can be combined into one or divided. It should be understood that these modifications are also included in the scope of the present disclosure.

[0084] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0085] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0086] The vehicle 70 according to the present disclosure may include, for example, automobiles, industrial vehicles, rail vehicles, utility vehicles, or fixed-wing aircraft traveling on runways. The automobile may include, for example, passenger cars, trucks, buses, motorcycles, or trolleybuses. The industrial vehicle may include, for example, industrial vehicles for agriculture or construction. The industrial vehicle may include, for example, forklifts or golf carts. The agricultural industrial vehicle may include, for example, tractors, cultivators, transplanters, binders, combines, or lawnmowers. The construction industrial vehicle may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, or road rollers. The vehicle 70 may include vehicles that are propelled by human power. The classification of the vehicle 70 is not limited to the above examples. For example, the automobile may include industrial vehicles that can travel on roads. The same type of vehicle 70 may be included in multiple classifications.

[0087] In addition to the vehicle 70, the moving body according to the present disclosure may also include an aircraft or a ship.

[0088] The above has described an embodiment of an information processing method using the information processing system 1, but embodiments of the present disclosure can also be embodied as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.), in addition to a method or program for implementing the device.

[0089] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0090] In one embodiment, (1) an information processing device includes: an acquisition unit that acquires a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, captured by an image sensor having both infrared light shooting pixels and visible light shooting pixels; and a processing unit that generates at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

[0091] (2) In the information processing device described in (1) above, the processing unit may calculate a distance to the object based on the positions of the high-intensity pixels.

[0092] (3) In the information processing device described in (1) or (2) above, the processing unit may calculate the distance to the object based on an area of ​​the high-intensity pixels in the visible image.

[0093] (4) In the information processing device described in any one of (1) to (3) above, the processing unit may detect a portion in the visible image where the absolute value of the difference in intensity between adjacent pixels is equal to or greater than a difference threshold, and set the predetermined intensity to a value between the higher intensity and the lower intensity of the adjacent pixels in the detected portion.

[0094] (5) In the information processing device described in any one of (1) to (4) above, the processing unit may input the visible image to a recognition model generated by performing learning using learning data that associates the high-intensity pixels with information regarding the distance to the object, and obtain information regarding the distance to the object output from the recognition model.

[0095] (6) In the information processing device described in any one of (1) to (5) above, the processing unit may input the visible image to a recognition model generated by performing learning using learning data that associates the high-intensity pixels with the risk level of the object, and obtain the risk level of the object output from the recognition model.

[0096] In one embodiment, (7) an information processing system includes the information processing device according to any one of (1) to (6) above, and an imaging device having the imaging element.

[0097] (8) The information processing system described in (7) above may further include an illumination device having the infrared light irradiating unit and the visible light irradiating unit.

[0098] (9) In the information processing system described in (8) above, the lighting device may be mounted on a moving body, and the infrared light irradiating unit may be configured to irradiate infrared light in the direction of the moving body's high beam, and the visible light irradiating unit may be configured to irradiate visible light in the direction of the moving body's low beam.

[0099] In one embodiment, (10) an information processing method includes an information processing device acquiring a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, using an image sensor having both infrared light shooting pixels and visible light shooting pixels; and the information processing device generating at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information of high-intensity pixels having a predetermined intensity or higher among the pixels included in the visible image.

[0100] In one embodiment, (11) an information processing program causes an information processing device to acquire a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, captured by an image sensor having both infrared light shooting pixels and visible light shooting pixels, and causes the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

[0101] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device (12: Acquisition unit, 14: Processing unit, 16: Output unit, 18: Storage unit) 20 Imaging device 30 Lighting device 70 Vehicle 80 Pedestrian 82 High-intensity pixel

Claims

1. An information processing device comprising: an acquisition unit that acquires a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an image sensor that has both infrared light shooting pixels and visible light shooting pixels; and a processing unit that generates at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area, based on information about high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

2. The information processing device according to claim 1, wherein the processing unit calculates the distance to the object based on the positions of the high-intensity pixels.

3. The information processing device according to claim 1 or 2, wherein the processing unit calculates the distance to the object based on the area of ​​the high-intensity pixels in the visible image.

4. An information processing device described in any one of claims 1 to 3, wherein the processing unit detects a portion in the visible image where the absolute value of the difference in intensity between adjacent pixels is equal to or greater than a difference threshold, and sets the value between the higher and lower intensities of the adjacent pixels in the detected portion as the predetermined intensity.

5. An information processing device described in any one of claims 1 to 4, wherein the processing unit inputs the visible image into a recognition model generated by performing learning using learning data that associates the high-intensity pixels with information regarding the distance to the object, and obtains information regarding the distance to the object output from the recognition model.

6. An information processing device described in any one of claims 1 to 5, wherein the processing unit inputs the visible image into a recognition model generated by performing learning using learning data that associates the high-intensity pixels with the risk level of the object, and obtains the risk level of the object output from the recognition model.

7. An information processing system comprising the information processing device according to any one of claims 1 to 6 and an imaging device having the imaging element.

8. The information processing system according to claim 7, further comprising an illumination device having the infrared light irradiating section and the visible light irradiating section.

9. The information processing system according to claim 8, wherein the lighting device is mounted on a moving body, and the infrared light irradiating unit is configured to irradiate infrared light in the direction of the high beam of the moving body, and the visible light irradiating unit is configured to irradiate visible light in the direction of the low beam of the moving body.

10. An information processing method comprising: an information processing device acquiring a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit and the visible light irradiation unit, using an image sensor having both infrared light imaging pixels and visible light imaging pixels; and the information processing device generating at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

11. An information processing program that causes an information processing device to acquire a visible image of an illumination area including a portion illuminated by both an infrared light irradiation unit and a visible light irradiation unit, and a portion illuminated by only one of the infrared light irradiation unit or the visible light irradiation unit, by using an image sensor that has both infrared light shooting pixels and visible light shooting pixels, and causes the information processing device to generate at least one of information regarding the presence or absence of an object in the illumination area or information regarding the distance to an object present in the illumination area based on information regarding high-intensity pixels of a predetermined intensity or higher among the pixels included in the visible image.

Images