Recognition system, recognition device, recognition method, recognition program

Abstract

To provide a recognition system effective for recognition of a traffic light from a mobile object.SOLUTION: A processor of a recognition system which implements recognition processing of a traffic light from a host mobile object is configured to execute steps of: acquiring infrared region photography data Di regarding a plurality of color light emission parts corresponding to light colors switching in the traffic light as data in which the traffic light is photographed by an infrared camera in the host mobile object; and outputting recognition data Dr obtained by recognizing the light colors correlated with a time variation pattern P among infrared region brightness of the respective color light emission parts in the infrared region photography data Di.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This disclosure relates to a recognition technology for performing recognition processing of traffic signals from a moving object.

Background Art

[0002] In the technology disclosed in Patent Document 1, the operation information of a traffic signal is transmitted from a wireless transmitter, and the operation information is recognized in a vehicle as a moving object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology disclosed in Patent Document 1, if a failure occurs in the wireless communication between the vehicle and the wireless transmitter, the recognition of traffic signals in the vehicle will be inhibited.

[0005] The problem of this disclosure is to provide a recognition system effective for recognizing traffic signals from a moving object. Another problem of this disclosure is to provide a recognition device effective for recognizing traffic signals from a moving object. Still another problem of this disclosure is to provide a recognition method effective for recognizing traffic signals from a moving object. Yet another problem of this disclosure is to provide a recognition program effective for recognizing traffic signals from a moving object.

Means for Solving the Problems

[0006] Hereinafter, the technical means of this disclosure for solving the problems will be described. Note that the reference numerals in parentheses described in the claims and this column indicate the correspondence with the specific means described in the embodiments to be described in detail later, and do not limit the technical scope of this disclosure.

[0007] The first aspect of this disclosure is, A recognition system having a processor (12) that performs recognition processing of a signal light (3) from a host mobile device (2), The processor is In a mobile host, infrared imaging data (Di) is acquired regarding multiple color light-emitting units (30) corresponding to the switching colors of the traffic lights, as data captured by an infrared camera (40) on the traffic lights. The system is configured to output recognition data (Dr) that recognizes the color of light corresponding to the time-varying pattern (P, Pf, Pn) of the infrared luminance of each color light-emitting part in the infrared imaging data.

[0008] A second aspect of this disclosure is, A recognition device having a processor (12), configured to be mounted on a host mobile device (2), and performing recognition processing of a signal light (3) from the host mobile device, The processor is In a mobile host, infrared imaging data (Di) is acquired regarding multiple color light-emitting units (30) corresponding to the switching colors of the traffic lights, as data captured by an infrared camera (40) on the traffic lights. The system is configured to output recognition data (Dr) that recognizes the color of light corresponding to the time-varying pattern (P, Pf, Pn) of the infrared luminance of each color light-emitting part in the infrared imaging data.

[0009] A third aspect of this disclosure is: A recognition method performed by a processor (12) in order to perform recognition processing of a signal light (3) from a host mobile body (2), In a mobile host, infrared imaging data (Di) is acquired regarding multiple color light-emitting units (30) corresponding to the switching colors of the traffic lights, as data captured by an infrared camera (40) on the traffic lights. This includes outputting recognition data (Dr) that recognizes the color of light corresponding to the time-varying pattern (P, Pf, Pn) of the infrared luminance of each color light-emitting part in the infrared imaging data.

[0010] The fourth aspect of this disclosure is: A recognition program that includes instructions to be executed by a processor (12) which performs the recognition process of a signal light (3) from a host mobile device (2), and which is stored in a storage medium (10) in order to perform the recognition process of the signal light (3), In a mobile host, infrared imaging data (Di) is acquired regarding multiple color light-emitting units (30) corresponding to the switching colors of the traffic lights, as data captured by an infrared camera (40) on the traffic lights. This includes instructions to output recognition data (Dr) that recognizes the color of light corresponding to the time-varying pattern (P, Pf, Pn) of the infrared luminance of each color light-emitting part in infrared imaging data.

[0011] In the first to fourth embodiments, infrared imaging data is acquired from a host mobile device using an infrared camera to capture images of the traffic light, specifically concerning multiple color light-emitting units corresponding to the switching colors of the traffic light. At this time, the infrared luminance of each color light-emitting unit in the infrared imaging data can track the emission temperature of each color light-emitting unit, which changes over time in accordance with the switching of the light colors. Therefore, according to the first to fourth embodiments, the light colors that correlate with the time-varying patterns of the infrared luminances of each color light-emitting unit in the infrared imaging data can be accurately recognized, and recognition data can be output. In other words, according to the first to fourth embodiments, it is possible to provide a recognition process that is effective for recognizing traffic lights from a host mobile device. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the overall configuration of the first embodiment. [Figure 2] This is a plan view showing the camera mounted on a host vehicle to which the first embodiment is applied. [Figure 3]It is a block diagram showing the functional configuration of the recognition system according to the first embodiment. [Figure 4] It is an external view showing a traffic signal recognized according to the first embodiment. [Figure 5] It is an external view showing a traffic signal recognized according to the first embodiment. [Figure 6] It is a flowchart showing the recognition flow according to the first embodiment. [Figure 7] It is a schematic diagram for explaining the recognition flow according to the first embodiment. [Figure 8] It is a schematic diagram for explaining the recognition flow according to the first embodiment. [Figure 9] It is a graph for explaining the recognition flow according to the first embodiment. [Figure 10] It is a graph for explaining the recognition flow according to the first embodiment. [Figure 11] It is a block diagram showing the overall configuration of the second embodiment. [Figure 12] It is a plan view showing the mounting state of a camera on a host vehicle to which the second embodiment is applied. [Figure 13] It is a block diagram showing the functional configuration of the recognition system according to the second embodiment. [Figure 14] It is a flowchart showing the recognition flow according to the second embodiment. [Figure 15] It is a schematic diagram for explaining the recognition flow according to the second embodiment. [Figure 16] It is a flowchart showing the recognition flow according to the third embodiment. [Figure 17] It is a graph for explaining the recognition flow according to the third embodiment. [Figure 18] It is a graph for explaining the recognition flow according to the third embodiment.

Modes for Carrying Out the Invention

[0013] Hereinafter, several embodiments of this disclosure will be described with reference to the drawings. In each embodiment, the same reference numerals will be used for corresponding components, and redundant explanations may be omitted. Furthermore, if only a part of the configuration is described in each embodiment, the configuration of other embodiments described earlier may be applied to the other parts of that configuration. Moreover, not only the combinations of configurations explicitly stated in the description of each embodiment, but also the configurations of multiple embodiments can be partially combined even if not explicitly stated, as long as there are no particular problems with the combination.

[0014] As shown in Figure 1, the recognition system 1 of the first embodiment performs recognition processing to recognize the traffic light 3 (see Figures 4 and 5 described later) from the host vehicle 2 shown in Figure 2, which is the host mobile object. The host vehicle 2 can be said to be the ego-vehicle from a viewpoint centered on the vehicle. The host vehicle 2 is a mobile object such as an automobile that can travel on a road with an occupant on board. Therefore, the directions in the following description are defined with reference to the host vehicle 2 on the horizontal plane.

[0015] In the host vehicle 2, an automated driving mode is provided, which is categorized into levels according to the degree of manual intervention by the occupant in dynamic driving tasks. The automated driving mode may be implemented by autonomous driving control, such as conditional driving automation, highly automated driving, or fully automated driving, in which the system performs all dynamic driving tasks when in operation. The automated driving mode may also be implemented by advanced driver assistance control, such as driver assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The automated driving mode may be implemented by either one of these autonomous driving controls or advanced driver assistance controls, in combination, or by switching between them.

[0016] The host vehicle 2 is equipped with a sensor system 4. As shown in Figures 1-3, the sensor system 4 includes an infrared camera 40. The infrared camera 40 comprises an infrared image sensor 400 and an infrared imaging circuit 402. The infrared image sensor 400 is a semiconductor element, such as a microbolometer, having multiple pixels arranged in the vertical, horizontal, and vertical directions. The infrared image sensor 400 captures infrared light images, pixel by pixel, from targets within the imaging field of view Ai, receiving light in the infrared region, particularly in the far-infrared region (e.g., wavelength range of 7-14 μm). The infrared imaging circuit 402 is a semiconductor chip, such as an image processing circuit, that processes the imaging signals from each pixel of the infrared image sensor 400. The infrared imaging circuit 402 outputs infrared imaging data Di by converting the infrared brightness of each pixel according to the received light energy of the infrared light image from within the imaging field of view Ai into two-dimensional data.

[0017] As shown in Figures 4 and 5, the traffic light 3 is equipped with multiple color-emitting units 30 that correspond to different light colors, providing different signal colors to traffic participants. In the traffic light 3, each color-emitting unit 30 turns on and off in a prescribed order stipulated in accordance with traffic laws, thereby switching the light color of the lit color-emitting unit 30, i.e., the color of the lit state. The recognition system 1 targets at least one of the traffic lights 3 shown in Figure 4 for vehicles and the traffic lights shown in Figure 5 for pedestrians, but in the following explanation, for ease of understanding, the recognition process for the former, the vehicle traffic light 3, will be used as a representative example.

[0018] As shown in Figure 1, the recognition system 1 is configured to include at least one dedicated computer. The recognition system 1 is connected to the sensor system 4 via at least one of the following: a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line. If the recognition system 1 consists of multiple dedicated computers, the connections between those dedicated computers are similar.

[0019] The dedicated computer constituting the recognition system 1 may be an electronic control unit (ECU) that controls the operation of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a navigation ECU that navigates the driving path of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a locator ECU that estimates the self-state quantities of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an actuator ECU that controls the driving actuators of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a human-machine interface (HMI) control unit (HCU) that controls the presentation of information in the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a computer other than the host vehicle 2 that constructs an external center or mobile terminal that can communicate via the host vehicle 2's communication system.

[0020] The dedicated computer constituting the recognition system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as semiconductor memory, magnetic media, and optical media, which non-temporarily stores programs and data that can be read by the computer. Here, storage may be accumulation in which data is retained even when the host vehicle 2 is turned off, or temporary storage in which data is erased when the host vehicle 2 is turned off. The processor 12 includes at least one type as a core, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), RISC (Reduced Instruction Set Computer)-CPU, DFP (Data Flow Processor), and GSP (Graph Streaming Processor).

[0021] In the recognition system 1, the processor 12 executes multiple instructions contained in the recognition program stored in the memory 10 in order to perform the recognition process of the traffic light 3 from the host vehicle 2. In this way, the recognition system 1 constructs multiple functional blocks for performing the recognition process of the traffic light 3 from the host vehicle 2. As shown in Figure 3, the multiple functional blocks constructed in the recognition system 1 include a data acquisition block 100 and a data output block 120.

[0022] Through the combined action of these blocks 100 and 120, the recognition method by which the recognition system 1 performs the recognition process of the traffic signal 3 from the host vehicle 2 is executed according to the recognition flow shown in Figure 6. This recognition flow is executed repeatedly while the host vehicle 2 is running. In this recognition flow, each "S" represents a step executed by multiple instructions included in the recognition program.

[0023] In S10, the data acquisition block 100 acquires infrared imaging data Di, which is captured by the infrared camera 40 within the imaging field of view Ai of the outside world of the host vehicle 2. In S20, following S10 in the recognition flow, the data acquisition block 100 extracts the infrared brightness for each pixel from the acquired infrared imaging data Di, as shown in Figure 7. At this time, by using the infrared imaging data Di acquired in the previous S10, the infrared brightness for each pixel is converted into time-series data, and noise may be reduced by, for example, low-pass filtering or moving average processing in the time-series direction.

[0024] In step S30 following S20 in the recognition flow shown in Figure 6, the data output block 120 determines whether or not the traffic light 3 was captured within the imaging field of view Ai based on the infrared luminance of each pixel in the infrared imaging data Di. At this time, it is preferable that the presence or absence of the traffic light 3 be determined by pattern matching between the recognition model stored in memory 10 for recognizing the outline 300 (see Figure 7) of the color light-emitting part 30 of the traffic light 3 and the infrared imaging data Di. Alternatively, the presence or absence of the traffic light 3 may be determined by inputting the infrared imaging data Di into a machine learning model stored in memory 10 for recognizing the outline 300 of the color light-emitting part 30 of the traffic light 3 and performing a recognition process.

[0025] As shown in Figure 6, if a negative result is obtained in S30, the current execution of the recognition flow ends. On the other hand, if a positive result is obtained in S30, the recognition flow proceeds to S40. In S40, the data output block 120 outputs recognition data Dr, which recognizes the current light color of the traffic light 3 based on the infrared luminance of each pixel in the infrared imaging data Di.

[0026] Specifically, in S40, the range of multiple pixels corresponding to the inside of the outline contour 300 recognized separately for each color light-emitting unit 30 in the infrared imaging data Di is identified as shown with cross-hatching in Figure 8 and set as the light-emitting pixel area 301. Then, in S40, the time-varying pattern P that appears in the infrared luminance of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared imaging data Di is monitored as shown in Figures 9 and 10. At this time, monitoring of the time-varying pattern P is preferably performed on the central part 301a (see Figure 8), which is the assumed location where the highest infrared luminance is expected to be, within the light-emitting pixel area 301 where the infrared luminance is distributed due to, for example, uneven emission on the outline contour 300 side, for each color light-emitting unit 30.

[0027] Now, as shown in Figure 9, regarding the infrared luminance of each color light-emitting unit 30, infrared luminance appears in two-color or three-color light-emitting units 30 during the transition between on and off states, making it difficult to accurately determine the color of the light based solely on that luminance. Therefore, in S40, the current color of the light, which is correlated with a specific time-varying pattern P as shown in Figure 10 and is noteworthy for appearing between the light-emitting pixel areas 301 of each color light-emitting unit 30, is recognized for the traffic light 3 captured in the infrared imaging data Di.

[0028] In detail, at S40, the change in the rate of change per unit time for infrared luminance is monitored as a time change pattern P. As a result, attention is paid to the combination of two time change patterns that appear in parallel: a time change pattern Pf in which the rate of decrease in time changes step from 0 to the upward side and then gradually decreases, and a time change pattern Pn in which the rate of increase in time changes step from 0 to the upward side and then gradually decreases. In particular, in the first embodiment, the time change pattern Pf appears in parallel with the time change pattern Pn appearing in a color light-emitting unit 30 that transitions from the on state to the off state, while the time change pattern Pn appears in parallel with the time change pattern Pn appearing in a different color light-emitting unit 30 that transitions from the off state to the on state. Note that in Figure 10, the vertical axis represents the rate of change in infrared luminance, with the rate of decrease in time represented by "-" and the rate of increase in time represented by "+".

[0029] Here, during period T1 in Figure 10, when the red light switches to a green light, a time-varying pattern Pf appears in the color light-emitting unit 30 that gives the signal color (red) of the source red light, while a time-varying pattern Pn appears in the color light-emitting unit 30 that gives the signal color (blue, green, or blue-green) of the destination blue light. During period T2 in Figure 10, when the green light switches to a yellow light, a time-varying pattern Pf appears in the color light-emitting unit 30 that gives the signal color (yellow) of the destination yellow light, in parallel with the appearance of the time-varying pattern Pf in the color light-emitting unit 30 that gives the signal color of the source blue light, while a time-varying pattern Pn appears in the color light-emitting unit 30 that gives the signal color (yellow) of the destination yellow light. During period T3 in Figure 10, when the yellow light switches to a red light, a time-varying pattern Pf appears in the color light-emitting unit 30 that gives the signal color (red light) of the source yellow light, in parallel with the appearance of the time-varying pattern Pf in the color light-emitting unit 30 that gives the signal color of the destination red light.

[0030] Based on the above, in S40 of Figure 6, when parallel combinations of time-varying patterns Pf and Pn appear, the light emitted by the color light-emitting unit 30 corresponding to the time-varying pattern Pn, where the time increase rate changes, is recognized as the current light color of the traffic light 3. At this time, the current light color may be determined using, for example, position information regarding the arrangement of each color light-emitting unit 30 and / or color information regarding the light emitted, which are stored in the memory 10 along with the recognition model described above.

[0031] Furthermore, in S40, the light color may be recognized using not only the time-varying pattern P, but also coordinated information obtained from an external center and / or infrastructure system via the communication unit of the host vehicle 2. In S10, if the traffic light 3 is for both vehicles and pedestrians, the recognition of the light color in one may also utilize the recognition result of the light color in the other.

[0032] Based on the above, in S40, when the current light color is recognized, the recognition data Dr is generated to represent the current light color and is output as shown in Figures 1 and 3. At this time, the output of the recognition data Dr may be stored in memory 10. The output of the recognition data Dr may be provided as data to, for example, the driving control ECU in the host vehicle 2. The output of the recognition data Dr may be transmitted as data to an external center through the communication unit of the host vehicle 2.

[0033] However, if it is determined that the lighting order of each color light-emitting unit 30 differs from the prescribed order based on the light color recognized in the past (i.e., previous) S40 and the light color recognized in the current (i.e., current) S40, recognition data Dr may be generated and output to notify a light color recognition error in response to this determination. The prescribed order in this case may be determined by comparing it with the recognized lighting order, for example, using order information regarding the prescribed order of each color light-emitting unit 30 stored in memory 10 along with the recognition model described above.

[0034] (Effects and Benefits) The effects and advantages of the first embodiment described above will be explained below.

[0035] In this first embodiment, infrared imaging data Di is acquired by the host vehicle 2 using an infrared camera 40 to capture images of the traffic light 3, relating to multiple color light-emitting units 30 that switch in the traffic light 3. At this time, the infrared luminance of each color light-emitting unit 30 in the infrared imaging data Di can follow the emission temperature of each color light-emitting unit 30, which changes over time in accordance with the switching of the light color. Therefore, according to the first embodiment, the light color that correlates with the time change pattern P of the infrared luminance of each color light-emitting unit 30 in the infrared imaging data Di can be accurately recognized, and recognition data Dr can be output. In other words, according to the first embodiment, it is possible to provide an effective recognition process for recognizing the traffic light 3 from the host vehicle 2.

[0036] According to the first embodiment, the light emission color from the color light emission unit 30 corresponding to the combination that appears in parallel from the time change pattern Pf of the color light emission unit 30 in which the time decrease rate of infrared luminance changes gradually, and the time change pattern Pn of the color light emission unit 30 in which the time increase rate of infrared luminance changes gradually, can be reflected as the light color. Therefore, in this case, the light color correlated with the time change pattern P can be accurately recognized.

[0037] According to the first embodiment, in the infrared imaging data Di, time-varying patterns P can be extracted from the light-emitting pixel areas 301 identified within the outer contour 300 recognized separately for each color light-emitting unit 30, focusing on the infrared luminance of each light-emitting unit 30. Therefore, the color of light correlated with the extracted time-varying pattern P can be accurately recognized. In particular, in the first embodiment, the time-varying pattern P can be extracted only from the assumed location where the highest infrared luminance is expected, even within the light-emitting pixel areas 301 where infrared luminance is distributed in the infrared imaging data Di, thereby improving the accuracy of color recognition.

[0038] According to the first embodiment, in response to the fact that the lighting order of each color light-emitting unit 30 recognized from the infrared region imaging data Di differs from the prescribed order defined for each color light-emitting unit 30, recognition data Dr is output to notify of a light color recognition error. This makes it possible to suppress the impact on the host vehicle 2 caused by misidentification of light colors.

[0039] (Second embodiment) The second embodiment is a modification of the first embodiment. As shown in Figures 11-13, the sensor system 2004 of the second embodiment includes an infrared camera 40 and a visible-range camera 2041. The imaging field of view Ai of the infrared camera 40 and the imaging field of view Av of the visible-range camera 2041 are superimposed as shown in Figure 12 to form a common field of view Ac that can be captured in common with each other. In particular, in the second embodiment, the common field of view Ac is set in a range where the imaging field of view Ai and the imaging field of view Av partially overlap with each other, both in the horizontal and vertical directions relative to the host vehicle 2.

[0040] As shown in Figures 11 and 13, the visible-range camera 2041 is composed of a visible-range image sensor 2410 and a visible-range imaging circuit 2412. The visible-range image sensor 2410 is a semiconductor element, such as a CMOS, having multiple pixels arranged in the vertical, horizontal, and vertical directions. The visible-range image sensor 2410 captures a visible-light image, pixel by pixel, from a target within the imaging field of view Av that receives light in the visible light range. The visible-range imaging circuit 2412 is a semiconductor chip, such as an image processing circuit, that processes the imaging signals from each pixel of the visible-range image sensor 2410. The visible-range imaging circuit 2412 outputs visible-range imaging data Dv by converting the visible-range brightness of each pixel according to the light reception intensity of the visible-light image from within the imaging field of view Av into two-dimensional data.

[0041] In the second embodiment in particular, the number of pixels used to capture the common field of view Ac in the visible-range image sensor 2410 and the number of pixels used to capture the common field of view Ac in the infrared-range image sensor 400 are different, with the latter being less than the former. As a result, the infrared-range image data Di is lower resolution than the visible-range image data Dv, while the infrared-range image data Di is higher resolution than the visible-range image data Dv.

[0042] In the recognition flow shown in Figure 14 of this second embodiment, S2010, S2020, S2030, and S2040 are executed, replacing S10, S20, and S40 of the first embodiment, respectively. Specifically, in S2010, the data acquisition block 100 acquires infrared imaging data Di, which is captured by the infrared camera 40 within the imaging field of view Ai of the outside world of the host vehicle 2, and visible imaging data Dv, which is captured by the visible camera 2041 within the imaging field of view Av of the same outside world. At this time, it is desirable that the acquisition timing of data Di and Dv be substantially synchronized.

[0043] In the recognition flow, in S2020 following S2010, the data acquisition block 100 extracts the infrared luminance for each pixel from the acquired infrared imaging data Di, similar to the first embodiment. At the same time, in S2020, the data acquisition block 100 extracts the visible luminance for each pixel from the acquired visible imaging data Dv, as shown in Figure 15.

[0044] In the recognition flow shown in Figure 14, at S2030 following S2020, the data output block 120 determines whether or not the traffic light 3 was captured within the common field of view Ac of the imaging field of view Ai, based on the infrared luminance of each pixel in the infrared imaging data Di. At this time, the determination of whether or not the traffic light 3 was captured is the same as in the first embodiment.

[0045] In the recognition flow, the data output block 120 in S2040, which follows the affirmative judgment in S2030, recognizes the outline contour 300 (see Figure 15) for each color light-emitting unit 30 based on the visible-range brightness of each pixel in the visible-range captured data Dv. At this time, it is preferable to perform pattern matching processing between the recognition model stored in memory 10 for recognizing the outline contour 300 of each color light-emitting unit 30 in the traffic light 3 and the visible-range captured data Do. Alternatively, recognition processing may be performed by inputting the visible-range captured data Dv into the machine learning model stored in memory 10 for recognizing the outline contour 300 of each color light-emitting unit 30 in the traffic light 3.

[0046] In this S2040, the range of multiple pixels within the outer contour 300 recognized separately for each color light-emitting unit 30 in the visible range imaging data Dv is set as the light-emitting pixel area 301 in the infrared range imaging data Di, where the range of multiple pixels corresponding to the coordinates is set. In S2040, the time-varying pattern P that appears in the infrared range brightness of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di is monitored in accordance with the first embodiment, and recognition data Dr that recognizes the light color correlated with the pattern P is output.

[0047] According to the second embodiment described above, within the outer contour 300 recognized separately for each color light-emitting unit 30 in the visible-range imaging data Dv, a time-varying pattern P can be extracted from the corresponding light-emitting pixel areas 301 in the infrared-range imaging data Di, focusing on the infrared luminance of each light-emitting unit 30. This allows for the effective use of the visible-range camera 2041 mounted on the host vehicle 2, enabling accurate recognition of the light color correlated with the extracted time-varying pattern P. In this second embodiment as well, the time-varying pattern P can be extracted from the assumed location where the highest infrared luminance is expected within the light-emitting pixel area 301 where infrared luminance is distributed in the infrared-range imaging data Di, thereby improving the accuracy of light color recognition.

[0048] (Third embodiment) The third embodiment is a modification of the first embodiment. As shown in Figure 16, in the recognition flow of the third embodiment, S3040 is executed instead of S40 in the first embodiment. Specifically, in S3030, the data output block 120 monitors the time-varying pattern P after subtraction correction, as shown in Figure 17, which subtracts the infrared luminance at a specific color light-emitting unit 30 that lights up with a reference specific light-emitting color (the signal color of a yellow signal in the third embodiment) from the infrared luminance that appears in the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared region imaging data Di. At this time, it is preferable that the monitoring of the time-varying pattern P be performed after subtraction correction of the infrared luminance at the central part 301a, where the highest luminance is expected to be in the light-emitting pixel area 301 for each color light-emitting unit 30.

[0049] In this third implementation, S3040, the combinations of the time-varying pattern Pf, in which the rate of decrease over time gradually decreases, and the time-varying pattern Pn, in which the rate of increase over time gradually decreases from zero, that appear in parallel differ depending on the lamp color, as shown in Figure 18. In Figure 18, the vertical axis represents the rate of change over time of infrared luminance, with the rate of decrease over time represented by "-" and the rate of increase over time represented by "+".

[0050] Here, during period T1 in Figure 18, when the red light switches to a green light, a time-varying pattern Pf appears on the color light-emitting unit 30 that gives the signal color of the red light, which is the source of the switch, while a time-varying pattern Pn appears on the color light-emitting unit 30 that gives the signal color of the blue light, which is the destination of the switch. During period T2 in Figure 18, when the green light switches to a yellow light, a time-varying pattern Pf appears on the color light-emitting unit 30 that gives the signal color of the blue light, which is the source of the switch, and in parallel, a time-varying pattern Pf also appears on the color light-emitting unit 30 that gives the signal color of the red light, which is not subject to the switch. During period T3 in Figure 18, when the yellow light switches to a red light, a time-varying pattern Pn appears on the color light-emitting unit 30 that gives the signal color of the red light, which is the destination of the switch, and in parallel, a time-varying pattern Pn also appears on the color light-emitting unit 30 that gives the signal color of the blue light, which is not subject to the switch. Furthermore, in all of these switching operations, the time-varying pattern P relating to the infrared luminance of the specific emission color (the signal color of the yellow traffic light in the third embodiment), which is used as the basis for the subtractive correction described above, will be held at a value of virtually zero.

[0051] Based on the above, in S3040 of Figure 16, the light-emitting color of the color light-emitting unit 30 corresponding to the difference in the combination relationship that appears in parallel from the time-changing patterns Pf and Pn is recognized as the current light color of the traffic light 3. The current light color at this time may also be determined using, for example, position information regarding the arrangement of each color light-emitting unit 30 and / or color information regarding the light-emitting color, which are stored in the memory 10 along with the recognition model described above.

[0052] According to this third embodiment, the influence of changes in lighting temperature due to, for example, air cooling can be suppressed in the time-varying pattern P of the infrared luminance of each color light-emitting unit 30, which has been subtractively corrected by the infrared luminance of the color light-emitting unit 30 that lights up in a specific color. Therefore, the light color correlated with the time-varying pattern P can be accurately recognized.

[0053] (Other embodiments) Although several embodiments have been described above, this disclosure is not limited to those embodiments and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0054] In the modified embodiments of the first to third embodiments, the dedicated computer constituting the recognition system 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Such a digital circuit may also have a memory that stores a program.

[0055] In a modified version of the second embodiment, in step S2030 of the recognition flow, it may be determined whether or not the traffic light 3 was captured within the common field of view Ac of the imaging field of view Av, based on the visible-range brightness of each pixel in the visible-range imaging data Do. In a modified version of the second embodiment, in step S2040 of the recognition flow, subtraction correction of infrared-range brightness, similar to that in step S3040 of the third embodiment, may be applied.

[0056] In a modified version of the second embodiment, the number of pixels used to capture the common field of view Ac in the visible-range image sensor 2410 and the number of pixels used to capture the common field of view Ac in the infrared-range image sensor 400 may be different, such that the former is less than the latter. In this case, the visible-range captured data Dv and the infrared-range captured data Di will be such that the former is low-resolution captured data with lower resolution relative to the common field of view Ac, while the latter is high-resolution captured data with higher resolution than the said low-resolution captured data.

[0057] In the modifications of the first to third embodiments, the host mobile body to which the recognition system 1 is applied may be, for example, an autonomous mobile robot capable of transporting luggage or collecting information by autonomous or remote driving. In addition to the embodiments described so far, the above embodiments and modifications may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a recognition device configured to be mounted on a host mobile body and having at least one processor 12 and one memory 10.

[0058] (Additional note) This specification discloses several technical concepts and several combinations thereof, as listed below. The symbols in parentheses in this supplementary section indicate correspondences with the specific means described in the embodiments detailed above, and do not limit the technical scope of this disclosure.

[0059] (Technical thought 1) A recognition system having a processor (12) that performs recognition processing of a signal light (3) from a host mobile device (2), The aforementioned processor, The host mobile device acquires infrared imaging data (Di) relating to multiple color light-emitting units (30) corresponding to the switching colors of the traffic lights, as data captured by the infrared camera (40) of the traffic lights. A recognition system configured to output recognition data (Dr) that recognizes the color of the light that correlates with the time-varying patterns (P, Pf, Pn) of the infrared luminance of each of the color light-emitting units in the infrared imaging data.

[0060] (Technical thought 2) Outputting the aforementioned recognition data means A recognition system according to technical concept 1, which includes recognizing the light emission color of the color light emission unit as the light color, based on a combination that appears in parallel from the time change pattern (Pf) of the color light emission unit in which the time decrease rate of the infrared range brightness changes gradually, and the time change pattern (Pn) of the color light emission unit in which the time increase rate of the infrared range brightness changes gradually.

[0061] (Technical Thought 3) Outputting the aforementioned recognition data means A recognition system according to technical idea 1 or 2, which includes monitoring the time-varying pattern of the infrared luminance between identified light-emitting pixel areas (301) within the outline contour (300) recognized separately for each color light-emitting part in the infrared imaging data.

[0062] (Technical Thought 4) The aforementioned processor, The host mobile device further includes acquiring visible-range imaging data (Dv) for each of the color light-emitting parts as data captured by a visible-range camera (2041) on the traffic light, Outputting the aforementioned recognition data means A recognition system according to technical idea 1 or 2, which includes monitoring the time-varying pattern of the infrared luminance between corresponding light-emitting pixel areas (301) in the infrared region, within the outline contour (300) recognized separately for each color light-emitting part in the visible region imaging data.

[0063] (Technical Thought 5) Extracting the aforementioned time-varying pattern is A recognition system according to technical concept 3 or 4, which includes monitoring the time-varying pattern at a location where the highest infrared luminance is expected to be present within the light-emitting pixel area where the infrared luminance is distributed in the infrared imaging data.

[0064] (Technical Thought 6) Outputting the aforementioned recognition data means A recognition system according to any one of the technical ideas 1 to 5, which includes monitoring the time-varying pattern of the infrared luminance of each of the color light-emitting units, which is corrected by subtraction based on the infrared luminance of the color light-emitting units that light up in a specific color.

[0065] (Technical Thought 7) Outputting the aforementioned recognition data means A recognition system according to any one of the technical ideas 1 to 6, which includes outputting recognition data in response to the fact that the lighting order of each of the color light-emitting units recognized from the infrared imaging data differs from a predetermined order specified for each of the color light-emitting units, thereby notifying a recognition error of the light color.

[0066] Furthermore, the technical concepts 1 to 7 described above may be understood within the respective technical concepts of the apparatus, method, and program. [Explanation of Symbols]

[0067] 1: Recognition system, 2: Host vehicle, 3: Traffic light, 10: Memory, 12: Processor, 30: Color light emitter, 40: Infrared camera, 300: Outline contour, 301: Light-emitting pixel area, 2041: Visible-range camera, Di: Infrared imaging data, Dr: Recognition data, Dv: Visible-range imaging data, P, Pf, Pn: Time-varying patterns

Claims

1. A recognition system having a processor (12) that performs recognition processing of a signal light (3) from a host mobile body (2), The aforementioned processor, The host mobile device acquires infrared imaging data (Di) relating to multiple color light-emitting units (30) corresponding to the switching colors of the traffic light, as data captured by the infrared camera (40) of the traffic light in the host mobile device. A recognition system configured to output recognition data (Dr) that recognizes the color of the light that correlates with the time-varying patterns (P, Pf, Pn) of the infrared luminance of each of the color light-emitting units in the infrared imaging data.

2. Outputting the aforementioned recognition data means The recognition system according to claim 1, which includes recognizing the light emission color of the color light emission unit as the light color, based on a combination that appears in parallel from the time change pattern (Pf) of the color light emission unit in which the time decrease rate of the infrared luminance changes gradually, and the time change pattern (Pn) of the color light emission unit in which the time increase rate of the infrared luminance changes gradually.

3. Outputting the aforementioned recognition data means The recognition system according to claim 1, further comprising monitoring the time-varying pattern of the infrared luminance between identified light-emitting pixel areas (301) within the outline contour (300) recognized for each of the color light-emitting units in the infrared imaging data.

4. The aforementioned processor, The host mobile device further includes acquiring visible-range imaging data (Dv) for each of the color light-emitting parts as data captured by a visible-range camera (2041) on the traffic light, Outputting the aforementioned recognition data means The recognition system according to claim 1, further comprising monitoring the time-varying pattern of the infrared luminance between corresponding light-emitting pixel areas (301) in the infrared region, within the outline contour (300) recognized separately for each color light-emitting part in the visible region imaging data.

5. Extracting the aforementioned time-varying pattern is The recognition system according to claim 3 or 4, further comprising monitoring the time-varying pattern at a location where the highest infrared luminance is expected to be present within the light-emitting pixel area where the infrared luminance is distributed in the infrared imaging data.

6. Outputting the aforementioned recognition data means The recognition system according to claim 1, which includes monitoring the time-varying pattern of the infrared luminance of each of the color light-emitting units, which is corrected by subtraction based on the infrared luminance of the color light-emitting units that light up in a specific color.

7. Outputting the aforementioned recognition data means The recognition system according to claim 1, further comprising outputting recognition data in response to the fact that the lighting order of each of the color light-emitting units recognized from the infrared imaging data differs from a predetermined order specified for each of the color light-emitting units, thereby notifying a recognition error of the light color.

8. A recognition device having a processor (12), configured to be mounted on a host mobile body (2), and performing recognition processing of a signal light (3) from the host mobile body, The aforementioned processor, The host mobile device acquires infrared imaging data (Di) relating to multiple color light-emitting units (30) corresponding to the switching colors of the traffic light, as data captured by the infrared camera (40) of the traffic light in the host mobile device. A recognition device configured to output recognition data (Dr) that recognizes the color of the lamp that correlates with the time-varying patterns (P, Pf, Pn) of the infrared luminance of each of the color light-emitting units in the infrared imaging data.

9. A recognition method performed by a processor (12) in order to perform recognition processing of a signal light (3) from a host mobile body (2), The host mobile device acquires infrared imaging data (Di) relating to multiple color light-emitting units (30) corresponding to the switching colors of the traffic light, as data captured by the infrared camera (40) of the traffic light in the host mobile device. A recognition method that includes outputting recognition data (Dr) that recognizes the color of the light that correlates with the time change pattern (P, Pf, Pn) of the infrared luminance of each of the color light-emitting parts in the infrared region imaging data.

10. A recognition program that includes instructions to be executed by a processor (12) which performs the recognition process of a signal light (3) from a host mobile body (2), and which is stored in a storage medium (10) in order to perform the recognition process of the signal light (3), The host mobile device acquires infrared imaging data (Di) relating to multiple color light-emitting units (30) corresponding to the switching colors of the traffic light, as data captured by the infrared camera (40) of the traffic light in the host mobile device. A recognition program including the command to output recognition data (Dr) that recognizes the color of the light that correlates with the time change pattern (P, Pf, Pn) of the infrared luminance of each of the color light-emitting units in the infrared imaging data.

Images