Recognition system, recognition device, recognition method, and recognition program

Abstract

According to the present invention, a processor of a recognition system that performs recognition processing of a traffic signal from a host mobile body is configured to acquire data obtained by capturing images of the traffic signal with an infrared camera at the host mobile body as infrared imaging data (Di) for a plurality of color light-emitting units that correspond to light colors that are switched at the traffic signal and to output recognition data (Dr) obtained by recognition of light colors that correlate with a change-over-time pattern (P) for the infrared brightness of the color light-emitting units in the infrared imaging data (Di).

Description

Recognition system, recognition device, recognition method, recognition program CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2024-43957 filed in Japan on March 19, 2024, the contents of which are incorporated by reference in their entirety.

[0002] The present disclosure relates to a recognition technology for performing recognition processing of traffic lights from a moving object.

[0003] In the technology disclosed in Patent Document 1, traffic signal operation information is transmitted from a wireless transmitter, and the operation information is recognized by a vehicle serving as a moving body.

[0004] Patent No. 2806801

[0005] However, with the technology disclosed in Patent Document 1, if a failure occurs in the wireless communication between the vehicle and the wireless transmitter, the vehicle is prevented from recognizing the traffic light.

[0006] An object of the present disclosure is to provide a recognition system that is effective for recognizing traffic lights from a mobile body. Another object of the present disclosure is to provide a recognition device that is effective for recognizing traffic lights from a mobile body. Yet another object of the present disclosure is to provide a recognition method that is effective for recognizing traffic lights from a mobile body. Yet another object of the present disclosure is to provide a recognition program that is effective for recognizing traffic lights from a mobile body.

[0007] The technical means of the present disclosure for solving the problems will be described below.

[0008] A first aspect of the present disclosure is a recognition system having a processor that performs traffic light recognition processing from a host mobile body, wherein the processor is configured to acquire infrared imaging data regarding a plurality of color light-emitting elements corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera in the host mobile body, and output recognition data that recognizes light colors that correlate with the time-varying pattern of the infrared brightness of each color light-emitting element in the infrared imaging data.

[0009] A second aspect of the present disclosure is a recognition device having a processor, configured to be mountable on a host mobile body, and performing traffic light recognition processing from the host mobile body, wherein the processor is configured to acquire infrared imaging data regarding a plurality of color light-emitting elements corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera in the host mobile body, and output recognition data that recognizes light colors that correlate with the time-varying pattern of the infrared brightness of each color light-emitting element in the infrared imaging data.

[0010] A third aspect of the present disclosure is a recognition method executed by a processor to perform traffic light recognition processing from a host mobile body, which includes acquiring infrared imaging data for a plurality of color light-emitting elements corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera in the host mobile body, and outputting recognition data that recognizes light colors that correlate with the time-varying pattern of the infrared brightness of each color light-emitting element in the infrared imaging data.

[0011] A fourth aspect of the present disclosure is a recognition program stored in a storage medium for performing traffic light recognition processing from a host mobile body, and including instructions for causing a processor that performs the recognition processing to execute the program, the instructions including instructions for acquiring infrared imaging data for a plurality of color light-emitting elements corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera in the host mobile body, and outputting recognition data that recognizes light colors that correlate with the time-varying pattern of the infrared brightness of each color light-emitting element in the infrared imaging data.

[0012] In the first to fourth aspects, infrared imaging data for multiple color light-emitting elements corresponding to the changing light colors of a traffic light is acquired as data of the traffic light captured by an infrared camera in a host mobile body. In this case, the infrared luminance of each color light-emitting element in the infrared imaging data can track the light emission temperature of each color light-emitting element, which changes over time as the light color changes. Therefore, according to the first to fourth aspects, it is possible to accurately recognize the light color that correlates with the time-varying pattern of the infrared luminance of each color light-emitting element in the infrared imaging data, and output recognition data. In other words, the first to fourth aspects make it possible to provide a recognition process that is effective for recognizing traffic lights from a host mobile body.

[0013] 1 is a block diagram showing the overall configuration of a first embodiment; FIG. 2 is a plan view showing a state in which a camera is mounted on a host vehicle to which the first embodiment is applied; FIG. 3 is a block diagram showing the functional configuration of a recognition system according to the first embodiment; FIG. 4 is an external view showing a traffic light recognized by the first embodiment; FIG. 5 is an external view showing a traffic light recognized by the first embodiment; FIG. 6 is a flowchart showing a recognition flow according to the first embodiment; FIG. 7 is a schematic view for explaining the recognition flow according to the first embodiment; FIG. 8 is a graph for explaining the recognition flow according to the first embodiment; FIG. 9 is a block diagram showing the overall configuration of a second embodiment; FIG. 10 is a plan view showing a state in which a camera is mounted on a host vehicle to which the second embodiment is applied; FIG. 11 is a block diagram showing the functional configuration of a recognition system according to the second embodiment; FIG. 12 is a flowchart showing the recognition flow according to the second embodiment; FIG. 13 is a schematic view for explaining the recognition flow according to the second embodiment; FIG. 14 is a flowchart showing the recognition flow according to a third embodiment; FIG. 15 is a graph for explaining the recognition flow according to the third embodiment; FIG. 16 is a graph for explaining the recognition flow according to the third embodiment.

[0014] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0015] As shown in Figure 1, the recognition system 1 of the first embodiment performs recognition processing to recognize a traffic light 3 (see Figures 4 and 5 described below) from a host vehicle 2 shown in Figure 2 as a host moving object. The host vehicle 2 can be said to be an ego-vehicle from a viewpoint centered on the host vehicle 2. The host vehicle 2 is a moving object, such as an automobile, that can travel on a roadway with an occupant on board. Therefore, directions in the following description are defined based on the host vehicle 2 on a horizontal plane.

[0016] The host vehicle 2 is provided with an autonomous driving mode that is classified into levels according to the degree of manual intervention by the occupant in the dynamic driving task. The autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, high driving automation, or full driving automation, in which the system performs all dynamic driving tasks when activated. The autonomous driving mode may also be realized by advanced driving assistance control, such as driving assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The autonomous driving mode may be realized by either autonomous driving control or advanced driving assistance control, or by a combination of these, or by switching between them.

[0017] The host vehicle 2 is equipped with a sensor system 4. As shown in FIGS. 1 to 3 , the sensor system 4 includes an infrared camera 40. The infrared camera 40 includes an infrared imaging element 400 and an infrared imaging circuit 402. The infrared imaging element 400 is a semiconductor element, such as a microbolometer, having a plurality of pixels arranged vertically and horizontally. The infrared imaging element 400 captures an infrared light image, pixel by pixel, of a target present within the imaging field of view Ai, receiving light in the infrared light range, particularly in the far-infrared light range (e.g., a wavelength range of 7 to 14 μm). The infrared imaging circuit 402 is a semiconductor chip, such as an image processing circuit, that processes the imaging signal from each pixel of the infrared imaging element 400. The infrared imaging circuit 402 converts the infrared light intensity of each pixel corresponding to the received light energy of the infrared light image from the imaging field of view Ai into two-dimensional data, thereby outputting infrared imaging data Di.

[0018] As shown in Figures 4 and 5, the traffic light 3 is equipped with color light-emitting units 30 that correspond to multiple light colors and provide different signal colors to traffic participants. In the traffic light 3, the color light-emitting units 30 turn on and off in a prescribed sequence regulated by traffic regulations, thereby switching the light color of the lit color light-emitting units 30, i.e., the light color in the lit state. Note that the traffic lights 3 targeted by the recognition system 1 are at least one of the traffic lights for vehicles shown in Figure 4 and the traffic lights for pedestrians shown in Figure 5. However, for ease of understanding, the following description will focus on the recognition process for the former traffic light 3 for vehicles as a representative example.

[0019] 1, the recognition system 1 includes at least one dedicated computer. The recognition system 1 is connected to a sensor system 4 via at least one of, for example, a local area network (LAN) line, a wire harness, an internal bus, or a wireless communication line. When the recognition system 1 includes multiple dedicated computers, the connections between these dedicated computers are similar.

[0020] The dedicated computer constituting the recognition system 1 may be a driving control ECU (Electronic Control Unit) that controls the driving of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a navigation ECU that navigates the driving route of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a locator ECU that estimates the self-state quantity of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an actuator ECU that controls the driving actuator of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an HCU (Human Machine Interface Control Unit (HMI)) that controls the presentation of information in the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a computer other than the host vehicle 2 that constitutes an external center or mobile terminal that can communicate via the communication system of the host vehicle 2.

[0021] The dedicated computer constituting the recognition system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs, data, and the like. Here, "storage" may refer to accumulation in which data is retained even when the host vehicle 2 is powered off, or may refer to temporary storage in which data is erased when the host vehicle 2 is powered off. The processor 12 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, a data flow processor (DFP), or a graph streaming processor (GSP).

[0022] In the recognition system 1, the processor 12 executes a plurality of instructions included in the recognition program stored in the memory 10 to perform recognition processing of the traffic light 3 from the host vehicle 2. In this way, the recognition system 1 constructs a plurality of function blocks for performing recognition processing of the traffic light 3 from the host vehicle 2. The plurality of function blocks constructed in the recognition system 1 include a data acquisition block 100 and a data output block 120, as shown in FIG.

[0023] The recognition method in which the recognition system 1 performs the recognition process of the traffic light 3 from the host vehicle 2 by cooperation of these blocks 100 and 120 is executed according to the recognition flow shown in Fig. 6. This recognition flow is executed repeatedly while the host vehicle 2 is running. Note that each "S" in this recognition flow represents each step executed by multiple commands included in the recognition program.

[0024] In S10, the data acquisition block 100 acquires infrared range imaging data Di obtained by capturing an image within an imaging field of view Ai of the outside world of the host vehicle 2 using the infrared range camera 40. In S20 following S10 in the recognition flow, the data acquisition block 100 extracts infrared range luminance for each pixel from the acquired infrared range imaging data Di as shown in Fig. 7. At this time, by using the infrared range imaging data Di acquired in the past in S10 to convert the infrared range luminance for each pixel into time-series data, noise may be reduced by, for example, low-pass filtering processing or moving average processing in the time-series direction.

[0025] In S30 following S20 in the recognition flow shown in Fig. 6 , the data output block 120 determines whether or not a traffic light 3 has been captured within the imaging field of view Ai, based on the infrared range luminance for each pixel in the infrared range imaging data Di. At this time, it is preferable to determine whether or not the traffic light 3 has been captured by pattern matching processing between the infrared range imaging data Di and a recognition model stored in the memory 10 for recognizing the outer contour 300 (see Fig. 7 ) of the color light-emitting unit 30 of the traffic light 3. Note that the infrared range imaging data Di may also be input to a machine learning model stored in the memory 10 for recognizing the outer contour 300 of the color light-emitting unit 30 of the traffic light 3, and the recognition processing may be performed to determine whether or not the traffic light 3 has been captured.

[0026] 6, if a negative determination is made in S30, the current execution of the recognition flow is terminated. On the other hand, if a positive determination is made in S30, the recognition flow proceeds to S40. In S40, the data output block 120 outputs recognition data Dr that recognizes the current light color of the traffic light 3 based on the infrared range luminance of each pixel in the infrared range shooting data Di.

[0027] Specifically, in S40, a range of multiple pixels corresponding to the inside of the outer contour 300 recognized for each color light-emitting unit 30 in the infrared range imaging data Di is identified as shown by cross-hatching in Fig. 8 and set as a light-emitting pixel area 301. Therefore, in S40, a time-varying pattern P appearing in the infrared range luminance of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di is monitored as shown in Figs. 9 and 10. At this time, the monitoring of the time-varying pattern P may be performed for a central portion 301a (see Fig. 8) as an expected location where the highest infrared range luminance is expected, within the light-emitting pixel area 301 where the infrared range luminance is distributed due to, for example, uneven light emission on the outer contour 300 side for each color light-emitting unit 30.

[0028] 9, since infrared luminance appears in two- or three-color light-emitting units 30 when they are switched on and off, it is difficult to accurately determine the light color based solely on that luminance. Therefore, in S40, the current light color that is noted for its appearance in the light-emitting pixel areas 301 of each color light-emitting unit 30 and that correlates with a specific time-varying pattern P as shown in FIG. 10 is recognized for the traffic light 3 captured in the infrared range photographed data Di.

[0029] Specifically, in S40, the transition of the time rate of change per unit time of infrared luminance is monitored as a time change pattern P. As a result, attention is paid to a combination of a time change pattern Pf, in which the time rate of decrease in the time rate of change stepwise increases from zero and then gradually decreases, and a time change pattern Pn, in which the time rate of increase in the time rate of change stepwise increases from zero and then gradually decreases, that appear in parallel. In this case, particularly in the first embodiment, the time change pattern Pf appears unique to a color light-emitting unit 30 transitioning from a lit state to an extinguished state, while the time change pattern Pn appears unique to another color light-emitting unit 30 transitioning from an extinguished state to a lit state. Note that the vertical axis in FIG. 10 represents the time rate of change of infrared luminance, with the time decrease rate indicated by "-" and the time increase rate indicated by "+."

[0030] 10 , during which the red light switches to a green light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (red) of the red light, which is the source of the switch, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (blue, green, or blue-green) of the green light, which is the destination of the switch. During period T2 in FIG. 10 , during which the green light switches to a yellow light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the destination of the switch, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the destination of the switch. During period T3 in FIG. 10 , during which the yellow light switches to a red light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the source of the switch, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (red) of the red light, which is the destination of the switch.

[0031] 6, when a parallel combination of time-change patterns Pf and Pn appears, the light color of the color light-emitting unit 30 corresponding to the time-change pattern Pn in which the time growth rate changes is recognized as the current light color of the traffic light 3. In this case, the current light color may be determined using, for example, positional information regarding the aligned positions of the color light-emitting units 30 and / or color information regarding the light colors, which are stored in memory 10 together with the above-described recognition model.

[0032] In addition to the time-varying pattern P, the light color in S40 may be recognized using coordination information acquired from an external center and / or infrastructure system via the communication unit of the host vehicle 2. In S10, if the traffic light 3 is for both vehicles and pedestrians, the light color recognition at one may also use the light color recognition result at the other.

[0033] As described above, in S40, when the current light color is recognized, the recognition data Dr is generated to represent the current light color and is output as shown in Figures 1 and 3. At this time, the recognition data Dr may be output by storing the data in the memory 10. The recognition data Dr may also be output by providing the data to, for example, a driving control ECU in the host vehicle 2. The recognition data Dr may also be output by transmitting the data to an external center via a communication unit in the host vehicle 2.

[0034] However, if it is determined that the lighting order of the color light-emitting units 30 differs from the specified order based on the light colors recognized in a past (i.e., previous or previous) S40 and the light colors recognized in the current (i.e., current) S40, recognition data Dr may be generated and output in response to the determination to notify a light color recognition error. The specified order in this case may be compared with the recognized lighting order using, for example, sequence information regarding the specified order of the color light-emitting units 30 stored in memory 10 together with the above-mentioned recognition model.

[0035] (Operations and Effects) Operations and effects of the first embodiment described above will be described below.

[0036] As described above, in the first embodiment, infrared imaging data Di for the multiple color light-emitting elements 30 corresponding to the changing light colors of the traffic light 3 is acquired as data of the traffic light 3 captured by the infrared camera 40 in the host vehicle 2. At this time, the infrared luminance of each color light-emitting element 30 in the infrared imaging data Di can track the light emission temperature of each color light-emitting element 30, which changes over time as the light color changes. Therefore, according to the first embodiment, it is possible to accurately recognize the light color that correlates with the time-varying pattern P between the infrared luminance of each color light-emitting element 30 in the infrared imaging data Di, and output recognition data Dr. In other words, according to the first embodiment, it is possible to provide a recognition process that is effective for recognizing the traffic light 3 from the host vehicle 2.

[0037] According to the first embodiment, the light color of the color light-emitting unit 30 corresponding to the combination of the time-varying pattern Pf of the color light-emitting unit 30 in which the rate of decrease in infrared luminance over time gradually changes and the time-varying pattern Pn of the color light-emitting unit 30 in which the rate of increase in infrared luminance over time gradually changes can be reflected as the light color. Therefore, in this case, the light color correlated with the time-varying pattern P can be accurately recognized.

[0038] According to the first embodiment, for light-emitting pixel areas 301 identified within the outer contour 300 recognized for each color light-emitting unit 30 in the infrared range imaging data Di, a time-varying pattern P can be extracted by narrowing down the infrared range luminance for each of those light-emitting units 30. Therefore, it is possible to accurately recognize the light color correlated with the extracted time-varying pattern P. Here, particularly in the first embodiment, the time-varying pattern P can be extracted by limiting it to an expected location where the highest infrared range luminance is expected, even among the light-emitting pixel areas 301 where infrared range luminance is distributed in the infrared range imaging data Di, and therefore the accuracy of light color recognition can be improved.

[0039] According to the first embodiment, recognition data Dr is output in response to the fact that the lighting order of the color light-emitting units 30 recognized from the infrared range shooting data Di differs from the predetermined order defined for each color light-emitting unit 30. This makes it possible to prevent the host vehicle 2 from being affected by a misrecognition of the light colors.

[0040] Second Embodiment The second embodiment is a modified example of the first embodiment. As shown in Figures 11 to 13, the sensor system 2004 of the second embodiment includes an infrared camera 40 and a visible camera 2041. The imaging field of view Ai of the infrared camera 40 and the imaging field of view Av of the visible camera 2041 are overlapped as shown in Figure 12 to form a common field of view Ac that can capture images in common with each other. In particular, in the second embodiment, the common field of view Ac is set in a range of the imaging field of view Ai that partially overlaps with the imaging field of view Av in both the horizontal and vertical views of the host vehicle 2.

[0041] 11 and 13 , the visible range camera 2041 includes a visible range image sensor 2410 and a visible range image sensor circuit 2412. The visible range image sensor 2410 is a semiconductor device, such as a CMOS, having a plurality of pixels arranged vertically and horizontally. The visible range image sensor 2410 captures a visible light image, received in the visible light range from a target present within the imaging field of view Av, pixel by pixel. The visible range image sensor circuit 2412 is a semiconductor chip, such as an image processing circuit, that processes the imaging signal from each pixel of the visible range image sensor 2410. The visible range image sensor circuit 2412 outputs visible range image data Dv by converting the visible range luminance of each pixel according to the received light intensity of the visible light image from within the imaging field of view Av into two-dimensional data.

[0042] In particular, in the second embodiment, the number of pixels for capturing an image of the common field of view Ac in the visible range image capture element 2410 is different from the number of pixels for capturing an image of the common field of view Ac in the infrared range image capture element 400, so that the number of pixels for capturing an image of the common field of view Ac in the visible range image capture element 2410 is smaller than the number of pixels for capturing an image of the common field of view Ac in the infrared range image capture element 400. As a result, the visible range image capture data Dv and the infrared range image capture data Di are low-resolution image capture data with a lower resolution than the low-resolution image capture data, while the former are high-resolution image capture data with a higher resolution than the low-resolution image capture data.

[0043] 14 of the second embodiment, S2010, S2020, S2030, and S2040 are executed instead of S10, S20, and S40 of the first embodiment, respectively. Specifically, in S2010, the data acquisition block 100 acquires infrared range imaging data Di obtained by capturing an image within an imaging field of view Ai of the outside world of the host vehicle 2 using the infrared range camera 40, and visible range imaging data Dv obtained by capturing an image within an imaging field of view Av of the same outside world using the visible range camera 2041. At this time, it is preferable that the acquisition timings of the data Di and Dv are substantially synchronized.

[0044] In S2020 following S2010 in the recognition flow, the data acquisition block 100 extracts infrared luminance for each pixel from the acquired infrared imaging data Di, as in the first embodiment. At the same time, the data acquisition block 100 in S2020 extracts visible luminance for each pixel from the acquired visible imaging data Dv, as shown in FIG.

[0045] 14, in S2030 following S2020, the data output block 120 determines whether or not the traffic light 3 has been captured within the common field of view Ac of the imaging field of view Ai based on the infrared luminance of each pixel in the infrared imaging data Di. At this time, the determination of whether or not the traffic light 3 has been captured conforms to the first embodiment.

[0046] In the recognition flow, the process proceeds from a positive determination in S2030 to S2040, where the data output block 120 recognizes the outer contour 300 (see FIG. 15 ) of each color light-emitting unit 30 from the visible range imaging data Dv based on the visible range luminance of each pixel in the data Dv. At this time, a pattern matching process may be performed between the visible range imaging data Do and a recognition model stored in memory 10 in order to recognize the outer contour 300 of each color light-emitting unit 30 in the traffic light 3. Note that the visible range imaging data Dv may be input to a machine learning model stored in memory 10 in order to recognize the outer contour 300 of each color light-emitting unit 30 in the traffic light 3, and the recognition process may be performed.

[0047] In S2040, for a range of multiple pixels within the outline 300 recognized for each color light-emitting unit 30 in the visible range imaging data Dv, a range of multiple pixels in the infrared range imaging data Di with corresponding coordinates is set as the light-emitting pixel area 301. In S2040, the time-varying pattern P appearing in the infrared range luminance of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di is monitored in accordance with the first embodiment, and recognition data Dr that recognizes the light color that correlates with the pattern P is output.

[0048] According to the second embodiment described above, for corresponding light-emitting pixel areas 301 in the infrared range imaging data Di within the outer contour 300 recognized for each color light-emitting unit 30 in the visible range imaging data Dv, a time-varying pattern P can be extracted by narrowing down the infrared range luminance for each of those light-emitting units 30. This makes it possible to accurately recognize the light color correlated with the extracted time-varying pattern P by effectively utilizing the visible range camera 2041 mounted on the host vehicle 2. Here, in the second embodiment as well, the time-varying pattern P can be extracted by limiting it to the expected location where the highest infrared range luminance is expected among the light-emitting pixel areas 301 where infrared range luminance is distributed in the infrared range imaging data Di, thereby improving the accuracy of light color recognition.

[0049] Third Embodiment The third embodiment is a modification of the first embodiment. As shown in FIG. 16 , the recognition flow of the third embodiment executes S3040 instead of S40 of the first embodiment. Specifically, in S3030, the data output block 120 monitors the time-varying pattern P after subtraction correction, as shown in FIG. 17 , in which the infrared range luminance of the specific color light-emitting unit 30 lit in the reference specific light-emitting color (the signal color of a yellow light in the third embodiment) is subtracted from the infrared range luminance appearing in the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di. In this case, the monitoring of the time-varying pattern P may be performed after subtraction correction of the infrared range luminance in the central portion 301 a, which is expected to have the highest luminance in the light-emitting pixel area 301 for each color light-emitting unit 30.

[0050] In S3040 of the third embodiment, the combinations that appear in parallel from among the time change pattern Pf in which the time decrease rate gradually decreases and the time change pattern Pn in which the time increase rate gradually decreases from 0 value differ depending on the light color, as shown in Fig. 18. Note that in Fig. 18 as well, the vertical axis represents the time change rate of infrared luminance, with the time decrease rate side represented by "-" and the time increase rate side represented by "+".

[0051] 18, during which the red light switches to a green light, a time-changing pattern Pf appears in the color light-emitting unit 30 that provides the signal color of the red light, which is the source of the switch, while a time-changing pattern Pn appears in the color light-emitting unit 30 that provides the signal color of the green light, which is the destination of the switch. During period T2 in FIG. 18, during which the green light switches to a yellow light, a time-changing pattern Pf appears in the color light-emitting unit 30 that provides the signal color of the green light, which is the source of the switch, while a time-changing pattern Pf also appears in the color light-emitting unit 30 that provides the signal color of the red light, which is not the target of the switch. During period T3 in FIG. 18, during which the yellow light switches to a red light, a time-changing pattern Pn appears in the color light-emitting unit 30 that provides the signal color of the red light, which is the destination of the switch, while a time-changing pattern Pn also appears in the color light-emitting unit 30 that provides the signal color of the green light, which is not the target of the switch. In addition, in either of these switching cases, the time change pattern P for the infrared range brightness of the specific light color (the signal color of a yellow light in the third embodiment) that is used as the basis for the above-mentioned subtraction correction will have its time change rate held at essentially zero.

[0052] 16, the light color of the color light-emitting unit 30 corresponding to the difference in the combination relationship that appears in parallel from the time-varying patterns Pf and Pn is recognized as the current light color of the traffic light 3. The current light color at this time may also be determined using, for example, positional information regarding the aligned positions of each color light-emitting unit 30 and / or color information regarding the light color, which are stored in memory 10 together with the above-mentioned recognition model.

[0053] According to the third embodiment, the influence of changes in lighting temperature due to, for example, wind chill, etc., can be suppressed in the time-varying pattern P of the infrared luminance of each color light-emitting unit 30 that has been subtractively corrected by using the infrared luminance of the color light-emitting unit 30 that lights up in a specific light color. Therefore, the light color that correlates with the time-varying pattern P can be accurately recognized.

[0054] (Other Embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0055] In the modifications of the first to third embodiments, the dedicated computer constituting the recognition system 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-a-chip (SOC), a programmable gate array (PGA), and a complex programmable logic device (CPLD). Such a digital circuit may also have a memory that stores a program.

[0056] In a modification of the second embodiment, in S2030 of the recognition flow, it may be determined whether or not the traffic light 3 has been captured within the common field of view Ac of the captured field of view Av based on the visible range luminance for each pixel in the visible range imaging data Do. In a modification of the second embodiment, in S2040 of the recognition flow, subtraction correction of the infrared range luminance similar to S3040 of the third embodiment may be applied.

[0057] In a modified example of the second embodiment, the number of pixels for capturing an image of the common field of view Ac in the visible range image capture element 2410 and the number of pixels for capturing an image of the common field of view Ac in the infrared range image capture element 400 may be different from each other so that the number of pixels for capturing an image of the common field of view Ac in the visible range image capture element 2410 is smaller than the number of pixels for capturing an image of the common field of view Ac in the infrared range image capture element 400. In this case, the visible range image capture data Dv and the infrared range image capture data Di are low-resolution image capture data with a lower resolution than the low-resolution image capture data, while the infrared range image capture data are high-resolution image capture data with a higher resolution than the low-resolution image capture data.

[0058] In the modifications of the first to third embodiments, the host mobile body to which the recognition system 1 is applied may be, for example, an autonomous robot that is capable of autonomously or remotely traveling to transport luggage or collect information, etc. In addition to the forms described so far, the above-described embodiments and modifications may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a recognition device that is configured to be mountable on a host mobile body and has at least one processor 12 and one memory 10.

[0059] (Additional Remarks) This specification discloses the following technical ideas and their combinations. Note that the reference symbols in parentheses in this Additional Remarks section indicate the correspondence with the specific means described in the above detailed embodiments, and do not limit the technical scope of the present disclosure.

[0060] (Technical Idea 1) A recognition system having a processor (12) that performs recognition processing of a traffic light (3) from a host mobile body (2), wherein the processor is configured to: acquire infrared imaging data (Di) regarding a plurality of color light-emitting elements (30) corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera (40) in the host mobile body; and output recognition data (Dr) that recognizes the light colors that correlate with the time-varying pattern (P, Pf, Pn) of the infrared range brightness of each of the color light-emitting elements in the infrared imaging data.

[0061] (Technical Idea 2) The output of the recognition data is a recognition system according to Technical Idea 1, which includes recognizing as the light color the light emission color of the color light-emitting unit corresponding to a combination that appears in parallel from the time change pattern (Pf) of the color light-emitting unit in which the rate of decrease in the infrared range luminance over time gradually changes and the time change pattern (Pn) of the color light-emitting unit in which the rate of increase in the infrared range luminance over time gradually changes.

[0062] (Technical Idea 3) A recognition system according to Technical Idea 1 or 2, wherein outputting the recognition data includes monitoring the time-varying pattern of the infrared range brightness between identified light-emitting pixel areas (301) within the outer contour (300) recognized for each color light-emitting element in the infrared range photography data.

[0063] (Technical Idea 4) The processor further includes acquiring visible range imaging data (Dv) for each of the color light-emitting elements as data of the traffic light photographed by a visible range camera (2041) in the host mobile body, and outputting the recognition data includes monitoring the time-varying pattern of the infrared range brightness between corresponding light-emitting pixel areas (301) in the infrared range imaging data within the outer contour (300) recognized for each of the color light-emitting elements in the visible range imaging data, in the recognition system described in Technical Idea 1 or 2.

[0064] (Technical Idea 5) A recognition system according to Technical Idea 3 or 4, wherein extracting the time-varying pattern includes monitoring the time-varying pattern at an expected location where the infrared range luminance is expected to be the brightest, within the light-emitting pixel area where the infrared range luminance is distributed in the infrared range shooting data.

[0065] (Technical Idea 6) A recognition system described in any one of Technical Ideas 1 to 5, wherein outputting the recognition data includes monitoring the time-varying pattern of the infrared range luminance of each of the color light-emitting units, which is subtracted and corrected by the infrared range luminance of the color light-emitting unit that is lit in a specific light color.

[0066] (Technical Idea 7) A recognition system described in any one of Technical Ideas 1 to 6, wherein outputting the recognition data includes outputting the recognition data so as to notify a recognition error of the light color in response to the lighting order of each of the color light-emitting elements recognized from the infrared range shooting data being different from the specified order specified for each of the color light-emitting elements.

[0067] The above-mentioned technical concepts 1 to 7 may be understood as the respective technical concepts of an apparatus, a method, and a program.

Claims

1. A recognition system having a processor (12) that performs recognition processing of a traffic light (3) from a host mobile body (2), wherein the processor is configured to: acquire infrared imaging data (Di) relating to a plurality of color light-emitting elements (30) corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera (40) in the host mobile body; and output recognition data (Dr) that recognizes the light colors that correlate with the time-varying pattern (P, Pf, Pn) of the infrared brightness of each of the color light-emitting elements in the infrared imaging data.

2. The recognition system of claim 1, wherein outputting the recognition data includes recognizing as the light color the light color of the color light-emitting unit corresponding to a combination that appears in parallel from the time-varying pattern (Pf) of the color light-emitting unit in which the rate of decrease in the infrared range luminance over time gradually changes and the time-varying pattern (Pn) of the color light-emitting unit in which the rate of increase in the infrared range luminance over time gradually changes.

3. The recognition system of claim 1, wherein outputting the recognition data includes monitoring the time-varying pattern of the infrared range brightness between identified light-emitting pixel areas (301) within the outline (300) recognized for each color light-emitting element in the infrared range photography data.

4. The processor further includes acquiring visible range imaging data (Dv) for each of the color light-emitting elements as data of the traffic light photographed by a visible range camera (2041) in the host mobile body, and outputting the recognition data includes monitoring the time-varying pattern of the infrared range brightness between corresponding light-emitting pixel areas (301) in the infrared range imaging data within the outer contour (300) recognized for each of the color light-emitting elements in the visible range imaging data. The recognition system described in claim 1.

5. A recognition system as described in claim 3 or 4, wherein extracting the time-varying pattern includes monitoring the time-varying pattern at an expected location where the infrared range luminance is expected to be the brightest within the light-emitting pixel area where the infrared range luminance is distributed in the infrared range photography data.

6. The recognition system of claim 1, wherein outputting the recognition data includes monitoring the time-varying pattern of the infrared range luminance of each of the color light-emitting units, corrected by subtraction using the infrared range luminance of the color light-emitting unit that is lit in a specific light color.

7. The recognition system of claim 1, wherein outputting the recognition data includes outputting the recognition data so as to notify a recognition error of the light color in response to the lighting order of each of the color light-emitting elements recognized from the infrared range shooting data being different from the specified order specified for each of the color light-emitting elements.

8. A recognition device having a processor (12), configured to be mountable on a host mobile body (2), and performing recognition processing of a traffic light (3) from the host mobile body, wherein the processor is configured to: acquire infrared imaging data (Di) relating to a plurality of color light-emitting elements (30) corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera (40) in the host mobile body; and output recognition data (Dr) that recognizes the light colors that correlate with the time-varying pattern (P, Pf, Pn) of the infrared brightness of each of the color light-emitting elements in the infrared imaging data.

9. A recognition method executed by a processor (12) to perform recognition processing of a traffic light (3) from a host mobile body (2), the recognition method comprising: acquiring infrared imaging data (Di) relating to a plurality of color light-emitting elements (30) corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera (40) in the host mobile body; and outputting recognition data (Dr) that recognizes the light colors that correlate with the time-varying pattern (P, Pf, Pn) of the infrared brightness of each of the color light-emitting elements in the infrared imaging data.

10. A recognition program stored in a storage medium (10) for performing recognition processing of a traffic light (3) from a host mobile body (2), the recognition program including instructions for causing a processor (12) that performs the recognition processing to execute the following: acquiring infrared imaging data (Di) for a plurality of color light-emitting elements (30) corresponding to the light colors that change in the traffic light, as data of the traffic light photographed by an infrared camera (40) in the host mobile body; and outputting recognition data (Dr) that recognizes the light colors that correlate with the time-varying pattern (P, Pf, Pn) of the infrared brightness of each of the color light-emitting elements in the infrared imaging data.

Images