Recognition system, recognition device, recognition method, and recognition program

Abstract

To provide a recognition system effective for recognition of a traffic light from a mobile object.SOLUTION: A processor of a recognition system which implements recognition processing of a traffic light from a host mobile object is configured to execute steps of: acquiring infrared region photography data Di regarding a plurality of color light emission parts corresponding to light colors switching in the traffic light as data in which the traffic light is photographed by an infrared camera in the host mobile object; and outputting recognition data Dr obtained by recognizing the light colors correlated with a time variation pattern P among infrared region brightness of the respective color light emission parts in the infrared region photography data Di.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a recognition technology for performing recognition processing of traffic lights from a moving object. [Background technology]

[0002] In the technology disclosed in Patent Document 1, traffic signal operation information is transmitted from a wireless transmitter, and the operation information is recognized by a vehicle serving as a moving body. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 2806801 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, if a failure occurs in the wireless communication between the vehicle and the wireless transmitter, the vehicle is prevented from recognizing the traffic light.

[0005] An object of the present disclosure is to provide a recognition system that is effective for recognizing traffic lights from a mobile body. Another object of the present disclosure is to provide a recognition device that is effective for recognizing traffic lights from a mobile body. Yet another object of the present disclosure is to provide a recognition method that is effective for recognizing traffic lights from a mobile body. Yet another object of the present disclosure is to provide a recognition program that is effective for recognizing traffic lights from a mobile body. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the correspondence with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A recognition system having a processor (12) for performing recognition processing of a traffic light (3) from a host mobile unit (2), The processor Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; The system is configured to output recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range luminance of each color light-emitting element in the infrared range shooting data.

[0008] A second aspect of the present disclosure is A recognition device having a processor (12), configured to be mountable on a host mobile body (2), and performing recognition processing of a traffic light (3) from the host mobile body, The processor Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; The system is configured to output recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range luminance of each color light-emitting element in the infrared range shooting data.

[0009] A third aspect of the present disclosure is A recognition method executed by a processor (12) to perform a recognition process of a traffic light (3) from a host mobile (2), comprising: Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; This includes outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range luminance of each color light-emitting element in the infrared range shooting data.

[0010] A fourth aspect of the present disclosure is A recognition program stored in a storage medium (10) for performing a recognition process of a traffic light (3) from a host mobile body (2), the program including instructions to be executed by a processor (12) that performs the recognition process, Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range luminance of each color light-emitting element in the infrared range shooting data.

[0011] In the first to fourth aspects, infrared imaging data for multiple color light-emitting elements corresponding to the changing light colors of a traffic light is acquired as data of the traffic light captured by an infrared camera in a host mobile body. In this case, the infrared luminance of each color light-emitting element in the infrared imaging data can track the light emission temperature of each color light-emitting element, which changes over time as the light color changes. Therefore, according to the first to fourth aspects, it is possible to accurately recognize the light color that correlates with the time-varying pattern of the infrared luminance of each color light-emitting element in the infrared imaging data, and output recognition data. In other words, the first to fourth aspects make it possible to provide a recognition process that is effective for recognizing traffic lights from a host mobile body. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of a first embodiment. [Figure 2] FIG. 2 is a plan view showing a state in which a camera is mounted on a host vehicle to which the first embodiment is applied. [Figure 3]FIG. 1 is a block diagram showing a functional configuration of a recognition system according to a first embodiment. [Figure 4] FIG. 1 is an external view showing a traffic light recognized by a first embodiment. [Figure 5] FIG. 1 is an external view showing a traffic light to be recognized according to a first embodiment. [Figure 6] 1 is a flowchart showing a recognition flow according to a first embodiment. [Figure 7] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 8] FIG. 4 is a schematic diagram for explaining a recognition flow according to the first embodiment. [Figure 9] 10 is a graph for explaining a recognition flow according to the first embodiment. [Figure 10] 10 is a graph for explaining a recognition flow according to the first embodiment. [Figure 11] FIG. 10 is a block diagram showing the overall configuration of a second embodiment. [Figure 12] FIG. 10 is a plan view showing a state in which a camera is mounted on a host vehicle to which a second embodiment is applied. [Figure 13] FIG. 10 is a block diagram showing the functional configuration of a recognition system according to a second embodiment. [Figure 14] 10 is a flowchart showing a recognition flow according to a second embodiment. [Figure 15] FIG. 10 is a schematic diagram for explaining a recognition flow according to a second embodiment. [Figure 16] 10 is a flowchart showing a recognition flow according to a third embodiment. [Figure 17] 10 is a graph for explaining a recognition flow according to a third embodiment. [Figure 18] 10 is a graph for explaining a recognition flow according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0014] As shown in FIG. 1, the recognition system 1 of the first embodiment performs recognition processing to recognize a traffic light 3 (see FIGS. 4 and 5 described later) from a host vehicle 2 shown in FIG. 2 as a host moving object. The host vehicle 2 can be said to be an ego-vehicle from a viewpoint centered on the host vehicle 2. The host vehicle 2 is a moving object, such as an automobile, that can travel on a roadway with an occupant on board. Therefore, directions in the following description are defined based on the host vehicle 2 on a horizontal plane.

[0015] The host vehicle 2 is provided with an autonomous driving mode that is divided into levels according to the degree of manual intervention by the occupant in the dynamic driving task. The autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, high driving automation, or full driving automation, in which the system performs all dynamic driving tasks when activated. The autonomous driving mode may also be realized by advanced driving assistance control, such as driving assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The autonomous driving mode may be realized by either autonomous driving control or advanced driving assistance control, or by a combination of these, or by switching between them.

[0016] The host vehicle 2 is equipped with a sensor system 4. As shown in FIGS. 1 to 3, the sensor system 4 includes an infrared camera 40. The infrared camera 40 includes an infrared imaging element 400 and an infrared imaging circuit 402. The infrared imaging element 400 is a semiconductor element, such as a microbolometer, having a plurality of pixels arranged vertically and horizontally. The infrared imaging element 400 captures an infrared light image, pixel by pixel, of a target present within the imaging field of view Ai, receiving light in the infrared light range, particularly in the far-infrared light range (e.g., a wavelength range of 7 to 14 μm, etc.). The infrared imaging circuit 402 is a semiconductor chip, such as an image processing circuit, that processes an imaging signal from each pixel of the infrared imaging element 400. The infrared imaging circuit 402 converts the infrared light intensity of each pixel according to the light receiving energy of the infrared light image from the imaging field of view Ai into two-dimensional data, thereby outputting infrared imaging data Di.

[0017] As shown in Figures 4 and 5, the traffic light 3 is equipped with color light-emitting units 30 that correspond to multiple light colors and provide different signal colors to traffic participants. In the traffic light 3, the color light-emitting units 30 turn on and off in a prescribed order defined by traffic regulations, thereby switching the light color of the lit color light-emitting units 30, i.e., the light color in the lit state. Note that the traffic lights 3 targeted by the recognition system 1 are at least one of the types for vehicles shown in Figure 4 and for pedestrians shown in Figure 5. However, to make the following explanation easier to understand, the recognition process for the former type of traffic light 3 for vehicles will be representatively discussed.

[0018] 1, the recognition system 1 is configured to include at least one dedicated computer. The recognition system 1 is connected to a sensor system 4 via at least one of, for example, a LAN (Local Area Network) line, a wire harness, an internal bus, or a wireless communication line. When the recognition system 1 is configured with multiple dedicated computers, the connections between these dedicated computers are similar.

[0019] The dedicated computer constituting the recognition system 1 may be a driving control ECU (Electronic Control Unit) that controls the driving of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a navigation ECU that navigates the driving route of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a locator ECU that estimates the self-state quantity of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an actuator ECU that controls the driving actuator of the host vehicle 2. The dedicated computer constituting the recognition system 1 may be an HCU (Human Machine Interface (HMI) Control Unit) that controls the presentation of information in the host vehicle 2. The dedicated computer constituting the recognition system 1 may be a computer other than the host vehicle 2 that constitutes an external center or mobile terminal that can communicate via the communication system of the host vehicle 2.

[0020] The dedicated computer constituting the recognition system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs, data, and the like. Here, "storage" may refer to accumulation in which data is retained even when the host vehicle 2 is powered off, or may refer to temporary storage in which data is erased when the host vehicle 2 is powered off. The processor 12 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, a data flow processor (DFP), or a graph streaming processor (GSP).

[0021] In the recognition system 1, the processor 12 executes a plurality of instructions included in a recognition program stored in the memory 10 to perform recognition processing of the traffic light 3 from the host vehicle 2. In this way, the recognition system 1 constructs a plurality of functional blocks for performing recognition processing of the traffic light 3 from the host vehicle 2. The plurality of functional blocks constructed in the recognition system 1 include a data acquisition block 100 and a data output block 120, as shown in FIG.

[0022] The recognition method in which the recognition system 1 performs the recognition process of the traffic light 3 from the host vehicle 2 by cooperation of these blocks 100 and 120 is executed according to the recognition flow shown in Fig. 6. This recognition flow is executed repeatedly while the host vehicle 2 is running. Note that each "S" in this recognition flow represents each step executed by multiple commands included in the recognition program.

[0023] In S10, the data acquisition block 100 acquires infrared range imaging data Di obtained by capturing an image within an imaging field of view Ai of the outside world of the host vehicle 2 using the infrared range camera 40. In S20 following S10 in the recognition flow, the data acquisition block 100 extracts infrared range luminance for each pixel from the acquired infrared range imaging data Di as shown in Fig. 7. At this time, by using the infrared range imaging data Di acquired in the past in S10 to convert the infrared range luminance for each pixel into time-series data, noise may be reduced by, for example, low-pass filtering processing or moving average processing in the time-series direction.

[0024] In S30 following S20 in the recognition flow shown in Fig. 6, the data output block 120 determines whether or not a traffic light 3 has been captured within the imaging field of view Ai, based on the infrared luminance of each pixel in the infrared range imaging data Di. At this time, it is preferable to determine whether or not the traffic light 3 has been captured by pattern matching processing between the infrared range imaging data Di and a recognition model stored in memory 10 for recognizing the outer contour 300 (see Fig. 7) of the color light-emitting unit 30 in the traffic light 3. Note that the infrared range imaging data Di may be input to a machine learning model stored in memory 10 for recognizing the outer contour 300 of the color light-emitting unit 30 in the traffic light 3, and the result may be a recognition processing, in which the infrared range imaging data Di is input to the machine learning model stored in memory 10 for recognizing the outer contour 300 of the color light-emitting unit 30 in the traffic light 3, to determine whether or not the traffic light 3 has been captured.

[0025] 6, if a negative determination is made in S30, the current execution of the recognition flow is terminated. On the other hand, if a positive determination is made in S30, the recognition flow proceeds to S40. In S40, the data output block 120 outputs recognition data Dr that recognizes the current light color of the traffic light 3 based on the infrared range luminance of each pixel in the infrared range shooting data Di.

[0026] Specifically, in S40, a range of multiple pixels corresponding to the inside of the outer contour 300 recognized for each color light-emitting unit 30 in the infrared range imaging data Di is identified as shown by cross-hatching in Fig. 8 and set as a light-emitting pixel area 301. Therefore, in S40, a time-varying pattern P appearing in the infrared range luminance of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di is monitored as shown in Figs. 9 and 10. At this time, the monitoring of the time-varying pattern P should be performed for a central portion 301a (see Fig. 8) as an expected location where the highest infrared range luminance is expected, within the light-emitting pixel area 301 where the infrared range luminance is distributed due to, for example, uneven light emission on the outer contour 300 side for each color light-emitting unit 30.

[0027] As shown in Figure 9, when the infrared luminance of each color light-emitting element 30 transitions between on and off, infrared luminance appears in two- or three-color color light-emitting elements 30, making it difficult to accurately determine the light color based solely on that luminance. Therefore, in S40, the current light color that correlates with a specific time-varying pattern P as shown in Figure 10, which is noticeable because it appears between the light-emitting pixel areas 301 of each color light-emitting element 30, is recognized for the traffic light 3 captured in the infrared luminance imaging data Di.

[0028] Specifically, in S40, the transition of the time change rate per unit time of infrared luminance is monitored as a time change pattern P. As a result, attention is paid to a combination that appears in parallel of a time change pattern Pf in which the time decrease rate of the time change rate gradually decreases after a step change from zero to an upward direction and a time change pattern Pn in which the time increase rate of the time change rate gradually decreases after a step change from zero to an upward direction. In this case, particularly in the first embodiment, the time change pattern Pf appears unique to a color light-emitting unit 30 that is transitioning from a lit state to an extinguished state, while the time change pattern Pn appears unique to another color light-emitting unit 30 that is transitioning from an extinguished state to a lit state. Note that the vertical axis in FIG. 10 represents the time change rate of infrared luminance, with the time decrease rate indicated by "-" and the time increase rate indicated by "+."

[0029] 10, during which the red light changes to a green light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (red) of the red light, which is the source of the change, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (blue, green, or blue-green) of the green light, which is the destination of the change. During period T2 in FIG. 10, during which the green light changes to a yellow light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the destination of the change, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the destination of the change. During period T3 in FIG. 10, during which the yellow light changes to a red light, a time-changing pattern Pf appears on the color light-emitting unit 30 that provides the signal color (yellow) of the yellow light, which is the source of the change, while a time-changing pattern Pn appears on the color light-emitting unit 30 that provides the signal color (red) of the red light, which is the destination of the change.

[0030] 6, when a parallel combination of time-change patterns Pf and Pn appears, the light color of the color light-emitting unit 30 corresponding to the time-change pattern Pn in which the time growth rate changes is recognized as the current light color of the traffic light 3. In this case, the current light color may be determined using, for example, positional information regarding the aligned positions of each color light-emitting unit 30 and / or color information regarding the light color, which are stored in memory 10 together with the above-mentioned recognition model.

[0031] In addition, in S40, the light color may be recognized using not only the time-varying pattern P but also coordination information acquired from an external center and / or infrastructure system via the communication unit of the host vehicle 2. In S10, if the traffic light 3 is for both vehicles and pedestrians, the recognition of the light color at one may also utilize the recognition result of the light color at the other.

[0032] As described above, in S40, when the current light color is recognized, the recognition data Dr is generated to represent the current light color and is output as shown in Figures 1 and 3. At this time, the recognition data Dr may be output by storing the data in memory 10. The recognition data Dr may also be output by providing the data to, for example, a driving control ECU in the host vehicle 2. The recognition data Dr may also be output by transmitting the data to an external center via a communication unit in the host vehicle 2.

[0033] However, if it is determined that the lighting order of the color light-emitting units 30 differs from the specified order based on the light colors recognized in a past (i.e., previous or previous) S40 and the light colors recognized in the current (i.e., current) S40, recognition data Dr may be generated and output in response to the determination to notify a light color recognition error. The specified order in this case may be compared with the recognized lighting order using, for example, sequence information regarding the specified order of the color light-emitting units 30 stored in memory 10 together with the above-mentioned recognition model.

[0034] (Action and effect) The effects of the first embodiment described above will be explained below.

[0035] As described above, in the first embodiment, infrared imaging data Di for the multiple color light-emitting elements 30 corresponding to the changing light colors of the traffic light 3 is acquired as data of the traffic light 3 captured by the infrared camera 40 in the host vehicle 2. At this time, the infrared luminance of each color light-emitting element 30 in the infrared imaging data Di can track the light emission temperature of each color light-emitting element 30, which changes over time as the light color changes. Therefore, according to the first embodiment, it is possible to accurately recognize the light color that correlates with the time-varying pattern P of the infrared luminance of each color light-emitting element 30 in the infrared imaging data Di, and output recognition data Dr. In other words, according to the first embodiment, it is possible to provide a recognition process that is effective for recognizing the traffic light 3 from the host vehicle 2.

[0036] According to the first embodiment, the light color of the color light-emitting unit 30 corresponding to the combination of the time-varying pattern Pf of the color light-emitting unit 30, in which the rate of decrease in infrared luminance over time changes gradually, and the time-varying pattern Pn of the color light-emitting unit 30, in which the rate of increase in infrared luminance over time changes gradually, that appear in parallel can be reflected as the light color. Therefore, in this case, the light color correlated with the time-varying pattern P can be accurately recognized.

[0037] According to the first embodiment, for light emitting pixel areas 301 identified within the outer contour 300 recognized for each color light emitting element 30 in the infrared range imaging data Di, a time change pattern P can be extracted by narrowing down the infrared range luminance for each of those light emitting elements 30. Therefore, it is possible to accurately recognize the light color correlated with the extracted time change pattern P. Here, particularly in the first embodiment, the time change pattern P can be extracted by limiting it to an expected location where the highest infrared range luminance is expected, even among the light emitting pixel areas 301 where infrared range luminance is distributed in the infrared range imaging data Di, and therefore the accuracy of light color recognition can be improved.

[0038] According to the first embodiment, recognition data Dr is output in response to the fact that the lighting order of the color light-emitting units 30 recognized from the infrared range shooting data Di differs from the predetermined order defined for each color light-emitting unit 30. This makes it possible to prevent the host vehicle 2 from being affected by a misrecognition of the light color.

[0039] Second Embodiment The second embodiment is a modified example of the first embodiment. As shown in Figures 11 to 13, a sensor system 2004 of the second embodiment includes an infrared camera 40 and a visible camera 2041. The imaging field of view Ai of the infrared camera 40 and the imaging field of view Av of the visible camera 2041 are overlapped as shown in Figure 12 to form a common field of view Ac that can be commonly imaged by each other. Here, particularly in the second embodiment, the common field of view Ac is set in a range of the imaging field of view Ai that partially overlaps with the imaging field of view Av in both the horizontal and vertical views of the host vehicle 2.

[0040] 11 and 13, the visible range camera 2041 includes a visible range imaging element 2410 and a visible range imaging circuit 2412. The visible range imaging element 2410 is a semiconductor element, such as a CMOS, having a plurality of pixels arranged vertically and horizontally. The visible range imaging element 2410 captures a visible light image, received in the visible light range from a target present within the imaging field of view Av, pixel by pixel. The visible range imaging circuit 2412 is a semiconductor chip, such as an image processing circuit, that processes imaging signals from each pixel of the visible range imaging element 2410. The visible range imaging circuit 2412 outputs visible range imaging data Dv by converting the visible range luminance for each pixel according to the received light intensity of the visible light image from within the imaging field of view Av into two-dimensional data.

[0041] Here, particularly in the second embodiment, the number of pixels for capturing the common field of view Ac in the visible range image capture element 2410 and the number of pixels for capturing the common field of view Ac in the infrared range image capture element 400 are different from each other so that the latter is smaller than the former. As a result, between the visible range image capture data Dv and the infrared range image capture data Di, the latter is low-resolution image capture data with lower resolution than the common field of view Ac, while the former is high-resolution image capture data with higher resolution than the low-resolution image capture data.

[0042] 14 of the second embodiment, steps S2010, S2020, S2030, and S2040 are executed instead of steps S10, S20, and S40 of the first embodiment. Specifically, in step S2010, the data acquisition block 100 acquires infrared range image data Di obtained by capturing an image field of view Ai of the outside world of the host vehicle 2 using the infrared camera 40, and visible range image data Dv obtained by capturing an image field of view Av of the outside world using the visible range camera 2041. At this time, the acquisition timings of the data Di and Dv may be substantially synchronized.

[0043] In S2020 following S2010 in the recognition flow, the data acquisition block 100 extracts the infrared range luminance for each pixel from the acquired infrared range imaging data Di, as in the first embodiment. At the same time, the data acquisition block 100 in S2020 extracts the visible range luminance for each pixel from the acquired visible range imaging data Dv, as shown in FIG.

[0044] 14, in S2030 following S2020, the data output block 120 determines whether or not a traffic light 3 has been captured within the common field of view Ac of the imaging field of view Ai, based on the infrared luminance of each pixel in the infrared imaging data Di. At this time, the determination of whether or not the traffic light 3 has been captured conforms to the first embodiment.

[0045] In the recognition flow, the process proceeds from a positive determination in S2030 to S2040, where the data output block 120 recognizes the outer contour 300 (see FIG. 15 ) of each color light-emitting unit 30 from the visible range imaging data Dv based on the visible range luminance of each pixel in the data Dv. At this time, it is preferable to perform a pattern matching process between the visible range imaging data Do and a recognition model stored in memory 10 in order to recognize the outer contour 300 of each color light-emitting unit 30 in the traffic light 3. Note that the recognition process may be performed by inputting the visible range imaging data Dv into a machine learning model stored in memory 10 in order to recognize the outer contour 300 of each color light-emitting unit 30 in the traffic light 3.

[0046] In S2040, for a range of multiple pixels within the outline 300 recognized for each color light-emitting unit 30 in the visible range imaging data Dv, a range of multiple pixels in the infrared range imaging data Di with corresponding coordinates is set as the light-emitting pixel area 301. In S2040, the time-varying pattern P appearing in the infrared range luminance of the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range imaging data Di is monitored in accordance with the first embodiment, and recognition data Dr that recognizes the light color that correlates with the pattern P is output.

[0047] According to the second embodiment described above, for corresponding light-emitting pixel areas 301 in the infrared range imaging data Di within the outer contour 300 recognized for each color light-emitting unit 30 in the visible range imaging data Dv, a time-varying pattern P can be extracted by narrowing down to the infrared range luminance for each of those light-emitting units 30. This makes it possible to accurately recognize the light color that correlates with the extracted time-varying pattern P by effectively utilizing the visible range camera 2041 mounted on the host vehicle 2. Here, in the second embodiment as well, the time-varying pattern P can be extracted by limiting it to the expected location where the highest infrared range luminance is expected to be, among the light-emitting pixel areas 301 where infrared range luminance is distributed in the infrared range imaging data Di, and therefore the accuracy of light color recognition can be improved.

[0048] (Third embodiment) The third embodiment is a modification of the first embodiment. As shown in Fig. 16 , the recognition flow of the third embodiment executes S3040 instead of S40 of the first embodiment. Specifically, in S3030, the data output block 120 monitors the time-varying pattern P after subtraction correction, as shown in Fig. 17 , in which the infrared range luminance of the specific color light-emitting unit 30 lit in the reference specific light-emitting color (the signal color of a yellow light in the third embodiment) is subtracted from the infrared range luminance appearing in the light-emitting pixel area 301 for each color light-emitting unit 30 in the infrared range shooting data Di. In this case, the monitoring of the time-varying pattern P may be executed after subtraction correction of the infrared range luminance in the central portion 301 a, which is expected to have the highest luminance in the light-emitting pixel area 301 for each color light-emitting unit 30.

[0049] In S3040 of the third embodiment, the combinations that appear in parallel from among a time-change pattern Pf in which the rate of decrease over time gradually decreases and a time-change pattern Pn in which the rate of increase over time gradually decreases from 0 vary depending on the light color, as shown in Fig. 18. Note that in Fig. 18 as well, the vertical axis represents the rate of change over time of infrared luminance, with the time-change rate side indicated by "-" and the time-change rate side indicated by "+".

[0050] 18, when the red light switches to a green light, a time-changing pattern Pf appears in the color light-emitting unit 30 that provides the signal color of the red light, which is the source of the switch, while a time-changing pattern Pn appears in the color light-emitting unit 30 that provides the signal color of the green light, which is the destination of the switch. During a period T2 in FIG. 18, when the green light switches to a yellow light, a time-changing pattern Pf appears in the color light-emitting unit 30 that provides the signal color of the green light, which is the source of the switch, while a time-changing pattern Pf also appears in the color light-emitting unit 30 that provides the signal color of the red light, which is not the target of the switch. During a period T3 in FIG. 18, when the yellow light switches to a red light, a time-changing pattern Pn appears in the color light-emitting unit 30 that provides the signal color of the red light, which is the destination of the switch, while a time-changing pattern Pn also appears in the color light-emitting unit 30 that provides the signal color of the green light, which is not the target of the switch. Furthermore, in either of these switching cases, the time change pattern P for the infrared range brightness of the specific light color (the signal color of a yellow light in the third embodiment) that is used as the basis for the above-mentioned subtraction correction will have its time change rate held at essentially zero.

[0051] 16, the light emission color of the color light-emitting unit 30 corresponding to the difference in the combination relationship that appears in parallel from the time-change patterns Pf and Pn is recognized as the current light color of the traffic light 3. The current light color at this time may also be determined using, for example, positional information regarding the aligned positions of each color light-emitting unit 30 and / or color information regarding the light emission color, which are stored in memory 10 together with the above-mentioned recognition model.

[0052] According to the third embodiment, the influence of changes in lighting temperature due to wind chill, for example, can be suppressed in the time-varying pattern P of the infrared luminance of each color light-emitting unit 30 that has been subtracted and corrected by using the infrared luminance of the color light-emitting unit 30 that lights up in a specific light color. Therefore, the light color that correlates with the time-varying pattern P can be accurately recognized.

[0053] (Other embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0054] In the modifications of the first to third embodiments, the dedicated computer constituting the recognition system 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), an SOC (System on a Chip), a PGA (Programmable Gate Array), and a CPLD (Complex Programmable Logic Device). Furthermore, such a digital circuit may have a memory that stores a program.

[0055] In a modification of the second embodiment, in S2030 of the recognition flow, it may be determined whether or not the traffic light 3 has been captured within the common field of view Ac of the imaging field of view Av based on the visible range luminance for each pixel in the visible range imaging data Do. In a modification of the second embodiment, in S2040 of the recognition flow, subtraction correction of the infrared range luminance similar to S3040 of the third embodiment may be applied.

[0056] In a modification of the second embodiment, the number of pixels for capturing the common field of view Ac in the visible range image capture element 2410 may be different from the number of pixels for capturing the common field of view Ac in the infrared range image capture element 400, so that the number of pixels for capturing the common field of view Ac in the visible range image capture element 2410 is smaller than the number of pixels for capturing the common field of view Ac in the infrared range image capture element 400. In this case, the visible range image capture data Dv and the infrared range image capture data Di are low-resolution image capture data with a lower resolution than the common field of view Ac, while the infrared range image capture data Di are high-resolution image capture data with a higher resolution than the low-resolution image capture data.

[0057] In the modifications of the first to third embodiments, the host moving body to which the recognition system 1 is applied may be, for example, an autonomously traveling robot that is capable of transporting luggage or collecting information by autonomous traveling or remote traveling. In addition to the forms described so far, the above-described embodiments and modifications may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) as a recognition device that is configured to be mountable on a host moving body and has at least one processor 12 and one memory 10.

[0058] (Additional remarks) This specification discloses the following technical ideas and their combinations. Note that the reference symbols in parentheses in this appendix indicate the correspondence with the specific means described in the above detailed embodiments, and do not limit the technical scope of the present disclosure.

[0059] (Technical thought 1) A recognition system having a processor (12) for performing recognition processing of a traffic light (3) from a host mobile unit (2), The processor: Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range brightness of each of the color light-emitting elements in the infrared range shooting data.

[0060] (Technical thought 2) outputting the recognition data A recognition system according to technical idea 1, which includes recognizing as the light color the light color emitted by the color light-emitting unit corresponding to a combination that appears in parallel from the time-varying pattern (Pf) of the color light-emitting unit in which the rate of decrease in the infrared range brightness over time gradually changes, and the time-varying pattern (Pn) of the color light-emitting unit in which the rate of increase in the infrared range brightness over time gradually changes.

[0061] (Technical Thought 3) outputting the recognition data The recognition system according to Technical Idea 1 or 2 includes monitoring the time-varying pattern of the infrared range luminance between specified light-emitting pixel areas (301) within the outline (300) recognized for each of the color light-emitting elements in the infrared range photography data.

[0062] (Technical Thought 4) The processor: The method further includes acquiring visible range imaging data (Dv) for each of the color light emitting units as data of the traffic light captured by a visible range camera (2041) in the host mobile body, outputting the recognition data The recognition system according to Technical Idea 1 or 2 includes monitoring the time-varying pattern of the infrared range luminance between corresponding light-emitting pixel areas (301) in the infrared range photography data within the outline (300) recognized for each of the color light-emitting elements in the visible range photography data.

[0063] (Technical Thought 5) Extracting the time change pattern includes: The recognition system according to Technical Idea 3 or 4 includes monitoring the time change pattern at an expected location where the infrared range luminance is expected to be the brightest within the light-emitting pixel area where the infrared range luminance is distributed in the infrared range photography data.

[0064] (Technical Thought 6) outputting the recognition data A recognition system described in any one of technical ideas 1 to 5, which includes monitoring the time-varying pattern of the infrared range luminance of each of the color light-emitting units, which is corrected by subtracting the infrared range luminance of the color light-emitting unit that is lit in a specific light color.

[0065] (Technical Thought 7) outputting the recognition data A recognition system described in any one of technical ideas 1 to 6, which includes outputting the recognition data so as to notify a recognition error of the light color in response to the lighting order of each of the color light-emitting units recognized from the infrared range shooting data being different from the specified order specified for each of the color light-emitting units.

[0066] The above-mentioned technical concepts 1 to 7 may be understood as the respective technical concepts of an apparatus, a method, and a program. [Explanation of symbols]

[0067] 1: Recognition system, 2: Host vehicle, 3: Traffic light, 10: Memory, 12: Processor, 30: Color light emitting unit, 40: Infrared camera, 300: Outline contour, 301: Light emitting pixel area, 2041: Visible camera, Di: Infrared imaging data, Dr: Recognition data, Dv: Visible imaging data, P, Pf, Pn: Time change pattern

Claims

1. A recognition system having a processor (12) for performing recognition processing of a traffic light (3) from a host mobile unit (2), comprising: The processor: Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range brightness of each of the color light-emitting elements in the infrared range shooting data.

2. outputting the recognition data 2. The recognition system according to claim 1, further comprising: recognizing, as the light color, the light color of the color light-emitting unit corresponding to a combination that appears in parallel from the time-varying pattern (Pf) of the color light-emitting unit in which the rate of decrease in the infrared range brightness gradually changes and the time-varying pattern (Pn) of the color light-emitting unit in which the rate of increase in the infrared range brightness gradually changes.

3. outputting the recognition data 2. The recognition system according to claim 1, further comprising monitoring the time-varying pattern of the infrared range luminance between specified light-emitting pixel areas (301) within an outer contour (300) recognized for each of the color light-emitting units in the infrared range photography data.

4. The processor: The method further includes acquiring visible range imaging data (Dv) for each of the color light emitting units as data of the traffic light captured by a visible range camera (2041) in the host mobile body, outputting the recognition data 2. The recognition system according to claim 1, further comprising monitoring the time-varying pattern of the infrared range luminance between corresponding light-emitting pixel areas (301) in the infrared range imaging data within an outline (300) recognized for each of the color light-emitting portions in the visible range imaging data.

5. Extracting the time change pattern includes:

5. The recognition system according to claim 3, further comprising monitoring the time-varying pattern at an assumed location where the infrared range luminance is assumed to be the brightest within the light-emitting pixel area in which the infrared range luminance is distributed in the infrared range photography data.

6. outputting the recognition data The recognition system according to claim 1, further comprising monitoring the time-varying pattern of the infrared range luminance of each of the color light-emitting units, corrected by subtraction using the infrared range luminance of the color light-emitting unit that is lit in a specific light color.

7. outputting the recognition data The recognition system of claim 1 further comprises outputting the recognition data so as to notify a recognition error of the light color in response to the lighting order of each of the color light-emitting units recognized from the infrared range shooting data being different from the specified order specified for each of the color light-emitting units.

8. A recognition device having a processor (12), configured to be mountable on a host mobile body (2), and configured to perform recognition processing of a traffic light (3) from the host mobile body, The processor: Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range brightness of each of the color light-emitting elements in the infrared range shooting data.

9. A recognition method executed by a processor (12) to perform a recognition process of a traffic light (3) from a host mobile (2), comprising: Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range brightness of each of the color light-emitting elements in the infrared range shooting data.

10. A recognition program stored in a storage medium (10) for performing a recognition process of a traffic light (3) from a host mobile body (2), the recognition program including instructions to be executed by a processor (12) that performs the recognition process, Acquiring infrared imaging data (Di) relating to a plurality of color light emitting units (30) corresponding to the light colors that change at the traffic light as data of the traffic light photographed by an infrared camera (40) at the host mobile body; and outputting recognition data (Dr) that recognizes the light color that correlates with the time-varying pattern (P, Pf, Pn) between the infrared range brightness of each of the color light-emitting elements in the infrared range shooting data.

Images