Information processing device, information processing method, information processing program, and recording medium

Abstract

To ensure reliable identification of the lighting and flashing states of a light emitter of traffic signals.SOLUTION: A signal housing detection unit (902) detects a housing (TS1) of a traffic signal (TS) from images acquired using a camera (2). A light detection unit (903) detects a light unit (TS3) of the traffic signal from the images. A positional relationship calculation unit (904) calculates the relative positional relationship between the detected housing and the light unit. A flashing score calculation unit (905) calculates a flashing score representing the likelihood that the light unit is in a flashing state, based on the relative positional relationship. A flashing determination unit (906) determines whether the light unit is in the flashing state based on the flashing score. The flashing score calculation unit continuously calculates the flashing score based on the relative positional relationship.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, an information processing program, and a computer-readable non-transient tangible recording medium on which such an information processing program is recorded. [Background technology]

[0002] Various technologies have been proposed to assist vehicle driving, such as slowing down or stopping, in accordance with traffic signals. An example of this type of driving assistance device is described in Patent Document 1. The device described in Patent Document 1 is configured to assist driving control, such as slowing down or stopping, in accordance with traffic signals. Specifically, the device acquires traffic signal displays as traffic information using a camera, and provides driving assistance such that, when the light is green, the vehicle drives in accordance with the surrounding driving environment, such as the vehicle ahead, and when the light is yellow or red, the vehicle stops in accordance with the vehicle ahead or at the stop line. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5936258 Summary of the Invention [Problem to be solved by the invention]

[0004] Traffic signals can sometimes indicate different things when they are solid and when they are flashing. For example, in Japan, a solid yellow light indicates that vehicles must stop and not proceed beyond the stopping point unless it is not safe to stop. On the other hand, a flashing yellow light indicates that vehicles can proceed while paying attention to other traffic. Also, a solid red light indicates that vehicles must stop and not proceed beyond the stopping point. On the other hand, a flashing red light indicates that vehicles must stop temporarily before proceeding.

[0005] In this regard, it is conceivable to configure the device to distinguish between the lit state and the flashing state of the lighting unit. However, the image may be obscured by leaves on roadside trees, cargo or the body of a preceding vehicle, etc. For this reason, the prior art has had a problem in reliably distinguishing between the lit state and the flashing state of the lighting unit. The present disclosure has been made in consideration of the circumstances exemplified above. [Means for solving the problem]

[0006] In one aspect of the present disclosure, an information processing device (9) a traffic signal housing detection unit (902) for detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); a lighting detection unit (903) that detects lighting units (TS3) of the traffic signal from the image; a positional relationship calculation unit (904) that calculates a relative positional relationship between the housing detected by the signal housing detection unit and the lighting unit detected by the lighting detection unit; a blinking score calculation unit (905) that calculates a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the relative positional relationship calculated by the positional relationship calculation unit; a blinking determination unit (906) that determines whether the lighting unit is in a blinking state according to the blinking score calculated by the blinking score calculation unit; Equipped with The blinking score calculation unit continuously calculates the blinking score based on the relative positional relationship. In another aspect of the present disclosure, an information processing method executed by an information processing device (9) includes: Detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); Detecting a lighting portion (TS3) of the traffic signal from the image; Calculating the relative positional relationship between the detected housing and the lighting unit; calculating a blinking score representing the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; determining whether the lighting unit is in a blinking state according to the calculated blinking score; In calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. In yet another aspect of the present disclosure, an information processing program executed by an information processing device (9) includes: The process executed by the information processing device includes: A process of detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); A process of detecting a lighting portion (TS3) of the traffic signal from the image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. In yet another aspect of the present disclosure, a computer-readable non-transient tangible recording medium having an information processing program executed by an information processing device (9) recorded thereon includes: The information processing program includes the following processes: A process of acquiring an image; A process of detecting a housing (TS1) of a traffic signal (TS) from the acquired image; A process of detecting a lighting portion (TS3) of the traffic signal from the acquired image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship.

[0007] In addition, in each section of the application documents, each element may be assigned a reference symbol in parentheses. However, such reference symbols merely indicate an example of the correspondence between the element and the specific means described in the embodiments below. Therefore, the present disclosure is not limited in any way by the above-mentioned reference symbols. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing a vehicle to which the present disclosure is applied while traveling; [Figure 2] 2 is a schematic diagram showing an example of a photographed image acquired using the camera shown in FIG. 1. FIG. [Figure 3] 2 is a block diagram showing a schematic device configuration of the in-vehicle system shown in FIG. 1. FIG. [Figure 4] 1 is a block diagram illustrating a schematic device configuration of an information processing device according to an embodiment of the present disclosure. [Figure 5] 4 is a block diagram showing a schematic functional configuration realized in the information processing device shown in FIG. 3. FIG. [Figure 6] 5 is a flowchart showing an outline of an example of the operation of the information processing device shown in FIGS. 3 and 4. [Figure 7] 6 is a schematic diagram showing an example of detection of a casing and lighting unit of a traffic light in the traffic light detection step shown in FIG. 5. FIG. [Figure 8] FIG. 6 is a schematic diagram showing an outline of processing content in a positional relationship calculation step shown in FIG. 5. [Figure 9] 6 is a schematic diagram showing an outline of processing content in a blinking score calculation step shown in FIG. 5. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment: Configuration) Hereinafter, exemplary embodiments and specific examples of the present disclosure will be described with reference to the accompanying drawings as appropriate. First, referring to Fig. 1, an in-vehicle system 1 is configured to be mounted on a vehicle V and to perform various operations in the vehicle V. Hereinafter, the vehicle V equipped with the in-vehicle system 1 will be referred to as the "host vehicle."

[0010] (In-vehicle system configuration) The in-vehicle system 1 is equipped with a camera 2 that captures images of the surroundings of the vehicle, and is configured to perform operations such as presenting information to occupants and controlling driving of the vehicle using images captured by this camera 2. "Information presentation" includes display and audio output. "Driving control" includes the execution of longitudinal vehicle motion control subtasks and / or lateral vehicle motion control subtasks. The longitudinal vehicle motion control subtasks are starting, accelerating / decelerating, and stopping. The lateral vehicle motion control subtask is steering. Typically, the in-vehicle system 1 has a configuration as, for example, a so-called driving automation system, i.e., an automated driving system and / or a driving assistance system.

[0011] The camera 2 is equipped with an image sensor such as a CCD or CMOS, and is mounted at a predetermined position on the host vehicle to capture images of targets around the host vehicle. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor. "Targets" include objects that may be obstacles to driving, such as pedestrians and other vehicles, and three-dimensional objects such as road signs, as well as two-dimensional road markings. In this embodiment, the host vehicle is equipped with at least a forward camera as the camera 2.

[0012] FIG. 2 shows an example of an image Pg of the area ahead of the vehicle captured by a front camera serving as camera 2. The front camera is configured to capture an image of the road Rd ahead of the vehicle, i.e., the road ahead, and objects (e.g., traffic signals TS) on or around the road Rd. The traffic signal TS includes a housing TS1 and a lightable area TS2. As shown in FIG. 2, the traffic signal TS has at least a plurality of, typically three, circular lightable areas TS2 arranged along the longitudinal direction of the housing TS1 in a front view. Although not shown, in addition to or instead of these circular lightable areas TS2, lightable areas TS2 shaped like upward or horizontal arrows may be provided. Furthermore, the lightable areas TS2 may be intended display areas within a rectangular display area (e.g., a number of light-emitting elements arranged two-dimensionally vertically and horizontally). The lightable areas TS2 that are actually emitting light, i.e., lit, are referred to as lighting sections TS3. "Lighting" includes not only continuous lighting within a predetermined period of time but also "flashing."

[0013] Referring to FIG. 3, the in-vehicle system 1 includes, in addition to a camera 2, an in-vehicle sensor 3, a navigation device 4, a communication device 5, a target detection device 6, an HMI device 7, and a vehicle control device 8. HMI stands for Human Machine Interface. The camera 2, the in-vehicle sensor 3, the navigation device 4, and the communication device 5 are connected to the target detection device 6 via an in-vehicle network so as to be able to exchange information or signals. The HMI device 7 and the vehicle control device 8 are also connected to the target detection device 6 via the in-vehicle network so as to be able to exchange information or signals. The in-vehicle network is configured to comply with a predetermined communication standard such as CAN (international registered trademark: International Registration Number 1048262A). CAN (international registered trademark) is an abbreviation for Controller Area Network. Note that the in-vehicle network may have, in addition to a main network conforming to CAN (international registered trademark), another main network or sub-network conforming to LIN, FlexRay, or the like. LIN stands for Local Interconnect Network.

[0014] The on-board sensors 3 are configured to detect various quantities related to the driving state of the host vehicle. The "driving state" includes the driving operation state, driving behavior state, and driving environment state of the host vehicle. The "driving operation state" refers to the state related to the driving operation input of the host vehicle by the driver of the host vehicle or the vehicle control device 8 described later, and includes, for example, the steering amount, throttle opening, brake operation amount, shift range, etc. In other words, the on-board sensors 3 include an accelerator pedal sensor, a brake pedal sensor, a shift position sensor, a steering angle sensor, etc. The "driving behavior state" refers to the state related to the motion, i.e., physical behavior, of the host vehicle, and includes, for example, vehicle speed, acceleration, yaw rate, etc. In other words, the on-board sensors 3 include a vehicle speed sensor, a yaw rate sensor, an acceleration sensor, etc. The "driving environment state" refers to the environment around the host vehicle, and includes, for example, the illuminance, weather, outside temperature, road surface condition, the presence of objects such as pedestrians and other vehicles, etc. That is, the on-board sensors 3 include an illuminance sensor, a raindrop sensor, an outside air temperature sensor, a radar sensor, a laser radar sensor, a sonar sensor, etc. Among the on-board sensors 3, a sensor related to the presence state of an object is called an ADAS sensor. ADAS stands for Advanced Driver-Assistance Systems. The ADAS sensor may include a camera 2.

[0015] The navigation device 4 is configured to be able to output information relating to the current position and driving route of the vehicle. The navigation device 4 also includes a map information database 41. That is, the navigation device 4 is configured to be able to output the planned driving route and attribute information of the road Rd on which the vehicle is currently traveling, along with the current position and map information of the vehicle, to the target detection device 6, the HMI device 7, and the vehicle control device 8. The attribute information of the road Rd includes the speed limit, road width, curve curvature, gradient, etc.

[0016] The communication device 5 is an in-vehicle communication module also referred to as DCM, and is configured to be able to communicate information with an external server Z via base stations around the vehicle using wireless communication compliant with communication standards such as LTE or 5G. DCM is an abbreviation for Data Communication Module. LTE is an abbreviation for Long Term Evolution. 5G is an abbreviation for 5th Generation. The communication device 5 is configured to be able to acquire various information such as road traffic information such as congestion information and the latest map information from the external server Z and output it to the target detection device 6, the HMI device 7, and the vehicle control device 8.

[0017] The target detection device 6 is configured to detect targets around the vehicle based on information and signals acquired from the camera 2, the on-board sensor 3, etc. Furthermore, the target detection device 6 generates and outputs signals required for information presentation in the HMI device 7 and driving control in the vehicle control device 8 based on the target detection results. In other words, the target detection device 6 is configured as an electronic circuit unit called an image processing ECU, a target recognition ECU, or a target detection ECU. ECU is an abbreviation for Electronic Control Unit.

[0018] The HMI device 7 includes a display device, an audio output device, and the like for presenting various types of information and warnings to the occupants of the vehicle. The display device may include a meter, a meter display, a center information display, a head-up display, an electronic mirror, and the like. The vehicle control device 8 is configured as a so-called driving ECU, which is an on-board computer that controls the driving force generation mechanism, driving force transmission mechanism, braking mechanism, steering mechanism, and the like of the vehicle. That is, the vehicle control device 8 is configured to execute longitudinal and / or lateral motion control of the vehicle. More specifically, the vehicle control device 8 is configured to be able to execute at least a part of the motion control of the vehicle, such as starting, acceleration / deceleration, braking, stopping, steering, and the like.

[0019] (Information processing device) Fig. 4 shows a schematic configuration of an information processing device 9 according to an embodiment of the present disclosure. The information processing device 9 is an on-board microcomputer provided in the camera 2 or the target detection device 6 shown in Fig. 3, and includes at least a processor 91 and a memory 92. For the sake of simplicity of explanation, in the present embodiment, the information processing device 9 is assumed to be realized in the target detection device 6 as an exemplary implementation form.

[0020] The processor 91 includes at least one arithmetic unit configured as a CPU or MPU and its peripheral circuits (e.g., a timer circuit, etc.). The memory 92 includes at least a RAM and a ROM or a nonvolatile rewritable memory among various non-transient physical storage media such as a ROM, a RAM, and a nonvolatile rewritable memory. The nonvolatile rewritable memory is a storage device that allows information to be rewritten while the power is on but retains information in an unrewritable manner while the power is off, such as a flash memory. The information processing device 9 is configured so that the processor 91 reads and executes a computer program from the memory 92 to realize a predetermined function for recognizing targets around the vehicle. The memory 92 stores the computer program as well as various data required to execute the program, such as initial values, maps, look-up tables, etc.

[0021] As shown in Fig. 5, the information processing device 9 has an image acquisition unit 901 to a blinking determination unit 906 as functional components realized on an in-vehicle microcomputer by executing a computer program. Each of these functional components will be described below. In the following description, unless otherwise noted, the camera 2 in this embodiment refers to the front camera. However, it goes without saying that the present disclosure is not limited to this aspect.

[0022] The image acquisition unit 901 is configured to acquire image data captured by the camera 2. That is, in this embodiment, the image acquisition unit 901 receives image data captured by the camera 2 from the camera 2 and stores a certain amount of the image data in chronological order. The traffic light casing detection unit 902 is configured to detect a casing TS1 of the traffic light TS from the image acquired by the image acquisition unit 901, i.e., the captured image Pg. As will be described later, in this embodiment, the traffic light casing detection unit 902 is configured to detect the casing TS1 using machine learning, i.e., RNN. RNN stands for Recurrent Neural Network. The lighting detection unit 903 is configured to detect a lighting unit TS3 from the captured image Pg. As will be described later, in this embodiment, the lighting detection unit 903 is configured to detect the casing TS1 using machine learning, i.e., RNN.

[0023] The positional relationship calculation unit 904 calculates the relative positional relationship between the housing TS1 detected by the signal housing detection unit 902 and the lighting unit TS3 detected by the lighting detection unit 903. The "relative positional relationship" here refers to the relative positional relationship in the captured image Pg or an image obtained by cropping and resizing it. Cropping is an image transformation that cuts out a partial image within a ROI region of a predetermined shape (e.g., rectangular) from the original image and creates an image by removing the surrounding area. ROI stands for Region of Interest. The blinking score calculation unit 905 calculates a blinking score that indicates the likelihood that the lighting unit TS3 is in a blinking state based on the relative positional relationship between the housing TS1 and the lighting unit TS3 calculated by the positional relationship calculation unit 904. The larger the blinking score, the higher the likelihood that the lighting unit TS3 is in a blinking state. The blinking determination unit 906 determines whether the lighting unit TS3 is in a blinking state based on the blinking score calculated by the blinking score calculation unit 905. Specifically, the blinking determination unit 906 determines that an image is blinking when the blinking score is greater than a threshold value, and determines that an image is not blinking when the blinking score is equal to or less than the threshold value.

[0024] The calculation of the blinking score will be described in detail below. The blinking score calculation unit 905 calculates the blinking score according to the distance between the housing TS1 and the lighting unit TS3 in the relative positional relationship calculated by the positional relationship calculation unit 904. Specifically, if the two are separated by a certain value or more, the blinking score is calculated to be lower than if the separation is not by the certain value or more. The blinking score calculation unit 905 also calculates the blinking score according to the blinking period in each of the multiple illuminable areas TS2 included in the housing TS1. Specifically, if the blinking period of the lighting unit TS3 is within a predetermined range, the blinking score calculation unit 905 calculates a higher blinking score than if it is not within the predetermined range.

[0025] (Example of operation) The following describes an overview of the information processing operation by the information processing device 9 according to this embodiment, i.e., a method for detecting the flashing state of the lighting element TS3 of a traffic signal TS ahead of the vehicle. In the flowchart shown in FIG. 6, "S" is an abbreviation for "step." The information processing device 9 according to this embodiment, the information processing method and information processing program executed thereby, and a computer-readable, non-transient, tangible recording medium on which such program is recorded may be collectively referred to as "this embodiment." Such a recording medium may be realized, for example, by a ROM, a non-volatile rewritable memory, a magnetic disk, an optical disk, or the like. Specifically, such a recording medium may be realized in any format, for example, by an external server Z, a portable terminal device, an optical disk such as a CD-ROM, a memory card detachable from a computer device, or the like.

[0026] The flowchart shown in Fig. 6 outlines a method for detecting lighting unit TS3 from captured images Pg and determining their blinking state. In this method, processor 91 executes a series of processes from step 101 onwards for N frames of captured images Pg stored in chronological order, as shown in Fig. 6, where N is an integer greater than 1. The processing at each step will be described below in order.

[0027] In step 101, the processor 91 acquires the kth frame of the captured image Pg, where k is an integer and 1≦k≦N. In step 102, the processor 91 detects the housing TS1 and lighting units TS3 of the traffic signal TS from the captured image Pg using machine learning. Figure 7 shows a housing detection rectangle B1 as the detection result of the housing TS1, and a lighting unit detection rectangle B2 as the detection result of the lighting unit TS3.

[0028] In step 103, the processor 91 calculates the relative positional relationship between the detected housing TS1 and the lighting unit TS3. The method for calculating the relative positional relationship will be described with reference to FIG. 8. First, the processor 91 generates a converted image Ph shown on the left side of FIG. 8 by cropping and resizing the traffic light area based on the housing detection rectangle B1. Next, the processor 91 generates a normalized image Ps based on the converted image Ph. The normalized image Ps is generated as follows. First, the entire converted image Ph is temporarily assigned a value of "0," indicating an unlit state. Next, the area corresponding to the lighting unit detection rectangle B2 is assigned a value of "1," indicating an lit state. In the normalized image Ps shown on the right side of FIG. 8, the black portions indicate a value of "0," and the white portions indicate a value of "1." This binary image represents the relative positional relationship between the housing TS1 and the lighting unit TS3. Using such a normalized binary image makes it possible to suppress the positional fluctuation of a traffic signal TS approaching from a distance, which may be caused by, for example, pitching of the vehicle.

[0029] In step 104, the processor 91 calculates a blinking score. The method for calculating the blinking score will be described while referring to FIG. 9. In this embodiment, an RNN is used to calculate the blinking score. That is, in this embodiment, the output of the RNN for the past frames is input to the RNN for the current frame, thereby processing the captured images Pg for a plurality of frames which are time-series data. Specifically, as shown in FIG. 9, the output of the RNN for the captured image Pg at time t1 is input to the RNN for the captured image Pg at the next time t2. Similarly, the output of the RNN for the captured image Pg at time t2 is input to the RNN for the captured image Pg at the next time t3. In this way, the blinking scores at times t1, t2, t3, t4... are calculated in order.

[0030] In step 105, the processor 91 determines whether the blinking score exceeds a threshold value. If the blinking score exceeds the threshold value (i.e., step 105 = YES), the processor 91 proceeds with the process to step 106. In step 106, the processor 91 determines that the detected lit portion TS3 is in a blinking state. On the contrary, if the blinking score is below the threshold value (i.e., step 105 = NO), the processor 91 proceeds with the process to step 107. In step 107, the processor 91 makes a non-blinking determination indicating that there is no lit portion TS3 in a blinking state in the traffic signal TS detected this time. If the processing for N frames has not all been completed (i.e., k < N), the processor 91 increments the value of k and returns the process to step 101. If the processing for N frames has all been completed (i.e., k = N), the processor 91 ends the series of processes shown in FIG. 6.

[0031] (Effect) As described above in detail, this embodiment first uses machine learning to detect the housing TS1 and the lighting unit TS3 of the traffic signal TS. Then, using machine learning, i.e., RNN, the presence or absence of blinking is determined based on time-series changes in the detected position of the lighting unit TS3 relative to the housing TS1. Thus, according to this embodiment, by detecting the lighting unit TS3 using machine learning, it is possible to exclude simply bright areas in the captured image Pg from the determination of whether or not the lighting unit TS3 is blinking, thereby effectively suppressing erroneous detections. Furthermore, in time-series images in which the lighting unit TS3 approaches from a distance, normalizing the time-series data when determining whether or not the lighting unit TS3 is blinking based on the relative positional relationship between the housing TS1 and the lighting unit TS3 can improve the accuracy of the determination.

[0032] (Variation) The present disclosure is not limited to the above-described embodiments and specific examples. Therefore, the above-described embodiments and the like can be modified as appropriate. Representative modifications will be described below. In the following description of the modifications, differences from the above-described embodiments and the like will be mainly described. Furthermore, the same reference numerals are used for parts that are identical or equivalent to each other in the above-described embodiments and the following modifications. Therefore, in the following description of the modifications, the explanations in the above-described embodiments and the like can be used as appropriate for components that have the same reference numerals as the above-described embodiments and the like, unless there is a technical contradiction or special additional explanation.

[0033] The present disclosure is not limited to the specific applications and device configurations shown in the above embodiments. For example, the host vehicle may be a so-called automobile or a motorcycle. There are no particular limitations on the type of automobile or motorcycle.

[0034] All or part of the information processing device 9 may be configured to include a digital circuit, such as an ASIC or FPGA, configured to be able to realize the above-described functions or operations. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field Programmable Gate Array. In other words, the information processing device 9 may include both an on-board microcomputer and a digital circuit.

[0035] A computer program according to the present disclosure that enables the execution of various operations, procedures, or processes described in the above embodiments can be downloaded or upgraded via V2X communication using the communication device 5. V2X stands for Vehicle to X. Alternatively, such a computer program can be downloaded or upgraded via a terminal device installed in a vehicle manufacturing plant, a repair shop, a dealer, or the like. Such a computer program may be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0036] In this way, each of the above functional configurations and processes may be realized by a special-purpose computer provided by configuring a processor 91 and memory 92 programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and processes may be realized by a special-purpose computer provided by configuring a processor 91 with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and processes may be realized by one or more special-purpose computers configured by combining one or more processors 91 programmed to execute one or more functions and one or more memories 92 with one or more other processors 91 configured with one or more hardware logic circuits. Furthermore, a computer program may be stored in a computer-readable, non-transitory storage medium as instructions to be executed by a computer. In other words, each of the above functional configurations and processes may be expressed as a computer program including procedures for implementing the same, or as a non-transitory storage medium storing the computer program.

[0037] The present disclosure is not limited to the specific functions and operational modes described in the above embodiments. That is, for example, the normalized image Ps shown in Fig. 8 etc. is not limited to a binarized image using a value "0" indicating a non-illuminated state and a value "1" indicating a lit state. That is, for example, such normalized image Ps may be composed of quantized data.

[0038] Similar expressions such as "acquire," "calculate," "estimate," "detect," and "sensing" may be substituted for each other as appropriate within the scope of technical inconsistency. Furthermore, "exceeding the threshold" and "above the threshold" may be substituted for each other as appropriate within the scope of technical inconsistency. The same applies to "below the threshold" and "below the threshold."

[0039] It goes without saying that the elements constituting the above-described embodiments are not necessarily essential unless expressly stated as essential or clearly considered essential in principle. Furthermore, when numerical values ​​such as the number, value, amount, and range of components are mentioned, the present disclosure is not limited to those specific numbers unless expressly stated as essential or clearly limited to a specific number in principle. Similarly, when the shape, direction, positional relationship, etc. of components are mentioned, the present disclosure is not limited to those shapes, directions, positional relationships, etc. unless expressly stated as essential or clearly limited to a specific shape, direction, positional relationship, etc. in principle.

[0040] The variations are not limited to the above examples. For example, all or part of one of the variations may be combined with all or part of another, provided that no technical contradiction exists. Furthermore, all or part of the above specific example and all or part of the variations may be combined with each other, provided that no technical contradiction exists.

[0041] (Disclosure perspective) As is clear from the above description of the embodiments and modifications, this specification discloses at least the following matters.

[0042] [Perspective A] A method for detecting a flashing state of a lighting unit (TS3) of a traffic signal (TS) ahead of a vehicle (V), comprising: Acquiring an image using a camera (2) mounted on the vehicle; Detecting a housing (TS1) of the traffic signal from the acquired image; Detecting the lighting unit from the acquired image; Calculating the relative positional relationship between the detected housing and the lighting unit; calculating a blinking score representing the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; determining whether the lighting unit is in a blinking state according to the calculated blinking score; In calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Blinking state detection method.

[0043] [Perspective B1] An information processing device (9), a traffic signal housing detection unit (902) for detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); a lighting detection unit (903) that detects lighting units (TS3) of the traffic signal from the image; a positional relationship calculation unit (904) that calculates a relative positional relationship between the housing detected by the signal housing detection unit and the lighting unit detected by the lighting detection unit; a blinking score calculation unit (905) that calculates a blinking score representing the likelihood that the lighting unit is in a blinking state based on the relative positional relationship calculated by the positional relationship calculation unit; a blinking determination unit (906) that determines whether the lighting unit is in a blinking state according to the blinking score calculated by the blinking score calculation unit; Equipped with The blinking score calculation unit continuously calculates the blinking score based on the relative positional relationship. Information processing device. [Perspective B2] the blinking score calculation unit calculates the blinking score to be higher when the blinking cycle of the lighting unit is within a predetermined range than when the blinking cycle is not within the predetermined range, An information processing device according to aspect B1. [Perspective B3] the blinking score calculation unit calculates the blinking score to be lower when the housing and the lighting unit are spaced apart by a certain value or more in the relative positional relationship calculated by the positional relationship calculation unit than when the housing and the lighting unit are not spaced apart by the certain value or more. An information processing device according to aspect B1 or B2. [Perspective B4] the lighting detection unit detects the lighting unit using machine learning. The information processing device according to any one of the aspects B1 to B3. [Perspective B5] the blinking score calculation unit calculates the blinking score according to a blinking period in each of a plurality of illuminable areas (TS2) included in the housing; The information processing device according to any one of the aspects B1 to B4.

[0044] [Perspective C1] An information processing method executed by an information processing device (9), Detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); Detecting a lighting portion (TS3) of the traffic signal from the image; Calculating the relative positional relationship between the detected housing and the lighting unit; calculating a blinking score representing the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; determining whether the lighting unit is in a blinking state according to the calculated blinking score; In calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Information processing methods. [Perspective C2] When the blinking cycle of the lighting unit is within a predetermined range, the blinking score is calculated to be higher than when the blinking cycle is not within the predetermined range. The information processing method according to aspect C1. [Perspective C3] When the housing and the lighting unit are spaced apart by a certain value or more in the relative positional relationship, the blinking score is calculated to be lower than when the housing and the lighting unit are not spaced apart by the certain value or more. The information processing method according to aspect C1 or C2. [Perspective C4] detecting the lighting unit using machine learning; The information processing method according to any one of the aspects C1 to C3. [Perspective C5] calculating the blinking score according to the blinking period in each of a plurality of illuminable areas (TS2) included in the housing; The information processing method according to any one of the aspects C1 to C4.

[0045] [Perspective D1] An information processing program executed by an information processing device (9), The process executed by the information processing device is A process of detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); A process of detecting a lighting portion (TS3) of the traffic signal from the image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Information processing program. [Perspective D2] the process of calculating the blinking score includes a process of calculating the blinking score higher when the blinking cycle of the lighting unit is within a predetermined range than when the blinking cycle is not within the predetermined range. An information processing program according to aspect D1. [Perspective D3] the process of calculating the blinking score includes a process of calculating the blinking score to be lower when the housing and the lighting unit are spaced apart by a certain value or more in the relative positional relationship than when the housing and the lighting unit are not spaced apart by the certain value or more. An information processing program according to aspect D1 or D2. [Perspective D4] the process of detecting the lighting unit detects the lighting unit using the machine learning; An information processing program according to any one of aspects D1 to D3. [Perspective D5] The process of calculating the blinking score includes a process of calculating the blinking score according to a blinking period in each of a plurality of illuminable areas (TS2) included in the housing. An information processing program according to any one of aspects D1 to D4.

[0046] [Perspective E1] A computer-readable non-transient tangible recording medium that records an information processing program executed by an information processing device (9), The process included in the information processing program is A process of detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); A process of detecting a lighting portion (TS3) of the traffic signal from the image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Recording medium. [Perspective E2] the process of calculating the blinking score includes a process of calculating the blinking score higher when the blinking cycle of the lighting unit is within a predetermined range than when the blinking cycle is not within the predetermined range. A recording medium according to aspect E1. [Perspective E3] the process of calculating the blinking score includes a process of calculating the blinking score to be lower when the housing and the lighting unit are spaced apart by a certain value or more in the relative positional relationship than when the housing and the lighting unit are not spaced apart by the certain value or more. A recording medium according to aspect E1 or E2. [Perspective E4] the process of detecting the lighting unit detects the lighting unit using the machine learning; A recording medium according to any one of aspects E1 to E3. [Perspective E5] The process of calculating the blinking score includes a process of calculating the blinking score according to a blinking period in each of a plurality of illuminable areas (TS2) included in the housing. A recording medium according to any one of aspects E1 to E4. [Explanation of symbols]

[0047] 9. Information processing equipment 92 Storage devices (recording media) 902 Signal box detector 903 Lighting detection unit 904 Positional relationship calculation unit 905 Blinking score calculation unit 906 Blinking detection unit TS traffic signal TS1 enclosure TS3 lighting unit

Claims

1. An information processing device (9), a traffic signal housing detection unit (902) for detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); a lighting detection unit (903) that detects lighting portions (TS3) of the traffic signal from the image; a positional relationship calculation unit (904) that calculates a relative positional relationship between the housing detected by the signal housing detection unit and the lighting unit detected by the lighting detection unit; a blinking score calculation unit (905) that calculates a blinking score representing the likelihood that the lighting unit is in a blinking state based on the relative positional relationship calculated by the positional relationship calculation unit; a blinking determination unit (906) that determines whether the lighting unit is in a blinking state according to the blinking score calculated by the blinking score calculation unit; Equipped with The blinking score calculation unit continuously calculates the blinking score based on the relative positional relationship. Information processing device.

2. the blinking score calculation unit calculates the blinking score to be higher when the blinking cycle of the lighting unit is within a predetermined range than when the blinking cycle is not within the predetermined range. The information processing device according to claim 1 .

3. the blinking score calculation unit calculates the blinking score to be lower when the housing and the lighting unit are spaced apart by a certain value or more in the relative positional relationship calculated by the positional relationship calculation unit than when the housing and the lighting unit are not spaced apart by the certain value or more. The information processing device according to claim 1 .

4. the lighting detection unit detects the lighting unit using machine learning. The information processing device according to claim 1 .

5. the blinking score calculation unit calculates the blinking score according to a blinking period in each of a plurality of illuminable areas (TS2) included in the housing; The information processing device according to claim 1 .

6. An information processing method executed by an information processing device (9), Detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); Detecting a lighting portion (TS3) of the traffic signal from the image; Calculating the relative positional relationship between the detected housing and the lighting unit; calculating a blinking score representing the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; determining whether the lighting unit is in a blinking state according to the calculated blinking score; In calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Information processing methods.

7. An information processing program executed by an information processing device (9), The process executed by the information processing device is A process of detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); A process of detecting a lighting portion (TS3) of the traffic signal from the image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Information processing program.

8. A computer-readable non-transient tangible recording medium that records an information processing program executed by an information processing device (9), The process included in the information processing program is A process of detecting a housing (TS1) of a traffic signal (TS) from an image acquired using a camera (2); A process of detecting a lighting portion (TS3) of the traffic signal from the image; A process of calculating the relative positional relationship between the detected housing and the lighting unit; A process of calculating a blinking score that indicates the likelihood that the lighting unit is in a blinking state based on the calculated relative positional relationship; a process of determining whether the lighting unit is in a blinking state according to the calculated blinking score; Including, In the process of calculating the blinking score, the blinking score is continuously calculated based on the relative positional relationship. Recording medium.

Images