Vehicle control device and light-emitting body identification method

Abstract

This vehicle control device comprises: a first light-emitting body identification unit that acquires a single frame image including a periodically changing light-emitting body imaged by an imaging device mounted on a vehicle, and identifies the type and state of the light-emitting body from the acquired single frame image; a time-series data acquisition unit that acquires and holds time-series data of the light-emitting body by using a first identification result obtained by the first light-emitting body identification unit; and a second light-emitting body identification unit that receives input of the held time-series data to identify the type and state of the light-emitting body from the input time-series data.

Description

Vehicle control device and lighting unit identification method

[0001] The present invention relates to a vehicle control device.

[0002] In order to reduce traffic accidents, Level 1 and Level 2 Advanced Driver Assistance Systems (ADAS), which partially automate either or both accelerator / brake operation and steering operation, are being developed and put into practical use. Furthermore, efforts are being made both domestically and internationally to develop and put into practical use Level 3 and Level 4 autonomous driving technology (AD), in order to support mobility for the elderly and others, address driver shortages in the transportation industry, and improve productivity.

[0003] One of the challenges for advanced driver assistance systems is to prevent sudden braking in order to maintain a safe distance between vehicles and ensure smooth vehicle operation. In particular, Level 4 autonomous driving requires cooperative vehicle control that adheres to road manners.

[0004] For cooperative vehicle control, there is a need for technology to identify lighting and traffic signals in order to predict the behavior of other vehicles.

[0005] The following prior art exists as background art in this technical field. Patent Literature 1 (JP 2023-85060 A) discloses a lighting state identification device including: a state identification unit that identifies the lighting state of each of a first lighting unit and a second lighting unit of a vehicle by inputting each of a series of images in time series depicting the vehicle into a classifier, the classifier including: a feature calculation unit that, each time one of the series of images is input along the time series, performs a convolution operation on the input image to calculate a feature map representing features of the first lighting unit and the second lighting unit; a first lighting state identification unit having a recursive structure that identifies the lighting state of the first lighting unit while updating a first internal state that is used recursively by inputting the feature map calculated for each of the series of images in chronological order; and a second lighting state identification unit having a recursive structure that identifies the lighting state of the second lighting unit while updating a second internal state that is used recursively by inputting the feature map calculated for each of the series of images in chronological order.

[0006] JP 2023-85060 A

[0007] In a configuration in which the calculation of image features based on a learning model and the update process of time-series data are shared, such as in the device disclosed in Patent Document 1, if the number of time-series data is increased to suppress misidentification, the time required for identification becomes longer, and identification may not be completed in time in situations where there is little time left until a collision with another vehicle, such as a sudden cut-in.

[0008] An object of the present invention is to provide a vehicle control device that can quickly and accurately identify the periodic lighting state of lighting devices provided on other vehicles.

[0009] A representative example of the invention disclosed in the present application is as follows: That is, a vehicle control device includes a first lighting element identification unit that acquires a single-frame image including a periodically changing lighting element captured by an imaging device mounted on a vehicle and identifies the type and state of the lighting element from the acquired single-frame image, a time-series data acquisition unit that acquires and stores time-series data of the lighting element using a first identification result by the first lighting element identification unit, and a second lighting element identification unit that receives the stored time-series data and identifies the type and state of the lighting element from the input time-series data.

[0010] According to one aspect of the present invention, the state of a lighting element can be determined at an early stage. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0011] FIG. 1 is a block diagram showing an example of an internal configuration of a vehicle control device and an example of a connection relationship with an imaging device according to a first embodiment. FIG. 2 is a flowchart of processing executed by a first lighting object identification unit according to the first embodiment. FIG. 3 is a diagram showing an example of a single frame image including a lighting object whose brightness changes periodically in the first embodiment. FIG. 4 is a flowchart showing an example of processing executed by a time series data acquisition unit according to the first embodiment. FIG. 5 is a flowchart showing an example of time series data storage processing of step S44. FIG. 6 is an explanatory diagram of processing executed by the time series data acquisition unit according to the first embodiment. FIG. 7 is an explanatory diagram of processing executed by the time series data acquisition unit according to the first embodiment. FIG. 8 is an explanatory diagram of processing executed by the time series data acquisition unit according to the first embodiment.

[0012] First Embodiment FIG. 1 is a block diagram showing an example of the internal configuration of a vehicle control device 2 according to a first embodiment and an example of the connection relationship between the vehicle control device 2 and an imaging device 1. As shown in FIG.

[0013] The imaging device 1 is connected to the vehicle control device 2. The vehicle control device 2 has a first lighting element identification unit 3, a time series data acquisition unit 4, a second lighting element identification unit 5, and a vehicle control unit 6. The vehicle control device 2 performs vehicle control for advanced driver-assistance systems (ADAS) and autonomous driving systems (AD). In this specification, the automobile on which the imaging device 1 is mounted is referred to as the "host vehicle."

[0014] Each functional block of the vehicle control device 2 of this embodiment is implemented in a control device having an arithmetic unit, a storage device, and a communication interface. The arithmetic unit is a processor (e.g., a microcomputer) that executes programs stored in the storage device. The arithmetic unit operates as a functional block that provides various functions by executing predetermined programs. The storage device includes a non-volatile storage area and a volatile storage area. The non-volatile storage area is accessible by the arithmetic unit and includes a program area that stores programs executed by the arithmetic unit, and a data area that temporarily stores data used by the arithmetic unit when executing programs. The volatile storage area stores data used by the arithmetic unit when executing programs. The communication interface connects to other electronic control units via a network such as CAN or Ethernet.

[0015] Fig. 2 is a flowchart of the process executed by the first lighting object identification unit 3 in Example 1. Fig. 2 explains the process executed by the first lighting object identification unit 3 using an example in which a single frame image including a lighting object whose brightness changes periodically is used.

[0016] First, the first lighting element identification unit 3 acquires a single-frame image including a lighting element whose brightness changes periodically (step S31). FIG. 3 is a diagram illustrating an example of a single-frame image including a lighting element whose brightness changes periodically in Example 1. For example, a target object is recognized using a machine learning model for target object recognition that has previously learned the characteristics of vehicles and traffic lights. A leading vehicle Car1 and an oncoming vehicle Car2 are cropped into a rectangle inscribed with the target object in three-dimensional space, resized, and a single-frame image assigned with a tracking ID that identifies the recognition target is acquired. Similarly, a pedestrian traffic light Sig1 and a vehicular traffic light Sig2 are cropped into a rectangle inscribed with the target object in two-dimensional space, resized, and a single-frame image assigned with a tracking ID that identifies the recognition target is acquired. When identifying the lighting state of a rotating light, a single-frame image including a lighting element may be acquired using a rectangle larger than the rectangle inscribed with the target object. Other methods for acquiring a single-frame image including a lighting element whose brightness changes periodically may also be used. For example, an image may be recognized pixel by pixel, and pixels including a lighting element assigned with a tracking ID that identifies the recognition target may be acquired. Furthermore, the order of recognition targets may be rearranged in order of priority according to the distance to other vehicles.

[0017] The first lighting object identification unit 3 inputs the lighting object feature quantities extracted from the single frame image acquired in step S31 into a first machine learning model that has been trained in advance using learning data labeled with the type and state of the lighting object (step S32). For example, at least one of the image width, height, and number of color channels may be specified and input to the input layer of the first machine learning model. The number of color channels may be specified using grayscale instead of RGB. The method is not limited to using a machine learning model, and brightness values ​​acquired by image processing may also be used to calculate the lighting object feature quantities.

[0018] The first lighting element identification unit 3 calculates lighting element features (step S33). For example, image features may be extracted using a first learning model that has been previously trained on single-frame learning images using a two-dimensional convolutional neural network (CNN). The gradient vanishing problem may be mitigated by using a learning model created using a ReLU function or the like in an intermediate layer. The single-frame learning images are, for example, data in which vehicle images with turn signals on are labeled as "turn signal on" and vehicle images with turn signals off are labeled as "turn signal off." In addition to using a learning model, the lighting element features may be calculated by extracting color information and brightness values ​​through image processing, for example.

[0019] The first lighting object identification unit 3 identifies the type and state of the lighting object (step S34). For example, the output value of the identification result is converted by binary classification using a sigmoid function into a range of 0 to 1 for each class, and a first score belonging to each class is output. The first score represents, for example, the probability that the turn signal and brake light are each in an on state when the turn signal and brake light are both on at the same time. The method of identifying the type and state of the lighting object may use a learning model, or may include, for example, statistical processing of color information and brightness values ​​to calculate statistical values ​​(e.g., mode, maximum value), and using the calculated statistical values ​​as the output value of the identification result.

[0020] FIG. 4 is a flowchart illustrating an example of processing executed by the time-series data acquiring unit 4 according to the first embodiment.

[0021] First, the time-series data acquisition unit 4 acquires a first identification result (step S41). For example, the first identification result is the type and state of the lighting element output in step S34. The first identification result may be a value obtained by threshold processing, which binarizes the output value to 0 or 1 based on whether the output value exceeds a specific threshold. The output value may also be normalized.

[0022] The time-series data acquisition unit 4 determines whether the first classification result is the first classification result that has not been recognized in the past (step S42). For example, if there is no array in which the first classification result is stored, it is determined that the first classification result is the first classification result.

[0023] If it is determined in step S42 that this is the first identification result, the time-series data acquisition unit 4 executes initial time-series data processing in which any data that indicates that the lighting object is in an unlit state is assigned to the future value of the first identification result (step S43), and then proceeds to step S44. For example, the unlit state of the turn signal may be represented by 0, and 0 may be assigned to the future value. If it is determined in step S42 that this is not the first identification result, then proceeds to step S44.

[0024] The time-series data acquisition unit 4 updates and stores the time-series data (step S44). Details of the process of step S44 will be described later with reference to FIG.

[0025] FIG. 5 is a flowchart showing an example of the time-series data storage process in step S44.

[0026] In the time-series data storage process, the time-series data acquisition unit 4 determines whether all the first identification results are present (step S441). For example, if a variable for counting the number of first identification results is prepared and the lighting state is identified based on 2 seconds of time-series data at 10 frames / second, it is determined whether the number of first identification results is 20.

[0027] If it is determined in step S441 that the number of classification results is complete, the time-series data acquisition unit 4 deletes past values ​​(step S442) and updates and stores the time-series data (step S443).If it is determined in step S441 that the number of classification results is not complete, the time-series data acquisition unit 4 updates and stores the time-series data without deleting past values ​​(step S443).

[0028] 6A to 6C are explanatory diagrams of the processing executed by the time-series data acquiring unit 4 of the first embodiment.

[0029] The time-series data range D is the length of the time-series data and indicates the number of classification results that are retained. The vertical axis represents the first classification result P, which is normalized, for example, between 0 and 1, with the maximum value being P1 and the minimum value being P2. The horizontal axis represents the frame number F.

[0030] For example, as shown in Fig. 6A, when the first identification result is obtained in C1[F], arbitrary data is substituted into the future value FV of C1[F] to generate data that is determined to be in the non-illuminated state. Note that the numerical value represented by [F] corresponds to the frame number.

[0031] As shown in FIG. 6B, when the second first identification result P is obtained in C2[F], the first identification result P is overwritten and updated to any data of the future value FV that was assigned to C2[F].

[0032] When the classification result is obtained in C3[F], if the number of first classification results is consistent for the time-series data range D, for example, the first classification result P, which is a past value stored in O1[F], is erased, the time-series data is shifted one by one, and the current first classification result P is stored and updated in C3[F], resulting in the state shown in FIG. 6C.

[0033] FIG. 7 is a flowchart showing an example of processing by the second lighting element identifying unit 5 of the first embodiment.

[0034] First, the second lighting object identification unit 5 inputs the time series data held by the time series data acquisition unit 4 into a second learning model that has been previously trained using learning data in which learning time series data is labeled with the type and state of a lighting object (step S51). For example, the second lighting object identification unit 5 specifies the number of time series data and the number of channels and inputs them into the input layer of the learning model. When identifying the lighting states of a turn signal, brake light, and rotating light, the number of channels may be, for example, three. The number of channels may also be one and input into separate learning models. The method is not limited to using a learning model, and the time series data may, for example, be input into a function that performs statistical processing on the time series data.

[0035] The second lighting element identification unit 5 calculates lighting element feature quantities (step S52). For example, the second lighting element identification unit 5 uses a second learning model that has been trained in advance with training time-series data using a one-dimensional convolutional neural network (1D-CNN). To mitigate the gradient vanishing problem, a learning model created using a ReLU function or the like in the intermediate layer may be used. In addition to using the learning model, for example, statistical processing may be used to analyze the features of the time-series data.

[0036] The second lighting element identification unit 5 identifies the type and state of the lighting element (step S53). For example, the second lighting element identification unit 5 uses a sigmoid function to perform binary classification for each class, converts the output value of the identification result into a range from 0 to 1, and outputs a second score belonging to each class. The second score represents, for example, the probability that each of the turn signal, brake light, and rotating light is in an on state. The method is not limited to using a learning model, and for example, statistical processing may be used to classify the features of the time series data.

[0037] The second lighting object identification unit 5 identifies the type and state of the lighting object (step S54). For example, the second lighting object identification unit 5 determines that the lighting object is in a lighting state when the probability score for the second identification result obtained in step S53 is 0.8 or higher, and stores values ​​indicating lighting in variables indicating the lighting state of the left turn signal, right turn signal, brake light, and rotating light. Note that the second lighting object identification unit 5 may also acquire direction information of other vehicles, discard the lighting of the brake light toward an oncoming vehicle as an erroneous identification, and store 0 in the variable indicating the lighting state of the brake light. In addition to the method using a learning model, the lighting state may also be determined based on the results of statistical processing, for example.

[0038] FIG. 8 is an explanatory diagram of learning time-series data that is pre-learned by the second learning model of the first embodiment.

[0039] The time-series data range D is time-series data of a predetermined length and includes a predetermined number of frames. A second learning model capable of distinguishing between a turn signal flashing state and an off state is created by learning using data obtained by shifting the frames of the time-series data range D one by one. In FIG. 8 , the vertical axis represents the first discrimination result P, which is a value ranging from 0 to 1, with P1 being the maximum value and P2 being the minimum value. The horizontal axis represents the frame number F. The learning time-series data is, for example, time-series data composed of first scores arranged in chronological order. For example, when time series data showing a vehicle's turn signal continuously flashing for two seconds is used for learning, the time series data from one second after the turn signal starts flashing (T3[F]) to one second after the flashing ends (T4[F]) is set to the turn signal flashing state, and learning can be performed to distinguish the time series data from the beginning of the learning time series data, T1[F], to 0.9 seconds after the turn signal starts flashing (T2[F]), and the time series data from 0.9 seconds after the flashing ends (T5[F]) to the end of the learning time series data, T6[F], as the turn signal off state. Depending on how early the flashing state needs to be identified, the time series data to be learned as the flashing state and the time series data to be learned as the off state can be selected.

[0040] Next, we will explain an example in which the initial time series data processing procedure (S43) of the time series data acquisition unit 4 and the second lighting element identification unit 5 suppress erroneous identification by the first lighting element identification unit 3 when sufficient data is not available as the first identification result.

[0041] FIG. 9 is a diagram illustrating a comparison example between the time-series data acquiring unit 4 of the first embodiment and a conventional method.

[0042] The time series data range D is time series data of a predetermined length and includes a predetermined number of frames. The time series data range D includes 20 frames. In FIG. 9 , the vertical axis represents the first identification result P, ranging from 0 to 1, with P1 being the maximum value and P2 being the minimum value. The horizontal axis represents the frame number F. When the first identification result is obtained for the time series data I of the first identification result that is actually observed and misidentified by the first lighting object identification unit 3 due to external factors such as sunlight filtering through the trees or reflection from the setting sun, for example, in the conventional method, arbitrary data (e.g., 0) is assigned to the past value OV. On the other hand, in this embodiment, the time series data acquisition unit 4 assigns arbitrary data (e.g., 0) to the future value FV in step S43. When arbitrary data is assigned to the past value OV, the time series data at the time of the first lighting object identification unit 3's misidentification resembles the time series data at the start of lighting, so arbitrary data is assigned to the future value FV.

[0043] This shows the results of the recognition performed by the second lighting object recognition unit 5 of Example 1 in C4[F] of the 11th frame of the first recognition result. In the conventional method, the probability score of the turn signal being on was approximately 0.829, which resulted in an erroneous recognition when the number of first recognition results was insufficient. In the method of this example, the probability score of the turn signal being on was approximately 0.038, which correctly recognized that the turn signal was not on.

[0044] The technical significance of inputting arbitrary data into the future value is that it suppresses erroneous identification when the number of first classification results is insufficient. Note that when increasing the number of time-series data to suppress erroneous identification, performing classification after all first classification results are available would require a long time to determine the state. Therefore, by inputting arbitrary data into the time-series data including the initial classification result, classification is performed before all first classification results are available.

[0045] FIG. 10 is a flowchart illustrating an example of processing by the vehicle control unit 6 according to the first embodiment.

[0046] The vehicle control unit 6 acquires the second identification result (step S61), for example, the value 1 stored in the variable indicating the lighting state of the left turn signal and the brake lamp.

[0047] The vehicle control unit 6 executes vehicle control of the host vehicle based on the acquired second identification result (step S62). For example, if the second identification result indicates that the other vehicle is about to change lanes to the left ahead and enter the host vehicle's lane, the vehicle control unit 6 executes vehicle control of the host vehicle so as to gently decelerate the host vehicle. The vehicle control unit 6 may acquire the distance to the other vehicle and control the braking strength according to the distance.

[0048] As described above, in the embodiment of the present invention, the value determined to be the non-illuminated state at the time of initial identification is substituted into the time series data, and the results of single-frame identification are sequentially acquired, thereby enabling the state of the lighting element to be determined early and preventing delays in vehicle control. Furthermore, by substituting non-illuminated state data into future values ​​of the time series data, even if a single-frame erroneous identification occurs immediately after the start of detection, erroneous identification is suppressed in the time series determination, thereby suppressing erroneous control of the vehicle.

[0049] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0050] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0051] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

[0052] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A vehicle control device comprising: a first lighting element identification unit that acquires a single frame image including a periodically changing lighting element captured by an imaging device mounted on a vehicle and identifies the type and state of the lighting element from the acquired single frame image; a time series data acquisition unit that acquires and stores time series data of the lighting element using a first identification result by the first lighting element identification unit; and a second lighting element identification unit that receives the stored time series data and identifies the type and state of the lighting element from the input time series data.

2. A vehicle control device as described in claim 1, wherein at least one of the first lighting element identification unit and the second lighting element identification unit is configured using a learning model that has previously learned the characteristics of the lighting elements.

3. A vehicle control device as described in claim 2, wherein the learning model used by the first lighting element identification unit is learned using learning data in which single-frame learning images are labeled with the type and state of the lighting element.

4. A vehicle control device as described in claim 2, wherein the learning model used by the second lighting element identification unit is learned using learning data in which learning time series data is labeled with the type and state of the lighting element.

5. A vehicle control device according to claim 1, wherein the time series data acquisition unit holds time series data in which arbitrary data is substituted for future values ​​of time series data including the first acquired first identification result, and holds time series data including the second or subsequent acquired first identification results.

6. A vehicle control device according to claim 5, wherein the arbitrary data is data indicating that the lighting element is in a non-lighting state.

7. A vehicle control device according to claim 1, comprising a vehicle control unit that executes control of the vehicle in accordance with the second identification result by the second lighting element identification unit.

8. A lighting element identification method executed by a vehicle control device, wherein the vehicle control device has an arithmetic device that executes a program and a storage device accessible by the arithmetic device, the lighting element identification method comprising: a first lighting element identification procedure in which the arithmetic device acquires a single-frame image including a periodically changing lighting element captured by an imaging device mounted on the vehicle, and identifies the type and state of the lighting element from the acquired single-frame image; a time series data acquisition procedure in which the arithmetic device acquires and stores time series data of the lighting element using a first identification result from the first lighting element identification procedure; and a second lighting element identification procedure in which the stored time series data is input to the arithmetic device, and identifies the type and state of the lighting element from the input time series data.

Images