Vehicle control device, vehicle control method, and vehicle control computer program

Abstract

A vehicle control device, a vehicle control method and a computer program for vehicle control are provided. The vehicle control device comprises: a state recognition unit (33) for inputting features extracted from an object region representing surrounding vehicles in an image obtained by a photographing unit of a vehicle (10) into a signal light state recognizer to recognize the state of the signal lights of surrounding vehicles; a first control instruction determination unit (34) for determining a first control value according to a predetermined rule based on the recognition result of the state of the signal lights of surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles; a second control instruction determination unit (35) for inputting the recognition result into a control instruction recognizer that has been pre-learned to output a second control value to determine a second control value for controlling the vehicle to avoid collision with surrounding vehicles; and a control instruction synthesis unit (36) for determining a synthesis control value used in controlling the vehicle based on the first and second control values.

Description

Technical Field

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a vehicle control computer program. Background Art

[0002] In order to control the vehicle autonomously or assist the driver in driving the vehicle, it is desirable to accurately predict the behavior of other vehicles traveling around the vehicle. Therefore, in order to predict the behavior of other vehicles, a technology using a machine learning method such as a so-called deep neural network (hereinafter referred to as DNN) has been proposed (see Japanese Patent Publication No. 2020-42786).

[0003] The method for processing automobile images disclosed in Japanese Patent Publication No. 2020-42786 obtains the image to be processed collected by the collection point of automobile images set in the autonomous driving device, processes the image to be processed using a deep learning model, and outputs the state parameters of the automobile in the image to be processed. In addition, the processing method determines the behavior of the automobile in the image to be processed based on the state parameters. In addition, the state parameters of the automobile in the image to be processed are used to represent one or more of the brake light state, the turn signal state, the door state, the trunk door state, and the wheel direction state. Summary of the invention

[0004] Sometimes, depending on the driving environment around the vehicle, it is difficult to accurately detect the status of the turn signal indicator or brake light of other vehicles traveling around the vehicle. In addition, when the detection of the status of the turn signal indicator or brake light fails, the vehicle control device may perform inappropriate vehicle control such as approaching the other vehicle in a manner that may cause a risk of collision with the other vehicle, or performing unnecessary deceleration or braking.

[0005] Therefore, an object of the present invention is to provide a vehicle control device that can appropriately control a vehicle even if it is difficult to recognize the states of signal lights of other vehicles around the vehicle.

[0006] According to one embodiment, a vehicle control device is provided. The vehicle control device comprises: an object detection unit that detects an object region representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided on the vehicle to photograph the surroundings of the vehicle; a state recognition unit that recognizes the state of the signal lights of the surrounding vehicles by inputting features extracted from the object region on the image into a signal light state recognizer, the signal light state recognizer being pre-learned to recognize the state of the signal lights of the surrounding vehicles; and a first control instruction determination unit that determines the state of the vehicle according to a control instruction for controlling the vehicle based on the recognition result of the state of the signal lights of the surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles. A first control value is determined according to a predetermined rule for avoiding a collision between the vehicle and surrounding vehicles; a second control instruction determination unit determines a second control value by inputting a recognition result of the state of the signal lights of surrounding vehicles into a control instruction identifier, the control instruction identifier is pre-learned to output a second control value, and the second control value is used to control the vehicle so as to avoid a collision between the vehicle and surrounding vehicles; a control instruction synthesis unit determines a comprehensive control value for avoiding a collision between the vehicle and surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; and a control unit controls the travel of the vehicle according to the comprehensive control value.

[0007] In the vehicle control device, preferably, the first control instruction determining unit outputs the first control value at a predetermined period, and the second control instruction determining unit outputs the second control value at a predetermined period, and when the period in which the absolute value of the difference between the first control value and the second control value is less than a certain value continues for more than a predetermined time threshold, the control instruction integrating unit uses the average value of any first control value output during the period and any second control value output during the period as the integrated control value.

[0008] Alternatively, preferably, the first control instruction determining unit outputs the first control value at a predetermined period, and the second control instruction determining unit outputs the second control value at a predetermined period, and when a period in which the absolute value of the difference between the first control value and the second control value is greater than a certain value continues for more than a predetermined time threshold, the control instruction integrating unit uses the larger absolute value of the average value of the first control value output during the period and the average value of the second control value output during the period as the integrated control value.

[0009] Furthermore, in the vehicle control device, preferably, when the absolute value of the second control value is equal to or greater than a predetermined upper limit value, the control instruction integrating unit uses the upper limit value or the first control value as the integrated control value.

[0010] Furthermore, in the vehicle control device, the second control instruction determination unit preferably determines the second control value by inputting at least one of the relative position relationship between the vehicle and surrounding vehicles, the relative vehicle speed, the relative acceleration and deceleration, the size of the object area, and the position of the object area together with the recognition result of the state of the signal lights of the surrounding vehicles into the control instruction identifier.

[0011] Furthermore, in the vehicle control device, the signal light state identifier preferably has an input layer to which features extracted from the object area are input, an output layer to which the state of the signal lights of surrounding vehicles is output, and at least one intermediate layer is sequentially arranged between the input layer and the output layer from the input layer to the output layer, and at least one intermediate layer performs predetermined operations on the output from the layer closer to the input layer than the intermediate layer, and the second control instruction determination unit determines the second control value by inputting the result of the predetermined operation of any one of the at least one intermediate layers of the signal light state identifier together with the recognition result of the state of the signal lights of surrounding vehicles into the control instruction identifier.

[0012] Furthermore, in the vehicle control device, it is preferred that the signal light state identifier has a recursive structure, and the state identification unit identifies the state of the signal lights of surrounding vehicles by inputting features extracted from the object area in each of one or more past images obtained by the shooting unit before the image is formed and features extracted from the object area on the image into the signal light state identifier in a time series order.

[0013] Alternatively, in the vehicle control device, the signal light state identifier is preferably a convolutional neural network having a convolutional layer that performs convolution operations in the time axis direction, and the state recognition unit identifies the state of the signal lights of surrounding vehicles by inputting features extracted from object areas in each of one or more past images obtained by the shooting unit before the image and features extracted from the object area on the image into the signal light state identifier.

[0014] According to other embodiments, a vehicle control method is provided. The vehicle control method includes: detecting an object area representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided on the vehicle for photographing the surroundings of the vehicle; identifying the state of the signal lights of the surrounding vehicles by inputting features extracted from the object area on the image into a signal light state recognizer, the signal light state recognizer being pre-learned so as to recognize the state of the signal lights of the surrounding vehicles; determining a first control value according to a predetermined rule for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles based on the recognition result of the state of the signal lights of the surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles; determining a second control value by inputting the recognition result of the state of the signal lights of the surrounding vehicles into a control indication recognizer, the control indication recognizer being pre-learned so as to output a second control value, the second control value being used to control the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles; determining a comprehensive control value for avoiding a collision between the vehicle and the surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; and controlling the travel of the vehicle according to the comprehensive control value.

[0015] According to still another embodiment, a computer program for vehicle control is provided. The vehicle control computer program is used to cause a processor mounted on a vehicle to execute: detecting an object region representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided on the vehicle for photographing the surroundings of the vehicle; recognizing the state of the signal lights of the surrounding vehicles by inputting features extracted from the object region on the image into a signal light state recognizer, the signal light state recognizer being pre-learned to recognize the state of the signal lights of the surrounding vehicles; determining a first control value according to a predetermined rule for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles based on the recognition result of the state of the signal lights of the surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles; determining a second control value by inputting the recognition result of the state of the signal lights of the surrounding vehicles into a control instruction recognizer, the control instruction recognizer being pre-learned to output a second control value, the second control value being used to control the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles; determining a comprehensive control value for avoiding a collision between the vehicle and the surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; and controlling the travel of the vehicle according to the comprehensive control value.

[0016] The vehicle control device according to the present invention achieves an effect of being able to appropriately control the vehicle even when it is difficult to recognize the states of signal lights of other vehicles around the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1This is a schematic diagram of the structure of a vehicle control system in which a vehicle control device is installed.

[0018] Figure 2 This is a hardware configuration diagram of an electronic control device as one embodiment of a vehicle control device.

[0019] Figure 3 This is a functional block diagram of a processor of an electronic control device related to vehicle control processing.

[0020] Figure 4 This is a diagram for explaining an outline of an example of the first control instruction determination.

[0021] Figure 5 This is a flow chart of vehicle control processing. DETAILED DESCRIPTION

[0022] Hereinafter, a vehicle control device, a vehicle control method executed on the vehicle control device, and a computer program for vehicle control are described with reference to the accompanying drawings. The vehicle control device recognizes the state of the signal lights of other vehicles (hereinafter referred to as surrounding vehicles for convenience of description) traveling around the vehicle, which are represented in a series of images obtained in a time series manner by a camera mounted on the vehicle, and controls the vehicle using the recognition result. Therefore, the vehicle control device detects the area containing the surrounding vehicles on the image (hereinafter sometimes referred to as the object area) by image by inputting the series of images to a recognizer for object detection, and the recognizer is previously learned so as to detect the surrounding vehicles. In addition, the vehicle control device tracks the surrounding vehicles detected from each image, thereby associating the object areas representing the same surrounding vehicles in each image with each other. In addition, the vehicle control device recognizes the state of the signal lights of the surrounding vehicles (hereinafter sometimes referred to as the signal light state) by inputting the features obtained based on the pixel values ​​in the object areas representing the same surrounding vehicles in a series of images in a time series order to the recognizer for signal light state recognition. Furthermore, the vehicle control device obtains a first control instruction including a first control value according to a predetermined rule for controlling the vehicle so as to avoid collision between the vehicle and the surrounding vehicles based on the recognition result of the signal light state of the surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles. Furthermore, the vehicle control device obtains a second control instruction including a second control value by inputting the recognition result of the signal light state of the surrounding vehicles into a recognizer for determining control instructions, and the recognizer for determining control instructions is pre-learned so as to output a control instruction for controlling the vehicle so as to avoid collision between the vehicle and the surrounding vehicles. Furthermore, the vehicle control device determines a comprehensive control instruction including a final control value (hereinafter referred to as a comprehensive control value) based on the first control value and the second control value, and controls the travel of the vehicle according to the comprehensive control instruction.

[0023] For example, when the surrounding vehicles change lanes, the turn signal will flash. In addition, the surrounding vehicles will light up the brake lights when they slow down. The lighting or flashing of these signals or lights indicates that the surrounding vehicles are likely to perform actions at that time or soon after. Therefore, the vehicle control device identifies the signal light state of the surrounding vehicles, and determines the comprehensive control instruction based on the control instruction obtained by the identifier for control instruction determination using the identification result as input, and the control instruction determined based on the identification result of the signal light state based on a predetermined rule. In this way, the vehicle control device evaluates the control instructions obtained by the two methods using the identification result of the signal light state, and determines the comprehensive control instruction with a high possibility of avoiding the collision between the vehicle and the surrounding vehicles. Therefore, the vehicle control device can determine the appropriate control instruction even in a driving environment where it is difficult to identify the signal light state of the surrounding vehicles. In addition, in the following examples, as the control value included in each control instruction, the acceleration or deceleration of the vehicle (hereinafter referred to as acceleration and deceleration) is required. However, each control value is not limited to acceleration and deceleration, and can also be a steering angle.

[0024] Figure 1 This is a schematic diagram of a vehicle control system equipped with a vehicle control device. Figure 2 This is a hardware structure diagram of an electronic control device as an embodiment of a vehicle control device. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 has a camera 2 for photographing the surroundings of the vehicle 10, and an electronic control unit (ECU) 3 as an example of a vehicle control device. The camera 2 and the ECU 3 are connected in a communicative manner via an in-vehicle network that complies with a standard such as a controller area network. In addition, the vehicle control system 1 may also have a storage device for storing maps used in the automatic driving control of the vehicle 10. Furthermore, the vehicle control system 1 may also have a ranging sensor such as LiDAR or radar, a receiver such as a GPS receiver for locating the self-position of the vehicle 10 according to a satellite positioning system, and a wireless terminal for wireless communication with other devices. Furthermore, the vehicle control system 1 may also have a navigation device for searching for a scheduled driving route of the vehicle 10.

[0025] The camera 2 is an example of a photographing unit, and has a two-dimensional detector composed of an array of photoelectric conversion elements such as CCD or C-MOS that are sensitive to visible light, and an imaging optical system that images an area that becomes a photographing object on the two-dimensional detector. Furthermore, the camera 2 is installed in the cabin of the vehicle 10 in a manner facing the front of the vehicle 10, for example. Furthermore, the camera 2 photographs the area in front of the vehicle 10 at a predetermined photographing period (for example, 1 / 30 second to 1 / 10 second) to generate an image showing the area in front. The image obtained by the camera 2 is preferably a color image. In addition, a plurality of cameras with different shooting directions or focal distances may also be provided in the vehicle 10.

[0026] Whenever the camera 2 generates an image, it outputs the generated image to the ECU 3 via the in-vehicle network.

[0027] The ECU 3 controls the vehicle 10. In the present embodiment, the ECU 3 performs automatic driving control on the vehicle 10 based on the signal light states of surrounding vehicles detected from a series of time-series images obtained by the camera 2, so as to avoid collision between the vehicle 10 and surrounding vehicles. Therefore, the ECU 3 has a communication interface 21, a memory 22, and a processor 23.

[0028] The communication interface 21 is an example of a communication unit and has an interface circuit for connecting the ECU 3 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network. And, each time the communication interface 21 receives an image from the camera 2, it passes the received image to the processor 23.

[0029] The memory 22 is an example of a storage unit, such as a volatile semiconductor memory and a non-volatile semiconductor memory. In addition, the memory 22 may also have a dedicated storage circuit for each operation unit when the processor 23 has a plurality of operation units as described later. In addition, the memory 22 stores various data and parameters used in the vehicle control processing executed by the processor 23 of the ECU 3, such as images received from the camera 2, various parameters for determining each identifier used in the vehicle control processing, etc. Furthermore, the memory 22 stores various data generated during the vehicle control processing, such as a detection object list representing information related to the detected surrounding vehicles, for a certain period of time. Furthermore, the memory 22 may also store information used in the driving control of the vehicle 10, such as map information.

[0030] The processor 23 is an example of a control unit, and has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may also have other computing circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit (GPU). In addition, whenever the processor 23 receives an image from the camera 2 during the driving of the vehicle 10, it detects surrounding vehicles based on the received image and performs automatic driving control on the vehicle 10 so that the vehicle 10 does not collide with surrounding vehicles.

[0031] Figure 3 1 is a functional block diagram of the processor 23 of the ECU 3 related to vehicle control processing. The processor 23 includes an object detection unit 31, a tracking unit 32, a state recognition unit 33, a first control instruction determination unit 34, a second control instruction determination unit 35, a control instruction integration unit 36, and a vehicle control unit 37. These units of the processor 23 are, for example, functional blocks implemented by a computer program running on the processor 23. Alternatively, these units of the processor 23 may be dedicated operation circuits provided in the processor 23. In addition, when the vehicle 10 is provided with a plurality of cameras, the processor 23 may also perform vehicle control processing based on the images obtained by the cameras for each camera.

[0032] The object detection unit 31 detects an area (i.e., object area) having a predetermined shape including surrounding vehicles represented in the image and identifies the types of surrounding vehicles by inputting the latest image received into the identifier for object detection each time an image is received from the camera 2.

[0033] In this embodiment, the object detection unit 31 uses a DNN that has been pre-learned to detect an object area including surrounding vehicles represented in an image and to identify the types of surrounding vehicles as an identifier for object detection. The DNN used by the object detection unit 31 can be, for example, a DNN having a convolutional neural network (hereinafter referred to as CNN) type architecture such as Single Shot MultiBox Detector (SSD) or Faster R-CNN.

[0034] For example, in the case where the recognizer for object detection is constructed as a DNN, the recognizer has a main body arranged on the input side of the input image, and a position detection unit and a type estimation unit arranged on the output side compared to the main body. The position detection unit outputs the circumscribed rectangle of the detection target object such as the surrounding vehicles represented on the image as the object area based on the output from the main body. In addition, the shape of the object area is not limited to a rectangular shape, for example, it can also be a circular, elliptical or a polygonal shape larger than a pentagon. The type estimation unit calculates the certainty of various types of surrounding vehicles (such as ordinary passenger cars, buses, trucks, two-wheeled vehicles, etc.) represented by the object area detected by the position detection unit based on the output from the main body. In addition, the position detection unit and the type estimation unit can also be formed integrally.

[0035] The backbone can be, for example, a CNN having multiple layers connected in series from the input side to the output side. The multiple layers include more than two convolutional layers. Furthermore, the multiple layers of the backbone can also include a pooling layer set according to one or more convolutional layers.

[0036] The main body performs operations of each layer on the image when the image is input, thereby outputting a feature map calculated from the image. In addition, the main body may also output multiple feature maps with different resolutions. For example, the main body may also input a feature map having the same resolution as the resolution of the input image, and one or more feature maps having a resolution lower than the resolution of the input image.

[0037] The feature maps output from the main body are input to the position detection unit and the type estimation unit, respectively. Furthermore, the position detection unit and the type estimation unit can be, for example, respectively set as CNNs having multiple layers connected in series from the input side to the output side. With respect to each part of the position detection unit and the type estimation unit, the multiple layers of the CNN include more than two convolutional layers. In addition, with respect to each part of the position detection unit and the type estimation unit, the multiple layers of the CNN may also include pooling layers provided according to one or more convolutional layers. In addition, the convolutional layers and pooling layers of the CNN may also be shared with respect to the position detection unit and the type estimation unit. Furthermore, with respect to each part of the position detection unit and the type estimation unit, more than one fully connected layer may be included in the multiple layers. In this case, the fully connected layer is preferably provided on the output side compared to each convolutional layer. In addition, the output from each convolutional layer may also be directly input to the fully connected layer. The output layer of the type estimation unit may be a Softmax layer that calculates the certainty of each type of the surrounding vehicles according to the Softmax function, or a Sigmoid layer that calculates the certainty of each type of the surrounding vehicles according to the Sigmoid function.

[0038] The position detection unit and the type estimation unit are trained, for example, for regions of various positions, various sizes, and various aspect ratios on the image so as to output the confidence of each type of the surrounding vehicle. Therefore, the recognizer outputs the confidence of each type of the surrounding vehicle for regions of various positions, various sizes, and various aspect ratios on the image by receiving the image as input. Furthermore, the position detection unit and the type estimation unit detect a region where the confidence of any type of surrounding vehicle is greater than a predetermined confidence threshold as an object region representing the type of surrounding vehicle.

[0039] For example, labels indicating the type of surrounding vehicles to be detected and the circumscribed rectangle of the surrounding vehicles representing the object region of the surrounding vehicles are added to images (teacher images) included in teacher data used for training the classifier.

[0040] The classifier is trained using a large number of teacher images as described above, for example, by a learning method such as the error back propagation method. The processor 23 can detect surrounding vehicles to be detected from images with high accuracy by using the classifier trained in this way.

[0041] According to a modified example, the object detection unit 31 may also detect surrounding vehicles from an image using a recognizer other than a DNN. For example, the object detection unit 31 may also use a support vector machine (SVM) as a recognizer for object detection, and the support vector machine is pre-learned so that it takes the feature quantity calculated from the window set on the learning image as input and outputs the confidence that the window indicates the presence of surrounding vehicles. The object detection unit 31 calculates the feature quantity (such as HOG) from the window while making various changes to the position, size, and aspect ratio of the window set on the image, and inputs the calculated feature quantity to the SVM, thereby obtaining the confidence for the window. Furthermore, the object detection unit 31 determines that a window whose confidence becomes above a predetermined confidence threshold indicates the presence of surrounding vehicles, and the window is regarded as an object area.

[0042] In addition, the object detection unit 31 can also detect objects other than surrounding vehicles that have an impact on the driving control of the vehicle 10. Such objects include, for example, people, road signs, traffic lights, road markings such as lane markings, and other objects on the road. In this case, the object detection recognizer is pre-learned so that it can also detect these objects. In addition, the object detection unit 31 can also detect these objects by inputting images to the object detection recognizer.

[0043] The object detection unit 31 may further perform non-maximum suppression (NMS) processing to select one object region estimated to represent the same object from two or more object regions that at least partially overlap.

[0044] The object detection unit 31 registers the position and range of each object region on the image and the type of object contained in the object region in the detection object list. And, the object detection unit 31 stores the detection object list in the memory 22. Furthermore, the object detection unit 31 stores, for each object region, a feature map calculated from each pixel contained in the object region by the main body of the object detection identifier and output to the state recognition unit 33 in the memory 22. In addition, the feature map output to the state recognition unit 33 can be set to have the same resolution as the resolution of the image input to the object detection identifier. In addition, in the case where a feature map with a resolution lower than the resolution of the input image is calculated by the pooling layer or the like of the main body of the object detection identifier, the feature map with the low resolution can also be output to the state recognition unit 33. Furthermore, a plurality of feature maps with different resolutions calculated by the main body of the object detection identifier can also be output to the state recognition unit 33.

[0045] The tracking unit 32 tracks the surrounding vehicles shown in each object region detected from the latest image by associating the surrounding vehicles shown in the object region with the surrounding vehicles detected from the past images with reference to the detected object list.

[0046] The tracking unit 32 tracks the surrounding vehicles represented in the object area by, for example, applying a tracking process based on optical flow, such as the Lucas-Kanade method, to the object area of ​​interest in the latest image and the object area in the past image. Therefore, the tracking unit 32 extracts a plurality of feature points from the object area by, for example, applying a filter for feature point extraction, such as SIFT or Harris operator, to the object area of ​​interest. And, the tracking unit 32 calculates the optical flow by determining the corresponding points in the object area in the past image according to the applied tracking method for each of the plurality of feature points. Alternatively, the tracking unit 32 may track the surrounding vehicles represented in the object area by applying other tracking methods used in tracking moving objects detected from images to the object area of ​​interest in the latest image and the object area in the past image.

[0047] The tracking unit 32 assigns an identification number different from that of other surrounding vehicles being tracked to a surrounding vehicle detected from the latest image that is not associated with the surrounding vehicle represented by the past image as a new tracking object, and registers the assigned identification number in the detection object list. On the other hand, the tracking unit 32 associates the surrounding vehicle detected from the latest image that is associated with the surrounding vehicle represented by the past image, that is, the surrounding vehicle being tracked, with the same identification number as the identification number assigned to the surrounding vehicle being tracked.

[0048] The tracking unit 32 notifies the state recognition unit 33 of the identification number of the surrounding vehicle being tracked as the object of state recognition. In addition, the tracking unit 32 estimates the relative positional relationship, relative speed, and deceleration of the surrounding vehicle being tracked as the object of state recognition and the vehicle 10, and registers the estimation result in the detection list.

[0049] The tracking unit 32 can determine the direction from the vehicle 10 to the surrounding vehicles based on the position of the object area including the surrounding vehicles on the image and the optical axis direction of the camera 2. In addition, the position of the lower end of the object area is assumed to indicate the position where the surrounding vehicles included in the object area are in contact with the road surface. Therefore, the tracking unit 32 can estimate the distance to the surrounding vehicles indicated in the object area based on the direction from the camera 2 corresponding to the lower end of the object area and the installation height of the camera 2. In addition, when the distance between the vehicle 10 and the surrounding vehicles in the direction orthogonal to the traveling direction of the vehicle 10 (hereinafter referred to as the lateral direction) is shorter than the distance equivalent to the width of one lane, the tracking unit 32 estimates that the vehicle 10 and the surrounding vehicles are traveling in the same lane. On the other hand, when the distance between the vehicle 10 and the surrounding vehicles in the lateral direction is the distance equivalent to the width of one lane, the tracking unit 32 estimates that the surrounding vehicles are traveling in a lane adjacent to the lane in which the vehicle 10 is traveling. Furthermore, based on the change in the distance and direction from the vehicle 10 to the surrounding vehicles and the speed and acceleration / deceleration of the vehicle 10 when each image is generated, the tracking unit 32 estimates the relative speed between the vehicle 10 and the surrounding vehicles and the deceleration of the surrounding vehicles. In addition, the change in the distance and direction from the vehicle 10 to the surrounding vehicles is estimated based on the change in the position of the object area in each image generated during the tracking of the surrounding vehicles.

[0050] The state recognition unit 33 inputs the features obtained from the pixel values ​​in the object area containing each of the surrounding vehicles being tracked into a recognizer for signal light state recognition having a recursive structure (hereinafter referred to as the signal light state recognizer) every time an image is obtained from the camera 2. That is, the features of the object area of ​​the surrounding vehicle in the latest image obtained by the camera 2 and the features of the object area of ​​the surrounding vehicle in each of one or more past images obtained by the camera 2 before the latest image are input to the signal light state recognizer in a time series order. In addition, the state recognition unit 33 recognizes the signal light state of the surrounding vehicle that changes in appearance with the time series as the output result of the recognizer.

[0051] The state recognition unit 33 can use, as a feature obtained based on the pixel values ​​in the object area representing the surrounding vehicles, for example, the features included in the object area in the feature map calculated by the main body of the recognizer for object detection. Thus, not only the features of the surrounding vehicles themselves, but also the features of the environment around the surrounding vehicles can be used for state recognition. In this embodiment, the influence caused by the relative positional relationship between the surrounding vehicles and other vehicles traveling in front and behind the surrounding vehicles, such as a situation where a part of the turn signal of the surrounding vehicles is blocked by other vehicles, is also considered, and the signal light state recognizer can recognize the state of the signal lights of the surrounding vehicles. For example, when the resolution of the feature map is the same as the resolution of the image input to the recognizer for object detection, each feature included in the area on the feature map corresponding to the object area on the image becomes a feature obtained based on the pixel values ​​in the object area. In addition, when the resolution of the feature map is lower than the resolution of the image input to the recognizer for object detection, the position and range after the coordinates of the object area are corrected according to the ratio of the resolution of the feature map to the resolution of the input image becomes the area on the feature map corresponding to the object area. For example, the upper left end position and the upper right end position of the object area on the input image are set to (tlX, tlY) and (brX, brY), respectively, and the input image is downsized by 1 / N (N is an integer greater than 2) to calculate the feature map. In this case, the upper left end position and the lower right end position of the area on the feature map corresponding to the object area on the image become (tlX / N, tlY / N) and (brX / N, brY / N), respectively.

[0052] According to a modified example, the state recognition unit 33 may use the pixel values ​​in the object region representing the surrounding vehicles on the image input to the object detection recognizer as the feature input to the signal light state recognizer and obtained based on the pixel values ​​in the object region representing the surrounding vehicles. Alternatively, the state recognition unit 33 may use the value obtained by performing a predetermined filtering process such as a convolution operation on each pixel in the object region as the feature input to the signal light state recognizer and obtained based on the pixel values ​​in the object region representing the surrounding vehicles.

[0053] The state recognition unit 33 adjusts (resizes) the extracted features to a predetermined size (e.g., 32×32) by downsampling or upsampling the extracted features for each object region. Thus, even if the relative distance between the vehicle 10 and the surrounding vehicles changes during tracking of the surrounding vehicles, or the size of the surrounding vehicles on the image changes, the signal light state recognizer processes the input features as a certain size, thereby simplifying the structure of the signal light state recognizer.

[0054] The state recognition unit 33 can use a neural network with a recursive structure such as a recurrent neural network (RNN), a long short-term memory network (LSTM) or a gated recurrent unit (GRU) as a signal light state recognizer with a recursive structure. Such a neural network has an input layer to which features obtained based on pixel values ​​in an object area are input, an output layer to which the recognition result of the signal light state is output, and at least one intermediate layer arranged in sequence from the input layer to the output layer between the input layer and the output layer. Moreover, any intermediate layer is configured to have the above-mentioned recursive structure. Furthermore, the intermediate layer may also include a layer that performs a predetermined operation such as a convolution operation on the output of the layer from the upstream side. The signal light state recognizer only needs to process the features contained in the object area, so the size of the input layer and the intermediate layer can be small compared to the recognizer for object detection, and the number of parameters such as weight coefficients used to define the signal light state recognizer can be small. Therefore, the amount of calculation of the signal light state recognizer is less than that of the recognizer for object detection, which can reduce the calculation load on the processor 23. Furthermore, the amount of calculation required for learning the traffic light state recognition device can also be reduced.

[0055] The signal light state recognizer has a recursive structure, so whenever features are input sequentially in a time series manner, the internal state is updated. Thus, the signal light state recognizer can recognize the lighting state of the signal light based on the time series changes of the signal lights of the surrounding vehicles being tracked. In this embodiment, the state recognition unit 33 recognizes whether any left or right turn signal or hazard warning light is in a flashing state, and whether the brake light is in a lit state or an extinguished state as the signal light state of the surrounding vehicles. Therefore, as the activation function of the output layer of the signal light state recognizer, for example, a sigmoid function can be used. Thus, the signal light state recognizer can output the confidence of each state. In addition, the state recognition unit 33 compares the confidence of each state with the corresponding threshold value, and determines that the signal light state of the surrounding vehicles is in a state where the confidence becomes above the corresponding threshold value. For example, the confidence of the flashing state of the left turn signal of the surrounding vehicles is set to 0.8, and on the other hand, the confidence of the non-flashing state of the left turn signal is 0.2. Furthermore, when the threshold is set to 0.5, the state recognition unit 33 determines that the signal light state of the surrounding vehicle is in the state where the left turn signal is flashing.

[0056] Alternatively, a softmax function may be used as an activation function of the output layer of the signal light state recognizer. In this case, the signal light state recognizer outputs a determination result of a left turn signal flashing, a right turn signal flashing, a hazard warning light flashing, a brake light on, or none of them as the signal light state of the surrounding vehicles. Therefore, the state recognition unit 33 may use the state represented by the determination result output from the signal light state recognizer as the signal light state of the surrounding vehicles.

[0057] According to a variation, the state recognition unit 33 may also use a neural network having a CNN-type architecture that performs convolution operations in the time axis direction as a traffic light state recognizer. In this case, the traffic light state recognizer, for example, has one or more convolution layers (hereinafter referred to as time feature convolution layers) that perform convolution operations on the feature map output from the previous layer along the time axis direction. The kernel size of each time feature convolution layer in the time axis direction is, for example, set so that the convolution operation is performed as a whole during a period including multiple features input at one time through all the time feature convolution layers. The time feature convolution layer, for example, can perform convolution operations in any direction of the time axis direction and the spatial direction (hereinafter referred to as three-dimensional convolution operations), or can perform convolution operations only in the time axis direction (hereinafter referred to as time dimension convolution operations). Furthermore, the time feature convolution layer can also perform convolution operations or fully connected operations in the channel direction. In addition, in the case where the signal light state recognizer has multiple time feature convolution layers, any one of the multiple time feature convolution layers can also perform a three-dimensional convolution operation, and the other layers of the multiple time feature convolution layers can perform a time dimension convolution operation. Further, the signal light state recognizer can also have one or more convolution layers (hereinafter referred to as spatial feature convolution layers) that do not perform convolution operations in the time axis direction but perform convolution operations in the spatial direction. The spatial feature convolution layer can also perform convolution operations or full connection operations in the channel direction. In the case where the signal light state recognizer has more than one spatial feature convolution layer, the order of the spatial feature convolution layer and the temporal feature convolution layer can also be in any order. For example, the convolution layers can also be arranged in the order of spatial feature convolution layer and temporal feature convolution layer from the input side to the output side, or in the opposite order. Further, the spatial feature convolution layer and the temporal feature convolution layer can also be arranged alternately. Further, the signal light state recognizer can also have more than one pooling layer. Further, the signal light state recognizer can also have more than one activation layer and more than one fully connected layer. The output layer of the traffic light state recognizer uses, for example, a Sigmoid function or a Softmax function as an activation function.

[0058] The state recognition unit 33 registers the recognition result or the certainty of the signal light state of each surrounding vehicle to be recognized in the detection object list, and notifies the first control instruction determination unit 34 and the second control instruction determination unit 35 .

[0059] The first control instruction determination unit 34 determines the first control instruction including the first control value according to a predetermined rule for controlling the vehicle 10 so as to avoid a collision between the vehicle 10 and the surrounding vehicles based on the recognition result of the signal light state of the surrounding vehicles and the relative positional relationship between the vehicle 10 and the surrounding vehicles. That is, the first control instruction determination unit 34 determines the first control value according to a so-called rule base. In the present embodiment, the first control value is the acceleration and deceleration of the vehicle 10 that is targeted to avoid a collision between the vehicle 10 and the surrounding vehicles.

[0060] For example, the predetermined rule may be set as follows: the period P during which the identification result of the signal light state of the surrounding vehicle is the lighting state of the brake light is greater than the time threshold T1, and the surrounding vehicle and the vehicle 10 are traveling in the same lane. In addition, as another predetermined rule, it may be set as follows: the period P during which the signal light state of the surrounding vehicle is the flashing state of the right turn signal is greater than the time threshold T1, and the surrounding vehicle is traveling in a lane adjacent to the left side of the driving lane of the vehicle 10 (hereinafter referred to as the own lane). Further, as another predetermined rule, it may be set as follows: the period P during which the signal light state of the surrounding vehicle is the flashing state of the left turn signal is greater than the time threshold T1, and the surrounding vehicle is traveling in a lane adjacent to the right side of the own lane. When any of the surrounding vehicles satisfies any of these control conditions, the first control instruction determination unit 34 determines that it is necessary to decelerate the vehicle 10. On the other hand, when none of the surrounding vehicles satisfies these control conditions, the first control instruction determination unit 34 determines that it is not necessary to decelerate the vehicle 10.

[0061] When it is determined that the vehicle 10 needs to be decelerated, the first control instruction determination unit 34 determines the deceleration for decelerating the vehicle 10 as the first control value. Therefore, the first control instruction determination unit 34 determines the deceleration of the vehicle 10 based on at least one of the distance between the vehicle 10 and the surrounding vehicle that is the cause of the determination, the relative speed of the surrounding vehicle with respect to the vehicle 10, and the deceleration of the surrounding vehicle. For example, the shorter the distance between the vehicle 10 and the surrounding vehicle, the larger the speed difference obtained by subtracting the speed of the surrounding vehicle from the speed of the vehicle 10, or the larger the deceleration of the surrounding vehicle, the greater the first control instruction determination unit 34 increases the deceleration of the vehicle 10. And, the first control instruction determination unit 34 outputs the first control instruction including the determined deceleration as the first control value to the control instruction integration unit 36.

[0062] On the other hand, when it is determined that the vehicle does not need to be decelerated, the first control instruction determination unit 34 outputs a first control instruction to the control instruction integration unit 36 ​​to instruct the vehicle 10 to maintain the current vehicle speed (ie, the acceleration / deceleration is 0).

[0063] Figure 44 is a diagram for explaining an overview of an example of the determination of the first control instruction. It is assumed that the period P for indicating that the state of the signal light of the surrounding vehicle 401 traveling in front of the vehicle 10 in the own lane among the surrounding vehicles shown in the image 400 is the lighting state of the brake light is greater than the time threshold T1. In this case, the control instruction for decelerating the vehicle 10 is output as the first control instruction. Similarly, in the case where the period P for indicating that the state of the signal light of the surrounding vehicle 402 traveling in the adjacent lane on the right side of the own lane is the flashing state of the left turn signal is greater than the time threshold T1, the control instruction for decelerating the vehicle 10 is also output as the first control instruction. On the other hand, if the state of the signal light of the surrounding vehicle traveling in the own lane is not the lighting state of the brake light, and the state of the signal light of the surrounding vehicle traveling in the adjacent lane is not the flashing state of the turn signal on the side of the own lane, the first control instruction for maintaining the current speed of the vehicle 10 is output.

[0064] In addition, the predetermined rules are not limited to the above-mentioned examples. For example, the predetermined rules may also include rules for accelerating the vehicle 10. For example, it is assumed that a camera for photographing the rear of the vehicle 10 is provided on the vehicle 10, which is different from the above-mentioned camera 2. It is assumed that the surrounding vehicles detected from a series of images generated by the camera for photographing the rear are traveling in an adjacent lane relative to the own lane behind the vehicle 10, and the signal light state of the surrounding vehicles is a flashing state of the turn signal on the side of the own lane. In this case, it is presumed that the surrounding vehicles are about to change lanes to the own lane behind the vehicle 10. Therefore, the first control instruction determination unit 34 sets the first control value included in the first control instruction to a value for accelerating the vehicle 10 so as to increase the distance between the vehicle 10 and the surrounding vehicles.

[0065] Furthermore, the predetermined rules may also include rules that utilize the output (e.g., feature map) from any intermediate layer of the signal light state recognizer. For example, it is assumed that the sum of the squares of the values ​​contained in the latest feature map output from any intermediate layer of the signal light state recognizer for surrounding vehicles traveling in front of the vehicle 10 on its own lane is smaller than a predetermined threshold. In this case, it is assumed that the contrast in the object area is low and the reliability of the signal light state recognition result of the signal light state recognizer is low. Therefore, the first control instruction determination unit 34 may also determine the first control value regardless of the signal light state. For example, in a case where the sum of the squares of the values ​​contained in the feature map of surrounding vehicles traveling on its own lane is smaller than a predetermined threshold, the first control instruction determination unit 34 may also generate a first control instruction including a greater deceleration as the inter-vehicle distance between the surrounding vehicles and the vehicle 10 is shorter.

[0066] As described above, when any one or more of the surrounding vehicles satisfies any rule, the first control instruction determination unit 34 outputs the first control instruction including the first control value for instructing the behavior of the vehicle 10 corresponding to the satisfied rule. Furthermore, when the inter-vehicle distance between the surrounding vehicles traveling in the own lane and the vehicle 10 becomes less than the predetermined distance, the first control instruction determination unit 34 may output the first control instruction for decelerating the vehicle 10 regardless of the signal light state of the surrounding vehicles. In addition, the greater the vehicle speed of the vehicle 10, the longer the predetermined distance is set.

[0067] The second control instruction determination unit 35 obtains the second control instruction including the second control value by inputting the recognition result of the signal light state of each surrounding vehicle to the recognizer for control instruction determination. In the following, the recognizer for control instruction determination is referred to as the control instruction recognizer. The control instruction recognizer is pre-learned so that it outputs a control value for controlling the vehicle 10 to avoid a collision between the vehicle 10 and the surrounding vehicles. In addition, in the present embodiment, the second control value is the acceleration and deceleration of the vehicle 10 that is targeted in order to avoid a collision between the vehicle 10 and the surrounding vehicles, similarly to the first control value. In addition, as an input parameter of the control instruction recognizer, at least one of the relative positional relationship between the vehicle 10 and the surrounding vehicles, the relative vehicle speed, the relative acceleration and deceleration, the size of the object area including the surrounding vehicles, and the position of the object area on the image can also be used together with the recognition result of the signal light state. Furthermore, the output (e.g., feature map) of any intermediate layer from the signal light state recognizer is used as one of the input parameters to the control instruction recognizer. Furthermore, the second control instruction determination unit 35 may input not only the latest parameter but a plurality of parameters obtained in the most recent predetermined period into the control instruction recognition unit regarding one or more of the above-mentioned input parameters. By inputting such parameters together with the recognition result of the signal light state, the control instruction recognition unit can output a more appropriate second control value even in a driving environment where it is difficult to recognize the signal light state of surrounding vehicles.

[0068] The second control instruction determination unit 35 can use a neural network with a recursive structure such as RNN, LSTM or GRU as a control instruction identifier. Alternatively, the second control instruction determination unit 35 can also use a neural network with other structures that do not have a recursive structure as a control instruction identifier. For example, the second control instruction determination unit 35 can use the following neural network as a control instruction identifier, which has an input layer, one or more intermediate layers as fully connected layers, and an output layer that uses a sigmoid function or a softmax function as an activation function and outputs any control output. Alternatively, the second control instruction determination unit 35 can also use an identifier according to a machine learning model other than a neural network as a control instruction identifier. The second control instruction determination unit 35 can use, for example, an SVM, a logistic regression model or a naive Bayes classifier as such an identifier.

[0069] The control instruction recognizer is pre-learned according to a learning method corresponding to the control instruction recognizer using a large amount of teacher data, and the large amount of teacher data includes combinations of input parameters and appropriate control outputs for avoiding collisions for those input parameters. In addition, appropriate control outputs for avoiding collisions also include control outputs such as avoiding unnecessary deceleration and unnecessary route changes by maintaining the vehicle speed and route of the vehicle 10 without the fear of collision. Therefore, the second control instruction determiner 35 can obtain appropriate control instructions corresponding to the signal light status of surrounding vehicles by using such a control instruction recognizer.

[0070] In addition, when the signal light state recognizer and the control instruction recognizer are respectively constructed as neural networks, the common teacher data can be used to learn the neural networks in an integrated manner by the error back propagation method. In this way, by learning the recognizers in an integrated manner, even if the environment around the vehicle 10 is such that it is difficult to recognize the signal light state of surrounding vehicles, the recognizers can determine appropriate control instructions. In the case where the recognizer for object detection is also constructed as a neural network, the recognizer for object detection can also be learned in an integrated manner together with the signal light state recognizer and the control instruction recognizer.

[0071] The second control instruction determination unit 35 outputs the determined second control instruction to the control instruction integration unit 36 ​​.

[0072] The control instruction integration unit 36 ​​determines an integrated control instruction including an integrated control value for avoiding a collision between the vehicle 10 and a surrounding vehicle based on a difference between a first control value included in the first control instruction and a second control value included in the second control instruction or an absolute value of the second control value.

[0073] For example, when the absolute value of the difference between the acceleration and deceleration specified by the first control instruction and the acceleration and deceleration specified by the second control instruction (hereinafter referred to as the acceleration and deceleration difference) is below a certain value, the reliability of those control instructions is considered to be high. Therefore, when the period P during which the acceleration and deceleration difference between the first control instruction and the second control instruction is below a certain value lasts for more than the second time threshold T2, the control instruction comprehensive unit 36 ​​determines the average value of the acceleration and deceleration included in each of the latest first control instruction and the second control instruction as the comprehensive control value. Alternatively, the control instruction comprehensive unit 36 ​​may also determine the average value of the acceleration and deceleration included in each of the multiple first control instructions and the multiple second control instructions received during the period P as the comprehensive control value. In this way, the acceleration and deceleration of the vehicle 10 can be suppressed from changing rapidly.

[0074] On the other hand, when the period P during which the difference in acceleration / deceleration between the first control instruction and the second control instruction is greater than a certain value continues for more than the third time threshold value T3, it is considered that the reliability of those control instructions is relatively low. Therefore, the control instruction comprehensive unit 36 ​​determines the average value of the acceleration / deceleration in the first control instruction received during the period P and the average value of the acceleration / deceleration in the second control instruction received during the period P, whichever has the larger absolute value, as the comprehensive control value included in the comprehensive control instruction. Thus, for example, although it is determined by the first control instruction determination unit 34 that deceleration control is not required, when deceleration control is actually required, the control instruction comprehensive unit 36 ​​can also generate a comprehensive control instruction that indicates deceleration control according to the output from the second control instruction determination unit 35. In addition, the third time threshold value T3 is preferably set to a time longer than the second time threshold value T2, for example.

[0075] In addition, it is assumed that: when the period P during which the difference in acceleration and deceleration between the first control instruction and the second control instruction is greater than a certain value is shorter than the second time threshold T2, the reliability of at least one of the first control instruction and the second control instruction is unstable. Therefore, the control instruction comprehensive unit 36 ​​can also include the acceleration and deceleration obtained by applying feedback control to the acceleration and deceleration included in the second control instruction as a comprehensive control value in the comprehensive control instruction. In addition, as feedback control, for example, PD control or PID control is used. As a result, the control instruction comprehensive unit 36 ​​can suppress the acceleration and deceleration of the vehicle 10 from changing sharply, and can make the vehicle 10 accelerate or decelerate smoothly.

[0076] It is further assumed that: when the variance value of the acceleration / deceleration of any one of the first control instruction and the second control instruction within a certain period is larger than a certain value, the reliability of at least one of the first control instruction and the second control instruction is unstable. Therefore, the control instruction comprehensive unit 36 ​​ignores the control instruction assumed to be unstable in reliability during a predetermined time period, and directly uses the acceleration / deceleration value of the other control instruction as the comprehensive control value during the period, so that the vehicle 10 can be accelerated or decelerated smoothly.

[0077] Alternatively, the control instruction comprehensive unit 36 ​​may also use the acceleration / deceleration included in the second control instruction as the comprehensive control value included in the comprehensive control instruction when the absolute value of the acceleration / deceleration included in the second control instruction is above a predetermined upper threshold. Alternatively, when the absolute value of the acceleration / deceleration included in the second control instruction is above a predetermined upper threshold, the control instruction identifier may output an abnormal value because the environment around the vehicle 10 is an unexpected environment. Therefore, in this case, the control instruction comprehensive unit 36 ​​uses the acceleration / deceleration included in the first control instruction as the comprehensive control value. Alternatively, the control instruction comprehensive unit 36 ​​may also be set to use the same control value as the comprehensive control value included in the comprehensive control instruction output not long ago as the comprehensive control value this time to continue the same control.

[0078] The control instruction integration unit 36 ​​notifies the vehicle control unit 37 of the integrated control instruction.

[0079] The vehicle control unit 37 controls each part of the vehicle 10 so that the vehicle 10 travels according to the integrated control value included in the integrated control instruction received from the control instruction integration unit 36. For example, the vehicle control unit 37 sets the accelerator opening or the braking amount so as to eliminate the difference between the current acceleration of the vehicle 10 measured by the acceleration sensor (not shown) and the acceleration / deceleration as the integrated control value included in the received integrated control instruction. At that time, the vehicle control unit 37 can also set the accelerator opening or the braking amount through feedback control such as PD control or PID control. In addition, the vehicle control unit 37 obtains the fuel injection amount according to the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 37 obtains the power supplied to the motor of the vehicle 10 according to the set accelerator opening, and controls the driving device of the motor so that the power is supplied to the motor. Alternatively, the vehicle control unit 37 outputs a control signal corresponding to the set braking amount to the brake of the vehicle 10.

[0080] Figure 5 The flowchart of the vehicle control process executed by the processor 23 is shown in FIG. Figure 5 The action flowchart shown executes the vehicle control process.

[0081] The object detection unit 31 of the processor 23 inputs the latest image obtained from the camera 2 to the identifier for object detection, and detects the surrounding vehicles represented in the image. That is, the object detection unit 31 detects the object area including the surrounding vehicles on the image (step S101). In addition, the object detection unit 31 registers the detected surrounding vehicles in the detection object list.

[0082] The tracking unit 32 of the processor 23 tracks the surrounding vehicles shown in each object region including the surrounding vehicles in the latest image based on the object region and the object regions in the past images (step S102 ).

[0083] The state recognition unit 33 of the processor 23 recognizes the signal light state of each of the surrounding vehicles being tracked by inputting a feature obtained based on pixel values ​​in an object region representing the surrounding vehicle into a signal light state recognizer (step S103).

[0084] Furthermore, the first control instruction determination unit 34 of the processor 23 determines the first control instruction according to a predetermined rule based on the signal light states of each surrounding vehicle being tracked and the relative positional relationship between each surrounding vehicle and the vehicle 10 (step S104 ).

[0085] Furthermore, the second control instruction determination unit 35 of the processor 23 inputs the recognition result of the signal light state of each surrounding vehicle being tracked into the control instruction recognition unit, thereby determining the second control instruction (step S105).

[0086] The control instruction integration unit 36 ​​of the processor 23 determines the integrated control instruction based on the first control instruction and the second control instruction (step S106). Furthermore, the vehicle control unit 37 of the processor 23 controls the vehicle 10 so that the vehicle 10 travels according to the integrated control instruction (step S107). Then, the processor 23 ends the vehicle control process.

[0087] As described above, the vehicle control device detects surrounding vehicles from a series of images obtained in a time series manner by a camera mounted on the vehicle, and tracks the surrounding vehicles. Furthermore, the vehicle control device recognizes the signal light state of the surrounding vehicles by inputting the features obtained from the object area representing the surrounding vehicles in each image into the signal light state recognizer. Furthermore, the vehicle control device obtains the first control instruction related to the driving of the vehicle according to a predetermined rule based on the recognition result of the signal light state of the surrounding vehicles and the relative positional relationship between the vehicle and the surrounding vehicles. Furthermore, the vehicle control device obtains the second control instruction related to the driving of the vehicle by inputting the recognition result of the signal light state of the surrounding vehicles into the control instruction recognizer. Furthermore, the vehicle control device determines the comprehensive control instruction based on the first control instruction and the second control instruction, and controls the driving of the vehicle according to the determined control instruction. In this way, the vehicle control device not only refers to the recognition result of the signal light state of the surrounding vehicles, but also refers to the relative positional relationship between the vehicle and the surrounding vehicles. Therefore, even in the case where it is difficult to accurately recognize the signal light state, the first control instruction can be determined more appropriately. Furthermore, the vehicle control device uses a control instruction identifier constructed according to a machine learning system using the recognition result of the signal light state as one of the input parameters. For the control instruction identifier, data representing the combination of the recognition result of the signal light state in a driving environment where sufficient recognition accuracy cannot be obtained and the vehicle control that should be performed is also used as teacher data for learning. Therefore, the control instruction identifier can determine a more appropriate second control instruction even when it is difficult to accurately recognize the signal light state. In addition, the vehicle control device can determine a control instruction that is considered more appropriate to avoid a collision between the vehicle and surrounding vehicles by referring to both the first control instruction and the second control instruction. Therefore, the vehicle control device can appropriately control the vehicle even if it is difficult to recognize the state of the signal lights of other vehicles around the vehicle. For example, the vehicle control device can suppress the situation where the surrounding vehicles do not merge in front of the vehicle but perform deceleration control, and on the contrary, the flashing of the direction indicator accompanying the merging of the surrounding vehicles or the lighting of the brake lights accompanying the deceleration of the surrounding vehicles is overlooked and the braking is not performed.

[0088] In addition, the computer program for implementing the functions of each part of the processor 23 of the vehicle control device involved in the above-mentioned embodiment or modification example can also be provided in the form of a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0089] As described above, those skilled in the art can make various modifications in accordance with the embodiments within the scope of the present invention.

Claims

1. A vehicle control device, comprising: an object detection unit that detects an object region representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided in the vehicle and photographing the surroundings of the vehicle; a state recognition unit that recognizes the state of the signal lights of the surrounding vehicles by inputting features extracted from the object region on the image into a signal light state recognizer, the signal light state recognizer being pre-learned so as to recognize the state of the signal lights of the surrounding vehicles; a first control instruction determination unit that determines a first control value according to a predetermined rule for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles based on a recognition result of the state of the signal lamp of the surrounding vehicle and a relative positional relationship between the vehicle and the surrounding vehicles; a second control instruction determination unit that determines a second control value by inputting a recognition result of the state of the signal lamp of the surrounding vehicle into a control instruction recognition unit that is pre-learned to output the second control value, the second control value being a control value for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicle; a control instruction integration unit that determines an integrated control value for avoiding a collision between the vehicle and the surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; and A control unit controls the travel of the vehicle according to the comprehensive control value.

2. The vehicle control device according to claim 1, The first control instruction determination unit outputs the first control value at a predetermined period, and the second control instruction determination unit outputs the second control value at the predetermined period, When the period in which the absolute value of the difference between the first control value and the second control value is less than a certain value continues for more than a predetermined time threshold, the control instruction comprehensive unit takes the average value of any one of the first control values ​​output during the period and any one of the second control values ​​output during the period as the comprehensive control value.

3. The vehicle control device according to claim 1, The first control instruction determination unit outputs the first control value at a predetermined period, and the second control instruction determination unit outputs the second control value at the predetermined period, When the period in which the absolute value of the difference between the first control value and the second control value is greater than a certain value continues for more than a predetermined time threshold, the control instruction comprehensive unit uses the larger absolute value of the average value of the first control value output during the period and the average value of the second control value output during the period as the comprehensive control value.

4. The vehicle control device according to claim 1, When the absolute value of the second control value is equal to or greater than a predetermined upper limit value, the control instruction comprehensive unit uses the predetermined upper limit value or the first control value as the comprehensive control value.

5. The vehicle control device according to any one of claims 1 to 4, The second control instruction determination unit determines the second control value by inputting at least one of the relative position relationship between the vehicle and the surrounding vehicles, the relative vehicle speed, the relative acceleration and deceleration, the size of the object area and the position of the object area together with the recognition result of the state of the signal lights of the surrounding vehicles into the control instruction identifier.

6. The vehicle control device according to any one of claims 1 to 4, The signal light state recognizer comprises an input layer to which features extracted from the object region are input, an output layer to which the states of signal lights of surrounding vehicles are output, and at least one intermediate layer arranged sequentially from the input layer to the output layer between the input layer and the output layer, wherein the at least one intermediate layer performs a predetermined operation on the output from a layer closer to the input layer than the intermediate layer, The second control instruction determination unit determines the second control value by inputting the result of the predetermined operation on any intermediate layer of the at least one intermediate layer of the signal light state recognizer into the control instruction recognizer together with the recognition result of the state of the signal light of the surrounding vehicle.

7. The vehicle control device according to any one of claims 1 to 4, The signal light state identifier has a recursive structure. The state recognition unit recognizes the state of the signal lights of the surrounding vehicles by inputting features extracted from the object area in each of one or more past images obtained by the shooting unit before the image and features extracted from the object area on the image into the signal light state identifier in a time series order.

8. The vehicle control device according to any one of claims 1 to 4, The signal light state recognizer is a convolutional neural network having a convolutional layer that performs a convolution operation in the time axis direction. The state recognition unit recognizes the state of the signal lights of the surrounding vehicles by inputting features extracted from the object area in each of one or more past images obtained by the shooting unit before the image and features extracted from the object area on the image into the signal light state recognizer.

9. A vehicle control method, comprising: detecting an object region representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided in the vehicle and photographing the surroundings of the vehicle; recognizing the state of the signal lights of the surrounding vehicles by inputting features extracted from the object region on the image into a signal light state recognizer, the signal light state recognizer being pre-learned to recognize the state of the signal lights of the surrounding vehicles; determining a first control value according to a predetermined rule for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles based on a recognition result of the states of the signal lights of the surrounding vehicles and a relative positional relationship between the vehicle and the surrounding vehicles; Determining a second control value by inputting a recognition result of the state of the signal lamp of the surrounding vehicle into a control instruction recognizer, the control instruction recognizer being pre-learned to output the second control value, the second control value being a control value for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicle; determining a comprehensive control value for avoiding a collision between the vehicle and the surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; The travel of the vehicle is controlled according to the integrated control value.

10. A computer program product, comprising a computer program for controlling a vehicle, the computer program being configured to cause a processor mounted on a vehicle to execute: detecting an object region representing surrounding vehicles traveling around the vehicle from an image obtained by a photographing unit provided in the vehicle and photographing the surroundings of the vehicle; recognizing the state of the signal lights of the surrounding vehicles by inputting features extracted from the object region on the image into a signal light state recognizer, the signal light state recognizer being pre-learned to recognize the state of the signal lights of the surrounding vehicles; determining a first control value according to a predetermined rule for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicles based on a recognition result of the states of the signal lights of the surrounding vehicles and a relative positional relationship between the vehicle and the surrounding vehicles; Determining a second control value by inputting a recognition result of the state of the signal lamp of the surrounding vehicle into a control instruction recognizer, the control instruction recognizer being pre-learned to output the second control value, the second control value being a control value for controlling the vehicle so as to avoid a collision between the vehicle and the surrounding vehicle; determining a comprehensive control value for avoiding a collision between the vehicle and the surrounding vehicles based on a difference between the first control value and the second control value or an absolute value of the second control value; as well as The travel of the vehicle is controlled according to the integrated control value.

Images