Travel assistance device and travel assistance method for travel assistance device

Abstract

A travel assistance device according to the present invention is provided with: a region extraction unit (11) that extracts, from image information of sensor data, a specific region including traffic participants around a host vehicle; a risk derivation unit (15) that derives the degree of risk with respect to the vehicle in safe driving, for the traffic participants in the extracted specific area; a priority level setting unit (16) that sets a processing priority level for the specific region in accordance with the risk level; a processing load setting unit (18) that sets the processing load of the information of the specific region in accordance with the processing priority; and an identification processing unit (19) that identifies the traffic participant on the basis of the processing load information from the processing load setting unit.

Description

Technical Field

[0001] The present invention relates to a driving assistance device for assisting the driving of a vehicle and a driving assistance method thereof, and in particular to a driving assistance device for a vehicle suitable for automatic driving and a driving assistance method thereof. Background Art

[0002] Recently, in the highly automated driving on highways and driving assistance in complex driving environments on general roads, the driving control is required to be more sophisticated, and for this purpose, high-precision external recognition is required. In order to improve the accuracy of external recognition, it is considered to improve the performance of external recognition sensors (for example, high resolution of cameras) and increase the number of external recognition sensors installed.

[0003] Furthermore, in order to perform the recognition process of the outside world recognition in real time, a high computing power of the computer performing the image processing is required, but the computing power of current computers is not sufficient. Therefore, a method of effectively and flexibly utilizing computer resources is required.

[0004] As a countermeasure for this, for example, as described in Japanese Patent Gazette No. 2018-568383 (Patent Document 1), it is proposed to set a priority for the object to be identified, set the computing power corresponding to the priority, and seek to improve the accuracy of external recognition and appropriately allocate the computing power.

[0005] In Patent Document 1, a captured image is divided into a plurality of rectangular regions, and different resolutions are set in the plurality of regions corresponding to the straight-forward direction, the left-turn direction, and the right-turn direction. Specifically, in the straight-forward direction, the resolution near the central region of the captured image is set high, and the resolution outside the central region is set low. In addition, in the left-turn direction, the resolution near the left region of the captured image is set high, and the resolution outside the left region is set low. Similarly, in the right-turn direction, the resolution near the right region of the captured image is set high, and the resolution outside the right region is set low.

[0006] By setting the priority of the resolution according to the traveling direction of the vehicle, it is possible to improve the accuracy of external recognition and appropriately allocate the computing power, thereby making effective use of computer resources.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: (Japanese) Patent Publication No. 2018-56838 Summary of the invention

[0010] Problems to be solved by the invention

[0011] However, in the outside world recognition described in Patent Document 1, different resolutions are set in a plurality of areas according to the traveling direction of the vehicle. However, this method is not expected to be able to cope with the following situation.

[0012] For example, when the vehicle is traveling in a straight line, the resolution near the central area of ​​the captured image is increased, so the resolution is not high in the area outside the central area. Therefore, when there are other vehicles approaching the vehicle in the left area or the right area, the recognition accuracy of the other vehicles is not high, so it takes time to recognize the other vehicles approaching, and there is a concern that the vehicle may come into contact with the vehicle. Of course, the same concern also exists when the vehicle is traveling in the left turn direction or the right turn direction.

[0013] As described above, it is required to achieve early recognition of safety risks such as contact and appropriate allocation of computing power, effectively utilize computer resources, and provide safe driving assistance.

[0014] An object of the present invention is to provide a driving assistance device and a driving assistance method of the driving assistance device that can effectively and flexibly utilize computer resources and perform safe driving assistance by seeking early recognition of danger (ensuring safety) and appropriate allocation of computing power.

[0015] Means for solving problems

[0016] The driving assistance device of the present invention comprises: an information processing unit for processing information of sensor data acquired by a sensor; and a vehicle control unit for performing driving assistance of a vehicle using the information processed by the information processing unit. In the driving assistance device,

[0017] The information processing department has:

[0018] An area extraction unit extracts a specific area of ​​a traffic participant including the surroundings of the vehicle from the image information of the sensor data;

[0019] A risk derivation unit, for extracting the traffic participants in the specific area, deriving the risk level of the vehicle in terms of safe driving;

[0020] A priority setting unit that sets a processing priority for a specific area according to a risk level;

[0021] a processing load setting unit that sets a processing load of information in a specific area according to a processing priority; and

[0022] The recognition processing unit recognizes the traffic participant based on the processing load information from the processing load setting unit.

[0023] Effects of the Invention

[0024] According to the present invention, it is possible to achieve early recognition of danger (security assurance) and appropriate allocation of computing power, thereby making effective use of computer resources and providing safe driving assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a configuration diagram of a driving assistance device according to a first embodiment of the present invention.

[0026] Figure 2 Yes Figure 1 Flowchart for explaining a control flow for setting a processing priority in the driving assistance device shown in FIG.

[0027] Figure 3 1 is an image of the front of the vehicle captured by a camera, and is an explanatory diagram showing an example of a traffic participant.

[0028] Figure 4 It is shown that Figure 3 An illustration of an example of a specific area of ​​a traffic participant in the image shown.

[0029] Figure 5 It is to seek Figure 4 A flow chart illustrating a process for controlling the collision risk of traffic participants in a specific area is shown.

[0030] Fig. 6A 1 is an explanatory diagram showing a first example of collision risk.

[0031] Figure 6B 2 is an explanatory diagram showing a second example of the collision risk.

[0032] Figure 6C : is an explanatory diagram showing a third example of the collision risk.

[0033] Figure 7 This is a flowchart for explaining a control flow for obtaining a processing priority based on the collision risk of traffic participants.

[0034] Figure 8 This is a flowchart illustrating a control process for adjusting the resolution based on the processing priority of traffic participants.

[0035] Fig. 9 It is shown based on Figure 8 An explanatory diagram of an example of an image of a specific area whose resolution is adjusted according to a control flow.

[0036] Fig.10 This is a flowchart for explaining a control flow for adjusting the level of recognition processing based on the processing priority of traffic participants.

[0037] Fig.11 It is an explanatory diagram showing parameters for detecting a dangerous driving state.

[0038] Fig.12 It is an explanatory diagram showing a specific example of parameters for detecting a dangerous driving state.

[0039] Fig.13A This is a flowchart for explaining the control flow of the first control example for avoiding a dangerous state.

[0040] Fig. 13B This is a flowchart for explaining the control flow of the second control example for avoiding a dangerous state.

[0041] Fig. 13C This is a flowchart for explaining the control flow of the third control example for avoiding a dangerous state.

[0042] Fig.14 This is a flowchart for explaining a control flow for adjusting a processing cycle based on the processing priority of traffic participants, which is a second embodiment of the present invention.

[0043] Fig.15 It is shown based on Fig.14 An explanatory diagram of a specific example of a processing cycle adjusted according to a control flow.

[0044] Fig.16 It is an explanatory diagram showing the structure of a neural network for image recognition used in recognition processing which is a third embodiment of the present invention.

[0045] Fig.17 This is a flowchart for explaining a control flow of adjusting the number of neurons based on the processing priority of traffic participants according to the third embodiment of the present invention.

[0046] Fig.18 It is shown by Fig.17 1 is an explanatory diagram of an example of a neural network for image recognition in which a pruning ratio is large and adjusted according to the control flow shown.

[0047] Fig.19 It is shown by Fig.17 An explanatory diagram of an example of a neural network for image recognition in which the reduction rate is small and adjusted according to the control flow shown.

[0048] Fig. 20 It is shown based on Fig.17An explanatory diagram of a specific example of processing of a processing slot with a reduction rate adjusted according to a control flow.

[0049] Fig.21 2 is an image of the front of the vehicle captured by a camera, and is an explanatory diagram showing an example of a specific area including an increased number of traffic participants.

[0050] Fig. 22 This is a flowchart for explaining a control flow of adjusting the number of neurons based on the processing priority of traffic participants, which is a modification of the third embodiment of the present invention.

[0051] Fig.23 It is shown based on Fig. 22 An explanatory diagram of a specific example of processing of a processing time slot with a reduction rate adjusted according to a control flow.

[0052] Fig.24 1 is an explanatory diagram showing an example of a front image of a vehicle captured by a camera and including a specific area including a blind-spot area.

[0053] Fig.25 This is a flowchart for explaining a control flow for obtaining the collision risk in a blind spot area according to a fourth embodiment of the present invention.

[0054] Fig.26 It is a configuration diagram of a driving assistance device according to a fifth embodiment of the present invention.

[0055] Fig. 27 is shown as input to Fig.26 FIG. 2 is an explanatory diagram of an example of an image of a camera of the processing load setting unit shown.

[0056] Fig.28 It is a configuration diagram of a driving assistance device according to a sixth embodiment of the present invention. DETAILED DESCRIPTION

[0057] Hereinafter, embodiments of the present invention will be described in detail using the drawings. However, the present invention is not limited to the following embodiments, and various modifications and application examples are also included in the technical concept of the present invention.

[0058] Example 1

[0059] [Description of the overall system structure]

[0060] Figure 1 Functional blocks showing the configuration of a driving assistance device according to an embodiment of the present invention are shown. The driving assistance device performs "automatic driving of a vehicle" and "driving assistance of a vehicle".

[0061] Furthermore, the vehicle is typically an automobile, but other traveling vehicles (for example, a cargo transport vehicle used in a warehouse or the like) may also be used as the target.

[0062] The driving assist device 10 includes at least an information processing unit 11 and a vehicle control unit 12, and a control output from the vehicle control unit 12 is provided to an actuator 13. The control output controls the driving state of the vehicle, and typically, control outputs such as brake control, accelerator control, and steering control are provided to the actuator 13 to control the driving of the vehicle. Actuators such as brake control, accelerator control, and steering control can use known actuators.

[0063] The information processing unit 11 is composed of, for example, a central processing unit, i.e., a CPU (Central Processing Unit). In addition to the CPU, the information processing unit 11 may also include a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc., or may be composed of any one of them.

[0064] In the present invention, the information processing unit 11 is used as an object, and thus the embodiment of the information processing unit 11 will be mainly described below. However, the basic concept of this embodiment is as follows.

[0065] In this embodiment, if Figure 1 As shown, the information processing unit 11 is characterized in that it has at least: an area extraction unit 14, which extracts a specific area including traffic participants around the vehicle from the image information of the sensor data; a risk derivation unit 15, which derives the risk level of the traffic participants in the extracted specific area for safe driving of the vehicle; a priority setting unit 16, which sets the processing priority of the specific area according to the risk level; and an image recognition unit 17, which sets the processing load of the information of the specific area according to the processing priority to recognize the image.

[0066] Here, the image recognition unit 17 includes a processing load setting unit 18 for setting the processing load of information, and a recognition processing unit 19. In addition, in the following, there are cases where the image recognition unit 17 is described, and cases where the processing load setting unit 18 and the recognition processing unit 19 are described for recognizing traffic participants.

[0067] As described above, the information processing unit 11 includes the region extracting unit 14 , the risk deriving unit 15 , the priority setting unit 16 , and the image recognizing unit 17 , but their control functions can be realized by executing a predetermined control program stored in a storage unit included in the information processing unit 11 .

[0068] Figure 2 A representative control flow for setting the processing priority level is shown, which is executed by the above-mentioned control program.

[0069] S01

[0070] In step S01, the entire screen of the captured image is set to the same priority. This priority is an initial setting, so it can be any priority. If the initial setting priority is set, the process moves to step S02.

[0071] S02

[0072] In step S02, it is detected whether there is a traffic participant who may come into contact with the vehicle in the entire screen of the captured image. If it is determined that there is a traffic participant who may come into contact with the vehicle, the area including the traffic participant is detected. If it is determined that there is no traffic participant who may come into contact with the vehicle, the process of step S02 is executed again. On the other hand, if it is determined that there is a traffic participant who may come into contact with the vehicle, the process transfers to step S03. In addition, the contact possibility can also be assigned a level such as "large / medium / small".

[0073] S03

[0074] In step S03, the processing priority of each area is set according to the level of contact possibility and the number of traffic participants. If the processing priority is set, the process moves to step S04. Here, if the priority is set, the calculation process corresponding to the processing priority is executed.

[0075] For example, as the calculation processing corresponding to the processing priority, the image resolution, the image processing cycle, the processing content of the recognition processing, and the reduction rate in the neural network can be changed according to the processing priority. In addition, these will be described later.

[0076] S04

[0077] In step S04, it is detected whether the number of traffic participants with contact potential has increased or decreased compared with the result of the previous processing action. If there is no increase or decrease in the number of traffic participants with contact potential, the processing of step S04 is executed again. On the other hand, if there is an increase or decrease in the number of traffic participants with contact potential, the process is transferred to step S05.

[0078] S05

[0079] In step S05, it is detected whether all the traffic participants with contact possibility have disappeared. If the traffic participants with contact possibility have not disappeared, the processing of step S03 is executed again. On the other hand, if the traffic participants with contact possibility have disappeared, it returns to step S01 and repeats the same processing.

[0080] This makes it possible to achieve early recognition of dangers such as contact (ensuring safety) and appropriate allocation of computing power, effectively utilize computer resources, and provide safe driving assistance.

[0081] Next, return to Figure 1 , the function of each component constituting the driving assistance device 10 will be described.

[0082] [Description of Region Extraction Unit]

[0083] The area extraction unit 14 receives sensor data from a sensor group 20 to 23 such as a camera, a radar (RADAR), and a laser radar (LiDAR). Figure 3 , the image information (sensor data) based on the camera is shown in FIG. This image information is a piece of image information at a certain moment in front of the vehicle observed from the vehicle, and captures and shows the parked vehicle 201, oncoming vehicles 202, 203, and bicycle 204 near the vehicle. Furthermore, the area extraction unit 14 extracts a specific area where a traffic participant (so-called object) exists from this piece of image information.

[0084] Here, the traffic participants include moving traffic participants and non-moving traffic participants. For example, the moving traffic participants are oncoming vehicles 202, 203, and bicycles 204. In addition, non-moving traffic participants are parked vehicles 201, road conditions (road shape, road components such as curbs, fallen goods, etc.).

[0085] Moreover, as a method of extracting specific areas where traffic participants exist from image information, for example, if methods such as (1) Region Proposal Network, (2) Selective Search, and (3) CPMC (Constrained Parametric Min-Cuts) are used, it is possible to determine the area including traffic participants.

[0086] Figure 4, for example, a rectangular specific area 301 including the parked vehicle 201, rectangular specific areas 302 and 303 including the oncoming vehicles 202 and 203, and a rectangular specific area 304 including the bicycle 204 are extracted and displayed. In addition, at this time, the entire area 305 of the image information is extracted together with the extraction of the specific areas. This will be described later, but the extraction is performed in order to distribute the computing power between the specific areas 301 to 304 and the entire area 305 excluding the specific areas.

[0087] When the specific regions are extracted by the region extracting unit 14 , the risk deriving unit 15 determines the risk levels of the specific regions.

[0088] [Explanation of the risk derivation section]

[0089] The risk derivation unit 15 determines the risk level, which is the risk level of whether the traffic participants 201 to 204 extracted by the region extraction unit 14 are performing dangerous actions against the vehicle. The risk level refers to the size of the risk to the safe driving of the vehicle, that is, the "danger level", such as other vehicles approaching the vehicle and contacting or colliding with it. Therefore, it is necessary to identify other vehicles that are about to contact or collide as soon as possible, and more computing power needs to be allocated to this identification.

[0090] In the present embodiment, the estimated contact time TTC (Time To Collision) is used to determine the degree of risk. Therefore, the risk derivation unit 15 inputs location information from the GPS unit 21, map information from the map (MAP) unit 22, and vehicle speed information from the vehicle speed sensor 23. In addition, information other than these can also be used. With this information, the estimated contact time until the other vehicle contacts and collides with the vehicle is calculated. And, the shorter the estimated contact time TTC is, the higher the risk level is. In addition, the calculation method of the estimated contact time TTC is well known, so the detailed description is omitted.

[0091] Next, use Figure 5 , and explain the control process for determining the risk level. In addition, the number of specific areas is (N). Figure 4 There are 4 specific areas.

[0092] <Step S10>

[0093] In step S10, "(i) = 1" is set to indicate that it is the first risk determination of the first specific area. Thus, the risk level of the first specific area is determined. If (i) is set to "1", the process moves to step S11.

[0094] <Step S11>

[0095] In step S11, the speed of the vehicle is obtained. The speed of the vehicle can be obtained by the output of the vehicle's own speed sensor. Alternatively, it can be obtained by the change of the position signal from the GPS unit 21, and the method is not limited. Once the speed of the vehicle is obtained, the process moves to step S12.

[0096] <Step S12>

[0097] In step S12, the speed of the specific area (i=1) is obtained. The speed of the specific area (i=1) can be obtained from the image information, and the method of obtaining the speed is also well known. Once the speed of the specific area (i=1) is obtained, the process proceeds to step S13.

[0098] <Step S13>

[0099] In step S13, the distance (inter-vehicle distance) between the specific area (i=1) and the vehicle is obtained. The inter-vehicle distance can also be obtained from image information, and the method of obtaining the inter-vehicle distance is also well known. Once the distance between the specific area (i=1) and the vehicle is obtained, the process proceeds to step S14.

[0100] <Step S14>

[0101] In step S14, the travel path of the specific area (i=1) is predicted. The travel path can be predicted based on map information. In addition, road conditions such as road shape, road area, white lines, and traffic signs can be obtained based on map information. If the travel path of the specific area (i=1) is predicted, the process moves to step S15.

[0102] <Step S15>

[0103] In step S15, the following judgment is performed: whether there is an intersection between the path of the specific area (i=1) obtained in the above control step and the path of the vehicle, that is, whether there is a concern of contact or collision. If it is judged that there is an intersection between the path of the vehicle and the path of the specific area (i=1), the process moves to step S16; if it is judged that there is no intersection between the paths, the process moves to step S19.

[0104] <Step S16>

[0105] In step S16, the estimated contact time TTC (i=1) between the specific area (i=1) and the host vehicle is calculated. The estimated contact time TTC (i=1) can be calculated based on the speed difference (relative speed) between the specific area (i=1) and the host vehicle obtained in the above control step, and the distance between the host vehicle and the specific area (i=1). If the estimated contact time TTC (i=1) is obtained, the process proceeds to step S17.

[0106] 《Step S17》

[0107] In step S17, it is determined whether the estimated contact time TTC (i=1) obtained in step S17 is shorter than a predetermined estimated contact time threshold TTCth. If it is determined that the estimated contact time TTC (i=1) is shorter than the predetermined estimated contact time threshold TTCth, it is considered that the risk of contact or collision is high and the process moves to step S18. If it is determined that the estimated contact time TTC (i=1) is longer than the predetermined estimated contact time threshold TTCth, it is considered that the risk of contact or collision is low and the process moves to step S19.

[0108] 《Step S18》

[0109] In step S18, since the risk of contact or collision is considered high in step S17, the risk level is set to "risk(i) = 1" and the process moves to step S20. Here, "1" on the right indicates high risk. Thus, the specific area (i = 1) is set as a high-risk specific area.

[0110] 《Step S19》

[0111] In step S19, since it is considered that there is no intersection of the travel path in step S15, or that the risk of contact or collision is low in step S17, the risk level is set to "risk(i) = 0" and the process moves to step S20. Here, the "0" on the right indicates a low risk. Thus, the specific area (i = 1) is set as a low-risk specific area.

[0112] <Step S20>

[0113] In step S20, it is determined whether the current risk level has been determined for all specific areas (N). Therefore, if the risk level has been determined for all specific areas (N), the process exits to the end.

[0114] On the other hand, when the risk level has not been determined for all the specific areas (N), the risk level of the specific area (i=1) has been determined, and therefore the process moves to step S21.

[0115] <Step S21>

[0116] In step S21, by setting (i) to "i+1", the process returns to step S12 again to determine the risk level of the second specific area. The above control steps are then executed until (i) reaches "N". Figure 3 In the example, it is presumed that the bicycle 204 avoids the parked vehicle 201 and approaches to the right, and there is a concern that the bicycle 204 may collide with the vehicle. Figure 4Specific area 304 is a high-risk specific area, and specific areas 301 to 302 are oncoming vehicles, so they will not affect the driving of the vehicle and there is no concern about collision, so they are low-risk specific areas. In addition, specific area 301 is a parked vehicle, so it will not affect the driving of the vehicle and there is no concern about collision, so it is a low-risk specific area.

[0117] Here, the determination of the risk of the specific area is described as two levels, high risk and low risk, but it is not limited thereto and can be set to three or more levels. For example, by setting the image information other than the specific area to the background level, such as Figure 4 As shown in FIG. 3 , the risk level can be set as “high-risk specific area 304>low-risk specific areas 301, 302, 303>background area 305”. Of course, the high-risk specific areas and the low-risk specific areas can be further divided into levels.

[0118] As described above, in this embodiment, the estimated contact time TTC can be used to determine the risk level. Figure 6A to Figure 6C Examples of high risk specific areas are shown in .

[0119] exist Fig. 6A In the example, there is a possibility that other vehicles may come into contact with the vehicle in order to avoid the parked vehicle. There are parked vehicles 301, oncoming vehicles 302, 303, and bicycle 304 around the vehicle. The risk of contact and collision between the parked vehicle 301, oncoming vehicles 302, 303 and the vehicle is low, so it is determined as a low-risk specific area. On the other hand, as mentioned above, it is predicted that the bicycle 304 will move in front of the vehicle in order to avoid the parked vehicle 301 located on the side, so it is determined as a high-risk specific area. In addition, oncoming vehicles 302, 303, and bicycle 304 are a type of "moving traffic participants".

[0120] Figure 6B This is an example of a situation where there is a possibility that the surrounding vehicle 305 will come into contact with the vehicle due to a change in road conditions. In front of the surrounding vehicle 305, there is a curved shape area 306 where the direction of the road changes toward the travel direction of the vehicle. Therefore, it is predicted that the surrounding vehicle 305 will enter the driving lane of the vehicle from the current driving lane and move in front of the vehicle, so it is determined to be a high-risk specific area. In addition, the road conditions such as the curved shape area 306 obtained based on the map information are a type of "non-moving traffic participants".

[0121] Figure 6CThis is an example of a situation where there is a possibility that another vehicle may come into contact with the vehicle due to overtaking from behind. When the vehicle and the surrounding vehicle 305 are moving side by side, it is predicted that a high-speed motorcycle 307 will overtake from behind and move in front of the vehicle, so it is determined as a high-risk specific area. In addition, the surrounding vehicles 305 and the motorcycle 307 are a type of "moving traffic participants".

[0122] As described above, the risk derivation unit 15 has a function of predicting whether the traffic participants in front of, on the sides of, and behind the vehicle extracted by the region extraction unit 14 are performing dangerous actions to the vehicle and determining the degree of risk.

[0123] In more detail, the risk derivation unit 15 has the function of predicting the movement of the mobile traffic participants based on the behavior of the mobile traffic participants and deriving the risk level. In addition, the risk derivation unit 15 has the following functions: in addition to the behavior of the mobile traffic participants, the risk derivation unit 15 also predicts the movement of the mobile traffic participants based on the non-moving non-mobile traffic participants and derives the above-mentioned risk level.

[0124] Furthermore, the risk derivation unit 15 has a function of estimating the estimated contact time until the mobile traffic participant becomes an obstacle to the safe driving of the vehicle, and deriving the risk corresponding to the estimated contact time. In addition, the estimated contact time estimated by the risk derivation unit 15 is the estimated contact time until the mobile traffic participant comes into contact with the vehicle, and has a function of deriving the risk corresponding to the estimated contact time.

[0125] Furthermore, the risk deriving unit 15 has a function of estimating the estimated contact time using the size, driving direction, driving speed, and driving acceleration of the mobile traffic participant. In addition, the non-mobile traffic participant is an obstacle and / or road condition around the mobile traffic participant, and the risk deriving unit 15 has a function of predicting the behavior of the mobile traffic participant based on the obstacle and / or road condition, predicting the movement of the mobile traffic participant, and deriving the risk level.

[0126] Here, in Figure 5 In the control process, the degree of risk is determined based on the estimated contact time TTC with the vehicle, with two levels of high risk and low risk. However, in addition to this, even if the possibility of contact with the vehicle is low, the risk can be set to high in the case of a mobile traffic participant who performs actions different from usual.

[0127] For example, the risk level can be determined based on behaviors such as (1) a moving traffic participant's shaky driving trajectory, (2) extremely squeezing the distance between vehicles relative to surrounding vehicles, (3) driving extremely faster or slower than surrounding vehicles, or (4) accelerating / decelerating drastically.

[0128] In addition, objects other than mobile road users can be set as objects for calculating the risk level. For example, (1) objects dropped on the road, (2) balls or animals running out of alleys connected to the road, and (3) cargo or tires dropped from surrounding vehicles.

[0129] Further, in Figure 5 In the present invention, the risk level is determined based on the estimated contact time TTC with the host vehicle, but even in a specific area where the risk of contact with the host vehicle is small, the risk level can be set high when the distance from the host vehicle is short. Therefore, when some changes in conditions occur, the priority of the recognition processing for the specific area can be set high.

[0130] When the risk level of a specific area is derived by the risk deriving unit 15 , a processing priority level is set according to the risk level.

[0131] [Explanation of the Priority Setting Unit]

[0132] The priority setting unit 16 performs a function of setting a processing priority of a specific area corresponding to the risk level of the specific area derived by the risk deriving unit 15. The processing priority is an indicator for allocating the computing power of the computer and is associated with the risk level. As will be described later, in a specific area with a high processing priority, the processing load of the recognition processing is large, but the recognition accuracy can be improved. On the contrary, in a specific area with a low processing priority, the recognition accuracy is not so high, but the processing load of the recognition measure can be reduced.

[0133] Next, use Figure 7 , the control flow for setting the processing priority is explained. In addition, the number of specific areas is (N). Figure 4 There are 4 specific areas.

[0134] <Step S30>

[0135] In step S30, "(i) = 1" is set to indicate that the processing priority of the first specific area is set. Thus, the processing priority of the first specific area is set. If (i) is set to "1", the process moves to step S31.

[0136] <Step S31>

[0137] In step S31, it is determined whether the risk level of the first specific area is high risk "1". Figure 5 If the risk level is determined to be high risk "1", the process moves to step S32. If the risk level is determined not to be high risk "1", in other words, if the risk level is determined to be low risk "0", the process moves to step S33.

[0138] <Step S32>

[0139] In step S31, the risk level is determined to be high risk "1", so in step S32, the processing priority is set to high processing priority "priority(i) = 1". Here, the "1" on the right indicates a high processing priority. As mentioned above, although the processing load of the recognition process is large, the computing power of the computer is allocated in a manner that can improve the recognition accuracy. If the processing priority is set, it will transfer to step S34.

[0140] <Step S33>

[0141] In step S31 , it is determined that the risk level is a low risk “0”, and therefore in step S33 , the processing priority is set to a low processing priority “priority(i)=0”.

[0142] Here, the "0" on the right indicates a low processing priority. As mentioned above, the recognition accuracy is not very high, but the computing power of the computer is allocated in a manner that can reduce the processing load of the recognition measure. If the processing priority is set, it will transfer to step S34.

[0143] <Step S34>

[0144] In step S34, it is determined whether the current processing priority has been set for all specific areas (N). Therefore, if the processing priority has been set for all specific areas (N), the process exits to the end.

[0145] On the other hand, when the setting of the processing priority is not executed for all the specific areas (N), the setting of the processing priority of the specific area (i=1) is executed here, and therefore the process moves to step S35.

[0146] <Step S35>

[0147] In step S35, by setting (i) to "i+1", the process returns to step S31 again to determine the risk level of the second specific area. The above control steps are then executed until (i) reaches "N". Figure 4 , the processing priority of the specific area 304 becomes higher, and the processing priority of the specific areas 301 to 303 becomes lower.

[0148] When the priority setting unit 16 sets the processing priority of the specific area, the image recognition unit 17 executes the priority processing corresponding to the processing priority.

[0149] [Description of the Image Recognition Unit]

[0150] The image recognition unit 17 is composed of at least a processing load setting unit 18 and a recognition processing unit 19 .

[0151] [(1) Description of the processing load setting unit]

[0152] The processing load setting unit 18 constituting the image recognition unit 17 has a function of setting and executing a processing load for a specific area according to the processing priority of the specific area set by the priority setting unit 16. The processing load setting unit 18 distributes the load of the recognition processing in the subsequent recognition processing unit 19 according to the processing priority of the specific area.

[0153] Next, use Figure 8 , the control flow for setting the processing load of the specific area and executing it is described. In addition, the number of specific areas is (N). Figure 4 The number of specific areas is 4. In addition, an example of adjusting the resolution of an image is described as a setting of the processing load.

[0154] <Step S40>

[0155] In step S40, "(i) = 1" is set to indicate that the processing load of the first specific area is set / executed. Thus, the following processing load of the first specific area is set and executed. If (i) is set to "1", the process moves to step S41.

[0156] <Step S41>

[0157] In step S41, it is determined whether the processing priority of the first specific area is a high processing priority "1". Figure 7 The judgment results of steps S32 to S33 are obtained. If it is judged that the processing priority is a high processing priority "1", it is transferred to step S42. If it is judged that the processing priority is not a high processing priority "1", in other words, it is judged to be a low processing priority "0", it is transferred to step S43.

[0158] <Step S42>

[0159] In step S41, it is determined that the processing priority is the high processing priority "1", so in step S42, the vertical and horizontal resolutions of the image are set to "1". This means that the recognition process is directly performed using the image information of the specific area. If the processing of the image information of the specific area is performed, it is transferred to step S44.

[0160] <Step S43>

[0161] In step S41, it is determined that the processing priority is the low processing priority "0", so in step S43, the image resolution is set to "1 / 2" in both the vertical and horizontal directions. This is to compress the image information of the specific area to 1 / 2 to reduce the image information and perform recognition processing. If the processing of the image information of the specific area is performed, it will transfer to step S44.

[0162] <Step S44>

[0163] In step S44, it is determined whether the current processing load has been set and executed for all specific areas (N). Therefore, when the processing load has been set and executed for all specific areas (N), the process exits to the end.

[0164] On the other hand, when the setting and execution of the processing loads have not been completed for all the specific areas (N), the setting and execution of the processing loads for the specific area (i=1) are performed here, and therefore the process moves to step S45.

[0165] <Step S45>

[0166] In step S45, by setting (i) to "i+1", the process returns to step S41 again to determine the processing priority of the second specific area. Hereinafter, the above control steps are executed until (i) reaches "N". Figure 4 , the resolution of the specific area 304 is "1 / 1", and the resolution of the specific areas 301 to 303 is "1 / 2".

[0167] pass Fig. 9 , through Figure 8 The processing of image information executed by the control flow shown is briefly described. Fig. 9 The specific area, the image extracted in the specific area, the setting of the processing load in the processing load setting unit 18, and the image of the specific area input to the recognition processing unit 19 are shown. Figure 4 The status shown is based on

[0168] The specific area 301 is a parked vehicle, and the risk level and processing priority are “0.” Therefore, the processing load setting unit 18 sets the resolution to “1 / 2” in both the vertical and horizontal directions, and reduces the image information to 1 / 2 and sends it to the recognition processing unit 19 .

[0169] Similarly, specific areas 302 and 303 are oncoming vehicles, and the risk level and processing priority are “0.” Therefore, the processing load setting unit 18 sets the resolution to “1 / 2” in both the vertical and horizontal directions, and reduces the image information to 1 / 2 before sending it to the recognition processing unit 19 .

[0170] On the other hand, the specific area 304 is a bicycle, and the risk level and processing priority are “1”. Therefore, the processing load setting unit 18 sets both the vertical and horizontal resolutions to “1” and directly sends the image information to the recognition processing unit 19 .

[0171] [(2) Description of the Recognition Processing Unit]

[0172] Next, in the recognition processing unit 19 constituting the image recognition unit 17, the class of the traffic participants in the specific area is identified from the image information sent from the processing load setting unit 18. Here, the class means, for example, vehicles, motorcycles, pedestrians, etc., and these types can be identified and recognized. Furthermore, the moving direction, moving speed, moving acceleration, etc. of these vehicles, motorcycles, pedestrians, etc. can also be identified and recognized.

[0173] Therefore, the processing load setting unit 18 sets the resolution (image size) of the specific area according to the processing priority. Therefore, since a high resolution is set in a specific area with a high priority, although the processing load of the recognition process is large, the recognition processing unit 19 can expect high recognition accuracy. On the other hand, since a low resolution is set in a specific area with a low priority, a reduction in the processing load can be expected.

[0174] In addition, in the recognition processing unit 19, it is also possible to adjust and set the items to be recognized according to the processing priority. Fig.10 The control flow is shown in FIG.

[0175] <Step S50>

[0176] In step S50, "(i) = 1" is set to indicate that the processing load of the first specific area is set / executed. Thus, the following recognition processing of the first specific area is set and executed. If (i) is set to "1", the process moves to step S51.

[0177] <Step S51>

[0178] In step S51, it is determined whether the processing priority of the first specific area is a high processing priority "1". Figure 7 The judgment result of steps S32 to S33. If it is judged that the processing priority is a high processing priority "1", it is transferred to step S52. If it is judged that the processing priority is not a high processing priority "1", in other words, it is judged to be a low processing priority "0", it is transferred to step S53.

[0179] <Step S52>

[0180] In step S51, it is determined that the processing priority is the high processing priority "1", so detailed recognition processing is performed in step S52. Detailed recognition processing refers to the identification of the type of traffic participants in a specific area (vehicles / motorcycles / pedestrians, etc.), and the processing of identifying the moving direction, moving speed, moving acceleration, etc. of vehicles / motorcycles / pedestrians, etc. If the recognition processing of traffic participants in a specific area is performed, it will be transferred to step S54.

[0181] <Step S53>

[0182] In step S51, it is determined that the processing priority is a low processing priority "0", so in step S53, a simple recognition process is performed. Simple recognition processing refers to the process of distinguishing the type of traffic participants (vehicles / motorcycles / pedestrians, etc.) in a specific area. The process of identifying the moving direction, moving speed, moving acceleration, etc. of vehicles / motorcycles / pedestrians, etc. is not performed. If the recognition process of traffic participants in a specific area is performed, it is transferred to step S54.

[0183] <Step S54>

[0184] In step S54, it is determined whether the recognition process has been executed for all the specific areas (N). Therefore, when the recognition process has been completed for all the specific areas (N), the process exits to the end.

[0185] On the other hand, when the recognition process has not been completed for all the specific areas (N), the recognition process for the specific area (i=1) is performed here, and therefore the process moves to step S55.

[0186] <Step S55>

[0187] In step S55, (i) is set to "i+1", and the process returns to step S51 again to execute the recognition process of the second specific area. Hereinafter, the above-mentioned control steps are executed until (i) reaches "N".

[0188] This makes it possible to further achieve appropriate allocation of computing power and effectively utilize computer resources.

[0189] [Description of Vehicle Control Unit and Actuator]

[0190] The vehicle control unit 12 calculates control signals to be sent to the actuators based on the behavior of the road users recognized by the recognition processing unit 19. Typically, the control outputs such as brake control, accelerator control, and steering control are provided to the actuators 13 to control the travel of the vehicle.

[0191] In addition, the vehicle control unit 12 can detect a dangerous driving state through the risk level of the risk derivation unit 15, the estimated contact time TTC, and the statistical values ​​of the vehicle acceleration, deceleration, steering angular velocity, etc. based on the actuator 13, and correct the control output for driving control or notify a warning to the driver, etc. A specific example is described below.

[0192] Fig.11 Parameters for detecting dangerous driving conditions are shown. The first parameter is a statistical value of the number of specific areas whose risk level is set to "high" obtained in the risk derivation unit 15 (for example, the average number per unit time, etc.), the second parameter is a statistical value of the estimated contact time TTC in the specific area whose risk level is set to "high" obtained in the risk derivation unit 15 (for example, the average value, etc.), and the third parameter is a statistical value of the acceleration, deceleration, and steering angular velocity in the actuator 13 (the average value, etc.).

[0193] Fig.12 A specific example is shown in FIG. Fig.12 In the figure, the average values ​​of the five items (1) driving area, (2) driving day, (3) average number of high-risk specific areas per unit time, (4) average estimated contact time TTC of high-risk specific areas, and (5) average acceleration of the vehicle at different dates and times are shown. In addition, the speed of the vehicle can also be set as a parameter. In addition, the overall statistical values ​​(average values) are calculated by the calculation formulas (1) to (3) shown in the figure.

[0194] Then, using these parameters, the state during normal operation is calculated or learned, and when the state deviates greatly from the normal state, it is possible to correct the control output or notify the driver of a warning as if the driving state has become dangerous.

[0195] Next, based on Figures 13A to 13C , and explain its specific control process. Fig.13A This is an example of notifying the driver of a warning.

[0196] <Step S60>

[0197] In step S60, by Fig.11 , Fig.12 The method shown in the figure obtains the average number Nave of specific areas whose risk level is set to "high", and further obtains the number Ncurr of specific areas whose risk level is set to "high" at the current moment. Once these information are obtained, the process proceeds to step S61.

[0198] <Step S61>

[0199] In step S61, it is determined whether the number Ncurr of specific areas whose risk level is set to "high" at the current moment is greater than the average number Nave of specific areas whose risk level is set to "high". Thus, when the number Ncurr of specific areas whose risk level is set to "high" is greater than that during normal driving, it can be determined to be a dangerous state.

[0200] Moreover, if the number Ncurr of specific areas where the risk level at the current moment is set to "high" is greater than the average number Nave of specific areas where the risk level is set to "high", transfer to step S69; if the number Ncurr of specific areas where the risk level at the current moment is set to "high" is less than the average number Nave of specific areas where the risk level is set to "high", exit to the end and end the processing.

[0201] <Step S69>

[0202] In step S61, it is determined that the number of specific areas Ncurr whose risk level is set to "high" at the current moment is greater than the average number Nave of specific areas whose risk level is set to "high", and it is considered to be a dangerous state. Therefore, in step S69, the display device and the speaker are used to notify the driver that there is danger in the current situation, and the driver is urged to pay attention.

[0203] Next, Fig. 13B This is an example of correcting the target vehicle interval between the host vehicle and other vehicles as a control output.

[0204] In addition, the vehicle interval uses the contact estimation time TTC as a substitute parameter.

[0205] <Step S63>

[0206] In step S63, by Fig.11 , Fig.12 The method shown in the figure acquires the average TTC value TTCave of the specific area whose risk level is set to "high", and further acquires the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment. Once these pieces of information are acquired, the process proceeds to step S64.

[0207] <Step S64>

[0208] In step S64, it is determined whether the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment is greater than the average TTC value TTCave of the specific area whose risk level is set to "high". Therefore, compared with normal driving, the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment is smaller, so it can be determined as a dangerous state. Moreover, if the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment is smaller than the average TTC value TTCave of the specific area whose risk level is set to "high", transfer to step S65, if the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment is greater than the average TTC value TTCave of the specific area whose risk level is set to "high", exit to the end and end the processing.

[0209] <Step S65>

[0210] In step S64, it is determined that the TTC value TTCcurr of the specific area whose risk level is set to "high" at the current moment is less than the average TTC value TTCave of the specific area whose risk level is set to "high", and it is considered to be a dangerous state. Therefore, in step S65, the correction value TTCcom is added to the average TTC value TCCave of the specific area whose risk level is set to "high", and a new target vehicle interval TTCtar is calculated. In this way, the vehicle can be prevented from traveling at a dangerous vehicle interval. If the processing is completed, exit to the end and the processing ends.

[0211] In addition, step S69 may be executed instead of step S65. In step S69, a display device or a speaker is used to notify that there is danger in the current situation, and the driver is urged to pay attention. In addition, both step S65 and step S69 may be controlled.

[0212] Next, Fig. 13C This is an example of correcting the target acceleration of the host vehicle as the control output.

[0213] <Step S66>

[0214] In step S66, by Fig.11 , Fig.12 The method shown in the figure acquires the normal acceleration ACCave and further acquires the acceleration ACCcurr which is the current control output. Once these pieces of information are acquired, the process proceeds to step S67.

[0215] <Step S67>

[0216] In step S67, it is determined whether the current acceleration ACCcurr is greater than the normal acceleration ACCave. Therefore, the current acceleration ACCcurr is greater than that during normal driving, and therefore it can be determined to be a dangerous state.

[0217] Then, if the current acceleration ACCcurr is greater than the normal acceleration ACCave, the process proceeds to step S68, and if the current acceleration ACCcurr is less than the normal acceleration ACCave, the process exits to the end and ends the process.

[0218] <Step S68>

[0219] In step S67, it is determined that the current acceleration ACCcurr is greater than the normal acceleration ACCave, which is considered a dangerous state. Therefore, in step S68, the correction value ACCcom is subtracted from the normal acceleration ACCave to obtain a new target acceleration ACCtar. This can prevent the vehicle from running at a dangerous acceleration. If the process is completed, exit to the end and end the process.

[0220] In addition, step S69 may be executed instead of step S68. In step S70, the display device and the speaker are used to notify that there is danger in the current situation, and the driver is urged to pay attention. In addition, both step S68 and step S69 may be controlled.

[0221] As described above, according to the present embodiment, early recognition of dangers such as contact (security assurance) and appropriate allocation of computing power can be achieved, computer resources can be utilized effectively, and safe driving assistance can be performed.

[0222] Next, other embodiments (including modified examples) related to the present embodiment will be described.

[0223] Example 2

[0224] In the first embodiment of the present invention, the processing load setting unit 18 adjusts the resolution of the input image of the specific area according to the processing priority, but in the second embodiment, the processing load setting unit 18 is different in that it adjusts the processing cycle of the input image according to the processing priority.

[0225] Here, the functions of the region extracting unit 14 , the risk deriving unit 15 , and the priority setting unit 16 are the same as those of the first embodiment, and thus their description is omitted here.

[0226] In this embodiment, the image information of the specific area input to the processing load setting unit 18 is processed in a short cycle when the processing priority is high, and in a long cycle when the processing priority is low. In addition, the image processing includes the capturing process and the transferring process of the input image.

[0227] exist Fig.14 The specific control process is shown in FIG. 1 , so the following is described using the accompanying drawings. Figure 3 , Figure 4 The status shown is based on

[0228] <Step S70>

[0229] In step S70, "(i) = 1" is set to indicate that the processing load of the first specific area is set / executed. Thus, the image processing of the following first specific area is set and executed. If (i) is set to "1", the process moves to step S71.

[0230] <Step S71>

[0231] In step S71, it is determined whether the processing priority of the first specific area is a high processing priority "1". Figure 7 If the processing priority is determined to be a high processing priority "1", the process moves to step S72. If the processing priority is determined not to be a high processing priority "1", in other words, if the processing priority is determined to be a low processing priority "0", the process moves to step S73.

[0232] <Step S72>

[0233] In step S71, it is determined that the processing priority is a high processing priority "1", so in step S72, a short processing cycle is set. The processing cycle refers to the cycle of image processing (including image acquisition processing, forwarding processing, etc.), and the shorter it is, the more image processing can be performed. If the cycle of image processing of a specific area is set, it is transferred to step S74.

[0234] <Step S73>

[0235] In step S71, it is determined that the processing priority is the low processing priority "0", so in step S73, a long processing cycle is set. The processing cycle refers to the cycle of image processing. The longer it is, the less image processing (including image acquisition processing, forwarding processing, etc.) is performed. If the cycle of image processing of a specific area is set, it will transfer to step S74.

[0236] <Step S74>

[0237] In step S74, it is determined whether the image processing cycle has been set for all specific areas (N). Therefore, if the image processing cycle has been set for all specific areas (N), the process exits to the end.

[0238] On the other hand, when the image processing cycle setting has not been completed for all the specific areas (N), the image processing of the specific area (i=1) is performed here, and therefore the process moves to step S75.

[0239] <Step S75>

[0240] In step S75, (i) is set to "i+1", and the process returns to step S71 again to set the processing cycle of the image of the second specific area. The above control steps are then executed until (i) reaches "N".

[0241] exist Fig.15 The difference between image processing in a short processing cycle and a long processing cycle is shown in FIG. 2. In addition, "T" in the figure represents a processing cycle, and processing of a processing slot is performed in this processing cycle.

[0242] The processing time slot SLT304 of the specific area 304 set with a high processing priority is executed in each processing cycle. In contrast, the processing time slots SLT301 to SLT303 of the specific areas 301 to 303 set with a low processing priority are processed once for two processing cycles. In this way, the specific areas assigned with a low processing priority have fewer processing execution times than the specific areas assigned with a high processing priority.

[0243] Therefore, the calculation load of the recognition processing unit 19 can be reduced, and computer resources can be effectively utilized while performing safe driving assistance.

[0244] Example 3

[0245] In the first embodiment of the present invention, the processing load setting unit 18 adjusts the resolution of the image of the input specific area according to the processing priority, and in the second embodiment, the processing load setting unit 18 adjusts the processing cycle of the input image according to the processing priority, but in the third embodiment, it is different in that the reduction rate of the neural network for image recognition is adjusted.

[0246] Here, the functions of the region extracting unit 14 , the risk deriving unit 15 , and the priority setting unit 16 are the same as those of the first embodiment, and thus their description is omitted here.

[0247] Furthermore, in the present embodiment, the image information of the specific area input to the image recognition neural network provided in the processing load setting unit 18 is processed at a reduced rate when the processing priority is high, and is processed at a increased rate when the processing priority is low. In addition, the image recognition neural network may be provided in the recognition processing unit 19.

[0248] Fig.16 shows a neural network for image recognition. Fig.16 In the image recognition neural network, the image recognition neural network is composed of an input layer Nu10, a subsequent first layer Nu11, a second layer Nu12, a third layer Nu13, and an output layer Nu15. Image information is input to the input layer Nu10, and the input information is convolved using a plurality of neurons Nu14 in the first layer Nu11 to the third layer Nu13, and output information is output from the output layer Nu15. Such a convolutional neural network is an effective method for image recognition processing.

[0249] Next, use Fig.17 , the control flow of adjusting the reduction rate according to the processing priority is described. Figure 3 , Figure 4 The status shown is based on

[0250] Here, the reduction rate is the ratio of the number of unused neurons to the total number of neurons. If the reduction rate is small (the number of neurons used is large), although it takes computing time, the recognition accuracy can be improved. If the reduction rate is large (the number of neurons used is small), although the recognition accuracy is reduced, the computing time can be shortened.

[0251] <Step S80>

[0252] In step S80, "(i) = 1" is set to indicate setting / execution of the processing load of the first specific area. Thus, the following image processing of the first specific area is set and executed. If (i) is set to "1", the process moves to step S81.

[0253] <Step S81>

[0254] In step S81, it is determined whether the processing priority of the first specific area is a high processing priority "1". Figure 7 If the processing priority is determined to be a high processing priority "1", the process moves to step S82. If the processing priority is determined not to be a high processing priority "1", in other words, if the processing priority is determined to be a low processing priority "0", the process moves to step S83.

[0255] <Step S82>

[0256] In step S81, it is determined that the processing priority is the high processing priority "1", so in step S82, a small reduction rate is set. As mentioned above, if the reduction rate is small, although the operation time is required, the recognition accuracy can be improved. If the reduction rate of the image processing of the specific area is set, it is transferred to step S84.

[0257] <Step S83>

[0258] In step S81, it is determined that the processing priority is the low processing priority "0", so in step S83, a large reduction rate is set. As mentioned above, if the reduction rate is large, although the recognition accuracy is reduced, the operation time can be shortened. If the reduction rate of the image processing of the specific area is set, it is transferred to step S84.

[0259] <Step S84>

[0260] In step S84, it is determined whether the image reduction ratio has been set for all specific areas (N). Therefore, if the image reduction ratio has been set for all specific areas (N), the process exits to the end.

[0261] On the other hand, when the image reduction rate has not been set for all the specific areas (N), the image processing of the specific area (i=1) is performed here, and therefore the process moves to step S85.

[0262] <Step S85>

[0263] In step S85, (i) is set to "i+1", and the process returns to step S81 to set the reduction rate of the image of the second specific area. The above control steps are then performed until (i) reaches "N".

[0264] exist Fig.18 , Fig.19 The difference in recognition processing at a large reduction rate and a small reduction rate is shown in FIG.

[0265] In addition, the dotted "gray circles" in the figure represent unused neurons. In addition, neurons suitable for recognition processing are appropriately selected to determine the neurons to be used.

[0266] Fig.18 In the case of a large reduction rate, the number of unused neurons Nu14del is large relative to the number of used neurons Nu14use, so although the recognition accuracy decreases, the computational load of neural operations can be reduced.

[0267] Fig.19In the case of a small reduction rate, the number of unused neurons Nu14del is small relative to the number of used neurons Nu14use, so although the computational load of neural calculation increases, the recognition accuracy can be improved.

[0268] Fig. 20 The time relationship between the processing cycle "T" and the processing time slot is shown. After the processing in the processing time slot SLT304 of the specific area 304 with a high processing priority is executed, the processing in the processing time slots SLT301 to SLT303 of the specific area 301 to 303 with a low processing priority is executed. The processing time slots SLT301 to SLT304 are executed within the interval of the processing cycle "T", so sufficient processing can be performed.

[0269] In this way, in a specific area with a low processing priority and a low risk of collision, the computation load of neural computation can be reduced, and computer resources can be effectively utilized while providing safe driving assistance.

[0270] Next, a modification of this embodiment will be described. Fig.21 In the example, a specific area 310 including a new bicycle (in this case, the processing priority is "1") and a specific area 311 including a new oncoming vehicle (in this case, the processing priority is "0") appear. Therefore, if the number of specific areas increases, there is a concern that the processing time slots of all specific areas may not be processed within the processing cycle "T".

[0271] Therefore, in this variant, the reduction rate of the image of the specific area is increased to reduce the processing load of the individual neural operation, so that the processing of the processing time slots of all specific areas can be completed within the processing cycle "T". Fig. 22 , the control flow for executing this action is described. Fig.17 The same control steps are described.

[0272] <Step S80> to <Step S85>

[0273] After the reduction rates of all specific areas are set in steps S80 to S85 , the process proceeds to step S86 .

[0274] <Step S86>

[0275] In step S86, it is determined whether the total processing time of the processing time slots of all specific areas is greater than the processing cycle "T", that is, whether all processing time slots can be processed within the processing cycle "T".

[0276] In this judgment, if it is judged that all processing time slots can be processed within the processing cycle "T", the control flow is terminated. On the other hand, if it is judged that all processing time slots cannot be processed within the processing cycle "T", the control flow is transferred to step S87.

[0277] 《Step S87》

[0278] In step S87, the reduction rate is further updated to be larger according to the current reduction rate to reduce the processing load of the neural operation. By repeating this process, all processing time slots can be processed within the processing cycle "T". Then, if it is determined in step S86 again that all processing time slots can be processed within the processing cycle "T", the control flow is exited to the end.

[0279] Fig.23 The time relationship between the processing cycle "T" and the processing time slot is shown, and the specific area 304 with high processing priority, the processing time slot SLT304 of the specific area 310, and the processing time slot SLT310 are Fig. 20 Compared with the processing time slot SLT304 in, the reduction rate is adjusted to be larger and the computational load of the neural operation is changed to be smaller.

[0280] Similarly, the processing time slots SLT301 to SLT303 of the specific areas 301 to 303 with low processing priority and the processing time slot SLT311 of the specific area 311 are the same as those of the specific areas 301 to 303 with low processing priority. Fig. 20 Compared with the processing time slots SLT301 to SLT303, the reduction rate is adjusted to be larger and the computational load of the neural operation is updated to be smaller.

[0281] In this way, even if the number of specific areas increases, adjustments are made so that the reduction rate of the neural operations of processing time slots SLT301 to SLT304 and processing time slots SLT310 to SLT311 within the processing cycle "T" becomes larger. That is, in order to complete the processing of the processing time slots of all specific areas, the processing load of the neural operations of a single specific area is reduced by increasing the reduction rate of the image of the specific area, so that the processing of the processing time slots of all specific areas can be completed within the processing cycle "T".

[0282] Example 4

[0283] In the first embodiment of the present invention, the risk derivation unit 15 determines the danger of collision of the traffic participants captured by the camera 20, but the fourth embodiment is different in that, in addition to this, the risk is also derived when there are traffic participants in the space that becomes a blind spot of the image.

[0284] For example, Fig.24In this way, it is assumed that a blind spot area 312 is predicted behind the specific area 304, and that there are traffic participants behind the specific areas 301 to 303, and blind spot areas 313 to 315 are predicted as specific areas. Therefore, in this embodiment, risks are also derived for these blind spots.

[0285] The risk derivation unit 15 determines the risk level of whether a road user assumed to be present in the blind spot area extracted by the area extraction unit 14 performs a dangerous action with respect to the vehicle.

[0286] Next, use Fig.25 , the control process for determining the risk level is described. In addition, the number of blind spots is (N). Figure 4 There are 4 blind spots.

[0287] <Step S90>

[0288] In step S90, to indicate that it is the first risk determination of the first blind spot area, "(i) = 1" is set. Thus, the risk level of the first blind spot area is determined. If (i) is set to "1", the process moves to step S91.

[0289] <Step S91>

[0290] In step S91, the speed of the vehicle is obtained. The speed of the vehicle can be obtained by the output of the vehicle's own speed sensor. Alternatively, it can be obtained by the change of the position signal from the GPS unit 21, and the method is not limited. If the speed of the vehicle is obtained, the process moves to step S92.

[0291] <Step S92>

[0292] In step S92, the speed of the blind spot area (i=1) is obtained. The speed of the blind spot area (i=1) can be obtained using the speed of the traffic participants in the specific area corresponding to the blind spot area, and can be obtained based on the image information as in the first embodiment. The method of obtaining the speed is also well known. In the following description, the blind spot area is also regarded as the traffic participant corresponding to the area and the processing is performed. If the speed of the blind spot area (i=1) is obtained, the process is transferred to step S93.

[0293] <Step S93>

[0294] In step S93, the distance (inter-vehicle distance) between the blind spot area (i=1) and the vehicle is obtained. The inter-vehicle distance can also be obtained from image information, and the method of obtaining the inter-vehicle distance is also well known. Once the distance between the blind spot area (i=1) and the vehicle is obtained, the process proceeds to step S94.

[0295] <Step S94>

[0296] In step S90, the path of the blind spot area (i=1) is predicted. The path can be predicted based on map information. In addition, the road conditions such as road shape, road area, white lines, and traffic signs can be obtained based on the map information. If the path of the blind spot area (i=1) is predicted, the process moves to step S95.

[0297] 《Step S95》

[0298] In step S95, it is determined whether there is an intersection between the path of the blind spot area (i=1) obtained in the above control step and the path of the vehicle, that is, whether there is a concern of contact or collision. If it is determined that there is an intersection between the path of the vehicle and the path of the blind spot area (i=1), the process moves to step S96, and if it is determined that there is no intersection between the paths, the process moves to step S99.

[0299] 《Step S96》

[0300] In step S96, the estimated contact time TTC (i=1) between the blind spot area (i=1) and the host vehicle is calculated. The estimated contact time TTC (i=1) can be calculated based on the speed difference (relative speed) between the blind spot area (i=1) and the host vehicle obtained in the above control step, and the distance between the host vehicle and the blind spot area (i=1). If the estimated contact time TTC (i=1) is obtained, the process moves to step S97.

[0301] 《Step S97》

[0302] In step S97, it is determined whether the estimated contact time TTC (i=1) obtained in step S97 is shorter than the predetermined estimated contact time threshold TTCth. If it is determined that the estimated contact time TTC (i=1) is shorter than the predetermined estimated contact time threshold TTCth, it is considered that the risk of contact or collision is high and the process moves to step S98. If it is determined that the estimated contact time TTC (i=1) is longer than the predetermined estimated contact time threshold TTCth, it is considered that the risk of contact or collision is low and the process moves to step S99.

[0303] 《Step S98》

[0304] In step S98, since the risk of contact or collision is considered high in step S97, the risk level is set to "risk(i) = 1" and the process moves to step S20. Here, the "1" on the right indicates a high risk. Thus, the blind spot area (i = 1) is set as a high-risk blind spot area.

[0305] 《Step S99》

[0306] In step S99, since it is considered that there is no intersection of the travel path in step S95, or that the risk of contact or collision is low in step S97, the risk level is set to "risk(i) = 0" and the process moves to step S20. Here, the "0" on the right indicates a low risk. Thus, the blind spot area (i = 1) is set to a low-risk blind spot area.

[0307] <Step S100>

[0308] In step S100, it is determined whether the current risk level has been determined for all blind spot areas (N). Therefore, if the risk level has been determined for all blind spot areas (N), the process exits to the end.

[0309] On the other hand, when the risk level is not determined for all blind spot areas (N), the risk level of the blind spot area (i=1) is determined here, and the process moves to step S21.

[0310] <Step S101>

[0311] In step S101, (i) is set to "i+1", and the process returns to step S92 to determine the risk level of the second blind spot area. The above control steps are then executed until (i) reaches "N".

[0312] exist Fig.24 In the example, it is estimated that the bicycle 304 avoids the parked vehicle 201 and approaches to the right side, and there is a concern about collision with the vehicle, so the blind spot area 312 also becomes a high-risk blind spot area, and the blind spots 313-314 are oncoming vehicles, so they will not affect the driving of the vehicle and there is no concern about collision, so they become low-risk blind spots. These risk information are sent to the priority setting unit 16 and used in the same way as in Example 1.

[0313] Example 5

[0314] In the first embodiment of the present invention, image information of the sensor group 20 to 23, here the camera 20, is input into the area extraction unit 14, and a specific area is extracted. However, in the fifth embodiment, the image information is input into the processing load setting unit 18 in addition to the area extraction unit 14, which is different.

[0315] Here, the functions of the region extracting unit 14 , the risk deriving unit 15 , and the priority setting unit 16 are the same as those of the first embodiment, and thus their description is omitted here.

[0316] Moreover, if Fig.26As shown, the image information of the camera 20 is input to the processing load setting unit 18 in parallel with the area extraction unit 14. The image information (the whole image in front of the vehicle) input to the processing load setting unit 18 is different from the above-mentioned specific area and has a low processing priority, so the action of reducing the whole and lowering the resolution is performed.

[0317] Fig. 27 The reduced whole image as shown is then input to the recognition processing unit 19 , and the recognition processing unit 19 performs processing such as extraction of white lines 400 on the travel road, extraction of road areas 401 , and extraction of traffic signs 402 .

[0318] In this way, when identifying information on a driving road that is not very dangerous to the vehicle, the entire image in front of the vehicle is reduced for identification, thereby reducing the calculation load of the recognition processing unit 19, making it possible to effectively and flexibly utilize computer resources and provide safe driving assistance.

[0319] Example 6

[0320] In the first embodiment of the present invention, a camera 20 is used as a sensor group, and the image information of the camera 20 is input into the area extraction unit 14 to extract a specific area. However, in the sixth embodiment, the signal of a laser distance sensor such as a laser radar (LiDAR) is input into the area extraction unit 14, and the image of the camera 20 is input into the recognition processing unit 19.

[0321] Here, the functions of the region extracting unit 14 , the risk deriving unit 15 , and the priority setting unit 16 are the same as those of the first embodiment, and thus their description is omitted here.

[0322] like Fig.28 As shown, the point cloud information of the laser radar 24 is input to the area extraction unit 14, and the area of ​​the traffic participant can be determined and extracted in the same manner as in the first embodiment. The information of the laser radar 24 is advantageous in information processing because it is point cloud information. In addition, the recognition accuracy of the traffic participant is high, so a more accurate area extraction can be performed.

[0323] However, it is difficult to determine color information such as white line information and sign information in the laser radar 24, so the image information of the camera 20 is input to the recognition processing unit 19. In this case, the image information of the camera 20 is processed in the recognition processing unit 19 to extract the white line 400 on the travel road, the road area 401, and the traffic sign 402.

[0324] As described above, the present invention is characterized in that the driving assistance device comprises: an information processing unit, which processes information of sensor data obtained by the sensor; and a vehicle control unit, which uses the information processed by the information processing unit to assist the vehicle in driving. In the driving assistance device, the information processing unit comprises: an area extraction unit, which extracts a specific area of ​​traffic participants around the vehicle from the image information of the sensor data; a risk derivation unit, which derives the risk level of safe driving for the vehicle with respect to the traffic participants in the extracted specific area; a priority setting unit, which sets the processing priority of the specific area according to the risk level; a processing load setting unit, which sets the processing load of the information of the specific area according to the processing priority; and an image recognition processing unit, which identifies the traffic participants based on the processing load information from the processing load setting unit.

[0325] According to the present invention, it is possible to achieve early recognition of danger (security assurance) and appropriate allocation of computing power, thereby making effective use of computer resources and providing safe driving assistance.

[0326] In addition, the present invention is not limited to the several embodiments described above, and includes various modifications. For example, the vehicle may also be a transport vehicle traveling in a factory, in which case the traffic participants may also be other transport vehicles, pedestrians walking in the factory, etc. The above-mentioned embodiments are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to having all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. For the structure of each embodiment, other structures can also be added, deleted, or replaced.

[0327] Description of Reference Numerals

[0328] 10 driving assistance device; 11 information processing unit; 12 vehicle control unit; 13 actuator vehicle control device; 14 area extraction unit; 15 risk derivation unit; 16 priority setting unit; 17 image recognition unit; 18 processing load setting unit; 19 recognition processing unit; 20 camera (sensor).

Claims

1. A driving assist device, characterized in that: The driving assistance device comprises: an information processing unit for processing information of sensor data obtained by a sensor; and a vehicle control unit for performing driving assistance of a vehicle using the information processed by the information processing unit. The information processing unit has: An area extraction unit extracts a specific area of ​​traffic participants around the vehicle from the information of the sensor data; A risk deriving unit, for deriving the risk level of the vehicle in terms of safe driving for the extracted traffic participants in the specific area; a priority setting unit for setting a processing priority of the specific area according to the risk level; a processing load setting unit that sets a processing load of the information of the traffic participant in the specific area according to the processing priority; as well as The recognition processing unit recognizes the traffic participant based on the processing load information from the processing load setting unit.

2. The driving assistance device according to claim 1, characterized in that: In the driving assistance device, The priority setting unit sets the specific area having a high risk level as a high-risk specific area, and sets the specific area having a lower risk level than the high-risk specific area as a low-risk specific area.

3. The driving assistance device according to claim 2, characterized in that: In the driving assistance device, The sensor data is based on the image information of the camera, The region extraction unit extracts the specific region including the traffic participant around the host vehicle from the image information.

4. The driving assistance device according to claim 3, characterized in that: In the driving assistance device, The region extraction unit extracts the specific region from one piece of the image information.

5. The driving assistance device according to claim 4, characterized in that: In the driving assistance device, The region extraction unit extracts an entire region and a specific region existing in the entire region from the image information. The priority setting unit sets the processing priority of the entire area to be lower than the processing priority of the specific area.

6. The driving assistance device according to claim 1, wherein: In the driving assistance device, The risk deriving unit derives the risk level by predicting the movement of the moving traffic participant based on the behavior of the moving traffic participant.

7. The driving assistance device according to claim 6, characterized in that: In the driving assistance device, The risk deriving unit predicts the movement of the mobile traffic participant based on non-moving non-moving traffic participants in addition to the behavior of the mobile traffic participant, and derives the risk level.

8. The driving assistance device according to claim 6, characterized in that: In the driving assistance device, The risk deriving unit estimates an estimated contact time until the moving traffic participant becomes an obstacle to safe driving of the host vehicle, and derives the risk level according to the estimated contact time.

9. The driving assistance device according to claim 8, characterized in that: In the driving assistance device, The estimated contact time estimated by the risk deriving unit is an estimated contact time until the mobile traffic participant comes into contact with the host vehicle, and the risk level is derived in accordance with the estimated contact time.

10. The driving assistance device according to claim 9, characterized in that: In the driving assistance device, The risk derivation unit estimates the estimated contact time using the size, driving direction, driving speed, and driving acceleration of the mobile traffic participant.

11. The driving assistance device according to claim 7, wherein: In the driving assistance device, The non-moving traffic participants are obstacles and / or road conditions around the moving traffic participants, The risk deriving unit predicts the behavior of the mobile traffic participant based on the obstacle and / or the road condition, and derives the risk level by predicting the movement of the mobile traffic participant.

12. The driving assistance device according to claim 1, wherein: The vehicle control unit detects a dangerous driving state using a statistical value related to the risk level of the risk deriving unit, and corrects a control output so as to escape from the dangerous driving state.

13. The driving assistance device according to claim 1, wherein: The vehicle control unit detects a dangerous driving state using the statistical value of the risk level of the risk deriving unit and issues a warning.

14. The driving assistance device according to claim 2, wherein: In the driving assistance device, The processing load setting unit sets a resolution of the image of the high-risk specific area to be higher than a resolution of images other than the high-risk specific area, according to the processing priority.

15. The driving assistance device according to claim 2, characterized in that: In the driving assistance device, The processing load setting unit sets a resolution of the image of the high-risk specific area to be higher than a resolution of the image of the low-risk specific area in accordance with the processing priority.

16. The driving assistance device according to claim 2, wherein: In the driving assistance device, The processing load setting unit sets a processing cycle of calculations in the high-risk specific area to be shorter than a processing cycle of calculations other than the high-risk specific area, according to the processing priority.

17. The driving assistance device according to claim 2, characterized in that: In the driving assistance device, The processing load setting unit sets a processing cycle of calculations in the high-risk specific area to be shorter than a processing cycle of calculations in the low-risk specific area in accordance with the processing priority.

18. The driving assistance device according to claim 2, characterized in that: In the driving assistance device, The image of the specific area is image recognized using a convolutional neural network, The processing load setting unit sets the reduction rate of the convolutional neural network in the high-risk specific area to be smaller than the reduction rate of the convolutional neural network in the area other than the specific area, corresponding to the processing priority.

19. The driving assistance device according to claim 18, characterized in that: In the driving assistance device, The processing load setting unit sets the reduction rate of the convolutional neural network so that the total processing time of the specific area and the area other than the specific area is less than or equal to a target processing time.

20. The driving assistance device according to claim 1, wherein: In the driving assistance device, The area extraction unit extracts a blind spot area from the image information of the sensor data. The risk deriving unit estimates an estimated contact time until the blind spot area becomes an obstacle to safe traveling of the host vehicle, and derives the risk level according to the estimated contact time.

21. The driving assistance device according to claim 1, characterized in that: In the driving assistance device, The sensors are laser distance sensors and cameras, The point cloud information of the laser distance sensor is input into the area extraction unit. Image information from the camera is input to the recognition processing unit.

22. A driving assistance method for a driving assistance device, characterized in that: The driving assistance device comprises: an information processing unit for processing information of sensor data acquired by a sensor; and a vehicle control unit for performing driving assistance of a vehicle using the information processed by the information processing unit. In the driving assistance method, The information processing unit Extracting a specific area of ​​traffic participants around the host vehicle from the information of the sensor data; For the extracted traffic participants in the specific area, deriving a risk level for the vehicle in terms of safe driving; Setting a processing priority for the specific area corresponding to the risk level; setting a processing load of the information of the traffic participant in the specific area corresponding to the processing priority; as well as The road user is identified based on the processing load.

Images