Vehicle control system and vehicle control method

Abstract

The invention relates to a vehicle control system and a vehicle control method. The present disclosure ensures safety when controlling a vehicle using a vehicle position estimated on the basis of a common camera and a plurality of different algorithms. The invention provides a vehicle control system for controlling a vehicle. The vehicle control system is configured to: acquire an image captured by a common camera; calculating a first vehicle position, which is the position of the vehicle, from the image captured by the common camera on the basis of a first position estimation algorithm; calculating a second vehicle position, which is the position of the vehicle, from the image captured by the common camera on the basis of a second position estimation algorithm; and decelerating or stopping the vehicle in response to a deviation state in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error.

Description

Technical Field

[0001] The present disclosure relates to a technology for controlling a vehicle using a camera. Background Art

[0002] Patent document 1 discloses a system for controlling the travel of a vehicle within a specified area. The control device in the system comprises: a first acquisition unit that acquires detection data of a vehicle within a specified area from a detection unit provided within the specified area; a first estimation unit that estimates a first position indicating the position of the vehicle based on the detection data; a second acquisition unit that acquires a second position indicating the position of the vehicle estimated by a second estimation unit provided in the vehicle from the vehicle; and a correction control unit that causes one of the first estimation unit and the second estimation unit to perform calibration based on a comparison result of the first position and the second position.

[0003] It should be noted that, as documents showing the technical level in the technical field related to the present disclosure, in addition to the above-mentioned Patent Document 1, the following Patent Documents 2 to 7 can be exemplified.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-081961

[0007] Patent Document 2: International Publication No. 2019 / 073795

[0008] Patent Document 3: Japanese Patent Application Publication No. 2019-135620

[0009] Patent Document 4: Japanese Patent Application Laid-Open No. 11-039589

[0010] Patent Document 5: Japanese Patent Application Laid-Open No. 11-096494

[0011] Patent Document 6: Japanese Patent Application Publication No. 2019-020786

[0012] Patent Document 7: Japanese Patent Application Publication No. 2019-130997

[0013] Various algorithms can be proposed as algorithms for estimating the position of a vehicle based on images captured by a camera. However, in the past, there has been little research into the technology of estimating the position of a vehicle based on a common image captured by a common camera using a plurality of different algorithms. Summary of the invention

[0014] The present disclosure provides a vehicle control system for controlling a vehicle. The vehicle control system includes one or more processors.

[0015] One or more processors are configured to: acquire an image captured by a shared camera; calculate a first vehicle position as the position of the vehicle based on the image captured by the shared camera based on a first position estimation algorithm; calculate a second vehicle position as the position of the vehicle based on the image captured by the shared camera based on a second position estimation algorithm; and decelerate or stop the vehicle in response to a deviation state in which the position difference between the first vehicle position and the second vehicle position exceeds an allowable error.

[0016] The present disclosure provides a vehicle control method for controlling a vehicle.

[0017] The vehicle control method includes: acquiring an image captured by a common camera; calculating a first vehicle position as the position of the vehicle based on the image captured by the common camera based on a first position estimation algorithm; calculating a second vehicle position as the position of the vehicle based on the image captured by the common camera based on a second position estimation algorithm; and decelerating or stopping the vehicle in response to a deviation state in which the position difference between the first vehicle position and the second vehicle position exceeds an allowable error.

[0018] Effects of the Invention

[0019] According to the technology disclosed in the present invention, multiple vehicle positions are calculated based on a common image based on multiple position estimation algorithms. The reliability of the position estimation can be evaluated by comparing the multiple calculated vehicle positions. Furthermore, if the reliability of the position estimation is considered to be reduced, the vehicle is slowed down or stopped. In this way, the safety of the vehicle's driving can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a diagram for explaining the outline of the vehicle control system according to the first embodiment.

[0021] Figure 2 This is a diagram for explaining the position difference calculated by the vehicle control system.

[0022] Figure 3 This is a block diagram showing an example of a processing unit included in the vehicle control system.

[0023] Figure 4 It is a diagram showing an example of environmental parameters in the third embodiment.

[0024] Figure 5 This is a graph showing an example of the relationship between environmental parameters and allowable errors.

[0025] Figure 6 It is a diagram for explaining the fourth embodiment.

[0026] Figure 7 This is a graph showing an example of the allowable error in the fourth embodiment.

[0027] Figure 8 It is a diagram for explaining the fifth embodiment.

[0028] Fig. 9 This is another diagram for explaining the fifth embodiment.

[0029] Fig.10 It is a diagram showing a configuration example of a vehicle according to a sixth embodiment.

[0030] Description of Reference Numerals

[0031] AR: specified area, CAM: camera, VID: image, INF: camera information, PV1: first vehicle position, PV2: second vehicle position, ΔPV: position difference, α: allowable error, 1: vehicle, 100: vehicle control system, 101: processor, 102: storage device, 111: first position estimating unit, 113: second position estimating unit, 115: vehicle control unit, 116: detection unit. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described with reference to the drawings.

[0033] 1. First Implementation

[0034] 1-1. Overview and basic structure of vehicle control system

[0035] Figure 1 This is a conceptual diagram for explaining the outline of the vehicle control system 100 of the present embodiment. The vehicle control system 100 controls the vehicle 1 within the specified area AR. The vehicle 1 may be an autonomous driving vehicle that can perform autonomous driving at least within the specified area AR. There may also be multiple vehicles 1 that are the objects of control by the vehicle control system 100. At least a part of the vehicle control system 100 may be included in a management server outside the vehicle 1. At least a part of the vehicle control system 100 may also be included in a vehicle-mounted system mounted on the vehicle 1. The vehicle control system 100 may also be distributed in the management server and the vehicle-mounted system.

[0036] At least one camera CAM is installed in the specified area AR. The camera CAM is an infrastructure camera that can shoot the specified area AR. The camera CAM shoots the conditions in the specified area AR and obtains an image VID representing the conditions in the specified area AR. The image VID is a collection of images obtained by the camera CAM continuously shooting within the field of view at each specified shooting cycle. The vehicle 1 traveling in the specified area AR may be presented in the image VID. Examples of the specified area AR in which the camera CAM is installed include parking lots, factories, lanes (ordinary roads, highways), etc. In the case where the specified area AR is a parking lot, the vehicle 1 can handle automatic valet parking (AVP). In the case where the specified area AR is a factory site, the vehicle 1 assembled in the assembly plant can automatically drive from the assembly plant to the yard. In this case, one or more cameras CAM are installed on the road from the assembly plant to the yard.

[0037] The vehicle control system 100 includes one or more processors 101 (hereinafter, referred to as only processor 101) and one or more storage devices 102 (hereinafter, referred to as only storage device 102). The processor 101 performs various processes. As the processor 101, examples include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. The storage device 102 stores one or more programs and various information. As the storage device 102, examples include a HDD (Hard Disk Drive), an SSD (Solid State Drive), a volatile memory, a non-volatile memory, etc. The functions of the vehicle control system 100 can be realized by the cooperation of the processor 101 executing one or more programs and the storage device 102. One or more programs can also be recorded in a computer-readable recording medium.

[0038] The vehicle control system 100 is communicatively connected to the camera CAM. The image VID captured by the camera CAM is sent to the vehicle control system 100 via a wired communication network or a wireless communication network. The vehicle control system 100 acquires the image VID captured by the camera CAM. The vehicle control system 100 estimates the position of the vehicle 1 in the specified area AR (hereinafter referred to as the vehicle position PV) based on the image VID. In more detail, the vehicle control system 100 estimates (calculates) the vehicle position PV based on the image VID based on a specified position estimation algorithm.

[0039] For example, the vehicle control system 100 detects (identifies) the vehicle 1 in the specified area AR according to the image VID based on the specified vehicle detection algorithm. For example, the vehicle control system 100 uses image recognition AI to identify the vehicle 1 presented in the image VID. The image recognition AI is generated in advance through machine learning. Then, the vehicle control system 100 calculates the position of the vehicle 1 in the image VID (image). In other words, the vehicle control system 100 calculates the relative position of the vehicle 1 observed from the camera CAM. On the other hand, the vehicle control system 100 maintains the camera information INF. The camera information INF is information indicating the setting position, setting direction, field of view, etc. of the camera CAM. The camera information INF is provided to the vehicle control system 100 in advance and is stored in the storage device 102. The camera information INF can also be sent from the camera CAM. The vehicle control system 100 can calculate the vehicle position PV based on the relative position of the vehicle 1 in the image and the camera information INF.

[0040] The vehicle control system 100 controls the vehicle 1 based on the vehicle position PV thus estimated. For example, the vehicle control system 100 performs automatic driving control based on the vehicle position PV. As another example, the vehicle control system 100 may perform driving assistance control based on the vehicle position PV. As driving assistance control, risk avoidance control, lane keeping control, etc. are given as examples. For example, the vehicle control system 100 generates a target trajectory of the vehicle 1. The target trajectory includes a target position and a target speed of the vehicle 1. Then, the vehicle control system 100 controls the travel of the vehicle 1 in a manner that follows the target trajectory.

[0041] The vehicle 1 includes a driving device. The driving device includes a driving device for driving the vehicle 1, a braking device for braking the vehicle 1, and a steering device for steering the vehicle 1. The control information is information for controlling the vehicle 1. The vehicle control system 100 controls the driving device according to the control information, thereby controlling the vehicle 1.

[0042] In this way, the vehicle 1 is controlled based on the vehicle position PV estimated based on the prescribed position estimation algorithm. However, the estimation accuracy of the vehicle position PV, that is, the reliability of the estimated vehicle position PV, may decrease depending on the distance between the vehicle 1 and the camera CAM and the environment around the vehicle 1. For example, when the distance between the vehicle 1 and the camera CAM is far, the reliability of the vehicle position PV may decrease. In addition, when using an image VID captured in a backlit environment, the reliability of the vehicle position PV may decrease. When the reliability of the vehicle position PV is low, the accuracy of the vehicle control based on the vehicle position PV may also decrease. From the perspective of safety, this is not preferable.

[0043] Therefore, according to the present embodiment, the reliability of the vehicle position PV is evaluated. In more detail, the vehicle control system 100 calculates a plurality of vehicle position PVs based on a plurality of different position estimation algorithms. For example, the plurality of position estimation algorithms are created and provided by different developers. In the case of a position estimation algorithm using an image recognition AI, the learning levels of the respective image recognition AIs may be different. It should be noted that, although a plurality of different position estimation algorithms are used, the input image VID is a common image VID captured by a common camera CAM. That is, the vehicle control system 100 calculates a plurality of vehicle position PVs based on a common image VID based on a plurality of different position estimation algorithms. Moreover, the vehicle control system 100 compares the plurality of vehicle position PVs with each other. If the difference between the plurality of vehicle position PVs is within an allowable range, it can be said that the reliability of the plurality of vehicle position PVs (that is, the plurality of position estimation algorithms) is high.

[0044] On the other hand, when the difference between the plurality of vehicle positions PV exceeds the permissible range, it is considered that the reliability of at least one of the vehicle positions PV (that is, at least one of the position estimation algorithms) has decreased. In other words, it is considered that the reliability of the position estimation has decreased. Therefore, in this case, the vehicle control system 100 decelerates or stops the vehicle 1. In this way, the safety of the vehicle's travel can be ensured.

[0045] Figure 2 1 is a diagram for explaining the processing of this embodiment. Hereinafter, a case where two position estimation algorithms are used will be explained. The same is true for a case where three or more position estimation algorithms are used.

[0046] The vehicle control system 100 calculates the first vehicle position PV1 based on the shared image VID based on the first position estimation algorithm. In addition, the vehicle control system 100 calculates the second vehicle position PV2 based on the shared image VID based on the second position estimation algorithm. The vehicle control system 100 compares the first vehicle position PV1 with the second vehicle position PV2 to calculate the position difference ΔPV between the two vehicle positions. The position difference ΔPV is calculated, for example, as the distance between the center of the first vehicle position PV1 and the center of the second vehicle position PV2. Then, when the position difference ΔPV is greater than the allowable error α, that is, when the difference between the two vehicle positions is greater than the range allowed as an error, the vehicle control system 100 decelerates or stops the vehicle 1.

[0047] Figure 2 2 shows two conditions, condition a and condition b, and condition a shows a condition where the position difference ΔPV is smaller than the allowable error α. When the position difference ΔPV between the two vehicle positions is smaller than the allowable error α, it is considered that the reliability of the position estimation is guaranteed. Therefore, the vehicle control system 100 controls the vehicle 1 as usual.

[0048] On the other hand, situation b shows a situation where the position difference ΔPV is larger than the allowable error α. When the position difference ΔPV is larger than the allowable error α, the reliability of the position estimation may decrease. Therefore, the vehicle control system 100 decelerates or stops the vehicle 1 .

[0049] Thus, according to the vehicle control system 100 of this embodiment, two vehicle positions are calculated based on two algorithms based on the images obtained from the common camera CAM, and they are compared, thereby estimating the reliability of the position estimation. In addition, the vehicle 1 is decelerated or stopped in a situation where the reliability of the position estimation may decrease, thereby ensuring the driving safety of the vehicle 1.

[0050] Figure 3 1 is a block diagram showing an example of a processing unit included in the vehicle control system 100. The vehicle control system 100 includes a first position estimating unit 111, a second position estimating unit 113, and a vehicle control unit 115. The vehicle control unit 115 includes a detection unit 116. These processing units are implemented by the processor 101 executing a program stored in the storage device 102.

[0051] The first position estimating unit 111 and the second position estimating unit 113 are processing units for estimating the position of the vehicle 1 within the predetermined area AR based on the image VID and the camera information INF acquired from the camera CAM. The processor 101 and the storage device 102 constituting the first position estimating unit 111 and the second position estimating unit 113 may be shared or different. Algorithms for position estimation are applied to the first position estimating unit 111 and the second position estimating unit 113, and the first position estimating unit 111 and the second position estimating unit 113 perform position estimation of the vehicle 1 based on the image VID and the camera information INF according to the respective algorithms.

[0052] When performing position estimation, the first position estimation unit 111 and the second position estimation unit 113 obtain a common image VID from a common camera CAM. In the case of multiple camera CAMs, the image VID is obtained from the same camera CAM among the multiple camera CAMs. However, different algorithms are used in the first position estimation unit 111 and the second position estimation unit 113. The first position estimation algorithm is applied to the first position estimation unit 111, but the second position estimation algorithm different from the first position estimation algorithm is applied to the second position estimation unit 113.

[0053] The algorithm applied to either or both of the first position estimation unit 111 and the second position estimation unit 113 may be a machine learning model. In the case where the machine learning model is applied to both, the machine learning models applied to the first position estimation unit 111 and the second position estimation unit 113 may be machine learning models with different learning levels. The first position estimation unit 111 calculates the first vehicle position PV1 according to the image VID based on the first position estimation algorithm. The second position estimation unit 113 calculates the second vehicle position PV2 according to the image VID based on the second position estimation algorithm.

[0054] The first vehicle position PV1 and the second vehicle position PV2 are input to the detection unit 116. The detection unit 116 detects the state where the first vehicle position PV1 and the second vehicle position PV2 deviate as a "deviation state". Specifically, the detection unit 116 calculates the position difference ΔPV between the first vehicle position PV1 and the second vehicle position PV2, and detects the state where the position difference ΔPV exceeds the allowable error α as a deviation state. Figure 2 For example, condition a is a condition in which no deviation is detected, and condition b is a condition in which a deviation is detected.

[0055] It should be noted that, as mentioned above, the image VID is a collection of images continuously captured by the camera CAM, and the images captured by the camera CAM are sequentially input to the first position estimation unit 111 and the second position estimation unit 113. The first position estimation unit 111 and the second position estimation unit 113 sequentially estimate the vehicle position based on the sequentially input images. The calculated first vehicle position PV1 and second vehicle position PV2 are sequentially input to the detection unit 116, and the detection results obtained by the detection unit 116 are sequentially output whenever the two vehicle positions are compared.

[0056] The vehicle control unit 115 controls the vehicle 1 based on the position estimation results obtained by the first position estimating unit 111 and the second position estimating unit 113 and the detection result obtained by the detection unit 116. As the result of the position estimation, it is arbitrary to use either the first vehicle position PV1 or the second vehicle position PV2, and either one may be used, or the vehicle control unit 115 may recalculate the position of the vehicle 1 based on both.

[0057] At this time, if the deviation state is not detected, the vehicle control unit 115 controls the vehicle 1 as usual based on the estimated position of the vehicle 1. On the other hand, if the deviation state is detected, the vehicle control unit 115 decelerates or stops the vehicle 1 in response to the deviation state.

[0058] 1-2. Control of the Vehicle in Response to Detection of a Deviation State

[0059] The vehicle control unit 115 decelerates or stops the vehicle 1 in response to the deviation state. The control of the vehicle 1 in response to the deviation state may be to decelerate or stop the vehicle 1 immediately when the deviation state is detected. Alternatively, the control of the vehicle 1 may be changed according to the length of time the deviation state is detected. Four examples of the control of the vehicle 1 in response to the deviation state are given below.

[0060] As a first example, when the deviation state is detected, the vehicle 1 may be decelerated or stopped regardless of the length of the detection time.

[0061] As a second example, if the deviation state is detected for a certain period of time, the vehicle 1 may be decelerated or stopped. That is, even if the deviation state is temporarily detected, if the deviation state is not detected again before a certain period of time has passed, the vehicle 1 may not be decelerated or stopped. According to the second example, it is possible to prevent the vehicle 1 from being unnecessarily decelerated or stopped when the deviation state is detected due to a temporary factor such as an erroneous detection.

[0062] As a third example, the vehicle control unit 115 may start to slowly decelerate the vehicle 1 when the deviation state is detected. Then, if the deviation state is detected for a certain period of time, the vehicle control unit 115 may further reduce the speed of the vehicle 1 to stop the vehicle 1. If the deviation state is no longer detected before the certain period of time has passed, the deceleration of the vehicle 1 is canceled and the control of the vehicle 1 as usual is resumed.

[0063] According to the third example, deceleration is started when a deviation state is detected, thereby ensuring the safety of the vehicle 1. On the other hand, when the detection of the deviation state is caused by a temporary factor such as false detection and is eliminated in a short time, deceleration is also cancelled, thereby preventing the vehicle 1 from stopping unnecessarily.

[0064] As a fourth example, the vehicle control unit 115 may temporarily stop the vehicle 1 when the deviation state is detected. Then, if the deviation state is detected for a certain period of time, the vehicle control unit 115 may transfer the vehicle 1 from the temporary stop state to the complete stop state, and then perform processing for restoring the vehicle 1 to a controllable state. If the deviation state is not detected before the certain period of time has passed, the vehicle control unit 115 cancels the stop of the vehicle 1 and resumes the control of the vehicle 1 that was performed before the temporary stop.

[0065] The process for restoring the vehicle 1 to a controllable state may be, for example, notifying the manager of the vehicle 1 that the reliability of the position estimation has decreased. Alternatively, if the vehicle 1 is a remote-drive vehicle that can be controlled by remote driving, the following process may be performed: requesting remote driving from the remote driver, thereby switching the vehicle 1 to remote driving.

[0066] 2. Second Embodiment (Stop of Vehicle in Response to Uncomputable State)

[0067] In the second embodiment, an uncalculated state in which neither the first vehicle position PV1 nor the second vehicle position PV2 can be calculated is considered. In the uncalculated state, neither the first vehicle position PV1 nor the second vehicle position PV2 can be calculated, so the detection unit 116 cannot calculate the position difference ΔPV and cannot detect the deviation state. In such a state, the reliability of the position estimation cannot be evaluated.

[0068] Therefore, the vehicle control unit 115 decelerates or stops the vehicle 1 when the incomputable state is detected. It should be noted that, at this time, the vehicle control unit 115 may first temporarily stop the vehicle 1, and release the stop of the vehicle 1 when the incomputable state is eliminated before the incomputable state continues for a certain period of time. In this case, when the incomputable state is detected for a certain period of time, the vehicle 1 is transferred to a completely stopped state.

[0069] By stopping the vehicle 1 when the incalculable state is detected, control for ensuring safety can be performed even when the deviation state is not detected. In addition, the vehicle 1 is released from the stop when the incalculable state is eliminated before continuing for a certain period of time, so that the control of the vehicle 1 can be restored to normal when the incalculable state is generated due to temporary factors such as temporary processing delay. In this way, the running of the vehicle 1 can be prevented from being excessively suppressed.

[0070] 3. Third Embodiment (Tolerance Based on Environmental Parameters)

[0071] The allowable error α set for the position difference ΔPV may be set to a uniform value in advance or may be set to a value that varies in response to the situation. In the third embodiment, the allowable error α is set to a variable value in response to the environment around the vehicle 1 .

[0072] The detection unit 116 calculates the environmental parameter EP, and sets the allowable error α so as to have a value that varies in response to the environmental parameter EP. Figure 4 1 is a diagram for explaining an example of a method for calculating an environmental parameter EP. The environmental parameter EP is a parameter representing the environment around the vehicle 1. The less the environment in which the vehicle 1 is located is affected by the reliability of the position estimation, the larger the environmental parameter EP is calculated to be. An environment that is less affected by the reliability of the position estimation is an environment that is considered to have little effect on the travel of the vehicle 1 even if an error occurs in the control of the vehicle 1.

[0073] For example, based on the lane width of the lane in which the vehicle 1 is traveling, the environmental parameter EP may be set such that the larger the lane width, the larger the environmental parameter EP. For example, the environmental parameter EP may be a value representing the road width itself. Even if an error occurs in the vehicle position PV, if the lane width is large, the possibility of the vehicle 1 exceeding the lane is small, and it can be said that the impact on the travel of the vehicle 1 is also small. Therefore, the larger the lane width, the larger the environmental parameter EP.

[0074] Alternatively, the environmental parameter EP may be calculated based on the distance to the obstacle existing around the vehicle 1 so that the greater the distance to the obstacle, the greater the environmental parameter EP. For example, the distance to the obstacle itself may be set as the environmental parameter EP. Even if an error occurs in the vehicle position PV, if the distance between the vehicle 1 and the obstacle is large, the possibility of the vehicle 1 colliding with the obstacle is small, and it can be said that the impact on the travel of the vehicle 1 is also small. Therefore, the greater the distance to the obstacle, the greater the environmental parameter EP.

[0075] It should be noted that the vehicle control system 100 can obtain information about the lane width and the distance to obstacles by analyzing the image VID obtained from the camera CAM. As another example, map information indicating the lane width may be pre-stored in the storage device 102, and the lane width at the position where the vehicle 1 is traveling may be obtained based on the map information and the approximate position of the vehicle 1. As another example, the vehicle control system 100 may also obtain this information from the vehicle 1. The vehicle 1 may obtain the lane width and the distance to obstacles around the vehicle 1 based on the information obtained by the vehicle-mounted sensors such as cameras and LIDAR (Light Detection And Ranging) that identify the surroundings of the vehicle 1.

[0076] The permissible error α is set so that the larger the environmental parameter EP is, the larger the permissible error α is. Figure 5 : is a graph showing an example of the allowable error α set based on the environmental parameter EP. The allowable error α can be set to a value that increases linearly as the environmental parameter EP increases. Alternatively, the allowable error α can also be set to a value that increases nonlinearly, for example, exponentially, as the environmental parameter EP increases. In addition, the increase in the allowable error α can be either a monotonic increase or a step-by-step increase.

[0077] According to the third embodiment, in an environment where the influence of the reliability of the position estimation is small, the permissible error α is increased. The increase in the permissible error α means that the deviation state is not easily detected. As a result, it is possible to suppress the unnecessary deceleration and stop of the vehicle 1 in a situation where the influence on the driving of the vehicle 1 is small. In this way, the deviation state is easily detected in a scenario where the influence of the reliability of the position estimation is large, thereby maintaining safety and improving the continuity of the driving of the vehicle 1.

[0078] 4. Fourth Embodiment (Tolerance Based on Time Difference)

[0079] 4-1. Calculation method of allowable error

[0080] The fourth embodiment is also an embodiment in which the permissible error α is set to a variable value. As described above, the image VID is a set of images continuously acquired in each shooting cycle. In the fourth embodiment, the permissible error α is set to be variable in response to the time difference of the images used for position estimation among the images included in the image VID.

[0081] Figure 6 This is a diagram for explaining the fourth embodiment. The video VID captured by the camera CAM during the period from time t1 to time t9 includes images IMG1 to IMG9 of 9 frames.

[0082] Images IMG1 to IMG9 are sequentially input to the first position estimation unit 111 and the second position estimation unit 113, and the first vehicle position PV1 and the second vehicle position PV2 calculated based on the input images are sequentially outputted respectively. However, although the input is the same, the first vehicle position PV1 and the second vehicle position PV2 are not limited to being outputted at the same timing. Different algorithms are applied to the first position estimation unit 111 and the second position estimation unit 113, so the operation time for position estimation may be different. In addition, processing times such as the communication time for sending images from the camera CAM to each processing unit and the communication time for sending the position estimation result from each processing unit to the detection unit 116 may also be different. Figure 6 In the example, the processing time of the second position estimating unit 113 is longer than the processing time of the first position estimating unit 111, and when the calculation results based on the images captured at the same timing are compared, the second vehicle position PV2 is output later than the first vehicle position PV1.

[0083] The detection unit 116 basically calculates the position difference ΔPV by comparing the latest vehicle positions with each other. For example, it is assumed that the position difference ΔPV is calculated at time t9. The latest first vehicle position PV1 is the first vehicle position PV1-8 calculated based on the image IMG8 captured at time t8. In addition, the latest second vehicle position PV2 is the second vehicle position PV2-7 calculated based on the image IMG7 captured at time t7. Therefore, the detection unit 116 compares the first vehicle position PV1-8 with the second vehicle position PV2-7 to calculate the position difference ΔPV.

[0084] However, since the shooting time is different, the actual position of the vehicle 1 may change between the image IMG7 and the image IMG8. If the actual position of the vehicle 1 is different, the first vehicle position PV1 and the second vehicle position PV2 will of course be different, and the position difference ΔPV may become larger. In the case where the first vehicle position PV1 and the second vehicle position PV2 to be compared are calculated based on images shot at different times, there is a possibility that the position estimation process itself is normal but is mistakenly determined to be a deviation state. However, from the perspective of the continuity of the travel of the vehicle 1, it is not preferable to slow down or stop the vehicle 1 in such a situation.

[0085] Therefore, in the fourth embodiment, the permissible error α is changed in consideration of the time difference between the time when the image used for the calculation of the position estimation is captured. Hereinafter, the image used for the calculation of the first vehicle position PV1 among the images included in the video VID is referred to as the first image, and the image used for the calculation of the second vehicle position PV2 is referred to as the second image. The detection unit 116 sets the permissible error α so that the permissible error α increases as the time difference between the time when the first image is captured and the time when the second image is captured increases.

[0086] For example, when the first vehicle position PV1-8 and the second vehicle position PV2-7 are compared to calculate the position difference ΔPV, the first image becomes image IMG8 and the second image becomes image IMG7. Furthermore, the detection unit 116 sets the permissible error α based on the time difference between the time t7 when the image IMG7 is captured and the time t8 when the image IMG8 is captured. In this case, the permissible error α is larger than when the first image and the second image are images captured at the same time.

[0087] Figure 7 This is an example of a method for setting the allowable error α based on the time difference ΔT between the time when the first image is captured and the time when the second image is captured. The allowable error α is set to increase as the time difference ΔT increases. The allowable error α can be set to increase monotonically as the time difference ΔT increases, or it can be set to increase in stages.

[0088] Thus, in the third embodiment, the times when the images used for calculating the two vehicle positions were captured are compared, and when the time difference ΔT between the captured times is large, the permissible error α is increased. The increase in the permissible error α means that the deviation state is not easily detected. Thus, it is possible to prevent the position estimation process itself from being erroneously determined as a deviation state even though it is normal. As a result, it is possible to prevent the vehicle 1 from being decelerated or stopped unnecessarily, and to prevent the passengers of the vehicle 1 from feeling uncomfortable.

[0089] It should be noted that the detection unit 116 can determine the time difference between the shooting times of the first image and the second image as follows. Time information about the time when the images were shot is associated with each image included in the image VID. The time information is a timestamp, a serial number, etc., and is stored in the frame header of each image. The first position estimating unit 111 and the second position estimating unit 113 send the time information of the image used in calculating the vehicle position to the detection unit 116 in association with the vehicle position. The detection unit 116 can calculate the time difference ΔT between the time when the images used for calculating the vehicle position were shot based on the time information of each image associated with the first vehicle position PV1 and the second vehicle position PV2.

[0090] 4-2. Application examples

[0091] In the third embodiment, the following application examples may also be adopted. In the application examples, the first vehicle position PV1 and the second vehicle position PV2 to be compared are not limited to the latest vehicle positions.

[0092] The calculated first vehicle position PV1 and second vehicle position PV2 are not immediately compared but are temporarily stored in a buffer. The buffer is a temporary storage area included in the storage device 102. The detection unit 116 selects the first vehicle position PV1 and the second vehicle position PV2 stored in the buffer, and the time difference between the shooting time of the first image and the shooting time of the second image is less than the time tolerance error. The time tolerance error is pre-set to a time such that even if the position of the vehicle 1 changes, it will become a change to a degree that is allowed as an error. The detection unit 116 compares the first vehicle position PV1 and the second vehicle position PV2 obtained from the buffer to calculate the position difference ΔPV.

[0093] 5. Fifth Implementation

[0094] The fifth embodiment is an embodiment in which the camera CAM includes a plurality of infrastructure cameras and performs switching (handover) of cameras that capture an image of the vehicle 1 . Figure 8 This is a diagram for explaining the fifth embodiment. The camera CAM includes a camera CAM1 and a camera CAM2 as a plurality of infrastructure cameras. The camera CAM2 is set at a position to shoot the vehicle 1 after the camera CAM1. When the vehicle 1 deviates from the field of view of the camera CAM1 as the vehicle 1 moves, the camera CAM used for calculating the vehicle position is switched. By switching the camera CAM, even if the vehicle 1 moves and no longer enters the shooting range of one camera CAM, the vehicle 1 can enter the shooting range of the next camera CAM.

[0095] The timing of switching the camera CAM that captures the image VID is considered. The switching conditions for switching the camera CAM are set for the first position estimation unit 111 and the second position estimation unit 113, respectively. The following examples are given as examples of switching conditions. The first example is that the difference between the installation position of the camera CAM that is currently capturing the vehicle 1 and the position of the vehicle 1 exceeds a threshold. The second example is that the vehicle 1 passes through a specified position within the field of view of the camera CAM currently in use. The third example is that the front end of the vehicle 1 passes through a specified position within the field of view of the camera CAM that is scheduled to be used next.

[0096] Here, since the algorithms applied in the first position estimation unit 111 and the second position estimation unit 113 are different, the camera switching conditions may also be different. For example, in the case where the difference between the installation position of the camera CAM and the position of the vehicle 1 exceeds a threshold value and is set as a switching condition, the threshold value set as the switching condition may be different. Or, for example, in the case where the vehicle 1 passes through a predetermined position within the field of view of the camera CAM currently in use and is set as a switching condition, the predetermined position set as the switching condition may be different. In the case of different switching conditions, there may be a period in which the camera CAM used for calculating the first vehicle position PV1 and the camera CAM used for calculating the second vehicle position PV2 are inconsistent. If the first vehicle position PV1 and the second vehicle position PV2 are calculated based on different camera CAMs, the position difference ΔPV may become larger. The deviation state detected due to such factors is not caused by an abnormality in the position estimation process itself, but is caused only by the inconsistency of the camera CAM used. That is, in the case where the camera CAM is switched at different timings in the first position estimation unit 111 and the second position estimation unit 113, there is a possibility that the position estimation process itself is normal but is mistakenly determined as a deviation state.

[0097] Therefore, the vehicle control system 100 makes the timing of switching the camera CAM consistent in the first position estimating unit 111 and the second position estimating unit 113. That is, the camera CAM used by the first position estimating unit 111 for calculating the first vehicle position PV1 and the camera CAM used by the second position estimating unit 113 for calculating the second vehicle position PV2 are switched from the camera CAM1 to the camera CAM2 in a manner consistent with each other.

[0098] Fig. 9, an example of switching of the camera CAM is shown in such a manner that the camera CAM used by the first position estimation unit 111 for calculating the first vehicle position PV1 is consistent with the camera CAM used by the second position estimation unit 113 for calculating the second vehicle position PV2. The vehicle control system 100 determines whether the first condition and the second condition are satisfied. Then, after both the first condition and the second condition are satisfied, the vehicle control system 100 switches the camera CAM used for calculating the first vehicle position PV1 and the second vehicle position PV2. The first condition is a condition that allows the camera CAM used by the first position estimation unit 111 for calculating the first vehicle position PV1 to be switched. The second condition is a condition that allows the camera CAM used by the second position estimation unit 113 for calculating the second vehicle position PV2 to be switched.

[0099] The first condition and the second condition are, for example, the same conditions as the switching condition described above. The first condition is different from the second condition, so either one may be satisfied first, but even if one is satisfied first, only the one will not be switched first.

[0100] exist Fig. 9 In the topmost state, both the first condition and the second condition are not satisfied, so both the first position estimation unit 111 and the second position estimation unit 113 obtain the image VID from the camera CAM1. After that, only the second condition is satisfied. At this stage, the first condition is not satisfied, so the second position estimation unit 113 does not switch the camera CAM, and the camera CAM1 continues to be used. After that, when the first condition is further satisfied, the camera CAM is switched to the camera CAM2 in both the first position estimation unit 111 and the second position estimation unit 113.

[0101] By making the switching timing of the camera CAM consistent in this way, the following effects can be obtained. The camera CAM used in the first position estimation unit 111 and the second position estimation unit 113 are consistent. That is, it is suppressed that the first vehicle position PV1 and the second vehicle position PV2 are unnecessarily deviated due to the inconsistency of the camera CAM. As a result, it is suppressed that the position estimation process itself is normal but is mistakenly determined to be a deviation state. As a result, unnecessary deceleration is suppressed, and the discomfort of the passengers is suppressed.

[0102] 6. Sixth Implementation

[0103] Fig.101 is a block diagram showing a configuration example of a vehicle 1 in a sixth embodiment. In the sixth embodiment, the camera CAM is a vehicle-mounted camera mounted on the vehicle 1. The vehicle control system 100 includes a control device 10 mounted on the vehicle 1. The vehicle control system 100 may further include a recognition sensor 11 for recognizing the surroundings of the vehicle 1, a vehicle state sensor 12 for detecting the state of the vehicle 1, a position sensor 13 for detecting the position of the vehicle 1, and a travel device 14.

[0104] The control device 10 includes a processor 101 and a storage device 102. The control device 10 may also be a part of the automatic driving system of the vehicle 1. That is, at least a part of the processor 101 and the storage device 102 may be shared with a processor and a storage device constituting the automatic driving system of the vehicle 1. In addition, the processor 101 may be, for example, one or more ECUs (Electronic Control Units).

[0105] In this case, the position estimation performed by either or both of the first position estimating unit 111 and the second position estimating unit 113 may be self-position estimation (Localization) by combining the recognition result based on the video VID of the camera CAM with map information to estimate the self-position.

[0106] As another example, when the camera CAM is a vehicle-mounted camera, the processor 101 and the storage device 102 may also be included in a server outside the vehicle 1. Alternatively, the processor 101 and part of the storage device 102 may be included in an external server, and the processing performed by the vehicle control system 100 may be performed in a distributed manner in the vehicle 1 and the external server.

[0107] The first to sixth embodiments have been described above. The fifth embodiment is limited to the case where the camera CAM is an infrastructure camera and cannot be combined with the sixth embodiment. However, for other embodiments, two or more embodiments can be combined arbitrarily.

Claims

1. A vehicle control system for controlling a vehicle, wherein: The vehicle control system comprises one or more processors, The one or more processors are configured to: Acquire images captured by a shared camera; calculating a first vehicle position as the position of the vehicle from the image based on a first position estimation algorithm; calculating a second vehicle position as the position of the vehicle from the image based on a second position estimation algorithm; as well as In response to a deviation condition in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error, the vehicle is decelerated or stopped.

2. The vehicle control system according to claim 1, wherein: The environmental parameter is the width of the lane where the vehicle is located or the distance between the vehicle and surrounding obstacles. The one or more processors increase the allowable error as the environmental parameter becomes larger.

3. The vehicle control system according to claim 1 or 2, wherein: The one or more processors decelerate the vehicle if the deviation state is detected or if the detection of the deviation state continues for a certain period of time.

4. The vehicle control system according to claim 1 or 2, wherein: The one or more processors are configured to: initiating deceleration of the vehicle upon detecting the deviation condition; If the detection of the deviation state continues for a certain period of time, stopping the vehicle; as well as When the detection of the deviation state is eliminated before the detection continues for the certain period of time, the deceleration of the vehicle is cancelled.

5. The vehicle control system according to claim 1 or 2, wherein: The one or more processors are configured to: When the deviation state is detected, stopping the vehicle; When the detection of the deviation state continues for a certain period of time, notifying a manager of the vehicle or switching to remote driving of the vehicle; as well as When the detection of the deviation state is eliminated before the state continues for the certain period of time, the stop of the vehicle is released.

6. The vehicle control system according to claim 1 or 2, wherein: The one or more processors are configured to: When an uncalculatable state in which any one of the first vehicle position and the second vehicle position cannot be calculated is detected, stopping the vehicle; as well as When the incalculable state is resolved before continuing for a certain period of time, the vehicle is released from the stop.

7. The vehicle control system according to claim 1 or 2, wherein: A first image is included in the image and is used for calculating the first vehicle position. The second image is included in the image and is used for calculating the position of the second vehicle. The one or more processors increase the allowable error as the time difference between the first image and the second image increases.

8. The vehicle control system according to claim 7, wherein: The first image is an image used for calculation of the latest first vehicle position, The second image is an image used for calculation of the latest second vehicle position, The one or more processors calculate a difference between the latest first vehicle position and the latest second vehicle position as the position difference.

9. The vehicle control system according to claim 7, wherein: The one or more processors are configured to: storing the information of the first vehicle position and the information of the second vehicle position in a buffer; as well as The position difference between the first vehicle position and the second vehicle position is calculated based on the first vehicle position and the second vehicle position at which the time difference between the first image and the second image is smaller than a time tolerance.

10. The vehicle control system according to claim 1 or 2, wherein: The common camera is an infrastructure camera external to the vehicle, The vehicle is presented in the image.

11. The vehicle control system according to claim 10, wherein: The shared camera includes a first shared infrastructure camera and a second shared infrastructure camera, wherein the second shared infrastructure camera is arranged at a position behind the first shared infrastructure camera to photograph the vehicle, The one or more processors also switch the common camera from the first common infrastructure camera to the second common infrastructure camera in a manner that causes the common camera used for calculation of the first vehicle position to be consistent with the common camera used for calculation of the second vehicle position.

12. The vehicle control system according to claim 11, wherein: The one or more processors are further configured to: When a first condition is satisfied, allowing the common camera used for calculation of the first vehicle position to be switched from the first common infrastructure camera to the second common infrastructure camera; allowing switching of the common camera used for calculation of the second vehicle position from the first common infrastructure camera to the second common infrastructure camera if a second condition is satisfied; using the first common infrastructure camera for calculating the first vehicle position and calculating the second vehicle position until both the first condition and the second condition are satisfied; as well as After both the first condition and the second condition are satisfied, the common camera used for calculation of the first vehicle position and calculation of the second vehicle position is switched from the first common infrastructure camera to the second common infrastructure camera.

13. A vehicle control method for controlling a vehicle, the vehicle control method comprising: Acquire images captured by a shared camera; calculating a first vehicle position as the position of the vehicle from the image based on a first position estimation algorithm; calculating a second vehicle position as the position of the vehicle from the image based on a second position estimation algorithm; as well as In response to a deviation condition in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error, the vehicle is decelerated or stopped.

Images