Vehicle control system and vehicle control method

Abstract

To ensure safety when controlling a vehicle by utilizing a vehicle position estimated based on a common camera and a plurality of different algorithms.SOLUTION: The present disclosure provides a vehicle control system for controlling a vehicle. The vehicle control system acquires an image captured by a common camera, calculates a first vehicle position, which is a position of the vehicle, from the image captured by the common camera based on a first position estimating algorithm, calculates a second vehicle position, which is a position of the vehicle, from the image captured by the common camera based on a second position estimating algorithm, and decelerates or stops the vehicle according to a divergence status in which a positional difference between the first vehicle position and the second vehicle position exceeds an allowable error.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a technology for controlling a vehicle using a camera. [Background technology]

[0002] Patent Document 1 discloses a system for controlling vehicle travel within a predetermined area. A control device in the system includes a first acquisition unit that acquires detection data of vehicles within a predetermined area from a detection unit provided within the predetermined area, a first estimation unit that estimates a first position indicating the position of the vehicle based on the detection data, a second acquisition unit that acquires from the vehicle a second position indicating the position of the vehicle estimated by a second estimation unit provided in the vehicle, and a calibration control unit that causes one of the first estimation unit and the second estimation unit to perform calibration based on a comparison result between the first position and the second position.

[0003] In addition to the above-mentioned Patent Document 1, the following Patent Documents 2 to 7 can be mentioned as examples of documents showing the state of the art in the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-081961 [Patent Document 2] International Publication No. 2019 / 073795 [Patent Document 3] JP 2019-135620 A [Patent Document 4] Japanese Patent Application Publication No. 11-039589 [Patent Document 5] Japanese Patent Application Publication No. 11-096494 [Patent Document 6] JP 2019-020786 A [Patent Document 7] JP 2019-130997 A Summary of the Invention [Problem to be solved by the invention]

[0005] There are many possibilities for proposing algorithms for estimating vehicle positions based on images captured by a camera. However, technology for estimating vehicle positions based on a plurality of different algorithms from a common image captured by a common camera has not been sufficiently studied. [Means for solving the problem]

[0006] The present disclosure provides a vehicle control system for controlling a vehicle. The vehicle control system includes one or more processors. The one or more processors Acquire the video captured by the common camera, Calculating a first vehicle position, which is the position of the vehicle, from an image captured by the common camera based on a first position estimation algorithm; Calculating a second vehicle position, which is the position of the vehicle, from the image captured by the common camera based on a second position estimation algorithm; The vehicle is slowed down or stopped in response to a deviation state in which the position difference between the first vehicle position and the second vehicle position exceeds an allowable error.

[0007] The present disclosure provides a vehicle control method for controlling a vehicle. The vehicle control method includes: Acquiring an image captured by a common camera; Calculating a first vehicle position, which is a position of the vehicle, from an image captured by the common camera based on a first position estimation algorithm; Calculating a second vehicle position, which is the position of the vehicle, from an image captured by the common camera based on a second position estimation algorithm; slowing down or stopping the vehicle in response to a deviation state in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error; Includes. Effect of the Invention

[0008] According to the technology of the present disclosure, multiple vehicle positions are calculated from a common image based on multiple position estimation algorithms. The reliability of the position estimation can be evaluated by comparing the multiple calculated vehicle positions. Then, if it is considered that the reliability of the position estimation has decreased, the vehicle is slowed down or stopped. In this way, the safety of the vehicle's travel can be ensured. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an overview of a vehicle control system according to a first embodiment. [Diagram 2] 10A and 10B are diagrams for explaining a position difference calculated by the vehicle control system; [Diagram 3] FIG. 2 is a block diagram showing an example of a processing unit included in the vehicle control system. [Figure 4] FIG. 13 is a diagram illustrating an example of environmental parameters in the third embodiment. [Diagram 5] 1 is a graph showing an example of a relationship between an environmental parameter and an allowable error. [Figure 6] FIG. 13 is a diagram for explaining a fourth embodiment. [Figure 7] 13 is a graph showing an example of an allowable error in the fourth embodiment. [Figure 8] FIG. 13 is a diagram for explaining a fifth embodiment. [Figure 9] FIG. 13 is another diagram for explaining the fifth embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of the configuration of a vehicle according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0011] 1. First embodiment 1-1. Overview and basic configuration of vehicle control system FIG. 1 is a conceptual diagram for explaining an overview of a vehicle control system 100 according to the present embodiment. The vehicle control system 100 controls a vehicle 1 within a predetermined area AR. The vehicle 1 may be an autonomous vehicle capable of autonomous driving at least within the predetermined area AR. There may be a plurality of vehicles 1 that are targets of control by the vehicle control system 100. At least a part of the vehicle control system 100 may be included in a management server external to the vehicle 1. At least a part of the vehicle control system 100 may be included in an in-vehicle system mounted on the vehicle 1. The vehicle control system 100 may be distributed between the management server and the in-vehicle system.

[0012] At least one camera CAM is installed in the predetermined area AR. The camera CAM is an infrastructure camera capable of photographing the inside of the predetermined area AR. The camera CAM photographs the situation in the predetermined area AR and obtains a video VID showing the situation in the predetermined area AR. The video VID is a set of images obtained by the camera CAM photographing the field angle continuously at each predetermined photographing period. The video VID may include a vehicle 1 traveling in the predetermined area AR. Examples of the predetermined area AR in which the camera CAM is installed include a parking lot, a factory, a roadway (general road, expressway), and the like. When the predetermined area AR is a parking lot, the vehicle 1 may be compatible with automated valet parking (AVP). When the predetermined area AR is a factory site, the vehicle 1 assembled in the assembly plant may automatically travel from the assembly plant to the yard. In that case, one or more cameras CAM are installed on the road from the assembly plant to the yard.

[0013] The vehicle control system 100 includes one or more processors 101 (hereinafter simply referred to as processor 101) and one or more storage devices 102 (hereinafter simply referred to as storage device 102). The processor 101 executes various processes. Examples of the processor 101 include a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). The storage device 102 stores one or more programs and various information. Examples of the storage device 102 include a hard disk drive (HDD), a solid state drive (SSD), a volatile memory, and a non-volatile memory. The functions of the vehicle control system 100 may be realized by cooperation between the processor 101, which executes one or more programs, and the storage device 102. The one or more programs may be recorded in a computer-readable recording medium.

[0014] The vehicle control system 100 is communicatively connected to the camera CAM. The video VID captured by the camera CAM is transmitted to the vehicle control system 100 via a wired communication network or a wireless communication network. The vehicle control system 100 acquires the video VID captured by the camera CAM. The vehicle control system 100 estimates the position of the vehicle 1 in a predetermined area AR (hereinafter referred to as the vehicle position PV) based on the video VID. More specifically, the vehicle control system 100 estimates (calculates) the vehicle position PV from the video VID based on a predetermined position estimation algorithm.

[0015] For example, the vehicle control system 100 detects (recognizes) the vehicle 1 in a predetermined area AR from the video VID based on a predetermined vehicle detection algorithm. For example, the vehicle control system 100 uses an image recognition AI to recognize the vehicle 1 shown in the video VID. The image recognition AI is generated in advance through machine learning. Then, the vehicle control system 100 calculates the position of the vehicle 1 in the video VID (image). In other words, the vehicle control system 100 calculates the relative position of the vehicle 1 seen from the camera CAM. Meanwhile, the vehicle control system 100 holds camera information INF. The camera information INF is information indicating the installation position, installation direction, angle of view, etc. of the camera CAM. The camera information INF is provided to the vehicle control system 100 in advance and stored in the storage device 102. The camera information INF may be sent from the camera CAM. The vehicle control system 100 can calculate the vehicle position PV based on the relative position of the vehicle 1 in the image and the camera information INF.

[0016] The vehicle control system 100 controls the vehicle 1 based on the vehicle position PV estimated in this manner. For example, the vehicle control system 100 performs automatic driving control based on the vehicle position PV. As another example, the vehicle control system 100 may perform driving assistance control based on the vehicle position PV. Examples of driving assistance control include risk avoidance control and lane keeping control. For example, the vehicle control system 100 generates a target trajectory for the vehicle 1. The target trajectory includes a target position and a target speed of the vehicle 1. Then, the vehicle control system 100 controls the traveling of the vehicle 1 so as to follow the target trajectory.

[0017] The vehicle 1 is equipped with a traveling device. The traveling device includes a driving device that drives the vehicle 1, a braking device that brakes the vehicle 1, and a steering device that steers the vehicle 1. The control information is information for controlling the vehicle 1. The vehicle control system 100 controls the vehicle 1 by controlling the traveling device in accordance with the control information.

[0018] In this way, the vehicle 1 is controlled based on the vehicle position PV estimated based on a predetermined position estimation algorithm. However, depending on the distance between the vehicle 1 and the camera CAM and the environment around the vehicle 1, the estimation accuracy of the vehicle position PV, that is, the reliability of the estimated vehicle position PV, may be reduced. For example, when the distance between the vehicle 1 and the camera CAM is long, the reliability of the vehicle position PV may be reduced. In addition, when a video VID captured in a backlit environment is used, the reliability of the vehicle position PV may be reduced. When the reliability of the vehicle position PV is low, the accuracy of the vehicle control based on the vehicle position PV may also be reduced. This is undesirable from the viewpoint of safety.

[0019] Therefore, according to this embodiment, the reliability of the vehicle position PV is evaluated. More specifically, the vehicle control system 100 calculates a plurality of vehicle positions PV based on a plurality of different position estimation algorithms. For example, the plurality of position estimation algorithms are created and provided by different developers. In the case of a position estimation algorithm using an image recognition AI, the learning levels of each image recognition AI may be different. Although a plurality of different position estimation algorithms are used, the input video VID is a common video VID captured by a common camera CAM. That is, the vehicle control system 100 calculates a plurality of vehicle positions PV from a common video VID based on a plurality of different position estimation algorithms. Furthermore, the vehicle control system 100 compares the plurality of vehicle positions PV with each other. If the difference between the plurality of vehicle positions PV is within an allowable range, it can be said that the reliability of the plurality of vehicle positions PV (that is, the plurality of position estimation algorithms) is high.

[0020] On the other hand, if the difference between the multiple vehicle positions PV exceeds the allowable range, it is considered that the reliability of at least one of the vehicle positions PV (i.e., at least one of the position estimation algorithms) has decreased. In other words, it is considered that a situation has occurred in which the reliability of the position estimation has decreased. In this case, the vehicle control system 100 decelerates or stops the vehicle 1. In this way, the safety of the vehicle's travel can be ensured.

[0021] 2 is a diagram for explaining the process according to the present embodiment. In the following, a case where two types of location estimation algorithms are used will be explained. The same applies to the case where three or more types of location estimation algorithms are used.

[0022] The vehicle control system 100 calculates a first vehicle position PV1 from the common image VID based on a first position estimation algorithm. The vehicle control system 100 also calculates a second vehicle position PV2 from the common image VID based on a second position estimation algorithm. The vehicle control system 100 compares the first vehicle position PV1 with the second vehicle position PV2, and calculates a position difference ΔPV between the two vehicle positions. The position difference ΔPV is calculated, for example, as the distance between the center of the first vehicle position PV1 and the center of the second vehicle position PV2. Then, when the position difference ΔPV is larger than the allowable error α, that is, when the difference between the two vehicle positions is larger than the range allowed as an error, the vehicle control system 100 decelerates or stops the vehicle 1.

[0023] 2 shows two situations, situation a and situation b, where situation a shows a situation in which the position difference ΔPV is smaller than the allowable error α. When the position difference ΔPV between the two vehicle positions is smaller than the allowable error α, it is considered that the reliability of the position estimation is guaranteed. Therefore, the vehicle control system 100 controls the vehicle 1 as usual.

[0024] In contrast, situation b shows a situation in which the position difference ΔPV is larger than the allowable error α. When the position difference ΔPV is larger than the allowable error α, the reliability of the position estimation may be reduced. Therefore, the vehicle control system 100 decelerates or stops the vehicle 1.

[0025] In this way, the vehicle control system 100 according to the present embodiment is able to estimate the reliability of position estimation by calculating two types of vehicle positions based on two types of algorithms from images acquired from a common camera CAM and comparing them. In a situation where the reliability of position estimation may be reduced, the vehicle 1 is slowed down or stopped, thereby ensuring the safety of the vehicle 1 during its travel.

[0026] 3 is a block diagram showing an example of a processing unit included in the vehicle control system 100. The vehicle control system 100 includes a first position estimation unit 111, a second position estimation unit 113, and a vehicle control unit 115. The vehicle control unit 115 includes a detection unit 116. These processing units are realized by the processor 101 executing a program stored in the storage device 102.

[0027] The first position estimation unit 111 and the second position estimation unit 113 are processing units for estimating the position of the vehicle 1 within a predetermined area AR based on the video VID and the camera information INF acquired from the camera CAM. The processor 101 and the storage device 102 constituting the first position estimation unit 111 and the second position estimation unit 113 may be the same or different. An algorithm for position estimation is applied to the first position estimation unit 111 and the second position estimation unit 113, and the position of the vehicle 1 is estimated based on the video VID and the camera information INF according to each algorithm.

[0028] When performing position estimation, the first position estimation unit 111 and the second position estimation unit 113 acquire a common video VID from a common camera CAM. When there are multiple camera CAMs, the video VID is acquired from the same one of the multiple camera CAMs. However, the first position estimation unit 111 and the second position estimation unit 113 apply different algorithms. The first position estimation unit 111 applies a first position estimation algorithm, and the second position estimation unit 113 applies a second position estimation algorithm different from the first position estimation algorithm.

[0029] An algorithm applied to either or both of the first position estimation unit 111 and the second position estimation unit 113 may be a machine learning model. When a machine learning model is applied to both, the machine learning models applied to the first position estimation unit 111 and the second position estimation unit 113 may be machine learning models with different learning levels. The first position estimation unit 111 calculates a first vehicle position PV1 from the video VID based on the first position estimation algorithm. The second position estimation unit 113 calculates a second vehicle position PV2 from the video VID based on the second position estimation algorithm.

[0030] The first vehicle position PV1 and the second vehicle position PV2 are input to the detection unit 116. The detection unit 116 detects a state in which the first vehicle position PV1 and the second vehicle position PV2 are separated from each other as a "separation state." Specifically, the detection unit 116 calculates a position difference ΔPV between the first vehicle position PV1 and the second vehicle position PV2, and detects a state in which the position difference ΔPV exceeds an allowable error α as a separation state. In the example of FIG. 2, situation a is a situation in which a separation state is not detected, and situation b is a situation in which a separation state is detected.

[0031] As described above, the video VID is a collection of images continuously captured by the camera CAM, and the images captured by the camera CAM are sequentially input to the first position estimation unit 111 and the second position estimation unit 113. The first position estimation unit 111 and the second position estimation unit 113 sequentially estimate the vehicle position based on the sequentially input images. The calculated first vehicle position PV1 and second vehicle position PV2 are sequentially input to the detection unit 116, and the detection results by the detection unit 116 are also sequentially output every time a comparison is made between the two vehicle positions.

[0032] Vehicle control unit 115 controls vehicle 1 based on the results of position estimation by first position estimation unit 111 and second position estimation unit 113, and the detection result by detection unit 116. It is arbitrary which of the first vehicle position PV1 and the second vehicle position PV2 is used as the result of position estimation, and either one of them may be used, or vehicle control unit 115 may recalculate the position of vehicle 1 based on both.

[0033] At this time, if a deviation state is not detected, the vehicle control unit 115 controls the vehicle 1 normally based on the estimated position of the vehicle 1. On the other hand, if a deviation state is detected, the vehicle control unit 115 decelerates or stops the vehicle 1 according to the deviation state.

[0034] 1-2. Vehicle control according to deviation detection The vehicle control unit 115 decelerates or stops the vehicle 1 depending on the deviation state. The control of the vehicle 1 depending on the deviation state may be to decelerate or stop the vehicle 1 immediately when the deviation state is detected. Alternatively, the control of the vehicle 1 may be changed depending on the length of time during which the deviation state is detected. Four examples of the control of the vehicle 1 depending on the deviation state are given below.

[0035] As a first example, when a deviation state is detected, the vehicle 1 may be decelerated or stopped regardless of the length of the detection time.

[0036] As a second example, the vehicle 1 may be decelerated or stopped when a deviation state is detected continuously for a certain period of time. In other words, even if a deviation state is detected temporarily, if the deviation state is not detected again before the certain period of time has elapsed, the vehicle 1 may not be decelerated or stopped. According to the second example, when a deviation state is detected due to a temporary factor such as erroneous detection, it is possible to prevent the vehicle 1 from being unnecessarily decelerated or stopped.

[0037] As a third example, the vehicle control unit 115 may start a gradual deceleration of the vehicle 1 when a deviation state is detected. Then, when the detection of the deviation state continues for a certain period of time, the vehicle control unit 115 may further reduce the speed of the vehicle 1 and stop it. When the detection of the deviation state is resolved before the certain period of time has elapsed, the deceleration of the vehicle 1 is released and normal control of the vehicle 1 is resumed.

[0038] According to the third example, when a deviation state is detected, deceleration is initiated, thereby ensuring the safety of the vehicle 1. On the other hand, when the detection of the deviation state is due to a temporary factor such as an erroneous detection and is resolved in a short time, the deceleration is released, thereby preventing the vehicle 1 from stopping unnecessarily.

[0039] As a fourth example, when a deviation state is detected, the vehicle control unit 115 may temporarily stop the vehicle 1. Then, when the detection of the deviation state continues for a certain period of time, the vehicle control unit 115 may transition the vehicle 1 from the temporarily stopped state to a completely stopped state, and perform processing to restore the vehicle 1 to a controllable state. When the deviation state is no longer detected before the certain period of time has elapsed, the vehicle control unit 115 releases the stop of the vehicle 1 and resumes the control of the vehicle 1 that was being performed before the temporary stop.

[0040] The process for returning the vehicle 1 to a controllable state may be, for example, notifying the manager of the vehicle 1 that the reliability of the position estimation has decreased. Alternatively, if the vehicle 1 is a remotely driven vehicle that can be controlled by remote driving, a request may be made to a remote driver to remotely drive the vehicle 1, and the vehicle 1 may be switched to remote driving.

[0041] 2. Second embodiment (stopping of vehicle in response to calculation impossible state) In the second embodiment, a calculation impossible state in which either the first vehicle position PV1 or the second vehicle position PV2 cannot be calculated is considered. In the calculation impossible state, since either the first vehicle position PV1 or the second vehicle position PV2 has not been calculated, the detection unit 116 cannot calculate the position difference ΔPV, and the deviation state is not detected. In such a state, the reliability of the position estimation cannot be evaluated.

[0042] Therefore, when the calculation impossible state is detected, the vehicle control unit 115 decelerates or stops the vehicle 1. At this time, the vehicle control unit 115 may first temporarily stop the vehicle 1, and if the calculation impossible state is resolved before the calculation impossible state continues for a certain period of time, the vehicle control unit 115 may release the stop of the vehicle 1. In this case, when the calculation impossible state is detected to continue for a certain period of time, the vehicle 1 is transitioned to a completely stopped state.

[0043] By stopping the vehicle 1 when a calculation impossible state is detected, control for ensuring safety can be performed even when a deviation state is not detected. Furthermore, by releasing the stop of the vehicle 1 when the calculation impossible state is resolved before it continues for a certain period of time, normal control of the vehicle 1 can be restored if the calculation impossible state is caused by a temporary factor such as a temporary processing delay. In this way, excessive suppression of the traveling of the vehicle 1 can be prevented.

[0044] 3. Third embodiment (tolerance based on environmental parameters) The allowable error α set for the position difference ΔPV may be preset as a uniform value, or may be a value that varies depending on the situation. In the third embodiment, the allowable error α is set to a variable value depending on the surrounding environment of the vehicle 1.

[0045] The detection unit 116 calculates the environmental parameter EP and sets the allowable error α so that the allowable error α is a value that varies according to the environmental parameter EP. FIG. 4 is a diagram for explaining an example of a method for calculating the environmental parameter EP. The environmental parameter EP is a parameter that represents the environment around the vehicle 1, and is calculated so that the allowable error α becomes larger as the vehicle 1 is in an environment in which the reliability of position estimation has a smaller effect. An environment in which the reliability of position estimation has a smaller effect is an environment in which the effect on the running of the vehicle 1 is considered to be small even if an error occurs in the control of the vehicle 1.

[0046] For example, the environmental parameter EP may be set based on the width of the lane on which the vehicle 1 is traveling, so that the larger the lane width, the larger the environmental parameter EP. For example, the environmental parameter EP may be a value that represents the road width itself. Even if an error occurs in the vehicle position PV, if the lane width is large, the possibility that the vehicle 1 will deviate from the lane is small, and it can be said that the impact on the traveling of the vehicle 1 is also small. Therefore, the larger the lane width, the larger the environmental parameter EP is set.

[0047] Alternatively, the environmental parameter EP may be calculated based on the distance to an obstacle present around the vehicle 1, so that the environmental parameter EP increases as the distance to the obstacle increases. For example, the distance to the obstacle itself may be used as the environmental parameter EP. Even if an error occurs in the vehicle position PV, if the distance between the vehicle 1 and the obstacle is large, the possibility of the vehicle 1 colliding with the obstacle is small, and the effect on the traveling of the vehicle 1 is also small. Therefore, the environmental parameter EP is increased as the distance to the obstacle increases.

[0048] The vehicle control system 100 can obtain information about the lane width and the distance to an obstacle by image analysis of the video VID obtained from the camera CAM. As another example, map information indicating the lane width may be stored in advance in the storage device 102, and the lane width at the position where the vehicle 1 is traveling may be obtained from the map information and the approximate position of the vehicle 1. As yet another example, the vehicle control system 100 may obtain such information from the vehicle 1. The vehicle 1 can obtain the lane width around the vehicle 1 and the distance to an obstacle from information obtained by an on-board sensor such as a camera or a LIDAR (Light Detection And Ranging) recognizing the surroundings of the vehicle 1.

[0049] The allowable error α is set so as to increase as the environmental parameter EP increases. FIG. 5 is a graph showing an example of the allowable error α set based on the environmental parameter EP. The allowable error α may be set as a value that increases linearly as the environmental parameter EP increases. Alternatively, the allowable error α may be set as a value that increases nonlinearly, for example, exponentially, as the environmental parameter EP increases. The increase in the allowable error α may be a monotonous increase or a stepwise increase.

[0050] According to the third embodiment, in an environment where the reliability of position estimation has a small effect, the allowable error α is increased. Increasing the allowable error α means that a deviation state is difficult to detect. Therefore, even when the effect on the traveling of the vehicle 1 is small, the vehicle 1 is prevented from being decelerated or stopped more than necessary. In this way, in a situation where the reliability of position estimation has a large effect, it is possible to easily detect a deviation state, and to increase the continuity of traveling of the vehicle 1 while maintaining safety.

[0051] 4. Fourth embodiment (tolerance based on time difference) 4-1.How to calculate the allowable error The fourth embodiment is also an embodiment in which the allowable error α is a variable value. As described above, the video VID is a collection of images continuously captured at each imaging period. In the fourth embodiment, the allowable error α is made variable according to the time difference between images used for position estimation among the images included in the video VID.

[0052] 6 is a diagram for explaining the fourth embodiment. A video VID captured by a camera CAM between times t1 and t9 includes nine frames of images IMG1-9.

[0053] Images IMG1 to IMG9 are sequentially input to the first position estimation unit 111 and the second position estimation unit 113, and the first vehicle position PV1 and the second vehicle position PV2 calculated based on the input images are sequentially output. However, even if the input is the same, the first vehicle position PV1 and the second vehicle position PV2 are not necessarily output at the same timing. Since different algorithms are applied to the first position estimation unit 111 and the second position estimation unit 113, the calculation time for position estimation may differ. In addition, the processing time such as the communication time for transmitting an image from the camera CAM to each processing unit and the communication time when the position estimation result is transmitted from each processing unit to the detection unit 116 may also differ. In the example of FIG. 6, the processing time in the second position estimation unit 113 is longer than the processing time in the first position estimation unit 111, and when the calculation results based on images captured at the same timing are compared, the second vehicle position PV2 is output later than the first vehicle position PV1.

[0054] The detection unit 116 basically calculates the position difference ΔPV by comparing the latest vehicle positions. For example, assume that the position difference ΔPV is calculated at time t9. The latest first vehicle position PV1 is the first vehicle position PV1-8 calculated based on the image IMG8 captured at time t8. The latest second vehicle position PV2 is the second vehicle position PV2-7 calculated based on the image IMG7 captured at time t7. Therefore, the detection unit 116 calculates the position difference ΔPV by comparing the first vehicle position PV1-8 and the second vehicle position PV2-7.

[0055] However, since the image capturing times are different, the actual position of the vehicle 1 may have changed between the images IMG7 and IMG8. If the actual position of the vehicle 1 is different, the first vehicle position PV1 and the second vehicle position PV2 will naturally also be different, and the position difference ΔPV may become large. In this way, when the first vehicle position PV1 and the second vehicle position PV2 to be compared are calculated based on images captured at different times, there is a risk that a deviation state will be erroneously determined even though the position estimation process itself is normal. However, it is not preferable from the viewpoint of the continuity of the traveling of the vehicle 1 to decelerate or stop the vehicle 1 even in such a case.

[0056] Therefore, in the fourth embodiment, the allowable error α is varied in consideration of the time difference between the images used to calculate the position estimation. Hereinafter, among the images included in the video VID, an image used to calculate the first vehicle position PV1 is called a first image, and an image used to calculate the second vehicle position PV2 is called a second image. The detection unit 116 sets the allowable error α so that the allowable error α increases as the time difference between the time the first image and the time the second image are captured increases.

[0057] For example, when calculating the position difference ΔPV by comparing the first vehicle position PV1-8 and the second vehicle position PV2-7, the first image is image IMG8 and the second image is image IMG7. Then, the detection unit 116 sets the allowable error α based on the time difference between the time t7 when image IMG7 is captured and the time t8 when image IMG8 is captured. In this case, the allowable error α is set larger than when the first image and the second image are captured at the same time.

[0058] 7 shows an example of a method for setting the allowable error α based on the time difference ΔT between the time when the first image is captured and the time when the second image is captured. In this way, the allowable error α is set to increase as the time difference ΔT increases. The allowable error α may be set to increase monotonically or stepwise as the time difference ΔT increases.

[0059] In this way, in the third embodiment, the times at which the images used to calculate the two types of vehicle positions were captured are compared, and if the time difference ΔT between the captured images is large, the allowable error α is increased. Increasing the allowable error α means that the deviation state is made more difficult to detect. This makes it possible to suppress erroneous determination of a deviation state even when the position estimation process itself is normal. As a result, it is possible to suppress unnecessarily decelerating or stopping the vehicle 1, and to suppress discomfort felt by the occupants of the vehicle 1.

[0060] The detection unit 116 can determine the time difference between the capture times of the first image and the second image as follows. Each image included in the video VID is linked with time information about the capture time. The time information is a time stamp, a sequence number, or the like, and is stored in the frame header of each image. The first position estimation unit 111 and the second position estimation unit 113 link the time information of the images used to calculate the vehicle position to the vehicle position and send it to the detection unit 116. The detection unit 116 can calculate the time difference ΔT between the times when the images used to calculate the vehicle position were captured, based on the time information of each image linked with the first vehicle position PV1 and the second vehicle position PV2.

[0061] 4-2.Application examples The third embodiment can also be applied as follows: In the application, the first vehicle position PV1 and the second vehicle position PV2 to be compared are not limited to the latest vehicle positions.

[0062] The calculated first vehicle position PV1 and second vehicle position PV2 are not immediately compared, but are temporarily stored in a buffer. The buffer is a temporary storage area included in the storage device 102. The detection unit 116 selects the first vehicle position PV1 and the second vehicle position PV2 stored in the buffer, which have a time difference between the imaging time of the first image and the imaging time of the second image that is less than the time allowable error. The time allowable error is set in advance as a time that results in a change in the position of the vehicle 1 that is an allowable change as an error. The detection unit 116 compares the first vehicle position PV1 and the second vehicle position PV2 acquired from the buffer to calculate the position difference ΔPV.

[0063] 5. Fifth embodiment The fifth embodiment is an embodiment in which the camera CAM includes a plurality of infrastructure cameras, and a switch (handover) of the camera capturing an image of the vehicle 1 is performed. FIG. 8 is a diagram for explaining the fifth embodiment. The camera CAM includes a camera CAM1 and a camera CAM2 as the plurality of infrastructure cameras. The camera CAM2 is installed at a position such that it captures an image of the vehicle 1 after the camera CAM1, and when the vehicle 1 moves out of the angle of view of the camera CAM1, the camera CAM used to calculate the vehicle position is switched. By switching the camera CAM, even if the vehicle 1 moves out of the imaging range of one camera CAM, the vehicle 1 can be brought into the imaging range of the next camera CAM.

[0064] Consider the timing for switching the camera CAM that captures the video VID. The first position estimation unit 111 and the second position estimation unit 113 each have a switching condition set for switching the camera CAM. Examples of the switching condition include the following. The first example is that the difference between the installation position of the camera CAM currently capturing the image of the vehicle 1 and the position of the vehicle 1 exceeds a threshold. The second example is that the vehicle 1 passes through a predetermined position within the angle of view of the camera CAM currently in use. The third example is that the front end of the vehicle 1 passes through a predetermined position within the angle of view of the camera CAM that is scheduled to be used next.

[0065] Here, since the algorithms applied to the first position estimation unit 111 and the second position estimation unit 113 are different, the camera switching conditions may also be different. For example, when the switching condition is that the difference between the installation position of the camera CAM and the position of the vehicle 1 exceeds a threshold, the threshold value set as the switching condition may be different. Alternatively, when the switching condition is that the vehicle 1 passes a predetermined position within the angle of view of the camera CAM currently in use, the predetermined position set as the switching condition may be different. When the switching conditions are different, a period may occur during which the camera CAM used to calculate the first vehicle position PV1 does not match the camera CAM used to calculate the second vehicle position PV2. When the first vehicle position PV1 and the second vehicle position PV2 are calculated based on different camera CAMs, the position difference ΔPV may become large. The deviation state detected due to such a factor is not due to an abnormality in the position estimation process itself, but is simply due to the camera CAM used not matching. In other words, if the camera CAM is switched at different times in the first position estimation unit 111 and the second position estimation unit 113, a deviation state may be erroneously determined even though the position estimation process itself is normal.

[0066] Therefore, the vehicle control system 100 synchronizes the timing of switching the camera CAM between the first position estimation unit 111 and the second position estimation unit 113. In other words, switching from camera CAM1 to camera CAM2 is performed so that the camera CAM used by the first position estimation unit 111 to calculate the first vehicle position PV1 and the camera CAM used by the second position estimation unit 113 to calculate the second vehicle position PV2 match.

[0067] 9 shows an example of camera CAM switching performed so that the camera CAM used by the first position estimation unit 111 to calculate the first vehicle position PV1 and the camera CAM used by the second position estimation unit 113 to calculate the second vehicle position PV2 are the same. The vehicle control system 100 judges whether the first condition and the second condition are satisfied. Then, the vehicle control system 100 switches the camera CAM used to calculate the first vehicle position PV1 and the second vehicle position PV2 after both the first condition and the second condition are satisfied. The first condition is a condition that allows the first position estimation unit 111 to switch the camera CAM used to calculate the first vehicle position PV1. The second condition is a condition that allows the second position estimation unit 113 to switch the camera CAM used to calculate the second vehicle position PV2.

[0068] The first and second conditions are, for example, the same conditions as the switching conditions described above. Since the first and second conditions are different, there is a possibility that either one will be satisfied first, but even if one of them is satisfied first, switching to only one of them will not be performed first.

[0069] In the top state of Fig. 9, neither the first nor the second condition is satisfied, so both the first position estimation unit 111 and the second position estimation unit 113 acquire the video VID from the camera CAM1. After that, only the second condition is satisfied. At this stage, since the first condition is not satisfied, the second position estimation unit 113 does not switch the camera CAM either, and continues to use the camera CAM1. After that, when the first condition is satisfied again, the camera CAM is switched to the camera CAM2 in both the first position estimation unit 111 and the second position estimation unit 113.

[0070] By aligning the timing of switching the camera CAM in this way, the following effects can be obtained. The camera CAM used by the first position estimation unit 111 and the second position estimation unit 113 is the same. In other words, the first vehicle position PV1 and the second vehicle position PV2 are prevented from unnecessarily deviating from each other due to a mismatch in the camera CAM. This prevents the position estimation process itself from being normal but being erroneously determined to be in a deviated state. As a result, unnecessary deceleration is prevented, and discomfort felt by the occupants is also prevented.

[0071] 6. Sixth embodiment 10 is a block diagram showing an example of the configuration of a vehicle 1 in the sixth embodiment. In the sixth embodiment, the camera CAM is an on-board camera mounted on the vehicle 1. The vehicle control system 100 includes a control device 10 mounted on the vehicle 1. The vehicle control system 100 may further include a recognition sensor 11 that recognizes the surroundings of the vehicle 1, a vehicle state sensor 12 that detects the state of the vehicle 1, a position sensor 13 that detects the position of the vehicle 1, and a driving device 14.

[0072] The control device 10 includes a processor 101 and a storage device 102. The control device 10 may be a part of an autonomous driving system of the vehicle 1. In other words, at least a part of the processor 101 and the storage device 102 may be common to a processor and a storage device that configure the autonomous driving system of the vehicle 1. Furthermore, the processor 101 may be, for example, one or more ECUs (Electronic Control Units).

[0073] In this case, the position estimation by either or both of the first position estimation unit 111 and the second position estimation unit 113 may be a self-position estimation (Localization) that estimates the self-position by combining the recognition result based on the image VID of the camera CAM and map information.

[0074] As another example, even when the camera CAM is an on-board camera, the processor 101 and the storage device 102 may be included in a server external to the vehicle 1. Alternatively, a part of the processor 101 and the storage device 102 may be included in an external server, and the processing by the vehicle control system 100 may be distributed between the vehicle 1 and the external server.

[0075] The first to sixth embodiments have been described above. Among these, the fifth embodiment is limited to the case where the camera CAM is an infrastructure camera, and cannot be combined with the sixth embodiment. However, the other embodiments can be arbitrarily combined with two or more of them. [Explanation of symbols]

[0076] AR...predetermined area CAM...camera VID...image INF...camera information PV1...first vehicle position PV2...second vehicle position ΔPV...position difference α...allowable error 1...vehicle 100...vehicle control system 101...processor 102...storage device 111...first position estimation unit 113...second position estimation unit 115...vehicle control unit 116...detection unit

Claims

1. A vehicle control system for controlling a vehicle, One or more processors; The one or more processors: Acquire the video captured by the common camera, Calculating a first vehicle position, which is a position of the vehicle, from the image based on a first position estimation algorithm; Calculating a second vehicle position, which is the position of the vehicle, from the image based on a second position estimation algorithm; The vehicle is decelerated or stopped in response to a deviation state in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error. It was configured as Vehicle control system.

2. 2. The vehicle control system according to claim 1, The environmental parameter is the width of the lane in which the vehicle is located or the distance between the vehicle and surrounding obstacles; The one or more processors increase the tolerance as the environmental parameter increases. Vehicle control system.

3. 3. A vehicle control system according to claim 1, The one or more processors decelerate the vehicle when the deviation state is detected or when the detection of the deviation state continues for a certain period of time. Vehicle control system.

4. 3. A vehicle control system according to claim 1, The one or more processors: when the deviation state is detected, deceleration of the vehicle is initiated; If the detection of the deviation state continues for a certain period of time, the vehicle is stopped; When the detection of the deviation state is eliminated before the certain period of time continues, the deceleration of the vehicle is released. Vehicle control system.

5. 3. A vehicle control system according to claim 1, The one or more processors: When the deviation state is detected, the vehicle is stopped; If the detection of the deviation state continues for a certain period of time, a notification is given to a manager of the vehicle or the vehicle is switched to remote driving; If the detection of the deviation state is eliminated before the certain period of time continues, the stop of the vehicle is released. Vehicle control system.

6. 3. A vehicle control system according to claim 1, The one or more processors: when a calculation impossible state in which either the first vehicle position or the second vehicle position cannot be calculated is detected, the vehicle is stopped; If the calculation impossible state is resolved before a certain period of time continues, the stop of the vehicle is released. Vehicle control system.

7. 3. A vehicle control system according to claim 1, the first image is an image included in the video and used to calculate the first vehicle position; The second image is an image included in the video and used to calculate the second vehicle position, The one or more processors increase the tolerance as the time difference between the first image and the second image increases. Vehicle control system.

8. 8. A vehicle control system according to claim 7, the first image is an image used in calculating the latest first vehicle position, the second image is an image used in calculating the latest second vehicle position, The one or more processors calculate a difference between the most recent first vehicle position and the most recent second vehicle position as the position difference. Vehicle control system.

9. 8. A vehicle control system according to claim 7, The one or more processors: storing the first vehicle position information and the second vehicle position information in a buffer; Calculating the position difference between the first vehicle position and the second vehicle position based on the first vehicle position and the second vehicle position at which the time difference between the first image and the second image is less than a time allowable error. Vehicle control system.

10. 3. A vehicle control system according to claim 1, the common camera is an infrastructure camera outside the vehicle, The image shows the vehicle. Vehicle control system.

11. 11. The vehicle control system of claim 10, the common camera includes a first common infrastructure camera and a second common infrastructure camera installed at a position so as to photograph the vehicle after the first common infrastructure camera, The one or more processors further switch the common camera from the first common infrastructure camera to the second common infrastructure camera so that the common camera used to calculate the first vehicle position and the second vehicle position match. Vehicle control system.

12. 12. A vehicle control system according to claim 11, The one or more processors further include: When a first condition is satisfied, the common camera used for calculating the first vehicle position is permitted to be switched from the first common infrastructure camera to the second common infrastructure camera; when a second condition is satisfied, allowing the common camera used for calculating the second vehicle position to be switched from the first common infrastructure camera to the second common infrastructure camera; using the first common infrastructure camera to calculate the first vehicle position and to calculate the second vehicle position until both the first condition and the second condition are satisfied; After both the first condition and the second condition are satisfied, the common camera used for calculating the first vehicle position and the second vehicle position is switched from the first common infrastructure camera to the second common infrastructure camera. Vehicle control system.

13. A vehicle control method for controlling a vehicle, comprising: Acquiring an image captured by a common camera; Calculating a first vehicle position, which is a position of the vehicle, from the image based on a first position estimation algorithm; calculating a second vehicle position, which is a position of the vehicle, from the image based on a second position estimation algorithm; slowing down or stopping the vehicle in response to a deviation state in which a position difference between the first vehicle position and the second vehicle position exceeds an allowable error; Includes A vehicle control method.

Images