Vehicle control device, vehicle control method, vehicle control system, and vehicle

Abstract

This vehicle control device is provided with: a recognition processing unit that obtains a recognition result based on a plurality of image frames obtained by imaging, by a camera of another vehicle, a field of view that includes the own vehicle; and a vehicle control unit that, on the basis of the recognition result, performs control for autonomously parking the own vehicle. This vehicle control method comprises: obtaining a recognition result based on a plurality of image frames obtained by imaging, by a camera of another vehicle, a field of view that includes the own vehicle; and, on the basis of the recognition result, performing control for autonomously parking the own vehicle.

Description

Vehicle control device, vehicle control method, vehicle control system, and vehicle

[0001] The present disclosure relates to an autonomously driven vehicle, a vehicle control device, a vehicle control method, and a vehicle control system.

[0002] Automated valet parking, or automated car transport services, have been proposed, which allow cars to automatically navigate the parking lot without a driver and park in a parking space.

[0003] Japanese Patent Application Laid-Open No. 2021-84626

[0004] Patent Document 1 discloses a technology in which a vehicle moves without a driver and autonomously parks in an available parking space by utilizing vehicle-to-infrastructure communication and vehicle-to-vehicle communication (paragraph 0027, Abstract).

[0005] In automated parking without driver intervention, the brakes cannot be applied even in situations where a malfunction occurs during parking, and high accuracy and safety are required. To achieve this, for example, it is possible to install many sensors and cameras on the vehicle and infrastructure (parking lot). However, if such conditions are required, it may take some time to realize a service that performs automated parking without driver intervention.

[0006] In view of the above circumstances, an object of the present disclosure is to realize highly accurate and safe automatic parking using existing equipment.

[0007] A vehicle control device according to one embodiment of the present disclosure includes a recognition processing unit that obtains recognition results based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle, and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

[0008] According to this embodiment, the situation around the vehicle, which cannot be monitored accurately by the vehicle's own camera, can be supplemented by using images from the cameras of other vehicles parked nearby and the recognition results obtained from those images.

[0009] The vehicle control unit may determine the vehicle speed during parking based on the amount of time lag between the time when each image frame is captured and the time when a recognition result based on each image frame is obtained.

[0010] The larger the time lag amount, the slower the vehicle speed can be made to stop in a shorter braking distance, making it possible to perform control for safe parking.

[0011] The vehicle control device may further include a time reference control unit that executes a time reference setting operation at the start of automatic parking, and the image capturing time and the recognition time may be times based on a time reference setting operation time.

[0012] By using the time reference setting operation point as the reference point for the subject vehicle and other vehicles, synchronization can be achieved between the vehicles.

[0013] The time reference control unit may blink a light of the host vehicle as the time reference setting operation.

[0014] Since the time difference between light emission and light reception is small enough to be ignored, synchronization can be achieved between each vehicle by setting the time reference setting operation point at which the light is flashed once as the reference time for the vehicle itself and other vehicles.

[0015] The vehicle control unit may determine the vehicle speed during parking based on a maximum value of the time lag amounts of a plurality of image frames within a recent predetermined period.

[0016] Even when there is only one other vehicle, the amount of time lag in transmission, reception, and processing varies due to various factors, but by adopting the maximum amount of time lag, the possibility of avoiding contact with obstacles increases, improving safety and accuracy.

[0017] The vehicle control unit may determine the vehicle speed during parking based on the maximum value of time lag amounts of image frames of a plurality of other vehicles.

[0018] The amount of time lag in transmission, reception, and processing varies due to various factors, but by adopting the maximum amount of time lag, the possibility of avoiding contact with obstacles increases, improving safety and accuracy.

[0019] The vehicle control unit may control the vehicle speed to be low from the time reference setting operation point in time to the recognition point in time, and to be at the determined vehicle speed after the recognition point in time.

[0020] At the start of parking, the vehicle has not yet received image frames and is therefore unable to fully recognize the surroundings, so it automatically drives at a sufficiently low speed for safety. The vehicle control unit determines the amount of time lag after the recognition point, and transitions to the determined vehicle speed to smoothly proceed with parking operations.

[0021] The time reference control unit may flash the lights of the vehicle multiple times at intervals specific to the vehicle before the time reference setting operation, and the recognition processing unit may receive the multiple image frames obtained by the other vehicle that has recognized the multiple flashes.

[0022] With the function of notifying and recognizing the vehicle ID in this way, this embodiment can be used with any host vehicle and other vehicles, regardless of the specific performance of the host vehicle and other vehicles.

[0023] The recognition processing unit may receive the plurality of image frames acquired by a plurality of other vehicles. By performing automatic parking based on the image frames of the plurality of other vehicles, safety and accuracy are increased.

[0024] The recognition processing unit may perform a recognition process of a surrounding environment of the host vehicle based on the plurality of image frames acquired by the plurality of other vehicles, and obtain the recognition result. The recognition result of the recognition processing unit may include a recognition result of the surrounding environment of the host vehicle and may be used for parking control of the host vehicle.

[0025] The other vehicle recognition processing unit may determine that the image frame including the region of interest with the highest brightness is an image frame captured while the light is blinking.

[0026] Since the time difference between light emission and light reception is negligibly small, this method allows other vehicles to accurately recognize the time when the vehicle starts flashing.

[0027] The camera may be a camera for a surround view monitor.

[0028] This means that the vehicle-mounted camera does not need to have special precision or resolution, and a resolution of around several megapixels, which is currently commonly used, is sufficient.

[0029] The host vehicle may be automatically parked in one parking space in an automated valet parking lot, and the other vehicle may be parked in another parking space in the automated valet parking lot from which the host vehicle can be imaged.

[0030] In this embodiment, safe automated valet parking can be achieved with simple equipment, without the need to significantly add equipment to either the automated valet parking lot or the vehicle.

[0031] The host vehicle and the other vehicle may transmit and receive the plurality of image frames by vehicle-to-vehicle communication or communication via a parking lot system.

[0032] This embodiment is not dependent on automated valet parking facilities and can also be applied to use cases other than parking lots.

[0033] The vehicle may be capable of acting as both the host vehicle and the other vehicle, thereby making it possible to apply this embodiment regardless of the vehicle's specific performance or environment.

[0034] A vehicle control method according to one embodiment of the present disclosure obtains a recognition result based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle, and performs control to automatically park the host vehicle based on the recognition result.

[0035] A vehicle control system according to one embodiment of the present disclosure includes a recognition processing unit that obtains recognition results based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle, and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

[0036] A vehicle according to one embodiment of the present disclosure includes a recognition processing unit that obtains recognition results based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle, and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

[0037] 1 shows an example of the arrangement of cameras on a vehicle. 2 shows an example of the camera angle. 3 shows the positional relationship between the host vehicle and other vehicles. 4 shows the functional configuration of a vehicle control system. 5 shows the operation sequence for automatic parking. 6 shows the operation flow of the host vehicle. 7 shows a schematic representation of light flashing, imaging, and time lag. 8 shows a schematic representation of a method by which other vehicles recognize light flashing. 9 shows a schematic representation of the transmission and reception of image frames. 10 shows a schematic representation of the relationship between time lag and vehicle speed. 11 shows a schematic representation of a method of vehicle speed control. 12 shows a schematic representation of a parking space allocation by a parking lot system. 13 shows a schematic representation of a parking space allocation by a parking lot system. 14 shows a schematic representation of a parking space allocation by a parking lot system. 15 shows a schematic representation of a parking space allocation by a parking lot system. 16 shows an example of a parking space layout. 17 shows the operation flow of a method for determining parking spaces and other vehicles by a parking lot system. 18 shows a schematic representation of a judgment step. 19 is a block diagram showing an example configuration of a vehicle control system. FIG. 2 is a diagram illustrating an example of a sensing region.

[0038] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0039] 1. Overview of this embodiment

[0040] In automated parking without driver intervention, the brakes cannot be applied even in situations where a malfunction occurs during parking, and high accuracy and safe parking are required. To achieve this, for example, vehicles equipped with many sensors and cameras, camera and sensor equipment installed in the infrastructure (parking lot), and special parking spaces with sufficient distance from adjacent vehicles may be required. There is a trade-off between safety, accuracy, and cost. Given these requirements, it may take time to realize a service that performs automated parking without driver intervention. In light of the above, we aim to achieve high accuracy and safe automated parking using existing equipment.

[0041] FIG. 1 shows an example of a camera arrangement on a vehicle.

[0042] Vehicles equipped with cameras for surround view monitoring are becoming commonplace. Each vehicle 1 is provided with about four cameras for surround view monitoring: a front camera 201, a rear camera 202, and left and right side cameras 203, 203.

[0043] FIG. 2 shows examples of camera angles.

[0044] The surround view monitor is a technology that processes and displays images captured by these cameras 201-203 into images that appear as if they are being viewed from above the vehicle. For this reason, as shown in (A), the front camera 201 and the rear camera 202 are installed facing downward (at a depression angle) from the horizontal in order to capture an image of the body 205 of the vehicle 1 that is located below the cameras. As shown in (B) and (C), the side cameras 203, 203 are also installed on the side mirrors 206 facing downward (at a depression angle) from the vertical or horizontal in order to capture an image of the body 205 of the vehicle 1 that is located below the cameras. A sonar sensor (not shown) is also installed on the vehicle 1, and this sonar sensor is also used to detect obstacles that are relatively below.

[0045] With the installation positions (height) and depression angles of the cameras 201 to 203, it is difficult to accurately detect the distance between an obstacle located higher than the cameras 201 to 203 and the body 205 of the vehicle 1. For this reason, it is difficult to accurately monitor the situation around the vehicle 1 using only the cameras 201 to 203.

[0046] FIG. 3 shows the positional relationship between the subject vehicle and other vehicles.

[0047] For example, as shown in (A), adjacent vehicle 1F is parked in a parking space next to host vehicle 1A, which is automatically traveling to back into a parking space. Because side camera 203A of host vehicle 1A is installed facing downward on side mirror 206A, the distance to side mirror 206F of adjacent vehicle 1F, which is located higher than side camera 203A, cannot be accurately detected, and there is a risk that side mirror 206A will come into contact with side mirror 206F.

[0048] In another example, as shown in (B), an adjacent vehicle 1G is parked in a parking space behind a host vehicle 1A that is automatically traveling to back into a parking space. Because the rear camera 202A of the host vehicle 1A is installed facing downward at the rear of the vehicle 1A, the distance to the spare tire 205G of the adjacent vehicle 1G, which is located higher than the rear camera 202A, cannot be accurately detected, and there is a risk that the rear camera 202A may come into contact with the spare tire 205G.

[0049] As described above, it is difficult to accurately monitor the situation around the vehicle 1A using only the four or so cameras 201 to 203 for a surround view monitor.

[0050] Therefore, in this embodiment, the situation around the vehicle 1, which cannot be monitored accurately by the cameras 201 to 203 of the vehicle 1, is supplemented by using images from the cameras 201 to 203 of other vehicles 1 parked nearby and the recognition results obtained from those images. By sharing the images from the cameras 201 to 203 of the other vehicles 1 and the recognition results, the vehicle 1 is able to perform automatic parking without any problems even with around four cameras 201 to 203 for a surround view monitor.

[0051] This eliminates the need for a system that uses a high-precision camera to control the vehicle's braking. Furthermore, the vehicle-mounted camera does not need to have special precision or resolution; a resolution of around a few megapixels, which is currently commonly used, is sufficient. In this way, this embodiment achieves safe automated valet parking with simple equipment, without requiring significant additional equipment on either the automated valet parking lot or the vehicle.

[0052] 2. Functional configuration of the vehicle control system

[0053] FIG. 4 shows the functional configuration of the vehicle control system.

[0054] The vehicle control device 200 constitutes the vehicle 1. The vehicle control device 200 includes a vehicle control unit 210, a time reference control unit 220, a recognition processing unit 230, a camera unit 240, and a wireless communication unit 250.

[0055] The vehicle control unit 210 performs analysis processing of the vehicle 1 and the surrounding situation, creates an action plan for the vehicle 1, and controls the operation of the vehicle 1 to realize the action plan.

[0056] The time reference control unit 220 blinks various lights of the vehicle 1. The lights to be controlled include, for example, headlights and backlights.

[0057] The recognition processing unit 230 has a video signal interface unit 231, an image processing unit 232, a non-volatile memory unit 233, a volatile memory unit 234, a control unit 235, and a communication interface unit 236. The image processing unit 232 has a brightness calculation unit 237 and an image recognition unit 238.

[0058] The video signal interface unit 231 transmits a control signal to the camera unit 240 and receives a video signal from the camera unit 240. The brightness calculation unit 237 calculates the brightness of the video signal. The image recognition unit 238 recognizes the image of the video signal. At this time, the image processing unit 232 reads firmware and various setting information from the non-volatile memory unit 233, executes them, and writes the various information to the volatile memory unit 234. The control unit 235 outputs a control signal to the image processing unit 232 and the video signal interface unit 231 and receives the image processing result from the image processing unit 232. The control unit 235 generates a camera signal processing result based on the image processing result and transmits it to other units of the vehicle control device 200 via the communication interface unit 236. The control unit 235 receives vehicle information and information from other units of the vehicle control device 200 via the communication interface unit 236. At this time, the control unit 235 reads firmware and various setting information from the non-volatile memory unit 233 and executes them. The communication interface unit 236 communicates with other units of the vehicle control device 200.

[0059] Camera unit 240 has a video signal interface unit 241, an imaging unit 242, and a lens unit 243. Video signal interface unit 241 receives a control signal from recognition processing unit 230 and transmits a video signal to recognition processing unit 230. Imaging unit 242 captures an image using lens unit 243 in accordance with the control signal and outputs the video signal to video signal interface unit 241.

[0060] The wireless communication unit 250 has a communication interface section 251 and a wireless communication section 252. The wireless communication section 252 performs wireless communication with other vehicles 1 and the parking lot system 2. The wireless communication section 252 wirelessly transmits vehicle information obtained via the communication interface section 251 and information of other units of the vehicle control device 200 to the outside. The wireless communication section 252 supplies various types of information wirelessly received from the outside to other units of the vehicle control device 200 via the communication interface section 251.

[0061] The vehicle 1 configured with the vehicle control device 200 can behave as both a host vehicle 1A and other vehicles 1B and 1C, which will be described later, depending on the position. This makes it possible to apply this embodiment regardless of the performance and environment specific to the vehicle.

[0062] 3. Automatic parking operation sequence

[0063] Fig. 5 shows the operation sequence of automatic parking, and Fig. 6 shows the operation flow of the host vehicle.

[0064] When the host vehicle 1A, which is an autonomous vehicle, enters the automated valet parking lot, it drives to a boarding and disembarking space by manual driving or by automatic driving controlled by the vehicle control unit 210, and stops in the boarding and disembarking space. The occupant disembarks from the host vehicle 1A parked in the boarding and disembarking space. The boarding and disembarking space is not a final parking space, but a space where the occupant disembarks when entering the parking lot and where the occupant gets on when leaving the parking lot.

[0065] After the occupant disembarks, the vehicle control unit 210 of the vehicle 1A transmits a parking entry request to the parking lot system 2 (step S11). The parking entry request is a request for the vehicle 1A to automatically drive from the boarding / exiting space to the parking space and park in the parking space. Note that the automatic driving and automatic parking behavior of each vehicle 1 is not controlled by the parking lot system 2, but is performed autonomously by each vehicle 1.

[0066] The parking lot system 2 is an infrastructure-side management system that communicates wirelessly with the vehicle to be parked, outputs engine on / off commands and commands for entering and leaving the parking lot, provides map information within the parking lot, etc. When the parking lot system 2 receives a parking request from the vehicle 1A, it determines a parking space in which the vehicle 1A should automatically park and one or more (two in this example) other vehicles 1B and 1C that will assist the vehicle 1A in parking (the determination method will be described later).

[0067] Here, the other vehicles 1B and 1C are parked in other parking spaces that can capture images of the host vehicle 1A. That is, the other vehicles 1B and 1C are vehicles already parked in other parking spaces that can capture images of the field of view including the host vehicle 1A parked in the determined parking space using any of the on-board cameras 201-203 (FIG. 2). For example, the other vehicles 1B and 1C are vehicles already parked in parking spaces directly in front of, diagonally in front of, adjacent to, or directly behind the parking space where the host vehicle 1A is to park. Parking assistance means that the cameras 201-203 of the other vehicles 1B and 1C that are parked or stopped capture images of the field of view including the host vehicle 1A and provide the captured images to the host vehicle 1A, thereby enabling the host vehicle 1A to perform autonomous parking control.

[0068] The parking lot system 2 transmits a parking assistance instruction to the determined other vehicles 1B, 1C, and outputs an engine-on command and an in-vehicle camera system activation command (step S21). The parking assistance instruction includes information identifying the subject vehicle 1A that is the target of parking assistance. The parking assistance instruction may include information identifying one or more cameras 201-203 that the other vehicles 1B, 1C will use to capture images from. The information identifying the subject vehicle 1A includes at least the blinking interval of the lights of the subject vehicle 1A, and may also include the number of blinks.

[0069] When the other vehicles 1B and 1C receive the parking assistance instruction, they turn on their engines and start their in-vehicle camera systems, and when preparation is complete, they transmit a preparation completion report to the parking lot system 2 (step S31). The parking lot system 2 transmits a parking position instruction indicating the determined parking space to the vehicle 1A (step S22).

[0070] When the vehicle 1A receives the parking position instruction from the parking lot system 2, the vehicle control unit 210 controls the vehicle 1A to depart from the boarding / alighting space, travel through the parking lot passage by automatic driving, and move to a position where the vehicle can back into the designated parking space (step S12). After the movement is completed, the vehicle control unit 210 controls the vehicle 1A to travel at a sufficiently slow vehicle speed V ０ Then, automatic driving back parking is started (step S13).

[0071] FIG. 7 shows a schematic diagram of light blinking, imaging, and time lag.

[0072] When the time reference control unit 220 of the host vehicle 1A starts backing into a parking space, it first flashes the lights of the host vehicle 1A multiple times at an interval specific to the host vehicle 1A. The flashing interval and the number of flashes specific to the host vehicle 1A function as a vehicle ID that uniquely identifies the host vehicle 1A. The host vehicle 1A may have a flashing interval and a number of flashes specific to the host vehicle 1A in advance before entering the parking lot. Alternatively, the parking lot system 2 may determine the flashing interval and the number of flashes when entering the parking lot and transmit information indicating this to the host vehicle 1A. In this way, the function of notifying and recognizing the vehicle ID allows this embodiment to be operated with any host vehicle 1A and other vehicles 1B, 1C, regardless of the specific performance, etc., of the host vehicle 1A and other vehicles 1B, 1C.

[0073] After the vehicle ID light flashes, the time reference control unit 220 of the vehicle 1A flashes the light of the vehicle 1A once as a time reference setting operation. The recognition processing unit 230 of the vehicle 1A starts counting time T0 (step S14), using the time reference setting operation point (T0=0) at which the light is flashed once as a reference time. Time T0 is determined by a counter of the vehicle 1A.

[0074] The other vehicles 1B and 1C recognize the flashing of the lights at intervals and frequency specific to the vehicle 1A, and thereby recognize that the vehicle 1A is the vehicle 1A designated by the parking assistance instruction. Next, the other vehicles 1B and 1C recognize the flashing of the lights once as the time reference setting operation.

[0075] FIG. 8 shows a schematic diagram of how other vehicles recognize the flashing of their lights.

[0076] The subject vehicle 1A flashes its lights twice at the interval shown in (A). The other vehicle 1B is parked in a parking space directly in front of the parking space where the subject vehicle 1A is to park. The other vehicle 1B captures a field of view including the subject vehicle 1A to obtain multiple image frames. As shown in (B), the recognition processing unit 230 of the other vehicle 1B sets an area within the multiple image frames in which the subject vehicle 1A's lights may be captured as a region of interest (ROI). As shown in (C), the recognition processing unit 230 of the other vehicle 1B integrates the brightness values ​​of each pixel within the ROI for each image frame to calculate an integrated brightness value for each image frame. As shown in (D), the recognition processing unit 230 of the other vehicle 1B interpolates (resamples) the integrated brightness value. As shown in (E), the recognition processing unit 230 of the other vehicle 1B A / D converts the integrated value. The recognition processing unit 230 of the other vehicle 1B determines that the image frame containing the region of interest with the highest brightness is the image frame captured when the headlights were flashing. Because the time difference between the light emission and light reception is negligibly small, this method allows the other vehicles 1B and 1C to accurately recognize the time when the headlights of the own vehicle 1A were flashing.

[0077] The other vehicles 1B and 1C determine the time when the lights flash once as a time reference setting operation. The other vehicles 1B and 1C start counting times T1 and T2, respectively, using the time when the lights flash (T1=0, T2=0) as a reference time (step S32). The times T1 and T2 are determined by the counters of the other vehicles 1B and 1C. The times T0=0, T1=0, and T2=0 are approximately equal. Because the time difference between light emission and light reception is negligibly small, synchronization can be achieved between the vehicle 1A and the other vehicles 1B and 1C by setting the time when the lights flash once as a time reference setting operation as a reference time for the vehicle 1A and the other vehicles 1B and 1C.

[0078] The other vehicles 1B and 1C start capturing images of the field of view including the vehicle 1A using the cameras 201 to 203 instructed by the parking assistance instruction. The imaging time when the other vehicles 1B and 1C capture each image frame is defined as T1=t ｂ , T2=t ｃ (Step S33). ｂ , T2=t ｃare points in time based on the points in time T1=0 and T2=0 when the other vehicles 1B and 1C recognize a single flashing of the light as the time reference setting operation.

[0079] FIG. 9 shows a schematic diagram of the transmission and reception of image frames.

[0080] As shown in (A), the other vehicle 1B captures an image at time T1=t ｂ A timestamp t indicating ｂ The other vehicle 1C transmits the image frame to the parking lot system 2 at the image capture time T2=t ｃ A timestamp t indicating ｃ and transmits the image frame to the parking lot system 2 (step S34).

[0081] The parking lot system 2 detects the timestamp t ｂ and the image frame of the other vehicle 1B with the time stamp t ｃ The image frame of the other vehicle 1C with the attached image frame is transferred to the own vehicle 1A (step S23).

[0082] The recognition processing unit 230 of the vehicle 1A receives the time stamp t ｂ and the image frame of the other vehicle 1B with the time stamp t ｃThe host vehicle 1A receives an image frame of the other vehicle 1C marked with "*" from the parking lot system 2. The recognition processing unit 230 of the host vehicle 1A processes each received image frame to generate a recognition result (step S15). The recognition processing unit 230 executes a detection process to detect the external situation of the vehicle 1 and a recognition process to recognize the external situation of the vehicle 1. The recognition processing unit 230 also executes a recognition process of the surrounding environment of the vehicle 1. Specifically, the recognition processing unit 230 executes a detection process and a recognition process of objects around the vehicle 1. The object detection process is, for example, a process to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition process is, for example, a process to recognize attributes such as the type of object or to identify a specific object. However, the detection process and the recognition process are not necessarily clearly separated and may overlap. The recognition result based on the image frame obtained by the other vehicle 1C capturing an image of the field of view including the host vehicle 1A includes the detection result of the external situation of the vehicle 1 and the recognition result of the surrounding environment of the vehicle 1, and is used for parking control of the host vehicle 1A. The time when the vehicle 1A processes each image frame is defined as the recognition time T0=t ａ The recognition time T0 = t ａ is a time point relative to the time reference setting operation time T0=0 when the vehicle 1A blinks its lights once. This imaging and time frame transmission / reception is repeated.

[0083] As a modified example, as shown in (B), one or both of the other vehicles 1B and 1C may transmit image frames to the vehicle 1A by vehicle-to-vehicle communication without using the parking lot system 2. In this case, the recognition time of each image frame of the vehicle 1A is also set to T0 = t ａ Let's say.

[0084] As shown in FIG. 7, the image capture time T1 of the other vehicles 1B and 1C is t ｂ , T2=t ｃ From this, the recognition time T0 of the image frame of the vehicle 1A is determined as t ａA time lag (delay) occurs until the image frames are captured. A time lag occurs both in the case of communication via the parking lot system 2 (A) and in the case of vehicle-to-vehicle communication (B). The amount of time lag is not constant for all image frames transmitted by the other vehicles 1B, 1C, but varies due to communication delays, recognition processing delays, etc. If the time at which the image frames were captured is unknown, the host vehicle 1A cannot use the image frames captured by the other vehicles 1B, 1C as parking assistance information for the host vehicle 1A.

[0085] Therefore, every time the recognition processing unit 230 of the vehicle 1A receives an image frame and performs recognition processing, the vehicle control unit 210 of the vehicle 1A determines the image capturing time T1=t ｂ , T2=t ｃ From the recognition time T0 = t ａ The time lag t ａ -t ｂ , t ａ -t ｃ The vehicle control unit 210 calculates the image frame at the recognition time t ａ Instead of the image of time lag t ａ -t ｂ , t ａ -t ｃ Based on the recognition result, the vehicle speed during back parking (currently the vehicle speed V ０ ) to perform control for automatic parking (step S16).

[0086] That is, the vehicle control unit 210 of the host vehicle 1A calculates the time lag amount t ａ -t ｂ , t ａ -t ｃ Specifically, the vehicle control unit 210 drives the vehicle at a speed that allows the vehicle to stop before hitting an obstacle, even if braking is initiated after receiving a signal requesting a stop for safety reasons.

[0087] FIG. 10 shows a schematic diagram of the relationship between the time lag and the vehicle speed.

[0088] The other vehicle 1B is photographed at the time t ｂThe recognition processing unit 230 of the vehicle 1A, which is traveling in reverse, receives the image frame and recognizes the image at the recognition time t ａ The recognition result is obtained at the image capture time t ｂ From the recognition time t ａ The time lag amount is t ａ -t ｂ Based on the recognition result, the vehicle control unit 210 of the host vehicle 1A detects an obstacle (the side mirror of an adjacent vehicle parked in an adjacent parking space) within a predetermined distance (several meters). Then, the host vehicle 1A starts to apply the brakes, and the brakes start to work after a predetermined time. The host vehicle 1A stops before coming into contact with the side mirror of the adjacent vehicle.

[0089] Time lag amount t ａ -t ｂ The larger the distance traveled (at the time of imaging t ｂ If the free running distance is long, the braking distance (the distance traveled from the time of image capture t ｂ The time lag t ａ -t ｂ The larger the time lag t, the shorter the braking distance. The free running distance is proportional to the vehicle speed, and the braking distance is proportional to the square of the vehicle speed. Therefore, the time lag t ａ -t ｂ The larger the value, the slower the vehicle speed can be made to stop in a short braking distance, and control for safe parking can be performed.

[0090] FIG. 11 shows a schematic diagram of a vehicle speed control method.

[0091] The vehicle speed control (step S16) by the vehicle control unit 210 of the vehicle 1A will be described in more detail. As shown in (A), every time the recognition processing unit 230 receives the latest image frame and obtains a recognition result, the vehicle control unit 210 ａ The time lag amount t of the past predetermined period Tm (ms) from ａ -t ｂ , t ａ -t ｃ The maximum value of d ｂ , d ｃWhen there are a plurality of other vehicles 1B and 1C as in this example, the vehicle control unit 210 determines the maximum value d ｂ , d ｃ The larger value d ｂ The vehicle control unit 210 determines the maximum time lag amount d ｂ The time lag amount d (ms) is defined as the amount of time lag in transmission, reception, and processing, not only when there are multiple other vehicles but also when there is only one other vehicle, varies due to various factors. ｂ By adopting this, the possibility of avoiding contact with obstacles etc. increases, improving safety and accuracy.

[0092] As shown in (B), the vehicle control unit 210 of the host vehicle 1A determines the vehicle speed V (km / h) based on the determined time lag d (ms). The specific intercept and slope of the linear function may differ for each vehicle, and may also be changed depending on the environment, such as road surface conditions and the presence or absence of nearby vehicles.

[0093] As shown in (C), the vehicle control unit 210 of the vehicle 1A has not yet received an image frame at the time of starting parking, and therefore is not able to fully recognize the surrounding situation. Therefore, for safety reasons, the vehicle speed V ０ The vehicle control unit 210 of the vehicle 1A automatically travels at the recognition time t ａ When the time lag amount d is determined after (B), the vehicle speed is transitioned to the vehicle speed V determined in (B) at an acceleration equal to or less than a certain value, and the parking operation proceeds smoothly. Thereafter, the vehicle control unit 210 repeats the operation of updating the vehicle speed V each time the time lag amount d is updated until parking is completed (step S17, No).

[0094] When the vehicle control unit 210 of the vehicle 1A completes parking (step S17, Yes), it transmits a parking completion report to the parking lot system 2 (step S18). When the parking lot system 2 receives the parking completion report from the vehicle 1A, it transmits a parking assistance stop instruction to the other vehicles 1B, 1C and outputs an engine off command (step S22). When the other vehicles 1B, 1C receive the parking assistance stop instruction, they shut down their in-vehicle camera systems and turn off their engines.

[0095] According to this embodiment, the situation around the vehicle 1A, which cannot be monitored accurately by the cameras 201-203 of the vehicle 1A, can be supplemented by using images from the cameras 201-203 of the other vehicles 1B and 1C parked nearby and the recognition results obtained from those images. By sharing the images from the cameras 201-203 of the other vehicles 1B and 1C and the recognition results among multiple vehicles 1, automatic parking can be performed without any problems even with about four cameras 201-203 for surround view monitoring.

[0096] This eliminates the need for a system that uses a high-precision camera to control the vehicle's braking. Furthermore, the vehicle-mounted camera does not need to have special precision or resolution; a resolution of around a few megapixels, which is currently commonly used, is sufficient. In this way, this embodiment makes it possible to achieve safe automated valet parking with simple equipment, without requiring significant expansion of equipment to either the automated valet parking lot or the vehicle.

[0097] 4. Modifications

[0098] In this embodiment, the host vehicle 1A receives image frames of the other vehicles 1B and 1C, performs recognition processing, and recognizes the image frames at a recognition time t ａ Alternatively, the other vehicles 1B and 1C may generate and transmit the recognition results to the host vehicle 1A, and the host vehicle 1A may receive the recognition results instead of the image frames. In this case, the host vehicle 1A also receives the recognition results at the recognition time t ａ Therefore, the vehicle 1A has a time lag amount t ａ -t ｂ , t ａ -t ｃ can be calculated.

[0099] In this embodiment, the vehicle 1A flashes its own lights multiple times at intervals specific to the vehicle 1A as a vehicle ID that uniquely identifies the vehicle 1A. This is to prevent interference between vehicles when multiple vehicles are parked simultaneously in a parking lot. If multiple vehicles are not parked simultaneously in the parking lot (if only one vehicle is parked), flashing the lights multiple times as a vehicle ID is not necessary, and the vehicle 1A may simply flash its own lights once as the time reference setting operation.

[0100] In this embodiment, the host vehicle 1A is parked in a parking space in an automated valet parking lot. However, this embodiment can also be implemented in locations other than automated valet parking lots, and can be applied not only to parking but also to stopping. Furthermore, this embodiment can be applied not only to automated vehicles, but also to manual parking of manually driven vehicles by the driver.

[0101] The host vehicle 1A may perform parking by automatic driving based on image frames captured by the host vehicle 1A in addition to image frames captured by the other vehicles 1B and 1C.

[0102] 5. How the parking lot system determines parking spaces and other vehicles

[0103] 12 to 17 show schematic diagrams of the allocation of parking spaces by the parking lot system.

[0104] As shown in FIG. 12, the vehicle 1 parks in a parking space designated by the parking lot system 2 using its own automatic parking function.

[0105] As shown in FIG. 13, the parking lot system 2 assigns parking slots in an order that prevents vehicles from coming adjacent to, in front of, or behind the parking slots while the parking lot is vacant, and instructs each vehicle to park.

[0106] As shown in Figure 14, if there are no available parking spaces adjacent to the vehicle 1A, the parking lot system 2 assigns a parking space according to a rule (for example, in ascending order of parking space number). At this time, there is a possibility that the vehicle 1A may come into contact with vehicles on the left, right, and rear. The parking lot system 2 determines the vehicle 1B directly in front of the vehicle 1A as the vehicle to assist the vehicle 1A in parking.

[0107] 15, when there is no other vehicle directly in front of the parking space of the vehicle 1A, the parking lot system 2 determines two other vehicles 1B and 1C diagonally in front as vehicles to assist the parking of the vehicle 1A. By performing automatic parking based on image frames of multiple other vehicles 1B and 1C, safety and accuracy are increased.

[0108] 16, when two adjacent parking spaces 20 and 21 are available, the parking lot system 2 preferentially determines the parking space 21 in which the other vehicle 1B is parked directly in front as the parking space for the vehicle 1A. The parking lot system 2 determines the other vehicle 1B directly in front as the vehicle that will assist the parking of the vehicle 1A.

[0109] 17, in addition to using an image of another vehicle 1B directly in front of the vehicle 1A, images taken by side cameras of other vehicles 1C and 1D on either side of the vehicle 1A, or an image taken by a rear camera of another vehicle 1E directly behind the vehicle 1A may also be used. Alternatively, depending on the parking space setting situation in the parking lot (for example, a parking lot with parallel parking), images of other vehicles in other directions may also be used (not shown).

[0110] Fig. 18 shows an example of a parking space layout, Fig. 19 shows an operation flow of a method for determining a parking space and other vehicles by a parking lot system, and Fig. 20 shows a schematic diagram of a determination step.

[0111] The parking lot system 2 sets parking space 1 (Zone=1, Column=1, Row=1 in FIG. 18) as a candidate parking space for the vehicle 1A (step S41). At this time, there are no candidates for other vehicles to assist the vehicle 1A in parking, so the designation is cleared (step S42). The parking lot system 2 determines whether there is a parked vehicle directly in front of the candidate parking space (FIG. 20(A), step S43).

[0112] If there is a parked vehicle directly in front of the candidate parking space (step S43, Yes), the parking lot system 2 determines whether there are parked vehicles on either side of the candidate parking space (FIG. 20(B), step S44). If there is a parked vehicle next to the candidate parking space (step S44, Yes), the parking lot system 2 designates the parked vehicle directly in front of the candidate parking space as the first other vehicle that will provide parking assistance to the host vehicle 1A (step S45). The parking lot system 2 activates the first other vehicle directly in front and sends a parking assistance instruction (step S48), and notifies the host vehicle 1A of the parking space (step S49).

[0113] On the other hand, if there is no parked vehicle directly in front of the candidate parking space (step S43, No), the parking lot system 2 determines whether there is a parked vehicle adjacent to the rear side of the candidate parking space (FIG. 20(C), step S46). If there is no parked vehicle adjacent to the rear side of the candidate parking space (step S46, No), the parking lot system 2 determines whether there is a parked vehicle adjacent to the front side of the candidate parking space (FIG. 20(D), step S47). If there is no parked vehicle adjacent to the front side of the candidate parking space (step S47, No), since there are no parked vehicles on either side of the candidate parking space, it can be said that it is safe, although the image of the parked vehicle directly in front cannot be used. Therefore, the parking lot system 2 determines that parking is possible without the assistance of other vehicles, does not send a parking assistance instruction to any other vehicles (no designation in step S48), and notifies the vehicle 1A of the parking space (step S49).

[0114] On the other hand, if there is a parked vehicle adjacent to the rear side of the candidate parking space (step S46, Yes), the parking lot system 2 determines whether there is a parked vehicle diagonally in front of the candidate parking space on the rear side in order to recognize interference with the adjacent parked vehicle on the rear side (FIG. 20(E), step S50). If there is a parked vehicle diagonally in front of the candidate parking space on the rear side (step S50, Yes), the parking lot system 2 designates the parked vehicle diagonally in front of the candidate parking space on the rear side as the first other vehicle to assist the host vehicle 1A in parking (step S51).

[0115] Next, the parking lot system 2 determines whether there is a parked vehicle adjacent to the near side of the candidate parking space (FIG. 20(D), step S47). If there is no parked vehicle adjacent to the near side of the candidate parking space (step S47, No), the image of the parked vehicle diagonally in front of the far side of the candidate parking space can be used to check for interference with the parked vehicle adjacent to the far side of the candidate parking space, so it can be said to be safe even though the image of the parked vehicle in front cannot be used. Therefore, the parking lot system 2 activates the first other vehicle diagonally in front of the far side and sends a parking assistance instruction (step S48), and notifies the vehicle 1A of the parking space (step S49).

[0116] On the other hand, if there is a parked vehicle adjacent to the near side of the candidate parking space (step S47, Yes), the parking lot system 2 determines whether there is a parked vehicle diagonally in front of the candidate parking space in order to recognize interference with the parked vehicle adjacent to the near side (FIG. 20(F), step S52). If there is a parked vehicle diagonally in front of the candidate parking space (step S52, Yes), the parking lot system 2 designates the parked vehicle diagonally in front of the near side as a second other vehicle that will provide parking assistance to the host vehicle 1A (step S53). The parking lot system 2 activates the first other vehicle diagonally in front of the far side and the second other vehicle diagonally in front of the near side to send a parking assistance instruction (step S48), and notifies the host vehicle 1A of the parking space (step S49).

[0117] On the other hand, if there is a parked vehicle adjacent to the rear side of the candidate parking space (Yes in step S46), but there is no parked vehicle diagonally in front of the candidate parking space (No in step S50), it is not possible to recognize interference with the adjacent parked vehicle on the rear side. Similarly, if there is a parked vehicle adjacent to the front side of the candidate parking space (Yes in step S47), but there is no parked vehicle diagonally in front of the candidate parking space (No in step S52), it is not possible to recognize interference with the adjacent parked vehicle on the front side. In these cases, safety and accuracy cannot be guaranteed when parking the host vehicle 1A in the candidate parking space, so the parking lot system 2 changes the candidate parking space.

[0118] If the current row number is less than the maximum value (RowMax = 8 in the example of FIG. 18 ) (Step S54, Yes), the parking lot system 2 increments the row number and updates the candidate parking space (Step S55). On the other hand, if the current row is the maximum value (Step S54, No), the parking lot system 2 determines the current column number (1 or 2) (Step S56). If the current column = 1 (Step S56, Yes), the parking lot system 2 increments the column to 2 and updates the candidate parking space (Step S57). On the other hand, if the current column = 2 (Step S56, No), the parking lot system 2 determines the current zone number (Step S58). If the current zone number is less than the maximum value (Step S58, Yes), the parking lot system 2 increments the zone number and updates the candidate parking space (Step S59).

[0119] The parking lot system 2 executes again the operation (step S43 and onwards) to determine another vehicle to provide parking assistance for the updated parking candidate slot. On the other hand, if the current zone number is the maximum value (step S58, No), this means that there are no other parking slot candidates, so the parking lot system 2 executes again the operation from determining the parking candidate slot (step S41). In either case, if another vehicle to provide parking assistance has already been designated (step S51), the parking lot system 2 clears the designation (step S42) and then executes the operation (step S43 and onwards) to determine another vehicle to provide parking assistance for the updated parking candidate slot.

[0120] 6. Example of vehicle control system configuration

[0121] FIG. 21 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0122] A vehicle control system 11 (FIG. 21) can be applied to realize a vehicle control device 200 (FIG. 4) constituting the vehicle 1. The vehicle control unit 210 corresponds to the driving automation control unit 29. The time reference control unit 220 corresponds to the light control unit 85. The recognition processing unit 230 corresponds to the recognition unit 73. The camera unit 240 corresponds to the camera 51. The wireless communication unit 250 corresponds to the communication unit 22.

[0123] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes automated driving of levels 1 to 5, and remote driving and remote assistance of the vehicle 1 by a remote driver.

[0124] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0125] The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 41 may be used depending on the type of data being transmitted. For example, CAN may be used for data related to vehicle control, and Ethernet may be used for large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0126] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0127] The vehicle control ECU 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0128] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can communicate using a plurality of communication methods.

[0129] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0130] Furthermore, for example, the communication unit 22 can communicate with a terminal located near the vehicle using P2P (Peer to Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0131] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (over the air). The communication unit 22 can also receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1 and the recognition result by the recognition unit 73. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0132] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0133] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 22. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or Wireless USB (WUSB). The communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High-Definition Link (MHL).

[0134] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a user in the vehicle, such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0135] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0136] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point cloud data). A vector map is a map adapted to automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0137] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters relating to the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0138] The location information acquisition unit 24 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires location information of the vehicle 1. The acquired location information is supplied to the driving automation control unit 29. Note that the method used by the location information acquisition unit 24 is not limited to using GNSS signals, and it may also acquire location information using a beacon, for example.

[0139] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0140] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is not particularly limited as long as the number is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0141] The imaging method of the camera 51 is not particularly limited. For example, cameras of various imaging methods capable of distance measurement, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera, can be applied to the camera 51 as needed. However, the camera 51 may simply acquire an image without distance measurement.

[0142] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0143] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the location of sound sources.

[0144] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of types and numbers that can be realistically installed in the vehicle 1.

[0145] For example, the interior sensor 26 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 26 may be a camera using any of various imaging methods capable of measuring distances, such as a Time of Flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 26 may also be a camera simply for acquiring captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat, a steering wheel, or the like, and detect various types of biometric information of the user.

[0146] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of types and numbers that can be realistically installed on the vehicle 1.

[0147] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that detects external impacts.

[0148] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an electrically erasable programmable read-only memory (EEPROM) and a random access memory (RAM). Examples of storage media that can be used include a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by each component of the vehicle control system 11. For example, the storage unit 28 includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0149] The driving automation control unit 29 controls the driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0150] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0151] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and matches the local map with the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0152] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy status of objects is indicated on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0153] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27 .

[0154] The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include compounding, integration, fusion, and association.

[0155] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1 .

[0156] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0157] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0158] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0159] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0160] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0161] For example, the recognition unit 73 can perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of estimation of the self-position by the self-position estimation unit 71, and the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 can recognize the positions and states of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled, etc.

[0162] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.

[0163] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route tracking processing.

[0164] The route planning includes global path planning and local path planning. Global path planning includes a process of planning a rough route from a start to a goal. Local path planning, also called trajectory planning, includes a process of generating a trajectory that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1 on the planned route, taking into account the motion characteristics of the vehicle 1.

[0165] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 62 can, for example, calculate a target speed and a target angular velocity of the vehicle 1 based on the results of this path following process.

[0166] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .

[0167] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in a vehicle control unit 32 described later, to perform lateral vehicle motion control and longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs control aimed at driver assistance functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, host vehicle collision warning, host vehicle lane departure warning, and driving automation such as driving without operation by the driver or a remote driver.

[0168] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 26 and input data input to the HMI 31 (described later). Examples of the driver's state to be recognized include physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0169] The DMS 30 may be configured to perform authentication processing for users other than the driver and recognition processing for the status of the users. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the status inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of the status inside the vehicle that may be recognized include temperature, humidity, brightness, and odor.

[0170] The HMI 31 inputs various data and instructions and presents various data to the user.

[0171] The following provides an overview of data input via the HMI 31. The HMI 31 includes input devices for a person to input data. The HMI 31 generates input signals based on data, instructions, and the like input via the input devices and supplies the signals to each component of the vehicle control system 11. The HMI 31 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 31 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 31 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 11.

[0172] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the user or the outside of the vehicle. The HMI 31 also performs output control, controlling the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information imparted to the user's sense of touch by, for example, force, vibration, movement, etc.

[0173] Examples of output devices that the HMI 31 uses to output visual information include a display device that displays an image on its own to present visual information and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the user's field of vision, such as a head-up display, a transmissive display, or a wearable device with an augmented reality (AR) function, in addition to a display device with a normal display. The HMI 31 may also use display devices included in a navigation system, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1 as output devices that output visual information.

[0174] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0175] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 31. The haptic element is provided on a part that the user touches, such as a steering wheel or a seat.

[0176] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0177] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0178] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0179] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0180] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0181] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0182] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0183] 22 is a diagram showing an example of a sensing area by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25. Note that in the figure, the vehicle 1 is shown as viewed from above, with the left end side being the front end (front) side of the vehicle 1 and the right end side being the rear end (rear) side of the vehicle 1.

[0184] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F covers the periphery of the front end of the vehicle 1 with multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of the vehicle 1 with multiple ultrasonic sensors 54.

[0185] The sensing results in the sensing area 101F and the sensing area 101B are used, for example, for parking assistance for the vehicle 1.

[0186] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the periphery behind the left side of the vehicle 1. Sensing area 102R covers the periphery behind the right side of the vehicle 1.

[0187] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0188] Sensing areas 103F to 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0189] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking assistance and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.

[0190] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0191] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.

[0192] A sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. The sensing area 105 covers a position further ahead of the vehicle 1 than the sensing area 104. On the other hand, the sensing area 105 has a narrower range in the left-right direction than the sensing area 104.

[0193] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0194] The sensing areas of the cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those shown in the figure. Specifically, the ultrasonic sensors 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the above-described examples. The number of each sensor may be one or more.

[0195] The present disclosure may have the following configurations.

[0196] (1) A vehicle control device comprising: a recognition processing unit that obtains a recognition result based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle; and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition result. (2) The vehicle control device described in (1) above, wherein the vehicle control unit determines a vehicle speed during parking based on a time lag between the image capture time of each image frame and the recognition time when a recognition result based on the each image frame is obtained. (3) The vehicle control device described in (2) above, further comprising a time reference control unit that performs a time reference setting operation at the start of automatic parking, wherein the image capture time and the recognition time are times relative to the time reference setting operation time. (4) The vehicle control device described in (3) above, wherein the time reference control unit flashes lights of the host vehicle as the time reference setting operation. (5) The vehicle control device according to any one of (2) to (4) above, wherein the vehicle control unit determines the vehicle speed during parking based on the maximum value within a most recent predetermined period among the time lag amounts of a plurality of image frames. (6) The vehicle control device according to any one of (2) to (5) above, wherein the vehicle control unit determines the vehicle speed during parking based on the maximum value among the time lag amounts of image frames of a plurality of other vehicles. (7) The vehicle control device according to (3) or (4) above, wherein the vehicle control unit sets the vehicle speed to a low speed from the time reference setting operation point in time to the recognition point in time, and controls the vehicle speed to the determined vehicle speed after the recognition point in time. (8) The vehicle control device according to (3), (4) or (7) above, wherein the time reference control unit flashes the lights of the host vehicle a plurality of times at intervals specific to the host vehicle before the time reference setting operation, and the recognition processing unit receives the plurality of image frames obtained by the other vehicle that has recognized the multiple flashes. (9) The vehicle control device according to any one of (1) to (8) above, in which the recognition processing unit receives the plurality of image frames acquired by a plurality of other vehicles.(10) The vehicle control device according to (9) above, wherein the recognition processing unit performs recognition processing of the surrounding environment of the host vehicle based on the plurality of image frames acquired by the plurality of other vehicles, and obtains the recognition result. (11) The vehicle control device according to any one of (1) to (10) above, wherein the recognition result of the recognition processing unit includes a recognition result of the surrounding environment of the host vehicle, and is used for parking control of the host vehicle. (12) The vehicle control device according to (4) above, wherein the recognition processing unit of the other vehicle determines that an image frame including a region of interest with maximum brightness is an image frame captured while the light was blinking. (13) The vehicle control device according to any one of (1) to (12) above, wherein the camera is a camera for a surround view monitor. (14) The vehicle control device according to any one of (1) to (13) above, wherein the host vehicle automatically parks in one parking space in an automated valet parking lot, and the other vehicle is parked in another parking space in the automated valet parking lot that can capture an image of the host vehicle. (15) The vehicle control device according to any one of (1) to (14) above, wherein the host vehicle and the other vehicle transmit and receive the multiple image frames by vehicle-to-vehicle communication or communication via a parking lot system. (16) The vehicle control device according to any one of (1) to (15) above, which is capable of acting as both the host vehicle and the other vehicle. (17) A vehicle control method, comprising: obtaining a recognition result based on multiple image frames obtained by a camera of the other vehicle capturing an image of a field of view including the host vehicle; and performing control to automatically park the host vehicle based on the recognition result. (18) A vehicle control system comprising: a recognition processing unit that obtains recognition results based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the vehicle itself; and a vehicle control unit that performs control to automatically park the vehicle itself based on the recognition results.(19) A vehicle comprising: a recognition processing unit that obtains recognition results based on multiple image frames obtained by a camera of another vehicle capturing an image of a field of view including the vehicle itself; and a vehicle control unit that performs control to automatically park the vehicle itself based on the recognition results.

[0197] Although the embodiments and modified examples of the present technology have been described above, the present technology is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present technology.

[0198] REFERENCE SIGNS LIST 1 vehicle 2 parking lot system 200 vehicle control device 201 front camera 202 rear camera 203 side camera 210 vehicle control unit 220 time reference control unit 230 recognition processing unit 240 camera unit 250 wireless communication unit

Claims

1. A vehicle control device comprising: a recognition processing unit that obtains recognition results based on multiple image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle; and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

2. A vehicle control device as claimed in claim 1, wherein the vehicle control unit determines the vehicle speed during parking based on the amount of time lag between the time when each image frame is captured and the time when a recognition result based on each image frame is obtained.

3. A vehicle control device according to claim 2, further comprising a time reference control unit that executes a time reference setting operation at the start of automatic parking, wherein the imaging time point and the recognition time point are times based on the time reference setting operation time point.

4. A vehicle control device according to claim 3, wherein the time reference control section blinks the lights of the host vehicle as the time reference setting operation.

5. A vehicle control device according to claim 2, wherein the vehicle control unit determines the vehicle speed during parking based on the maximum value within a recent predetermined period among the time lag amounts of a plurality of image frames.

6. A vehicle control device according to claim 2, wherein the vehicle control unit determines the vehicle speed during parking based on the maximum value of the time lag amounts of image frames of a plurality of other vehicles.

7. A vehicle control device as claimed in claim 3, wherein the vehicle control unit controls the vehicle speed to be a low speed from the time reference setting operation point to the recognition point, and to be the determined vehicle speed after the recognition point.

8. A vehicle control device as described in claim 3, wherein the time reference control unit blinks the lights of the vehicle a number of times at an interval specific to the vehicle before the time reference setting operation, and the recognition processing unit receives the multiple image frames obtained by the other vehicle that recognizes the multiple blinks.

9. A vehicle control device according to claim 1, wherein the recognition processing unit receives the plurality of image frames acquired by a plurality of other vehicles.

10. A vehicle control device according to claim 9, wherein the recognition processing unit performs recognition processing of the surrounding environment of the host vehicle based on the plurality of image frames acquired by the plurality of other vehicles, and obtains the recognition result.

11. A vehicle control device according to claim 1, wherein the recognition result of the recognition processing unit includes a recognition result of the surrounding environment of the host vehicle and is used for parking control of the host vehicle.

12. A vehicle control device according to claim 4, wherein the recognition processing unit of the other vehicle determines that the image frame including the area of ​​interest with the maximum brightness is an image frame captured when the light is blinking.

13. A vehicle control device according to claim 1, wherein the camera is a camera for a surround view monitor.

14. A vehicle control device as described in claim 1, wherein the host vehicle is automatically parked in one parking space in an automated valet parking lot, and the other vehicle is parked in another parking space in the automated valet parking lot that can capture an image of the host vehicle.

15. A vehicle control device according to claim 1, wherein the host vehicle and the other vehicle transmit and receive the plurality of image frames by vehicle-to-vehicle communication or communication via a parking lot system.

16. A vehicle control device according to claim 1, capable of acting as both the host vehicle and the other vehicle.

17. A vehicle control method, comprising: obtaining a recognition result based on a plurality of image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle; and performing control for automatically parking the host vehicle based on the recognition result.

18. A vehicle control system comprising: a recognition processing unit that obtains recognition results based on multiple image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle; and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

19. A vehicle equipped with a recognition processing unit that obtains recognition results based on multiple image frames obtained by a camera of another vehicle capturing an image of a field of view including the host vehicle, and a vehicle control unit that performs control to automatically park the host vehicle based on the recognition results.

Images