Control device and control method

Abstract

To provide a control device that can support a safe transition from autonomous driving to manual driving. [Solution] The control device 1 according to this disclosure comprises an image acquisition unit 11 that acquires an image of the vehicle taken by the vehicle, and a display control unit 12 that controls the display of the vehicle image. The display control unit 12 displays the vehicle image on the display unit 15 for a predetermined period of time after a switch from automatic driving to manual driving is requested in the vehicle.

Description

[Technical Field]

[0001] The present disclosure relates to a control device and a control method. [Background Art]

[0002] In vehicles having an automatic driving function, technologies for assisting switching from automatic driving to manual driving are known. As a related technology, Patent Document 1 discloses a driving assistance device that can more reliably guide a driver to an awake state when switching from automatic driving to manual driving. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2018-192958 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In a vehicle having such an automatic driving function, the driver's state changes from a relaxed state to a tense state at the timing of switching from automatic driving to manual driving. Therefore, at the moment of switching, the driver may overlook surrounding dangers. For example, a driver who has been relaxed during automatic driving becomes tense when switching to manual driving because the driver tries to concentrate consciousness on driving. At this time, the driver starts manual driving without sufficiently confirming the surrounding safety, and there is a possibility of causing an accident with surrounding vehicles due to unintended steering wheel operation.

[0005] An object of the present disclosure is to provide a control device and a control method capable of assisting safe switching from automatic driving to manual driving in view of the above-described problems. [Means for Solving the Problem]

[0006] The control device according to the present disclosure is An image acquisition unit that acquires an image of the vehicle taken by the vehicle itself, A display control unit that controls the display of the vehicle image, Equipped with, The display control unit displays an image of the vehicle on the display unit for a predetermined period of time after a request is made for the vehicle to switch from automatic driving to manual driving.

[0007] The control device relating to this disclosure is An image acquisition unit that acquires an image of the vehicle taken by the vehicle itself, A switching control unit that controls the switching from automatic driving to manual driving in the vehicle, Equipped with, The switching control unit suppresses the switching from automatic driving to manual driving when a switch from automatic driving to manual driving is requested and an object of attention is detected from the vehicle image.

[0008] The control methods relating to this disclosure are as follows: Computers Image acquisition step to acquire an image of the vehicle by taking an image of the vehicle itself, A display control step that controls the display of the vehicle image on the display unit, Execute, In the aforementioned display control step, the vehicle image is displayed on the display unit for a predetermined period of time after a request is made for the vehicle to switch from automatic driving to manual driving.

[0009] The control methods relating to this disclosure are as follows: Computers Image acquisition step to acquire an image of the vehicle by taking an image of the vehicle itself, A switching control step that controls the switching from automatic driving to manual driving in the vehicle, Execute, In the aforementioned switching control step, if a switch from automatic driving to manual driving is requested and an object of attention is detected from the vehicle image, the switch from automatic driving to manual driving is suppressed. [Effects of the Invention]

[0010] The control device and control method according to the present disclosure can support safe switching from automatic driving to manual driving. [Brief Description of the Drawings]

[0011] [Figure 1] Figure 1 is a diagram showing the overall configuration of an image system. [Figure 2] Figure 2 is a block diagram showing the detailed configuration of an image system. [Figure 3] Figure 3 is a diagram showing an example of a registration information DB. [Figure 4] Figure 4 is a diagram showing a situation ahead of a vehicle traveling behind, as seen from the perspective of the driver of said vehicle. [Figure 5] Figure 5 is a diagram showing a situation ahead of a vehicle traveling ahead, as seen from the perspective of the driver of said vehicle. [Figure 6] Figure 6 is a flowchart showing the flow of image transmission processing performed by an image transmission device. [Figure 7] Figure 7 is a flowchart showing the flow of image transfer processing performed by an image transfer device. [Figure 8] Figure 8 is a flowchart showing the flow of image display processing performed by a control device. [Figure 9] Figure 9 is a block diagram showing the detailed configuration of an image system. [Figure 10] Figure 10 is a flowchart showing the flow of switching processing performed by a control device. [Mode for Carrying Out the Invention]

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are assigned the same reference numerals. For the sake of clarity of explanation, redundant description is omitted as necessary.

[0013] <Embodiment 1> (Image system 10) Figure 1 is a diagram showing the overall configuration of the image system 10 according to this disclosure. The image system 10 comprises a control device 1, an image transmission device 2, an image transfer device 3, a surveillance camera C, and a drone D. The control device 1 is mounted on a vehicle V1 and includes a display unit 15 for displaying images. The image transmission device 2 is mounted on a vehicle V2 and includes an imaging unit 21 for capturing images of surrounding vehicles.

[0014] Vehicles V1 and V2 are, for example, vehicles traveling in the same lane. Vehicles V1 and V2 are traveling from right to left in Figure 1, with vehicle V2 traveling immediately behind vehicle V1. Figure 1 shows an example where vehicles V1 and V2 are traveling in the same lane and there are no other vehicles between them, but it is not limited to this. There may be other vehicles between vehicle V1 and vehicle V2. Also, the image system 10 may be applied to three or more vehicles, and these multiple vehicles may be traveling in different lanes.

[0015] Furthermore, for illustrative purposes, the following example describes a vehicle V1 equipped with a control device 1 and a vehicle V2 equipped with an image transmission device 2, but each vehicle may be equipped with both the control device 1 and the image transmission device 2. Also, for example, vehicles V1 and V2 may each be equipped with a device in which the control device 1 and the image transmission device 2 are integrated.

[0016] The control device 1, image transmission device 2, surveillance camera C, and drone D are connected to the image transfer device 3 via network N. Network N includes, for example, an internet connection or a wireless communication network, but the type of communication is not limited to these. The control device 1, image transmission device 2, surveillance camera C, and drone D each have a communication unit (not shown in Figure 1) and communicate with each other via network N.

[0017] The image system 10 is a system that provides an image transfer service that transfers images acquired by imaging devices installed in vehicles, etc., to a destination vehicle via communication means. Specifically, the image system 10 stores images acquired by the imaging unit 21 of vehicle V2, images acquired by the surveillance camera C, and images acquired by the drone D in the image transfer device 3 via the network N, and transfers these images to the control device 1 of vehicle V1.

[0018] In the following, images acquired by these imaging devices capturing images of vehicle V1 may be referred to as "self-vehicle images" of vehicle V1. Self-vehicle images are images acquired by imaging devices surrounding the vehicle capturing images of the vehicle. "Self-vehicle" refers to the vehicle equipped with the control device 1. In the example in Figure 1, vehicle V1 equipped with the control device 1 is considered the self-vehicle.

[0019] The vehicle image includes the vehicle itself. The vehicle image may also include objects surrounding the vehicle. For example, the vehicle image may include other vehicles, pedestrians, or obstacles present around the vehicle. In the example in Figure 1, the vehicle image may include the exterior of vehicles other than vehicle V1. The vehicle image may also include, for example, other vehicles traveling in front of, behind, to the left of, or to the right of vehicle V1.

[0020] Furthermore, vehicles other than one's own vehicle may be referred to as "other vehicles." In the example in Figure 1, vehicle V2, which is traveling behind vehicle V1, is an other vehicle. However, other vehicles are not limited to vehicles behind. Other vehicles may be vehicles that are in the vicinity of one's own vehicle. Other vehicles may be, for example, vehicles that are in front of, to the left of, or to the right of vehicle V1.

[0021] The imaging unit 21, surveillance camera C, and drone D shown in Figure 1 are examples of imaging devices located around the vehicle V2. The imaging unit 21 is an imaging device installed on the vehicle V2. The imaging unit 21 images the area around the vehicle V2. The imaging unit 21 is mounted on the vehicle V2 and performs imaging in a predetermined direction. As a result, the imaging unit 21 images the vehicle V1 traveling around the vehicle V2. The imaging unit 21 may be a forward-facing camera capable of imaging the area in front of the vehicle V2. For example, the imaging unit 21 is installed above the interior of the vehicle V2 in a position that allows it to image the area in front of the vehicle V2.

[0022] The imaging unit 21 captures images of the vehicle V1 traveling ahead at predetermined time intervals and transmits the images of the vehicle acquired through imaging to the image transfer device 3 via the communication unit. Further details about the imaging unit 21 will be described later.

[0023] Surveillance camera C is an imaging device installed around a road. Surveillance camera C may be installed on infrastructure, including facilities located near the road. Infrastructure includes, but is not limited to, traffic lights, streetlights, or road signs. Surveillance camera C images vehicle V1 from a predetermined position. Surveillance camera C transmits the vehicle image acquired through imaging to image transfer device 3 via a communication unit.

[0024] Drone D is a mobile device equipped with an imaging device (not shown) that images vehicle V1 from above. In Figure 1, Drone D is shown as an example of a mobile device equipped with an imaging device that moves by flight, but the method of movement of Drone D is not limited to flight. Drone D may also be a ground-based device. Furthermore, the type of mobile device is not limited to a drone. Any mobile device may be used as the mobile device that images vehicle V1. Drone D transmits the image of itself acquired by imaging to the image transfer device 3 via the communication unit.

[0025] The imaging device used to image the vehicle V1 is not limited to the example described above. Any imaging device capable of imaging the vehicle V1 may be used.

[0026] When a request to switch from automatic to manual driving is received from vehicle V1, the image system 10 transfers an image of the vehicle from the image transfer device 3 to the control device 1 of vehicle V1. The control device 1 displays the transferred image on the display unit 15. The driver of vehicle V1 can view the image of their own vehicle on the display unit 15, making it easy to understand the status of their vehicle as seen from vehicle V2, surveillance camera C, or drone D.

[0027] The image system 10 provides an image transfer service to pre-registered users. Users of the image system 10 register the license plate number of their vehicle and the destination for transferring images of their vehicle to the image system 10. For example, a user registers their vehicle's license plate number and the destination for transferring images of their vehicle using a terminal device such as a PC (Personal Computer) or smartphone. For example, the destination for transferring images of vehicle V1 is associated with the control device 1 installed in vehicle V1.

[0028] (Overview of the processing performed by the image system 10) The overview of the processing performed by the image system 10 will be explained with reference to Figure 1. Note that the white, one-way arrows shown in Figure 1 simply illustrate the flow of information (data or signals, etc.) and do not exclude the possibility of bidirectional information flow. Furthermore, in the following explanation, the image transmission device 2 will be used as a representative example of the vehicle V2's image transmission device 2, surveillance camera C, and drone D.

[0029] First, the image transmission device 2 uses the imaging unit 21 to image the vehicle V1 and acquire an image of the vehicle itself (S1). Next, the image transmission device 2 transmits the image of the vehicle to the image transfer device 3 (S2). The image transfer device 3 receives the image of the vehicle from the image transmission device 2 and transfers the image of the vehicle to the control device 1 (S3). The image transfer device 3 can identify the control device 1 as the destination for the transfer of the image of the vehicle based on the license plate information contained in the image of the vehicle.

[0030] The control device 1 receives the vehicle image transmitted from the image transfer device 3 (S4) and displays the vehicle image on the display unit 15 (S5). As a result, the driver of vehicle V1 can view the vehicle image captured by vehicle V2 from inside the vehicle.

[0031] Figure 2 is a block diagram showing the detailed configuration of the image system 10. Referring to Figure 2, the configurations of the control device 1, the image transmission device 2, and the image transfer device 3 will be described. Note that the control device 1 and the image transmission device 2 may be configured as dedicated hardware mounted on the vehicle, or they may be configured as devices with functions such as a drive recorder. The image transfer device 3 may be configured as a server device installed outside the vehicle.

[0032] (Image transmission device 2) First, let's describe the image transmission device 2. The image transmission device 2 transmits the vehicle image it has captured of the vehicle V1 to the image transfer device 3. The image transmission device 2 comprises an imaging unit 21, an image acquisition unit 22, a license plate reading unit 23, a registration determination unit 24, and a communication unit 27.

[0033] The image transmission device 2 includes a processor, memory, and storage device (not shown). The storage device stores a computer program on which the processing described herein is implemented. The processor can load the computer program from the storage device into memory and execute the computer program. In this way, the processor realizes the functions of the image acquisition unit 22, the number reading unit 23, and the registration determination unit 24.

[0034] Furthermore, the image acquisition unit 22, the number reading unit 23, and the registration determination unit 24 may each be implemented with dedicated hardware. Some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the above-mentioned circuits, etc., and programs. In addition, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc. can be used as the processor.

[0035] The imaging unit 21 is an imaging device that images the area around the vehicle V2. The imaging unit 21 is attached to the vehicle V2 and performs imaging in a predetermined direction. As a result, the imaging unit 21 images vehicles traveling around the vehicle V2. In the example in Figure 1, the imaging unit 21 images vehicle V1 traveling in front of vehicle V2. Therefore, the imaging unit 21 may be a front camera capable of imaging the area in front of vehicle V2. The imaging unit 21 is installed, for example, above the interior of the vehicle V2, in a position that allows it to image the area in front of vehicle V2.

[0036] The imaging unit 21 images the vehicle V1 traveling ahead at predetermined time intervals and outputs the image of the vehicle itself acquired by imaging to the image acquisition unit 22. The image of the vehicle itself may be a moving image or a still image. In the following description, the imaging unit 21 will be described as capturing a moving image of the area in front of the vehicle V2.

[0037] Although Figure 2 shows the imaging unit 21 inside the image transmission device 2, the imaging unit 21 may be provided separately from the image transmission device 2. For example, if the vehicle V2 is equipped with a drive recorder, the camera of the drive recorder may be used as the imaging unit 21. Furthermore, multiple imaging units 21 may be provided in the vehicle V2. For example, the imaging unit 21 may include a left-side camera that images the left side of the vehicle V2, a right-side camera that images the right side of the vehicle V2, and a rear-side camera that images the rear of the vehicle V2. In addition, the imaging unit 21 may be, for example, a panoramic camera capable of capturing images around the vehicle V2 with a 360-degree field of view.

[0038] The image acquisition unit 22 acquires an image of the vehicle from the imaging unit 21. The image acquisition unit 22 outputs the image of the vehicle to the license plate reading unit 23. The image acquisition unit 22 may also store the image of the vehicle in a storage unit (not shown).

[0039] The license plate reading unit 23 extracts a license plate image from the vehicle image. The license plate image is the image region of the vehicle image taken of the vehicle V1 that corresponds to the license plate attached to the vehicle V1. Here, the license plate displays the registration number of the vehicle V1. The license plate may be, for example, a vehicle registration number plate, a vehicle number plate, or a sign.

[0040] The license plate reading unit 23 reads license plate information from the license plate image if the vehicle image includes a license plate image. The license plate information is information obtained from the license plate image. The license plate reading unit 23 may use any image analysis technique to extract the license plate image and read the license plate information.

[0041] The license plate information may include, for example, the registration place name, classification number, kana characters, and serial designation number as inscribed on the license plate in Japan. The registration place name indicates the name of the place where the vehicle was registered. The classification number indicates the classification of the vehicle V1 by type and use. The kana characters indicate the use, such as commercial or private use. The serial designation number is part of the vehicle number and is a four-digit number inscribed in a relatively conspicuous location on the license plate.

[0042] The number reading unit 23 may acquire this character information using image recognition processing such as OCR (Optical Character Recognition). The number reading unit 23 outputs the read number information to the communication unit 27. Although this example shows the number reading unit 23 reading the number information, the number may be read by a device other than the image transmission device 2, and the image transmission device 2 may acquire the reading result.

[0043] The registration determination unit 24 determines whether the license plate information of the vehicle V1 read by the license plate reading unit 23 is registered in the image system 10. For example, the registration determination unit 24 makes a registration confirmation request to the image transfer device 3 to check whether the license plate information is already registered. For example, the registration determination unit 24 makes a registration confirmation request by transmitting the license plate information and the registration confirmation request to the image transfer device 3. As a result, the registration confirmation unit 31 in the image transfer device 3 checks whether the license plate information is registered or not.

[0044] The image transfer device 3 stores a registration information database (not shown), and the registration confirmation unit 31 checks whether the number information is registered in the registration information database. In response to the registration confirmation request, the registration confirmation unit 31 transmits the confirmation result to the image transmission device 2.

[0045] The registration determination unit 24 receives a response to the registration confirmation request from the image transfer device 3 via the communication unit 27 and determines, based on the confirmation result, whether or not the license plate information is registered in the image system 10. The fact that the license plate information is registered in the image transfer device 3 indicates that the vehicle associated with that license plate information is eligible for the image transfer service provided by the image system 10. If the license plate information of vehicle V1 is registered in the image transfer device 3, then vehicle V1 is eligible for the image transfer service.

[0046] If the registration determination unit 24 determines that the license plate information is registered, it transmits the license plate information and the image of the vehicle captured by the imaging unit 21 to the image transfer device 3. Therefore, the registration determination unit 24 transmits the vehicle image if the vehicle V1 is eligible for the service, and does not transmit the vehicle image if the vehicle V1 is not eligible for the service. In this way, by checking whether the vehicle is registered in the image system 10 before transmitting the vehicle image to the image transfer device 3, the image transmission device 2 can suppress unnecessary image transmission to the image transfer device 3.

[0047] Although this example describes registration verification being performed in the registration verification unit 31 of the image transfer device 3, the method of registration verification is not limited to this. For example, the image transmission device 2 may be equipped with the function of the registration verification unit 31 and perform registration verification in the image transmission device 2. In this case, the storage unit of the image transmission device 2, or an external storage device, should store the registration information DB.

[0048] For example, suppose the registration determination unit 24 of the image transmission device 2 also includes the function of the registration confirmation unit 31. In this case, the registration determination unit 24 refers to the registration information DB and checks whether the number information read by the number reading unit 23 is registered in the registration information DB. If there is number information in the registration information DB that matches the number information read, the registration determination unit 24 determines that the number information is registered. By performing registration confirmation of number information in this way in the image transmission device 2, the time required for registration confirmation can be reduced.

[0049] The communication unit 27 transmits and receives information to and from the outside world via a communication means. The communication means may be, for example, mobile communication technology or the internet. As for mobile communication technology, for example, low-latency 5G (5th Generation Mobile Communication System) or technologies with even lower latency may be used. However, the communication means are not limited to these.

[0050] For example, if the vehicle image includes a license plate image, the communication unit 27 transmits the license plate information read by the license plate reading unit 23 and a registration confirmation request to the image transfer device 3. Also, if the registration determination unit 24 determines that the license plate information is registered in the image system 10, the communication unit 27 transmits the license plate information and the vehicle image to the image transfer device 3. The communication unit 27 also receives a response from the image transfer device 3 to the registration confirmation request for the license plate information.

[0051] (Image transfer device 3) Next, the image transfer device 3 will be described. The image transfer device 3 receives the vehicle image from the image transmission device 2 and transfers the received vehicle image to the control device 1. The image transfer device 3 may also check whether the license plate information of the vehicle V1 is registered, and if registration is confirmed, it may transfer the vehicle image to the destination corresponding to the license plate information. The image transfer device 3 may be implemented by, for example, a server device or cloud system that provides an image transfer service. The image transfer device 3 includes a registration confirmation unit 31, a transfer processing unit 32, and a communication unit 37.

[0052] The image transfer device 3 includes a processor, memory, and storage device (not shown). The storage device stores a computer program on which the processing described herein is implemented. The processor can load the computer program from the storage device into memory and execute the computer program. In this way, the processor realizes the functions of the registration confirmation unit 31 and the transfer processing unit 32.

[0053] Furthermore, the registration confirmation unit 31 and the transfer processing unit 32 may each be implemented with dedicated hardware. Some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be composed of a single chip or multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the above-mentioned circuits, etc., and programs. In addition, a CPU, GPU, FPGA, quantum processor, etc. can be used as the processor.

[0054] The registration confirmation unit 31 checks whether the license plate information received from the image transmission device 2 is registered in the image system 10. When the registration confirmation unit 31 receives license plate information and a registration confirmation request from the image transmission device 2, it refers to the registration information DB stored in a storage unit (not shown). The registration information DB associates license plate information with the destination for transferring the vehicle image.

[0055] Figure 3 shows an example of a registration information database. In this example, the registration information database includes user ID, license plate information, and transfer destination information. The user ID is information that identifies the user receiving the image transfer service. The license plate information is the information written on the license plate of the vehicle to which the image transfer service is provided. The transfer destination is information indicating the destination to which the vehicle's images are transferred.

[0056] The destination is, for example, the address of control device 1. The address of control device 1 may be, for example, an IP address (Internet Protocol address) used for communication in network N. Alternatively, the destination may be a device ID that identifies control device 1. Furthermore, if image transfer device 3 is a cloud server, the destination may be an endpoint of a dedicated system. In addition to these, the destination may be any information that identifies the destination to which the vehicle image is transferred. If the registration confirmation unit 31 confirms the registration of license plate information, it uses the registration information DB to identify the destination associated with the license plate information.

[0057] Returning to Figure 2, the transfer processing unit 32 performs the transfer processing of the vehicle image. If the registration confirmation unit 31 determines that the license plate information is registered in the registration information DB, the transfer processing unit 32 waits until it receives an image transfer request from the control device 1. The image transfer request is a request to the image transfer device 3 to transfer the vehicle image. When the transfer processing unit 32 receives an image transfer request from the control device 1, it transfers the vehicle image to the control device 1.

[0058] The communication unit 37 transmits and receives information to and from the outside via communication means. The communication means may be, for example, mobile communication technology or the internet. For example, the communication unit 37 receives a registration confirmation request from the image transmission device 2. In response to the registration confirmation request from the image transmission device 2, the communication unit 37 transmits the result of the registration confirmation. If the communication unit 37 receives an image transfer request from the control device 1, it transfers the vehicle image to the control device 1.

[0059] (Control device 1) Next, the control device 1 will be described. The control device 1 receives the vehicle image transferred from the image transfer device 3 and displays the received vehicle image. The control device 1 comprises an image acquisition unit 11, a display control unit 12, a switching control unit 14, a display unit 15, and a communication unit 17.

[0060] The control device 1 includes a processor, memory, and storage device (not shown). The storage device stores a computer program on which the processing described herein is implemented. The processor can load the computer program from the storage device into memory and execute the computer program. In this way, the processor realizes the functions of the image acquisition unit 11, the display control unit 12, and the switching control unit 14.

[0061] Furthermore, the image acquisition unit 11, the display control unit 12, and the switching control unit 14 may each be implemented with dedicated hardware. Also, some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the above-mentioned circuits, etc., and programs. In addition, a CPU, GPU, FPGA, quantum processor, etc. can be used as the processor.

[0062] When the switching control unit 14 receives a manual switching request, the image acquisition unit 11 requests the image transfer device 3 to transfer the vehicle image. The image acquisition unit 11 requests the transfer of the vehicle image by transmitting the image transfer request to the image transfer device 3 via the communication unit 17. A manual switching request is a request to switch from automatic driving to manual driving (hereinafter sometimes simply referred to as "switching"). In contrast, an automatic switching request is a request to switch from manual driving to automatic driving.

[0063] Furthermore, the image acquisition unit 11 acquires an image of the vehicle itself. The image acquisition unit 11 may acquire an image of the vehicle from any device capable of capturing images of the vehicle. For example, the image acquisition unit 11 acquires an image of the vehicle transferred from the image transfer device 3. In the example shown in Figure 1, the image acquisition unit 11 acquires an image of the vehicle captured by the vehicle V2, the surveillance camera C, or the drone D via the image transfer device 3. The image acquisition unit 11 may also store the acquired image of the vehicle in a storage unit (not shown).

[0064] The display control unit 12 controls the display of the vehicle image acquired by the image acquisition unit 11. The display control unit 12 displays the vehicle image on the display unit 15 for a predetermined period of time after a request is made for the vehicle to switch from automatic driving to manual driving.

[0065] Here, "predetermined time" refers, for example, to the time elapsed from the time a switch is requested until a predetermined time has elapsed. This predetermined time may be a set time (for example, 5 to 10 seconds). For example, if the set time is 5 seconds, the display control unit 12 will display the vehicle image on the display unit 15 for 5 seconds from the time a switch is requested. By viewing the vehicle image displayed on the display unit 15, the driver can understand the situation of their vehicle from the perspective of the surroundings.

[0066] Furthermore, if the driver performs a predetermined operation before the set time has elapsed, the time between the request for switching and the performance of said predetermined operation may be the predetermined interval described above. Such predetermined operation is, for example, pressing the "display end button" to end the display of the vehicle image. The predetermined operation may also be pressing the "cancel switch button" to cancel the request to switch to manual driving.

[0067] When a switch is requested, the display control unit 12 displays these buttons on the display unit 15, and terminates the display of the vehicle image when the driver presses any of these buttons. These buttons may also be physical buttons.

[0068] Here, with reference to Figures 4 and 5, an example of a vehicle image displayed on the display unit 15 when processing related to the image system 10 is performed will be explained. Figure 4 is a diagram showing the situation in front of vehicle V2 as seen from the driver's perspective of vehicle V2. Figure 5 is a diagram showing the situation in front of vehicle V1 as seen from the driver's perspective of vehicle V1. In Figures 4 and 5, similar to the example shown in Figure 1, vehicle V2 is traveling behind vehicle V1.

[0069] As shown in Figure 4, vehicle V1 is about to turn left at an intersection in front of vehicle V2. In front of vehicle V1 is pedestrian P1 about to cross the crosswalk. Also, motorcycle P2 is approaching from behind vehicle V1. The driver of vehicle V2 can easily see pedestrian P1 and motorcycle P2. However, motorcycle P2 may be in the blind spot of the driver of vehicle V1. In this situation, if the driving mode is simply switched from automatic to manual, even a slight steering input could cause vehicle V1 to wobble and potentially collide with the motorcycle that is passing on the left.

[0070] In the image system 10 according to this disclosure, as shown in Figure 5, the display unit 15 of the vehicle V1 displays a vehicle image 151. The vehicle image 151 is an image acquired by the imaging unit 21 of the vehicle V2 imaging the vehicle V1 and transferred to the control device 1 via the image transfer device 3.

[0071] As shown in Figure 5, the vehicle image 151 includes not only pedestrians P1 visible from vehicle V1, but also vehicles V1 and motorcycle P2 which are not visible from vehicle V1. By viewing the vehicle image 151, the driver of vehicle V1 can easily understand the situation even in areas that are blind spots and difficult to see from their own field of view. This allows the driver of vehicle V1 to check the safety around their vehicle and prevent accidents from occurring. In addition, vehicle V2 shares its own vehicle image with vehicle V1 by transferring the image it has captured to vehicle V1. This allows vehicle V2 to contribute to improving the safety around its vehicle.

[0072] Returning to Figure 2, the switching control unit 14 controls the switching from automatic driving to manual driving in the vehicle. The switching control unit 14 receives manual switching requests and automatic switching requests from the driver and switches from the current driving mode to the driving mode corresponding to the received request.

[0073] For example, the switching control unit 14 receives a request to switch the driving mode when the "manual driving button" or "automatic driving button" is pressed during automatic or manual driving. In response to the received switching request, the switching control unit 14 instructs the vehicle V1's ECU (Electric Control Unit) to change the driving mode.

[0074] The display unit 15 displays the vehicle image transferred from the image transfer device 3. The display unit 15 is, for example, a display. The display unit 15 may also be a touch panel with input functionality. If the vehicle V1 is equipped with a drive recorder, the display of the drive recorder may be used as the display unit 15. The display may be one provided on the drive recorder itself, or it may be a separate display provided on the vehicle V1. For example, the display unit 15 may be the display of a terminal device such as a smartphone used by the driver of the vehicle V1.

[0075] The communication unit 17 transmits and receives information with external devices via communication means. For example, the communication unit 17 sends an image transfer request to the image transfer device 3. The communication unit 17 also receives images of the vehicle from the image transfer device 3.

[0076] The configurations of the image system 10 have been described above. While a detailed description of the configurations of the surveillance camera C and drone D has been omitted here, the surveillance camera C and drone D may be configured to transmit images of their own vehicle to the image transfer device 3, for example, by having a configuration equivalent to that of the image transmission device 2. Alternatively, the surveillance camera C and drone D may be configured to include at least the imaging unit 21, the image acquisition unit 22, and the communication unit 27, which are components of the image transmission device 2. In this case, the surveillance camera C and drone D transmit all acquired images of their own vehicle to the image transfer device 3, where the image transfer device 3 can then read license plate information and perform other necessary operations.

[0077] If some or all of the components of the control device 1, image transmission device 2, image transfer device 3, surveillance camera C, and drone D are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. Also, for example, the functions of the image transfer device 3 may be provided in SaaS (Software as a Service) format.

[0078] (Processing by image system 10) Next, the processing performed by each device in the image system 10 will be explained with reference to Figures 6 to 8. Figure 6 is a flowchart showing the flow of the image transmission process performed by the image transmission device 2. Figure 7 is a flowchart showing the flow of the image transfer process performed by the image transfer device 3. Figure 8 is a flowchart showing the flow of the image display process performed by the control device 1. Note that the processing shown in Figures 6 to 8 uses the example described using Figures 1 to 5, in which a vehicle V2 traveling behind captures an image of a vehicle V1 traveling in front.

[0079] (Image transmission process of image transmission device 2) Referring to Figure 6, the image transmission process of the image transmission device 2 will be explained. First, the imaging unit 21 captures an image of the vehicle V1 (S11). The image acquisition unit 22 then acquires the image of the vehicle obtained through imaging.

[0080] Next, the license plate reading unit 23 determines whether or not the vehicle image contains a license plate image (S12). If it is determined that the vehicle image does not contain a license plate image (NO in S12), the process in step S11 is repeated. If it is determined that the vehicle image contains a license plate image (YES in S12), the license plate reading unit 23 reads the license plate information from the vehicle image (S13).

[0081] Next, the registration determination unit 24 requests the image transfer device 3 to confirm the registration of the license plate information (S14). For example, the registration determination unit 24 makes the registration confirmation request by transmitting the license plate information and the registration confirmation request to the image transfer device 3. As a result, the image transfer device 3 confirms the registration of the license plate information. The registration confirmation verifies whether the vehicle is eligible for the image transfer service. The communication unit 27 receives the registration confirmation result from the image transfer device 3 (S15).

[0082] The registration determination unit 24 determines whether or not the license plate information is registered based on the registration confirmation result (S16). If it is determined that the license plate information is not registered (NO in S16), the process returns to step S11 and the subsequent processing is repeated. If it is determined that the license plate information is registered (YES in S16), the communication unit 27 transmits the license plate information and the vehicle image to the image transfer device 3 (S17).

[0083] Furthermore, the image transmission device 2 may have a function to follow a vehicle whose license plate information it has recognized. For example, even if the license plate is not visible in the image of its own vehicle, the image transmission device 2 may follow vehicle V1 by using the features and movement patterns of vehicle V1. The features of vehicle V1 may be, for example, the shape, color, or size of vehicle V1. In that case, the image transmission device 2 may proceed to step S17 after step S11 without performing steps S12 to S16.

[0084] (Image transfer processing of image transfer device 3) Referring to Figure 7, the image transfer process of the image transfer device 3 will be explained. First, the communication unit 37 determines whether or not it has received a registration confirmation request from the image transmission device (S21). If it is determined that a registration confirmation request has not been received (NO in S21), the process in step S21 is repeated. If it is determined that a registration confirmation request has been received (YES in S21), the registration confirmation unit 31 performs registration confirmation of the number information (S22).

[0085] Next, the registration confirmation unit 31 determines whether or not the number information is registered (S23). Specifically, the registration confirmation unit 31 refers to the registration information DB in the storage unit to check whether the number information is registered. If it is determined that the number information is not registered (NO in S23), the registration confirmation unit 31 sends a message to the image transmission device 2 indicating that the number information is not registered (S25). After that, the process returns to step S21.

[0086] If it is determined that the license plate information is registered (YES in S23), the registration confirmation unit 31 transmits a message to the image transmission device 2 indicating that the license plate information is registered (S24). The registration confirmation unit 31 also identifies the transfer destination associated with the registered license plate information. Subsequently, the license plate information and the vehicle image are transmitted from the image transmission device 2 to the image transfer device 3. The communication unit 37 receives the license plate information and the vehicle image from the image transmission device 2 (S26). The storage unit of the image transfer device 3 stores the license plate information and the vehicle image (S27).

[0087] Next, the transfer processing unit 32 determines whether the communication unit 37 has received an image transfer request from the control device 1 (S28). If it determines that an image transfer request has not been received (NO in S28), the process ends. If it determines that an image transfer request has been received (YES in S28), the transfer processing unit 32 transfers the vehicle image to the control device 1 (S29).

[0088] (Image display processing of control device 1) Referring to Figure 8, the image display processing of the control device 1 will be explained. As a prerequisite, it is assumed that at the start of this process, the driver of vehicle V1 is operating in automatic driving mode. First, the switching control unit 14 receives a request to switch from automatic driving to manual driving (S31). For example, the switching control unit 14 receives a switching request in response to the pressing of the "manual driving button" displayed on the display unit 15.

[0089] The image acquisition unit 11 requests the image transfer device 3 to transfer an image of its own vehicle via the communication unit 17 (S32). For example, the image acquisition unit 11 requests the transfer of an image of its own vehicle by sending an image transfer request to the image transfer device 3 that includes identification information for identifying its own vehicle. The identification information may be information written on the license plate, or a vehicle ID for identifying the vehicle. Alternatively, the identification information may be a device ID for identifying the control device 1.

[0090] The image transfer device 3 identifies the vehicle based on the identification information and identifies the transfer destination associated with the vehicle. When the vehicle image is transferred from the image transfer device 3, the communication unit 17 receives the vehicle image.

[0091] Next, the image acquisition unit 11 determines whether the communication unit 17 has received an image of the vehicle (S33). If it is determined that an image of the vehicle has not been received (NO in S33), the process proceeds to step S38. If it is determined that an image of the vehicle has not been received, the image acquisition unit 11 may repeat the process in step S33 up to a predetermined number of times.

[0092] If it is determined that an image of the vehicle has been received (YES in S33), the image acquisition unit 11 acquires the image of the vehicle (S34). The display control unit 12 displays the acquired image of the vehicle on the display unit 15 (S35).

[0093] Next, the display control unit 12 determines whether or not the set time has elapsed (S36). The set time may be set in advance by the driver of the vehicle or the like. If it is determined that the set time has not elapsed (NO in S36), the process in step S36 is repeated.

[0094] If it is determined that the set time has elapsed (YES in S36), the display control unit 12 terminates the display of the vehicle image (S37). In this way, the display control unit 12 can display the vehicle image on the display unit 15 for a predetermined period of time after a request to switch from automatic driving to manual driving is made in the vehicle. However, if a predetermined operation is performed by the driver before the set time has elapsed, the display control unit 12 may terminate the display of the vehicle image without waiting for the set time to elapse. The predetermined operation is, for example, pressing the "display termination button" or the "switch cancellation button".

[0095] If the display of the vehicle image ends in step S37, or if it is determined in step S33 that the vehicle image has not been received (NO in S33), the switching control unit 14 switches the driving mode from automatic driving to manual driving (S38). This completes the process. However, if the driver presses the "cancel switching button" before the switch is made, the switching control unit 14 will terminate the process without making the switch.

[0096] As described above, in the image system 10 of this disclosure, the control device 1 displays an image of the vehicle on the display unit 15 for a predetermined period of time after a request is made for the vehicle to switch from automatic driving to manual driving. This allows the driver to see the image of the vehicle displayed on the display unit 15 and understand the situation of the vehicle from the perspective of the surroundings. As a result, the driver can check for hazards around the vehicle, thereby avoiding dangers at the time of switching from automatic driving to manual driving and preventing accidents. With this configuration, the image system 10 can support a safe switch from automatic driving to manual driving.

[0097] <Embodiment 2> Next, Embodiment 2 will be described. Embodiment 1 described an example in which the vehicle displays an image of itself taken by surrounding vehicles when switching from autonomous driving to manual driving, thereby enabling a safe transition. This embodiment describes an example in which, when a driver requests a switch, an image of the vehicle is acquired and analyzed, and a decision is made on whether or not to switch based on the surrounding conditions of the vehicle. In this embodiment, if the control device detects a caution target from the vehicle image as a result of the analysis of the vehicle image, it suppresses the switch from autonomous driving to manual driving. The caution target will be described later.

[0098] The overall configuration of the image system 10a according to this embodiment is generally the same as that of the image system 10 described in Embodiment 1. The diagram showing the overall configuration of the image system 10a can be explained by substituting the image system 10 shown in Figure 1 with the image system 10a and the control device 1 with the control device 1a. Therefore, the illustration of the overall configuration of the image system 10a is omitted. The image system 10a comprises a control device 1a, an image transmission device 2, an image transfer device 3, a surveillance camera C, and a drone D. In the following, the differences from Embodiment 1 will be the main focus of the explanation, and overlapping points will be omitted as appropriate.

[0099] Figure 9 is a block diagram showing the detailed configuration of the image system 10a according to this disclosure. The image system 10a includes a control device 1a instead of the control device 1 of Embodiment 1. The control device 1a differs from the control device 1 in that, in addition to the image acquisition unit 11, display control unit 12, switching control unit 14, display unit 15, and communication unit 17, it also includes an image analysis unit 13. Except for the image analysis unit 13, the other units correspond to the respective functional units of the control device 1. Furthermore, the switching control unit 14 has a different function from that of the control device 1.

[0100] The image analysis unit 13 analyzes the vehicle image acquired by the image acquisition unit 11. The image analysis unit 13 may analyze the vehicle image using any image analysis technique. For example, the image analysis unit 13 detects attention objects from the vehicle image. Attention objects are things that the driver should pay attention to if they are present around the vehicle. The presence of attention objects can affect the safe transition from autonomous driving to manual driving. Attention objects are, but are not limited to, other vehicles, pedestrians, bicycles, motorcycles, or obstacles. For example, pedestrian crossings and intersections with poor visibility may be set as attention objects. Attention objects may be set in advance by the driver of vehicle V1 or the like.

[0101] If the image analysis unit 13 detects a target of attention in the vehicle image as a result of the analysis, it outputs a target of attention detection notification to the switching control unit 14 indicating that a target of attention has been detected in the vehicle image. The image analysis unit 13 may include information such as the location of the target of attention, the type of target of attention, or the status of the target of attention in the target of attention detection notification. For example, the target of attention detection notification may include information such as "left rear, motorcycle, approaching the vehicle".

[0102] The image analysis unit 13 also determines whether the object of attention is located in the vehicle's blind spot. The range of the vehicle's blind spot may be set in advance. The range of the blind spot may be set based on the arrangement of cameras and sensors mounted on the vehicle. For example, if the vehicle is equipped with a dashcam, the range of the blind spot may be set considering the imaging range of the dashcam. The range of the blind spot may also change depending on the presence or absence of the vehicle's front camera, rear camera, side cameras, or ultrasonic sensors. The range of the blind spot may also be dynamically set using AI (Artificial Intelligence) according to the surrounding environment and driving conditions of the vehicle.

[0103] The image analysis unit 13 may include in the attention detection notification the result of determining whether or not the attention target is located in the blind spot of the vehicle. In this case, the attention detection notification will include information such as "in blind spot, left rear, motorcycle, approaching the vehicle".

[0104] The switching control unit 14 suppresses the switch from automatic to manual driving when a switch from automatic to manual driving is requested and a target of attention is detected from the vehicle image. Here, "suppressing the switch from automatic to manual driving" means that the switch is restricted by performing a predetermined switching suppression process before the switch is made.

[0105] The switching suppression process may be a process that invalidates the driver's switching request, or a process that alerts the driver to the object of attention before the switching occurs. By invalidating the switching request, the switching control unit 14 controls the switching so that it does not switch from automatic driving to manual driving. In addition, by alerting the driver to the object of attention in the surroundings, the risk of accidents can be reduced.

[0106] The switching control unit 14 may, for example, alert the driver by outputting a warning. For example, the switching control unit 14 may display an alert message on the display unit 15 such as, "Caution is required when switching to manual driving at this time." The switching control unit 14 may also include information about the object of attention included in the vehicle image in the alert message, such as, "A motorcycle is approaching from the left rear." The switching control unit 14 may also output the alert message as sound from a speaker (not shown).

[0107] In this way, when the vehicle is in the vicinity of the vehicle, the switching control unit 14 does not immediately switch to manual driving at the driver's request, but rather suppresses the switch by disabling the switch or warning the driver. In this way, the switching control unit 14 allows the driver to continue driving safely compared to when the switch is performed immediately.

[0108] Furthermore, the switching control unit 14 may, when a switching is requested and a target of attention is detected from the vehicle image, determine whether the target of attention is located in the vehicle's blind spot, and if it determines that the target of attention is located in the vehicle's blind spot, it may suppress the switching. In this way, the switching control unit 14 can suppress the switching only when the target of attention is located in the vehicle's blind spot and the driver is unaware of it. This reduces the risk of collision between the vehicle and the target of attention located in the blind spot.

[0109] In this example, the image analysis unit 13 of the control device 1a performs the analysis of the vehicle's image, but this is not the only example. If the processing speed and communication environment requirements are met, for example, the image transfer device 3 may perform the analysis of the vehicle's image. Also, although Figure 9 shows an example in which the control device 1a includes a display control unit 12 and a display unit 15, the control device 1a does not need to include these if there is no need for the vehicle V1 to visually perceive the vehicle's image.

[0110] (Switching process of control device 1a) Figure 10 is a flowchart showing the flow of the switching process performed by the control device 1a. Steps S41 to S44 correspond to steps S31 to S34 shown in Figure 8. Therefore, the explanation is omitted here.

[0111] After step S44, the image analysis unit 13 analyzes the vehicle image acquired by the image acquisition unit 11 (S45). The image analysis unit 13 also determines whether or not a target of attention has been detected in the vehicle image (S46). If it is determined that no target of attention has been detected (NO in S46), the process proceeds to step S49. If it is determined that a target of attention has been detected (YES in S46), the image analysis unit 13 determines whether or not the target of attention is located in the blind spot of the vehicle (S47). If it is determined that the target of attention is not located in the blind spot (NO in S47), the process proceeds to step S49.

[0112] If the image analysis unit 13 determines that the object of attention is located in a blind spot (YES in S47), it outputs an object of attention detection notification to the switching control unit 14 indicating that the object of attention has been detected in the vehicle image. Here, the image analysis unit 13 outputs the object of attention detection notification including that the object of attention is located in the vehicle's blind spot. The switching control unit 14 performs a switching suppression process (S48). Here, as a switching suppression process, the switching control unit 14 performs a process to invalidate the switching request. As a result, the switching control unit 14 controls the switching so that it does not switch from automatic driving to manual driving. After that, the process returns to step S41. Alternatively, the subsequent process may return to step S42.

[0113] The switching control unit 14 may, as step S48, display an alert message on the display unit 15. The switching control unit 14 may also output the alert message as sound from a speaker (not shown). This allows the switching control unit 14 to alert the driver before making a switch. In this case, the process proceeds from step S48 to step S49.

[0114] If the vehicle image is not received (NO in S43), if the target of attention is not detected (NO in S46), or if the target of attention is detected but is not located in a blind spot (NO in S47), the switching control unit 14 switches the driving mode from automatic driving to manual driving (S49). This completes the process. However, if the driver presses the "cancel switching button" before the switch is made, the switching control unit 14 will terminate the process without making the switch.

[0115] In the example shown in Figure 10, step S47 determines whether the object of attention is located in the blind spot of the vehicle, but the image analysis unit 13 does not need to perform this determination. In that case, the control device 1a should perform step S48 if the answer to step S46 is YES, and step S49 if the answer to step S46 is NO.

[0116] As described above, in the image system 10a according to this disclosure, the control device 1a suppresses the switch from automatic driving to manual driving when a switch from automatic driving to manual driving is requested and a target of attention is detected from the vehicle image. Alternatively, the control device 1a suppresses the switch depending on the determination result of whether or not the target of attention is located in the vehicle's blind spot. As a result, the control device 1a can suppress the switch from automatic driving to manual driving when a target of attention is present around the vehicle, thereby avoiding danger at the time of switching from automatic driving to manual driving and preventing accidents. With this configuration, the image system 10a can support a safe switch from automatic driving to manual driving.

[0117] Each of the functional components described above—control device 1, control device 1a, image transmission device 2, image transfer device 3, surveillance camera C, and drone D—may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and a program to control them). For example, this disclosure can also be implemented by having a CPU execute a computer program to perform any processing.

[0118] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in various types of non-transitory computer-readable medium or tangible storage medium. Examples, but not limited to, include RAM (Random-Access Memory), ROM (Read-Only Memory), flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may also be transmitted over various types of transient computer-readable medium or communication medium. Examples, but not limited to, include transient computer-readable medium or communication medium, including electrically, optically, acoustically, or otherwise propagating signals.

[0119] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the embodiments described above can be combined in any way. For example, Embodiment 1 and Embodiment 2 may be combined and implemented. [Explanation of symbols]

[0120] 1, 1a Control device 2 Image transmission device 3. Image Transfer Device 10, 10a Image System 11 Image acquisition unit 12 Display Control Unit 13 Image Analysis Department 14 Switching Control Unit 15 Display 17 Communications Department 21 Imaging Department 22 Image acquisition unit 23 Number reading unit 24 Registration Determination Unit 27 Communications Department 31 Registration Confirmation Department 32 Transfer Processing Unit 37 Communications Department 151 Image of my own car C Surveillance Camera D Drone N Network P1 Pedestrian P2 Motorcycle V1, V2 vehicles

Claims

1. An image acquisition unit that acquires an image of the vehicle taken by the vehicle itself, A display control unit that controls the display of the vehicle image, Equipped with, The display control unit causes the vehicle image to be displayed on the display unit for a predetermined period of time after a request is made to switch from automatic driving to manual driving in the vehicle. Control device.

2. An image acquisition unit that acquires an image of the vehicle taken by the vehicle itself, A switching control unit that controls the switching from automatic driving to manual driving in the vehicle, Equipped with, The switching control unit suppresses the switch from automatic to manual driving when a switch from automatic to manual driving is requested and an object of attention is detected from the vehicle image. Control device.

3. When a switch from automatic driving to manual driving is requested and a target of attention is detected from the vehicle image, the switching control unit determines whether the target of attention is located in the vehicle's blind spot, and if it determines that the target of attention is located in the vehicle's blind spot, it suppresses the switch from automatic driving to manual driving. The control device according to claim 2.

4. Computers Image acquisition step to acquire an image of the vehicle by taking an image of the vehicle itself, A display control step that controls the display of the vehicle image on the display unit, Execute, In the display control step, the vehicle image is displayed on the display unit for a predetermined period of time after a request is made to switch from automatic driving to manual driving in the vehicle. Control method.

5. Computers Image acquisition step to acquire an image of the vehicle by taking an image of the vehicle itself, A switching control step that controls the switching from automatic driving to manual driving in the vehicle, Execute, In the aforementioned switching control step, if a switch from automatic driving to manual driving is requested and an object of attention is detected from the vehicle image, the switch from automatic driving to manual driving is suppressed. Control method.

Images