Remote monitoring devices, remote monitoring systems, and remote monitoring methods

Abstract

A remote monitoring device, system, and method are provided, which improve the convenience of remote monitoring performed by a remote operator by providing a surround view at appropriate timing. A remote operator refers to the remote monitoring terminal to remotely support the operation of an autonomous vehicle. The remote monitoring device acquires data from multiple sensors mounted on the autonomous vehicle, performs image generation processing to generate a surround view of the autonomous vehicle by processing the data, calculates the lane width of the lane the autonomous vehicle is traveling in based on map information, calculates the vehicle width ratio of the autonomous vehicle relative to the lane width, and provides the surround view generated by the image generation processing to the remote monitoring terminal if the vehicle width ratio is greater than a predetermined threshold.

Description

Technical Field

[0001] This invention relates to a remote monitoring device, a remote monitoring system, and a remote monitoring method. Background Technology

[0002] Patent Document 1 discloses a technology related to a vehicle remote monitoring system. According to this technology, the vehicle remote monitoring system treats objects within a predetermined distance as obstacles and transforms an image including the obstacles into an image showing a top-down view. The transformed image is then displayed on an image display unit in a control center.

[0003] Existing technical documents

[0004] Patent Document 1: Japanese Patent Application Publication No. 2020-3890 Summary of the Invention

[0005] When a remote operator provides remote support for the driving of an autonomous vehicle, there is a challenge that the remote operator may have difficulty grasping the vehicle's width and spacing. An effective countermeasure is to display an around-view, as described in the aforementioned techniques. However, providing the remote operator with more displays is not necessarily better. This is because providing multiple types of displays at once can confuse the remote operator. Therefore, it is preferable to provide displays to the remote operator at appropriate times to clarify the necessary conditions. In the aforementioned techniques, the timing of providing the around-view to the remote operator has not been sufficiently investigated.

[0006] This disclosure was made in view of the above-mentioned issues, and its object is to provide a technique that can improve the convenience of remote monitoring performed by a remote operator by providing a surround view at appropriate timing.

[0007] This disclosure provides a remote monitoring device for achieving the above-mentioned objectives. The remote monitoring device disclosed herein provides a surround view of an autonomous vehicle to a remote monitoring terminal referenced by a remote operator remotely monitoring the operation of the autonomous vehicle. The remote monitoring device includes: one or more memories storing one or more programs; and one or more processors combined with one or more memories. When executing one or more programs, the one or more processors acquire data from multiple sensors mounted on the autonomous vehicle, perform image generation processing to process the data and generate a surround view of the autonomous vehicle, calculate the lane width of the lane in which the autonomous vehicle is traveling based on map information, calculate the vehicle width ratio of the autonomous vehicle relative to the lane width, and, if the vehicle width ratio is greater than a predetermined threshold, provide the surround view generated by the image generation processing to the remote monitoring terminal.

[0008] In the remote monitoring device disclosed herein, the higher the speed of the autonomous vehicle, the smaller the threshold value set by one or more processors.

[0009] In the remote monitoring device disclosed herein, one or more processors may calculate the distance of the autonomous vehicle to the obstacle ahead based on data from multiple sensors. If the distance is less than a predetermined judgment distance, a surround view generated through image generation processing is provided to the remote monitoring terminal. In the remote monitoring device disclosed herein, the higher the speed of the autonomous vehicle, the larger the judgment distance will be set by one or more processors.

[0010] The remote monitoring device disclosed herein can be mounted on an autonomous vehicle or on a server connected to the autonomous vehicle via a communication network.

[0011] Furthermore, this disclosure provides a remote monitoring system for achieving the aforementioned objectives. The remote monitoring system disclosed herein displays a surround view of an autonomous vehicle to a remote monitoring terminal referenced by a remote operator remotely supporting the operation of the autonomous vehicle. The remote monitoring system includes: an onboard computer mounted in the autonomous vehicle; and a server connected to the onboard computer via a communication network. The onboard computer and the server cooperate to acquire data from multiple sensors mounted in the autonomous vehicle, perform image generation processing to process the data and generate a surround view of the autonomous vehicle, calculate the lane width of the lane in which the autonomous vehicle is traveling based on map information, calculate the vehicle width ratio of the autonomous vehicle relative to the lane width, and, if the vehicle width ratio is greater than a predetermined threshold, display the surround view generated by the image generation processing on the remote monitoring terminal.

[0012] In the remote monitoring system disclosed herein, if the vehicle-mounted computer sends a surround view generated through image generation processing to the server when the vehicle width ratio is greater than a predetermined threshold, the server may also receive the surround view and display it on the remote monitoring terminal.

[0013] Furthermore, this disclosure provides a remote monitoring method for achieving the aforementioned objectives. The remote monitoring method disclosed herein involves a computer performing processing to display a surround view of an autonomous vehicle on a remote monitoring terminal referenced by a remote operator remotely supporting the operation of the autonomous vehicle. In this remote monitoring method, a computer acquires data from multiple sensors mounted on the autonomous vehicle, performs image generation processing to generate a surround view of the autonomous vehicle by processing the data, calculates the lane width of the lane the autonomous vehicle is traveling in based on map information, calculates the vehicle width ratio of the autonomous vehicle relative to the lane width, and, if the vehicle width ratio is greater than a predetermined threshold, displays the surround view generated by the image generation processing on the remote monitoring terminal.

[0014] According to the technology disclosed herein, data from multiple sensors mounted on an autonomous vehicle are processed to generate a surround view of the autonomous vehicle. Then, the vehicle width ratio (the ratio of the autonomous vehicle's width to the lane width of the lane in which the autonomous vehicle is traveling) is calculated. If this ratio is greater than a predetermined threshold, the generated surround view is provided to a remote monitoring terminal. Thus, according to the technology disclosed herein, a surround view can be provided to a remote monitoring terminal referenced by a remote operator at appropriate timing, thereby improving the convenience of remote monitoring performed by a remote operator. Attached Figure Description

[0015] Figure 1 It is a diagram that roughly shows the structure of a remote monitoring system.

[0016] Figure 2 This is a view of an autonomous vehicle from above.

[0017] Figure 3 This is a block diagram illustrating an example of the structure of an autonomous vehicle.

[0018] Figure 4 This is a block diagram illustrating an example of the structure of a remote monitoring center.

[0019] Figure 5 This is a block diagram used to illustrate the functions of the vehicle-mounted computer and the server according to the first embodiment.

[0020] Figure 6 It is a mapping diagram that specifies the relationship between vehicle speed V and the judgment threshold TH1.

[0021] Figure 7 This is a block diagram used to illustrate the functions of the vehicle-mounted computer and the server involved in the second embodiment.

[0022] Figure 8It is a mapping diagram that specifies the relationship between vehicle speed V and judgment distance TH2.

[0023] (Symbol Explanation)

[0024] 10: Communication network; 20: Autonomous vehicle; 21: Onboard computer; 21a: Processor; 21b: Memory; 21c: Program; 22: External sensor; 22b: Camera; 22c: Millimeter-wave sensor; 23: Internal sensor; 24: Actuator; 25: Communication device; 30: Remote monitoring center; 32: Server; 32a: Processor; 32b: Memory; 32c: Program; 34: Remote monitoring terminal; 34a: Information output unit; 34b: Operation input unit; 38: Communication device; 100: Remote monitoring system; 2 11: Vehicle width ratio calculation and processing unit; 212: Judgment threshold setting processing unit; 213: Image generation processing unit; 214: Surround view necessity determination processing unit; 215: Surround view transmission processing unit; 216: Distance calculation processing unit; 217: Judgment threshold setting processing unit; 218: Surround view necessity determination processing unit; 221: Wide-angle camera; 221f: Front camera; 221b: Rear camera; 221l: Left side camera; 221r: Right side camera; 321: Surround view receiving processing unit; 322: Surround view display processing unit. Detailed Implementation

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Where the number, quantity, quantity, range, etc., of each element are mentioned in the embodiments shown below, the technical concept involved in this disclosure is not limited to the mentioned numbers, unless specifically stated or clearly determined in principle. Furthermore, the structures, etc., described in the embodiments shown below are not necessarily essential to the technical concept involved in this disclosure, unless specifically stated or clearly determined in principle.

[0026] 1. General Structure of a Remote Monitoring System

[0027] Figure 1 This is a schematic diagram illustrating the structure of a remote monitoring system common to all embodiments described below. The remote monitoring system 100 is a system for remotely monitoring the autonomous vehicle 20 from a remote monitoring center 30. The autonomous driving level of the autonomous vehicle 20 is envisioned, for example, as Level 4 or Level 5 in the SAE (Society of Automotive Engineers) level definition. Hereinafter, the autonomous vehicle 20 capable of remote monitoring will be simply referred to as vehicle 20.

[0028] The remote monitoring in this disclosure includes monitoring for remotely driving the vehicle 20, monitoring for providing remote support to the autonomously driving vehicle 20 as needed, and monitoring for confirming the surrounding environment of the autonomously driving vehicle 20. Remote monitoring is performed by a remote operator (OP) when it becomes difficult for the vehicle 20 to continue autonomous driving or when prediction becomes difficult.

[0029] Figure 2 This is a view of the autonomous vehicle from above. The vehicle 20 includes a wide-angle camera 221, serving as an external sensor 22, to capture images of the area surrounding the vehicle 20. Typically, the wide-angle camera 221 includes a front camera 221f, a rear camera 221b, a right-side camera 221r, and a left-side camera 221l. The front camera 221f is positioned at the center of the front end of the vehicle 20. The field of view (IAF) of the front camera 221f covers at least the area in front of the vehicle 20 in its direction of travel. The rear camera 221b is positioned at the center of the rear end of the vehicle 20. The field of view (IAB) of the rear camera 221b covers at least the area behind the vehicle 20 in its direction of travel. The right-side camera 221r ​​and the left-side camera 221l are positioned below the left and right rearview mirrors of the vehicle 20, respectively. The field of view (IAR) and the field of view (IAL) of the right-side camera 221r ​​and the left-side camera 221l cover at least the right and left sides of the vehicle 20, respectively. For example, when vehicle 20 is traveling in driving lane DL, the front, rear, right, and left sides of driving lane DL are respectively included in the photography range IAF, IAB, IAR, and IAL.

[0030] Return to Figure 1 The remote monitoring center 30 is equipped with a server 32 and a remote monitoring terminal 34. The vehicle 20 is connected to the server 32 via a communication network 10, including 4G and 5G networks. The number of vehicles 20 that can communicate with the server 32 can be one or more.

[0031] The remote monitoring terminal 34 is an operating terminal used for remote monitoring, monitored by a remote operator (OP). At least one remote monitoring terminal 34 is provided, preferably multiple terminals. The remote monitoring center 30 is equipped with a number of remote operators (OPs) corresponding to the number of remote monitoring terminals 34.

[0032] Each remote monitoring terminal 34 is connected to the server 32 via a communication network, including LAN and the Internet. Furthermore, the remote monitoring center 30 does not necessarily need to be a physical facility. Here, the system that connects the remote monitoring terminals 34 and the server 32 via a communication network is referred to as the remote monitoring center 30. Therefore, the server 32 can be set up in the cloud, and the remote monitoring terminals 34 can be set up in satellite offices in various locations or in the residences of remote operators (OPs).

[0033] Vehicle 20 is equipped with an onboard computer 21. Images captured by wide-angle camera 221 are input to onboard computer 21. Based on the input images, onboard computer 21 generates a surround view as an image of vehicle 20 viewed from above. This process is called "image generation processing". In image generation processing, onboard computer 21 generates an omnidirectional image encompassing the entire surroundings of vehicle 20 by combining images from the IAF, IAB, IAR, and IAL ranges. Onboard computer 21 retains vehicle image data from a top-down view of vehicle 20. Onboard computer 21 generates the surround view by synthesizing vehicle image data from the omnidirectional images.

[0034] The surround view generated in the image generation process is sent from the vehicle-mounted computer 21 to the server 32 of the remote monitoring center 30 via the communication network 10. In the remote monitoring center 30, the surround view sent from the vehicle-mounted computer 21 to the server 32 is displayed on the information output unit 34a of the remote monitoring terminal 34. The information output unit 34a can be, for example, a liquid crystal display, an organic EL display, a head-mounted display, a touch panel, or other display device. Since the remote operator OP does not actually ride in the vehicle 20, it is difficult to experience the width of the vehicle. The remote operator OP can efficiently monitor the surroundings of the vehicle 20 by viewing the surround view projected on the information output unit 34a of the remote monitoring terminal 34. For example, when the vehicle 20 is traveling in the driving lane DL, the remote operator OP can identify the relative position of the vehicle 20 with respect to the center line CL and the outer lane line OL on the information output unit 34a of the remote monitoring terminal 34.

[0035] 2. Research on Remote Monitoring Systems

[0036] As described above, according to the remote monitoring system 100, the remote operator (OP) can monitor the vehicle 20 by referring to a surround view in the remote monitoring center 30. However, the remote operator (OP) does not necessarily need to refer to the surround view all the time. That is, regarding the surround view, it may be useful information in scenarios such as driving on narrow roads where it is necessary to know the lateral position of the vehicle 20 relative to the driving lane. However, in scenarios such as driving in a straight line along a wide driving lane, the necessity of referring to the surround view is considered to be lower. In addition, the screen size of the information output unit 34a referred to by the remote operator (OP) is limited. Therefore, in scenarios where the necessity of the surround view is low, the remote operator (OP) may also find the display of the surround view cumbersome.

[0037] 3. Overview of the characteristics of a remote monitoring system

[0038] In the remote monitoring system 100, control is performed to provide a surround view at appropriate timings corresponding to the road structure. Specifically, the on-board computer 21 calculates the vehicle width ratio (%) of the vehicle 20 relative to the lane width of the driving lane. This process will be referred to as the "vehicle width ratio calculation process". In the vehicle width ratio calculation process, the vehicle width ratio is calculated based on the current position of the vehicle 20, the road width contained in the map information, and the number of driving lanes.

[0039] The onboard computer 21 determines whether a surround view needs to be displayed by comparing the calculated vehicle width ratio (%) with a predetermined threshold (%). This process will be referred to as the "surround view necessity determination process." The larger the vehicle width ratio (%), the more difficult it is to maintain the driving lane. That is, the larger the vehicle width ratio of vehicle 20, the greater the necessity of a surround view. Therefore, in the surround view necessity determination process, the onboard computer 21 determines whether the calculated vehicle width ratio (%) is greater than the predetermined threshold (%).

[0040] Regarding the judgment threshold, a predetermined fixed value can be used as the threshold for determining whether there is concern that vehicle 20 may be leaving its driving lane DL during remote monitoring. Alternatively, the judgment threshold can be set based on the speed of vehicle 20. This process will be referred to as the "judgment threshold setting process" below. In the judgment threshold setting process, the higher the speed of vehicle 20, the smaller the percentage (%) the judgment threshold will be set. The higher the speed of vehicle 20, the more difficult it is to maintain its driving lane. That is, the higher the speed of vehicle 20, the more necessary a surround view becomes. Therefore, according to the judgment threshold setting process, a judgment threshold corresponding to the necessity of a surround view can be set.

[0041] In the surround view necessity determination process, if it is determined that the vehicle width ratio (%) is greater than the determination threshold (%), the onboard computer 21 sends the surround view to the server 32 of the remote monitoring center 30. The server 32 displays the received surround view on the information output unit 34a of the remote monitoring terminal 34. According to such control, the surround view can be displayed to the remote operator (OP) at an appropriate timing corresponding to the road structure.

[0042] The detailed structure and function of the remote monitoring system 100 are described below. As explained herein, the remote monitoring system 100 is technically designed to provide a surround view at appropriate timings.

[0043] 4. The structure of the vehicle according to the first embodiment

[0044] Figure 3This is a block diagram illustrating an example of the structure of a vehicle 20 in the remote monitoring system 100 according to the first embodiment. The vehicle 20 includes an on-board computer 21. The on-board computer 21 is an assembly of multiple ECUs (Electronic Control Units) mounted on the vehicle 20. Additionally, the vehicle 20 includes external sensors 22, internal sensors 23, actuators 24, and communication devices 25. These are connected to the on-board computer 21 via an in-vehicle network such as CAN (Controller Area Network).

[0045] The vehicle-mounted computer 21 includes one or more processors 21a (hereinafter referred to as processor 21a) and one or more memories 21b (hereinafter referred to as memory 21b) combined with the processor 21a. The memory 21b stores one or more programs 21c (hereinafter referred to as programs 21c) that can be executed by the processor 21a and various information associated therewith.

[0046] The processor 21a performs various processing tasks by executing program 21c. Program 21c includes, for example, programs for implementing autonomous driving and programs for implementing remote driving or remote support. Additionally, program 21c includes a remote monitoring program that enables the onboard computer 21 to function as a remote monitoring device (described later). Program 21c is stored in memory 21b. Furthermore, memory 21b may also store a map database for managing map information used in autonomous driving.

[0047] In addition to the wide-angle camera 221 mentioned above, the external sensor 22 also includes other cameras that capture images of the surroundings of the vehicle 20, especially the front. These cameras can be either monocular or stereo cameras. Multiple cameras can be installed, and in addition to capturing images of the front of the vehicle 20, images can also be captured of the sides and rear. Furthermore, cameras can be shared for both autonomous driving and remote operation, or separate cameras can be installed for autonomous driving and remote operation.

[0048] External sensors 22 include identification sensors other than cameras. Identification sensors are used to identify the surrounding conditions of the vehicle 20. Examples of identification sensors other than cameras include LiDAR (Laser Imaging Detection and Ranging) and millimeter-wave radar. Additionally, external sensors 22 include position sensors that detect the position and orientation of the vehicle 20. Examples of position sensors include GPS (Global Positioning System) sensors. Information obtained from external sensors 22 is transmitted to the onboard computer 21.

[0049] The internal sensors 23 include state sensors that acquire information related to the motion of the vehicle 20. Examples of state sensors include wheel speed sensors, acceleration sensors, angular velocity sensors, and steering angle sensors. Acceleration sensors and angular velocity sensors may also be IMUs (Integrated Mutual Detectors). The information obtained from the internal sensors 23 is transmitted to the onboard computer 21.

[0050] Actuator 24 includes a steering mechanism for steering vehicle 20, a drive mechanism for driving vehicle 20, and a braking mechanism for braking vehicle 20. The steering mechanism may include, for example, a power steering system, a steer-by-wire system, and a rear-wheel steering system. The drive mechanism may include, for example, an engine, an EV system, and a hybrid system. The braking mechanism may include, for example, a hydraulic brake and a regenerative braking system. Actuator 24 operates according to control signals sent from onboard computer 21.

[0051] The communication device 25 is a device for controlling wireless communication with the outside world of the vehicle 20. The communication device 25 communicates with the server 32 via the communication network 10. Information processed by the onboard computer 21 is sent to the server 32 using the communication device 25. Information processed by the server 32 is retrieved into the onboard computer 21 using the communication device 25. Furthermore, in situations where vehicle-to-vehicle communication or road-to-road communication with infrastructure is required for autonomous driving, the communication device 25 also communicates with these external devices.

[0052] 5. Structure of the remote monitoring center according to the first embodiment

[0053] Figure 4This is a block diagram illustrating an example of the structure of a remote monitoring center 30 in the remote monitoring system 100 according to the first embodiment. The remote monitoring center 30 includes a server 32. The server 32 is a computer or a collection of multiple computers connected by a communication network. Additionally, the remote monitoring center 30 includes remote monitoring terminals 34 and communication devices 38. These are connected to the server 32 via a communication network. As described above, one or more remote monitoring terminals 34 may be connected to the server 32.

[0054] Server 32 includes one or more processors 32a (hereinafter referred to as processor 32a) and one or more memories 32b (hereinafter referred to as memory 32b) combined with processor 32a. In memory 32b, one or more programs 32c (hereinafter referred to as program 32c) that can be executed by processor 32a and various information associated therewith are stored.

[0055] The processor 32a performs various processing tasks by executing program 32c. Program 32c may include, for example, programs for enabling remote driving or remote support. Program 32c can be stored in memory 32b or a computer-readable recording medium serving as auxiliary storage. Additionally, memory 32b may store a map database for managing map information used in autonomous driving. The map database may be stored in at least one of server 32 and vehicle computer 21.

[0056] The remote monitoring terminal 34 includes an information output unit 34a. The information output unit 34a is a device that outputs information required for remote monitoring of the vehicle 20 to a remote operator (OP). The information output unit 34a includes a display device for outputting images. In addition to the aforementioned surround view, the display device can, for example, display images of the front of the vehicle 20 captured by a camera. As a display method, for example, the same view as observed from the driver's seat of the vehicle 20 can be displayed on the screen. The screen can also have multiple display screens and can display a surround view of the vehicle 20.

[0057] The remote monitoring terminal 34 includes an operation input unit 34b. The operation input unit 34b includes devices for inputting remote driving operations for the remote operator (OP). To simulate the operations required when actually driving the vehicle 20, the operation input unit 34b includes a steering wheel for steering operations, an accelerator pedal for acceleration operations, and a brake pedal for deceleration operations. Additionally, if the vehicle 20 has a transmission, the operation input unit 34b may also include a transmission control lever or switch. Furthermore, the operation input unit 34b includes a lever for operating the vehicle 20's direction indicator, a lever for operating the windshield wipers, and other devices for inputting operations required for safe driving. Alternatively, the operation input unit 34b may include devices for inputting remote support operations for the remote operator (OP). Specific examples of input devices include buttons, levers, and touch panels. For example, depending on the direction the lever is pushed, the vehicle 20 may be instructed to move forward / stop or to move laterally. Lateral movement may include, for example, avoiding deviations from obstacles ahead, lane changes, or overtaking of oncoming vehicles.

[0058] Communication device 38 is a device for controlling communication with the outside world of remote monitoring center 30. Communication device 38 communicates with one or more vehicles 20 via communication network 10. Information processed by server 32 is sent to vehicle 20 using communication device 38. Information processed by vehicle 20 is retrieved from server 32 using communication device 38.

[0059] In the case of remote driving, the remote operator (OP) operates the operation input unit (34b) while referring to the image displayed on the information output unit (34a). The server (32) of the remote monitoring center (30) obtains the remote operation information operated by the remote operator (OP) on the operation input unit (34b). Specifically, the remote operation information is transmitted to the vehicle (20) via the communication network (10). The on-board computer (21) of the vehicle (20) operates the actuators (24) of the vehicle (20) according to the remote operation information.

[0060] 6. Description of the remote monitoring method of the remote monitoring system according to the first embodiment

[0061] Able to have Figure 5 The remote monitoring system 100 with the structure shown implements the remote monitoring method of the remote monitoring system. Figure 5 The functions of the vehicle-mounted computer 21 and the server 32 are illustrated using modules. The remote monitoring system 100 will be described below focusing on the functions of the vehicle-mounted computer 21 and the server 32. Explanations of already described structures and functions will be omitted or simplified.

[0062] The vehicle-mounted computer 21 includes a vehicle width ratio calculation processing unit 211, a determination threshold setting processing unit 212, an image generation processing unit 213, a surround view necessity determination processing unit 214, and a surround view transmission processing unit 215. These functions of the vehicle-mounted computer 21 are implemented by the processor 21a executing the program 21c stored in the memory 21b of the vehicle-mounted computer 21.

[0063] The vehicle width ratio calculation processing unit 211 is a functional block for performing vehicle width ratio calculation processing. In this process, the vehicle width ratio calculation processing unit 211 identifies the position of the vehicle 20 on the map based on the position information of the vehicle 20 received from GPS 22d, information related to the movement of the vehicle 20 detected by the internal sensor 23, and map information obtained from a map database. Furthermore, the vehicle width ratio calculation processing unit 211 can also infer the position of the vehicle 20 based on the relative positions of features detected by LiDAR 22a, camera 22b, or millimeter-wave sensor 22c relative to the vehicle 20, information related to the movement of the vehicle 20 detected by the internal sensor 23, and the positions of the detected features on the map. The map information stores information on the road width and the number of lanes corresponding to the position on the map. The vehicle width ratio calculation processing unit 211 obtains the road width and the number of lanes at the location of the vehicle 20 on the map. Then, the vehicle width ratio calculation processing unit 211 calculates the average lane width of one lane by dividing the obtained road width by the number of lanes. The memory 21b stores vehicle width information of the vehicle 20. The vehicle width ratio calculation processing unit 211 calculates the vehicle width ratio WP (%) relative to the calculated lane width as the vehicle width ratio. The calculated vehicle width ratio WP is sent to the surround view necessity determination processing unit 214.

[0064] The determination threshold setting processing unit 212 is a function block for performing determination threshold setting processing. In the determination threshold setting processing, the determination threshold setting processing unit 212 calculates the current vehicle speed V of the vehicle 20 based on information related to the movement of the vehicle 20 detected by the internal sensor 23. Figure 6 This is a mapping diagram that defines the relationship between vehicle speed V and the judgment threshold TH1. The memory 21b stores... Figure 6 The mapping diagram shown. In this mapping diagram, it is stipulated that the larger the vehicle speed V, the smaller the determination threshold TH1 becomes. The determination threshold setting processing unit 212 follows... Figure 6 The mapping diagram shown is used to set a determination threshold TH1 corresponding to the vehicle speed V. Alternatively, the determination threshold setting processing unit 212 may set a fixed value independent of the vehicle speed V as the determination threshold TH1. The set determination threshold TH1 is sent to the surround view necessity determination processing unit 214.

[0065] Image generation processing unit 213 is a functional block for performing image generation processing. In image generation processing, image generation processing unit 213 acquires images (IMGF, IMGB, IMGR, IMGL) of the photographic ranges IAF, IAB, IAR, and IAL captured by the front camera 221f, rear camera 221b, right-side camera 221r, and left-side camera 221l. Then, image generation processing unit 213 generates an omnidirectional image surrounding the entire perimeter of vehicle 20 by combining these images using known image processing techniques. Memory 21b stores vehicle image data from a top-down view of vehicle 20. Image generation processing unit 213 generates a surround view AIMG by synthesizing vehicle image data from the omnidirectional image. The generated surround view AIMG is sent to surround view transmission processing unit 215.

[0066] The surround view necessity determination processing unit 214 is a functional block for performing surround view necessity determination. In the surround view necessity determination, the surround view necessity determination processing unit 214 determines whether the vehicle width ratio WP is greater than a determination threshold TH1. As a result, if the vehicle width ratio WP is less than or equal to the determination threshold TH1, the surround view necessity determination processing unit 214 determines that a surround view is not needed and sends its determination result signal DS to the surround view transmission processing unit 215. On the other hand, if the vehicle width ratio WP is greater than the determination threshold TH1, the surround view necessity determination processing unit 214 determines that a surround view is needed and sends its determination result signal DS to the surround view transmission processing unit 215.

[0067] When the determination result signal DS sent from the surround view necessity determination processing unit 214 is a result signal indicating that a surround view is needed, the surround view transmission processing unit 215 uses the communication device 25 to send the surround view AIMG to the server 32 of the remote monitoring center 30. As described above, the vehicle computer 21 according to the first embodiment functions as a remote monitoring device that provides the surround view AIMG of the vehicle 20 to the remote monitoring terminal 34.

[0068] Server 32 includes a surround view receiving processing unit 321 and a surround view display processing unit 322. They are implemented as the functions of server 32 when processor 32a executes program 32c stored in memory 32b of server 32.

[0069] The surround view receiving and processing unit 321 uses the communication device 38 to receive the surround view AIMG. The received surround view AIMG is sent to the surround view display processing unit 322.

[0070] The surround view display processing unit 322 displays the surround view AIMG on the information output unit 34a of the remote monitoring terminal 34. With this control, the surround view can be displayed to the remote operator (OP) at an appropriate timing corresponding to the road structure.

[0071] 7. Description of the remote monitoring method of the remote monitoring system according to the second embodiment

[0072] Next, use Figure 7 as well as Figure 8 The remote monitoring method of the remote monitoring system according to the second embodiment of this disclosure will be described. In the remote monitoring method according to the first embodiment, a surround view necessity determination process is performed to determine whether a surround view AIMG of the vehicle 20 needs to be provided based on whether the vehicle width ratio WP is greater than a determination threshold TH1. In the remote monitoring method according to the second embodiment, the determination of whether a surround view AIMG needs to be provided is based on the distance between the vehicle 20 and an obstacle in its direction of travel.

[0073] exist Figure 7 In this document, the functions of the vehicle-mounted computer 21 and the server 32 according to the second embodiment are illustrated using modules. Hereinafter, the remote monitoring system according to the second embodiment will be described focusing on the functions of the vehicle-mounted computer 21 and the server 32. Wherein, structures and functions already described are omitted or simplified by using common symbols.

[0074] The vehicle-mounted computer 21 involved in the second embodiment is relative to Figure 5 The vehicle-mounted computer 21 according to the first embodiment shown also includes a distance calculation processing unit 216, a determination threshold setting processing unit 217, and a surround view necessity determination processing unit 218. These functions of the vehicle-mounted computer 21 are implemented when the processor 21a executes the program 21c stored in the memory 21b of the vehicle-mounted computer 21.

[0075] The distance calculation processing unit 216 is a functional block for performing distance calculation processing. In this process, the distance calculation processing unit 216 uses methods such as pattern matching and deep learning to identify objects around the vehicle 20 based on information received from the LiDAR 22a, camera 22b, or millimeter-wave sensor 22c, and calculates the distance OD from the vehicle 20. The objects identified by the distance calculation processing unit 216 include, for example, moving objects such as vehicles, motorcycles, bicycles, and pedestrians, as well as stationary objects. The calculated distance OD is then sent to the surround view necessity determination processing unit 218.

[0076] The determination threshold setting processing unit 217 is a function block that performs setting processing to set the determination threshold for distance OD. In the determination threshold setting processing, the determination threshold setting processing unit 217 calculates the current vehicle speed V of the vehicle 20 based on information related to the movement of the vehicle 20 detected by the internal sensor 23. Figure 8 This is a mapping diagram that defines the relationship between vehicle speed V and determination distance TH2. The memory 21b stores... Figure 8 The mapping diagram shown. In this mapping diagram, it is stipulated that the larger the vehicle speed V, the smaller the determination distance TH2 becomes. The determination threshold setting processing unit 217 follows... Figure 8 The mapping diagram shown sets a determination distance TH2 corresponding to the vehicle speed V. Alternatively, the determination threshold setting processing unit 217 may set a fixed value independent of the vehicle speed V as the determination distance TH2. The set determination distance TH2 is sent to the surround view necessity determination processing unit 218.

[0077] The surround view necessity determination processing unit 218 is a function block for performing surround view necessity determination. In the surround view necessity determination, the surround view necessity determination processing unit 218 determines whether the distance OD is less than the determination distance TH2. As a result, if the distance OD is greater than or equal to the determination distance TH2, the surround view necessity determination processing unit 218 determines that a surround view is not needed and sends its determination result signal DS to the surround view transmission processing unit 215. On the other hand, if the distance OD is less than the determination distance TH2, the surround view necessity determination processing unit 218 determines that a surround view is needed and sends its determination result signal DS to the surround view transmission processing unit 215.

[0078] Based on the necessity determination of the surround view involved in this second embodiment, it can be determined that a surround view is needed even when the vehicle 20 is close to an obstacle. Therefore, the remote operator (OP) can refer to the surround view to understand the relative positional relationship between the obstacle and the vehicle 20.

[0079] 8. Other implementation methods

[0080] The server 32 can also be configured as a remote monitoring device. That is, the server 32 can have all the functions of the vehicle-mounted computer 21 as a remote monitoring device. Alternatively, the server 32 can have some of the functions of the vehicle-mounted computer 21 as a remote monitoring device, with the vehicle-mounted computer 21 and the server 32 cooperating in performing various processes.

Claims

1. A remote monitoring device that provides a surround view of an autonomous vehicle to a remote monitoring terminal referenced by a remote operator who remotely monitors the operation of the autonomous vehicle, characterized in that, The remote monitoring device includes: One or more storage devices that store one or more programs; as well as One or more processors, combined with one or more memories, When the one or more processors execute the one or more programs Data is acquired from multiple sensors mounted on the autonomous vehicle. An image generation process is performed to process the data and generate a surround view of the autonomous vehicle. The lane width of the lane in which the autonomous vehicle travels is calculated based on map information. Calculate the ratio of the width of the autonomous vehicle to the width of the lane. The higher the speed of the autonomous vehicle, the smaller the threshold value for determining the vehicle width ratio will be set by the one or more processors. If the vehicle width ratio is greater than the determination threshold, the surround view generated by the image generation process will be provided to the remote monitoring terminal. The one or more processors calculate the distance of the autonomous vehicle up to the obstacle ahead based on the data from the multiple sensors. If the distance is less than a predetermined judgment distance, the surround view generated by the image generation process is provided to the remote monitoring terminal. The higher the speed of the autonomous vehicle, the larger the value that the one or more processors will set for the determination distance.

2. An autonomous vehicle, characterized in that, It is equipped with the remote monitoring device as described in claim 1.

3. A server, characterized in that, Equipped with the remote monitoring device as described in claim 1, The server is connected to the autonomous vehicle via a communication network.

4. A remote monitoring system that displays a surround view of an autonomous vehicle to a remote monitoring terminal referenced by a remote operator remotely supporting the operation of the autonomous vehicle, characterized in that, The remote monitoring system has the following features: Onboard computer, mounted in the autonomous vehicle; as well as The server is connected to the onboard computer via a communication network. The onboard computer and the server work together. Data is acquired from multiple sensors mounted on the autonomous vehicle. An image generation process is performed to process the data and generate a surround view of the autonomous vehicle. The lane width of the lane in which the autonomous vehicle travels is calculated based on map information. Calculate the ratio of the width of the autonomous vehicle to the width of the lane. The higher the speed of the autonomous vehicle, the smaller the threshold value that the onboard computer sets for determining the vehicle width ratio. If the vehicle width ratio is greater than the determination threshold, the surround view generated by the image generation process is displayed on the remote monitoring terminal. The onboard computer calculates the distance of the autonomous vehicle up to the obstacle ahead based on data from the multiple sensors. If the distance is less than a predetermined judgment distance, the surround view generated by the image generation process is provided to the remote monitoring terminal. The higher the speed of the autonomous vehicle, the higher the value that the onboard computer will set for the determination distance.

5. The remote monitoring system according to claim 4, characterized in that, If the vehicle width ratio is greater than the threshold, the onboard computer will send the surround view generated by the image generation process to the server. The server receives the surround view and displays the received surround view on the remote monitoring terminal.

6. A remote monitoring method, comprising a computer performing processing to display a surround view of an autonomous vehicle on a remote monitoring terminal referenced by a remote operator remotely supporting the operation of the autonomous vehicle, characterized in that, In the computer, Data is acquired from multiple sensors mounted on the autonomous vehicle. An image generation process is performed to process the data and generate a surround view of the autonomous vehicle. The lane width of the lane in which the autonomous vehicle travels is calculated based on map information. Calculate the ratio of the width of the autonomous vehicle to the width of the lane. The higher the speed of the autonomous vehicle, the smaller the threshold value for determining the vehicle width ratio will be. If the vehicle width ratio is greater than the determination threshold, the surround view generated by the image generation process is displayed on the remote monitoring terminal. The distance of the autonomous vehicle up to the obstacle ahead is calculated based on data from the multiple sensors. If the distance is less than a predetermined judgment distance, the surround view generated by the image generation process is provided to the remote monitoring terminal. The higher the speed of the autonomous vehicle, the larger the value of the determination distance will be.

Images