Remote monitoring system, remote monitoring method, remote monitoring server, and in-vehicle information processing device

Abstract

To provide a remote monitoring technique capable of improving the accuracy of detection results of an object displayed on a monitoring screen while reducing communication costs.SOLUTION: An on-vehicle information processing device 20 compresses an image IMG from a camera 12 and transmits first data DT1 including the compressed image CIMG and a time stamp TS indicating acquisition time of the image IMG. The on-vehicle information processing device 20 also transmits second data DT2 including an object detection result ODR based on the image IMG and the time stamp TS indicating the acquisition time of the image IMG. A remote monitoring server 40 receives the first data DT1 and the second data DT2 and stores them in a memory. The remote monitoring server 40 extracts the compressed image CIMG and the object detection result ODR that both have the time stamp TS of the same time from the memory in chronological order, and superimposes a restored image DIMG obtained by restoring the compressed image CIMG and the object detection result ODR and displays them on a monitoring screen.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a remote monitoring system, a remote monitoring method, a remote monitoring server, and an in-vehicle information processing device that are suitable for use in remote monitoring of a vehicle. [Background technology]

[0002] Patent Document 1 discloses a technology related to a remote control device that remotely controls an autonomous vehicle that is no longer capable of autonomous driving. When an image captured by a camera mounted on the autonomous vehicle is displayed on a display unit, the remote control device according to this prior art analyzes the captured image and highlights specific objects in the image.

[0003] Images captured by a camera are compressed before being transmitted from the autonomous vehicle to the remote control device. Image compression is essential for reducing communication costs, but compression reduces the image resolution. As a result, the above-mentioned conventional technology is unable to accurately analyze images, and there is a risk that specific objects will not be highlighted appropriately.

[0004] In addition to Patent Document 1 above, Patent Documents 2 and 3 below can be cited as examples of prior art documents that represent the technical level of the technical field to which the present application pertains. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-161039 [Patent Document 2] Japanese Patent Application Publication No. 2018-56838 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-96063 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and has as its first object to provide a remote monitoring system and a remote monitoring method that can improve the accuracy of object detection results displayed on a monitoring screen while reducing communication costs.

[0007] A second object of the present disclosure is to provide a remote monitoring server that can improve the accuracy of object detection results displayed on the monitoring screen while reducing communication costs with an in-vehicle information processing device.

[0008] Furthermore, a third object of the present disclosure is to provide an in-vehicle information processing device that can improve the accuracy of object detection results displayed on the monitoring screen of the remote monitoring server while reducing communication costs between the remote monitoring server and the in-vehicle information processing device. [Means for solving the problem]

[0009] The remote monitoring system according to the present disclosure includes a camera mounted on a vehicle, an in-vehicle information processing device mounted on the vehicle and connected to the camera, and a remote monitoring server connected to the in-vehicle information processing device via a communication network. The in-vehicle information processing device performs the following object detection processing, compression processing, first data transmission processing, and second data transmission processing. The remote monitoring server performs the following first data reception processing, second data reception processing, extraction processing, and display processing.

[0010] In the object detection process by the in-vehicle information processing device, an object ahead of the vehicle is detected from an image captured by a camera. In the compression process, the image is compressed to generate a compressed image. In the first data transmission process, first data including the compressed image and the image capture time of the image before compression is transmitted to a remote monitoring server. In the second data transmission process, second data including the object detection result and the image capture time of the image used for object detection is transmitted to a remote monitoring server.

[0011] In the first data reception process by the remote monitoring server, first data transmitted from the in-vehicle information processing device is received and stored in memory. In the second data reception process, second data transmitted from the in-vehicle information processing device is received and stored in memory. In the extraction process, compressed images and object detection results having the same image acquisition time are extracted in chronological order from the memory. In the display process, a restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on the monitoring screen.

[0012] In this remote monitoring system, the in-vehicle information processing device may further execute a third data transmission process, and the remote monitoring server may further execute a third data reception process. The third data transmission process involves transmitting third data including sensor information obtained from a sensor signal of an in-vehicle sensor and a signal acquisition time of the sensor signal. The third data reception process involves receiving the third data transmitted from the in-vehicle information processing device and storing it in memory. In this case, the extraction process by the remote monitoring server involves extracting, from the memory, compressed images and object detection results having the same image acquisition time in chronological order, and extracting sensor information having the same signal acquisition time as the image acquisition time in chronological order. Furthermore, the display process involves superimposing the extracted sensor information together with the extracted object detection results on the restored image and displaying them on the monitoring screen.

[0013] A remote monitoring method according to the present disclosure is a method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network. The remote monitoring method causes the in-vehicle information processing device to detect an object ahead of the vehicle from images captured by a camera mounted on the vehicle, compress the images to generate compressed images, transmit first data including the compressed image and the image capture times of the uncompressed images, and transmit second data including object detection results and the image capture times of images used for the object detection. The remote monitoring method also causes the remote monitoring server to receive and store the first data in a memory, receive and store the second data in a memory, extract from the memory compressed images and object detection results that have the same image capture times in chronological order, and display on a monitoring screen a restored image obtained by restoring the extracted compressed images and the extracted object detection results superimposed on the restored images.

[0014] The remote monitoring method may include causing the in-vehicle information processing device to transmit third data including sensor information obtained from a sensor signal of the in-vehicle sensor and a signal acquisition time of the sensor signal. The remote monitoring method may also include causing the remote monitoring server to receive the third data and store it in a memory, extracting from the memory, in chronological order, compressed images and object detection results having the same image acquisition time as the image acquisition time, and extracting, in chronological order, sensor information having the same signal acquisition time as the image acquisition time, and displaying the extracted sensor information and the object detection result superimposed on the restored image on a monitoring screen.

[0015] The remote monitoring server according to the present disclosure is a server connected to an in-vehicle information processing device via a communication network. The remote monitoring server includes one or more memories storing one or more programs, and one or more processors coupled to the one or more memories. When the one or more programs are executed, the one or more processors perform the following processes:

[0016] In the process executed by one or more processors of the remote monitoring server, first data including a compressed image generated by compressing an image captured by an in-vehicle camera and the image capture time of the uncompressed image is received from the in-vehicle information processing device and stored in one or more memories. Second data including an object detection result based on the image and the image capture time of the image used for the object detection is received from the in-vehicle information processing device and stored in one or more memories. Furthermore, compressed images and object detection results having the same image capture time are extracted in chronological order from the one or more memories. Furthermore, a restored image obtained by restoring the extracted compressed image and the extracted object detection result are displayed superimposed on a monitoring screen.

[0017] The in-vehicle information processing device according to the present disclosure is a device connected to a remote monitoring server via a communication network. The in-vehicle information processing device includes one or more memories storing one or more programs, and one or more processors coupled to the one or more memories. When the one or more programs are executed, the one or more processors perform the following processes:

[0018] The processing executed by one or more processors of the in-vehicle information processing device includes detecting an object ahead of the vehicle from an image captured by an in-vehicle camera, compressing the image to generate a compressed image, transmitting first data including the compressed image and the image capture time of the image before compression to a remote monitoring server, and transmitting second data including the object detection result and the image capture time of the image used for the object detection to the remote monitoring server. [Effects of the Invention]

[0019] According to the present disclosure, object detection is performed from images before compression, thereby preventing a decrease in detection accuracy that occurs when object detection is performed from compressed images. This makes it possible to improve the accuracy of object detection results displayed on the monitoring screen while reducing communication costs with the in-vehicle information processing device. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating a schematic configuration of a remote monitoring system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a diagram showing an example of an image before compression for explaining a problem with the remote monitoring system. [Figure 3] FIG. 10 is a diagram illustrating an example of an object detection result from an image before compression, for explaining a problem with the remote monitoring system. [Figure 4] FIG. 10 is a diagram showing an example of a compressed image for explaining a problem with the remote monitoring system. [Figure 5] FIG. 10 is a diagram illustrating an example of an object detection result from a compressed image for explaining a problem with a remote monitoring system. [Figure 6] FIG. 1 is a diagram illustrating an overview of features of a remote monitoring system according to an embodiment of the present disclosure. [Figure 7] 1 is a block diagram showing a configuration of a vehicle according to a first embodiment of the present disclosure. [Figure 8] 1 is a block diagram showing a configuration of a remote monitoring center according to a first embodiment of the present disclosure. [Figure 9] 2 is a block diagram showing processes executed by each processor of the in-vehicle information processing device and the remote monitoring server according to the first embodiment of the present disclosure. FIG. [Figure 10] FIG. 4 is a block diagram showing the configuration of a vehicle according to a second embodiment of the present disclosure. [Figure 11] FIG. 10 is a block diagram showing the configuration of a remote monitoring center according to a second embodiment of the present disclosure. [Figure 12] FIG. 10 is a block diagram showing processes executed by each processor of an in-vehicle information processing device and a remote monitoring server according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, when the numbers, quantities, amounts, ranges, etc. of each element are mentioned in the embodiments described below, the concept of the present disclosure is not limited to the mentioned numbers unless otherwise specified or clearly specified in principle. Furthermore, the structures, etc. described in the embodiments described below are not necessarily essential to the concept of the present disclosure unless otherwise specified or clearly specified in principle.

[0022] 1. Outline of the remote monitoring system 1 is a diagram schematically illustrating the configuration of a remote monitoring system according to an embodiment of the present disclosure. The remote monitoring system 2 is a system in which a vehicle 10 traveling on a road is remotely monitored by a remote operator 50. The remote monitoring in the present disclosure includes monitoring for remotely driving the vehicle 10, monitoring for remotely assisting the autonomously traveling vehicle 10 according to the situation, and monitoring for checking the surrounding environment of the autonomously traveling vehicle 10.

[0023] A vehicle 10 to be remotely monitored is equipped with a camera 12. The camera 12 may be a stereo camera or a monocular camera. The imaging range IA of the camera 12 covers at least the area ahead in the direction of travel of the vehicle 10. For example, when the vehicle 10 is traveling in a driving lane DL, the area ahead of the driving lane DL and both the left and right sides of the driving lane DL are included in the imaging range IA of the camera 12.

[0024] The vehicle 10 is equipped with an on-board information processing device 20. Images captured by a camera 12 are input to the on-board information processing device 20. The on-board information processing device 20 is connected to a remote monitoring server 40 of a remote monitoring center 30 via a communication network 8. The communication network 8 includes mobile communications such as 4G and 5G. The images captured by the camera 12 are sent from the on-board information processing device 20 to the remote monitoring server 40 via the communication network 8.

[0025] At the remote monitoring center 30, the image from the camera 12 sent from the in-vehicle information processing device 20 to the remote monitoring server 40 is displayed on the monitoring screen 32a of the display device 32. The display device 32 may be, for example, a liquid crystal display, an organic EL display, a head-mounted display, or a touch panel. The remote operator 50 can monitor the area ahead of the vehicle 10 using the image from the camera 12 displayed on the monitoring screen 32a of the display device 32. For example, if an obstacle OJ is present within the shooting range IA of the camera 12, the remote operator 50 can recognize the presence of the obstacle OJ on the monitoring screen 32a of the display device 32.

[0026] The remote monitoring system 2 has a function to assist the remote operator 50 in recognizing an obstacle OJ. An obstacle OJ captured in an image from the camera 12 can be automatically detected using image analysis techniques such as pattern matching and deep learning. By using this object detection function to identify the obstacle OJ captured in the image, it is possible to inform the remote operator 50 of where the obstacle OJ is located on the monitoring screen 32a.

[0027] On the other hand, a reduction in communication costs is required for the remote monitoring system 2. Images captured by the camera 12 are encoded in the in-vehicle information processing device 20, and the encoded image data is transmitted to the remote monitoring server 40. In this encoding, the images captured by the camera 12 are compressed using a predetermined compression method. The image data transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is decoded in the remote monitoring server 40. The image restored by decoding by the remote monitoring server 40 is displayed on the display device 32.

[0028] There are no limitations on the image transmission format or image compression format. If the image transmission format is a video stream format, the image compression format can be, for example, H.264, H.265, VP8, VP9, ​​VP10, or AV1. If the image transmission format is a format in which images are transmitted one by one, the image compression format can be, for example, JPEG, Motion JPEG, or PNG. Preferably, H.264, VP8, or VP9 is used. Motion JPEG, VP10, and AV1 are also suitable compression formats for the remote monitoring system 2.

[0029] 2. Issues with remote monitoring systems As described above, the remote monitoring system 2 is required to assist the remote operator 50 by detecting objects from images captured by the camera 12, and to reduce communication costs by compressing the images captured by the camera 12. However, it is not easy to simultaneously satisfy these two requirements.

[0030] Here, four images are shown. FIG. 2 is a diagram showing an example of an original image captured by camera 12, i.e., an image before compression. FIG. 3 is a diagram showing an example of an object detection result from the original image shown in FIG. 2. FIG. 4 is a diagram showing an example of an image obtained by compressing the original image shown in FIG. 2, i.e., an example of an image after compression. FIG. 5 is a diagram showing an example of an object detection result from the image after compression shown in FIG. 4. Note that these images are originally color images, but have been binarized in the figures for convenience as application documents.

[0031] The image displayed on the monitoring screen 32a of the display device 32 is the compressed image shown in Fig. 4. However, the resolution of the compressed image is lower than that of the original image. This is also evident from the binarized image, as can be seen from a comparison between the image shown in Fig. 2 and the image shown in Fig. 4. For this reason, while two people and two vehicles are detected in the high-resolution original image as shown in Fig. 3, the two vehicles cannot be detected in the compressed image as shown in Fig. 5.

[0032] As in the above example, there are cases where an object that should be detected cannot be detected from the compressed image. Also, although not shown in the above example, there are cases where a non-existent object is erroneously detected from the compressed image. In other words, although the image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 needs to be compressed, it is not possible to detect an object ahead of the vehicle 10 with high accuracy from the compressed image.

[0033] 3. Overview of remote monitoring system features The remote monitoring system 2 solves the above problems as follows: FIG.

[0034] 6 shows the flow of information in the remote monitoring system 2. In the remote monitoring system 2, a compressed image is obtained from the original image before compression in the in-vehicle information processing device 20. Furthermore, in parallel with this, an object detection result is obtained from the original image before compression in the in-vehicle information processing device 20. The separately obtained compressed image and the object detection result are separately transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 via the communication network 8. In other words, in the remote monitoring system 2, the process of detecting an object from an image from the camera 12 is performed in the in-vehicle information processing device 20 before transmitting the image to the remote monitoring server 40, and moreover, this process is performed on the original image before compression, not on the compressed image.

[0035] The compressed image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is restored by the remote monitoring server 40. The object detection result transmitted from the in-vehicle information processing device 20 separately from the compressed image is superimposed on the restored image by the remote monitoring server 40. As a result, a superimposed display screen is obtained in which the object detection result is superimposed on the restored image. To explain this using a specific example, the compressed image shown in Fig. 4 is displayed on the monitoring screen 32a of the display device 32, and the object detection result shown in Fig. 3 is superimposed on it.

[0036] According to the remote monitoring system 2 having the above-described features, the in-vehicle information processing device 20 performs object detection on the original image before compression, thereby enabling highly accurate detection of objects ahead of the vehicle 10. Then, by displaying the object detection results together with the compressed image on the monitoring screen 32a of the display device 32, remote monitoring by the remote operator 50 can be assisted.

[0037] Although the compression process of the original image and the object detection process from the original image are performed by the in-vehicle information processing device 20, they are executed separately. Therefore, even if the original image is the same, there may be a difference between the time when the compressed image is obtained and the time when the object detection result is obtained. If the display device 32 displays the image with this time difference, it may cause a misunderstanding by the remote operator 50.

[0038] The following chapters will explain in detail the configuration and functions of the remote monitoring system 2. As will be explained there, the remote monitoring system 2 is technically designed to prevent a time lag from occurring between the image displayed on the display device 32 and the object detection result.

[0039] 4. Configuration of the vehicle according to the first embodiment 7 is a block diagram showing the configuration of a vehicle 10 equipped with an on-board information processing device 20 according to the first embodiment. The vehicle 10 is equipped with a camera 12, on-board sensors 14, a communication device 16, a driving device 18, and the on-board information processing device 20. Details of the camera 12 equipped in the vehicle 10 are as described above.

[0040] The on-board sensors 14 include status sensors that acquire information about the movement of the vehicle 10. The status sensors include, for example, at least one of a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The on-board sensors 14 also include a position sensor that detects the position and orientation of the vehicle 10. An example of the position sensor is a GPS (Global Positioning System) sensor. Furthermore, the on-board sensors 14 may include a recognition sensor other than the camera 12. The recognition sensor recognizes (detects) the situation around the vehicle 10. Examples of the recognition sensor include LiDAR (Laser Imaging Detection and Ranging), millimeter-wave radar, and an ultrasonic sensor.

[0041] The communication device 16 communicates with the outside of the vehicle 10. The communication device 16 communicates with the remote monitoring server 40 via the communication network 8.

[0042] The traveling device 18 includes a steering device that steers the vehicle 10, a drive device that drives the vehicle 10, and a braking device that brakes the vehicle 10. The steering device includes, for example, a power steering system, a steer-by-wire steering system, and a rear-wheel steering system. The drive device includes, for example, an engine, an EV system, and a hybrid system. The braking device includes, for example, a hydraulic brake and a regenerative brake.

[0043] The on-vehicle information processing device 20 is an ECU (Electronic Control Unit) mounted on the vehicle 10, or a collection of multiple ECUs. The on-vehicle information processing device 20 includes one or more processors 21 (hereinafter simply referred to as processors 21) and one or more memories 22 (hereinafter simply referred to as memories 22) coupled to the processors 21. The memory 22 includes a main storage device and an auxiliary storage device. The memory 22 stores programs executable by the processor 21 and various pieces of information related to the programs. The processor 21 executes the programs to realize various processes by the processor 21. The programs can be stored in the main storage device, or can be stored in a computer-readable recording medium that is an auxiliary storage device.

[0044] At least three storage areas 22a, 22b, and 22c are allocated in the memory 22. The processor 21 stores the unprocessed data DT0 in the first storage area 22a of the memory 22. The unprocessed data DT0 includes an image IMG captured by the camera 12 and a timestamp TS indicating the time the image IMG was captured. The processor 21 reads the unprocessed data DT0 and performs compression processing. The processor 21 also reads the unprocessed data DT0 and performs object detection processing.

[0045] The processor 21 stores first processing data (hereinafter referred to as first data) DT1 in the second storage area 22b of the memory 22. The first data DT1 includes a compressed image CIMG obtained by compressing the image IMG and a timestamp TS associated with the uncompressed image IMG. The processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0046] The processor 21 stores second processing data (hereinafter referred to as second data) DT2 in the third storage area 22c of the memory 22. The second data DT2 includes an object detection result ODR obtained by performing object detection processing on the image IMG and a timestamp TS associated with the image IMG used for object detection. The processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0047] 5. Configuration of the remote monitoring center according to the first embodiment 8 is a block diagram showing the configuration of a remote monitoring center 30 in which a remote monitoring server 40 according to the first embodiment is located. In addition to the remote monitoring server 40 and a display device 32, the remote monitoring center 30 is also provided with an input device 34 and a communication device 36. Like the display device 32, the input device 34 and the communication device 36 are connected to the remote monitoring server 40.

[0048] The input device 34 is an interface for receiving input from the remote operator 50. Examples of the input device 34 include a touch panel, a keyboard, and a mouse. When the remote monitoring is monitoring for remote driving, the input device 34 includes driving operation members for the remote operator 50 to drive (steer, accelerate, and decelerate) the vehicle 10.

[0049] The communication device 36 communicates with the outside of the vehicle 10. The communication device 36 communicates with the in-vehicle information processing device 20 via the communication network 8.

[0050] The remote monitoring server 40 is a computer or a collection of multiple computers installed in the remote monitoring center 30. The remote monitoring server 40 includes one or more processors 41 (hereinafter simply referred to as processor 41) and one or more memories 42 (hereinafter simply referred to as memory 42) coupled to the processor 41. The memory 42 includes a main storage device and an auxiliary storage device. The memory 42 stores programs executable by the processor 41 and various pieces of information related to the programs. The processor 41 executes the programs to realize various processes by the processor 41. The programs can be stored in the main storage device or in a computer-readable recording medium that is an auxiliary storage device.

[0051] The memory 42 is configured with at least two queues 42a and 42b. A plurality of first data DT1 are accumulated in the first queue 42a. The processor 41 puts the latest first data DT1 into the first queue 42a and takes out the oldest first data DT1 at the top from the first queue 42a. The first queue 42a stores the i-th to (i+j)-th first data DT1 in the order in which the timestamps TS are assigned.

[0052] A plurality of second data DT2 are accumulated in the second queue 42b. The processor 41 puts the latest second data DT2 into the second queue 42b and removes the oldest second data DT2 at the top from the second queue 42b. The second queue 42b stores the second data DT2 from the i-th to the (i+j)-th in the order in which the timestamps TS were assigned. The processor 41 removes the first data DT1 and the second data DT2 having the same time stamps TS from the first queue 42a and the second queue 42b, respectively.

[0053] 6. Processing executed in the remote monitoring system according to the first embodiment 9 is a block diagram showing the processing executed in the remote monitoring system 2 according to the first embodiment, more specifically, the processing executed by the processors 21 and 41 of the in-vehicle information processing device 20 and the remote monitoring server 40. The processing flow shown in this block diagram also represents the remote monitoring method according to the first embodiment of the present disclosure.

[0054] The processor 21 of the in-vehicle information processing device 20 executes a timestamp generation process 211, an image compression process 212, an object detection process 213, a first data transmission process 214, and a second data transmission process 215. These processes are executed by the processor 21 when a program stored in the memory 22 is executed in the processor 21.

[0055] In the timestamp generation process 211, the processor 21 generates a timestamp TS indicating the time when the image IMG was captured by the camera 12. The generated timestamp TS is stored in the memory 22 in association with the image IMG.

[0056] In the image compression process 212, the processor 21 reads the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 performs a predetermined compression process on the image IMG to generate a compressed image CIMG. The generated compressed image CIMG is associated with a timestamp TS and stored in the memory 22.

[0057] In the object detection process 213, the processor 21 reads the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 detects an object ahead of the vehicle 10 from the image IMG using an image analysis technique such as pattern matching or deep learning. The object detection result ODR is stored in the memory 22 in association with the timestamp TS.

[0058] In the first data transmission process 214, the processor 21 reads out the first data DT1 including the compressed image CIMG and the timestamp TS from the memory 22. Then, the processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0059] In the second data transmission process 215, the processor 21 reads out the second data DT2 including the object detection result ODR and the timestamp TS from the memory 22. Then, the processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0060] The processor 41 of the remote monitoring server 40 executes a first data reception process 411, a second data reception process 412, a simultaneous data extraction process 413, a compressed image restoration process 414, and a superimposed display process 415. These processes are executed by the processor 41 when a program stored in the memory 42 is executed in the processor 41.

[0061] In the first data reception process 411, the processor 41 receives the first data DT1 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received first data DT1 in the first queue 42a.

[0062] In the second data reception process 412, the processor 41 receives the second data DT2 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received second data DT2 in the second queue 42b.

[0063] In the simultaneous data extraction process 413, the processor 41 extracts, in chronological order, compressed images CIMG and object detection results ODR having the same time stamps TS from the queues 42a and 42b, respectively.

[0064] In the compressed image restoration process 414, the processor 41 restores the compressed image CIMG extracted together with the object detection result ODR using a restoration method corresponding to the compression method, to generate a restored image DIMG.

[0065] In the superimposed display process 415, the processor 41 superimposes the restored image DIMG and the object detection result ODR and displays them on the monitoring screen 32a of the display device 32. Since the time stamps TS of the restored image DIMG and the object detection result ODR are the same, there is no time lag between the image displayed on the display device 32 and the object detection result.

[0066] 7. Configuration of vehicle according to second embodiment Fig. 10 is a block diagram showing the configuration of a vehicle 10 equipped with an on-board information processing device 20 according to the second embodiment. In Fig. 10, the same elements as those equipped in the vehicle 10 according to the first embodiment are assigned the same reference numerals. Explanations of the configurations and functions that have already been explained will be omitted or simplified.

[0067] The information obtained from the sensor signals of the on-board sensors 14 (hereinafter referred to as sensor information) includes information that is useful to present to the remote operator 50. For example, LiDAR images obtained by imaging LiDAR sensor signals, and information on the distance and direction of an object obtained from sensor signals of millimeter-wave radar or ultrasonic sensors, are important information for understanding the driving environment of the vehicle 10. In the second embodiment, processing for obtaining sensor information from sensor signals is performed in the on-board information processing device 20. In the second embodiment, the sensor information is transmitted from the on-board information processing device 20 to the remote monitoring server 40. The remote monitoring server 40 superimposes the sensor information on a restored image together with the object detection result and displays it on the display device 32.

[0068] In the second embodiment, at least four storage areas 22a, 22b, 22c, and 22d are allocated in the memory 22. As in the first embodiment, the first storage area 22a stores unprocessed data DT0 including an image IMG captured by the camera 12 and a timestamp TS indicating the time the image IMG was captured. The second storage area 22a stores first data DT1 including a compressed image CIMG obtained by compressing the image IMG and a timestamp TS associated with the uncompressed image IMG. The third storage area 22c stores second data DT2 including an object detection result ODR obtained by object detection processing on the image IMG and a timestamp TS associated with the image IMG used for object detection.

[0069] The processor 21 stores third processing data (hereinafter referred to as third data) DT3 in the fourth storage area 22d of the memory 22. The third data DT3 includes sensor information SI obtained from the sensor signal of the on-board sensor 14 and a timestamp TS indicating the acquisition time of the sensor signal. The processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0070] 8. Configuration of the remote monitoring center according to the second embodiment Fig. 11 is a block diagram showing the configuration of a remote monitoring center 30 in which a remote monitoring server 40 according to the second embodiment is located. In Fig. 10, the same elements as those in the remote monitoring center 30 according to the first embodiment are denoted by the same reference numerals. Explanations of the configuration and functions that have already been explained will be omitted or simplified.

[0071] In the second embodiment, at least three queues 42a, 42b, and 42c are configured in the memory 42. As in the first embodiment, the first queue 42a stores the first data DT1 from the i-th to the (i+j)-th in the order in which the timestamps TS are assigned. The second queue 42b stores the second data DT2 from the i-th to the (i+j)-th in the order in which the timestamps TS are assigned.

[0072] A plurality of third data DT3 are accumulated in a third queue 42b newly configured in the second embodiment. The processor 41 places the latest third data DT3 in a third queue 42c and removes the oldest third data DT3 at the top from the third queue 42c. The third queue 42c stores the i-th to (i+j)-th third data DT3 in the order in which the timestamps TS were assigned. The processor 41 removes the first data DT1, second data DT2, and third data DT3, which have the same time stamps TS, from the first queue 42a, second queue 42b, and third queue 42c, respectively.

[0073] 9. Processing executed in the remote monitoring system according to the second embodiment 12 is a block diagram showing the processing executed in the remote monitoring system 2 according to the second embodiment, more specifically, the processing executed by the processors 21, 41 of the in-vehicle information processing device 20 and the remote monitoring server 40. The processing flow shown in this block diagram also represents the remote monitoring method according to the second embodiment of the present disclosure. In FIG. 10, the same processes as those executed in the first embodiment are denoted by the same reference numerals. The description of the processes already explained will be omitted or simplified.

[0074] The processor 21 of the in-vehicle information processing device 20 according to the second embodiment executes, in addition to the processes executed in the first embodiment, a timestamp generation process 216, a sensor signal process 217, and a third data transmission process 218. As with the other processes, these processes are processes executed by the processor 21 when a program stored in the memory 22 is executed in the processor 21.

[0075] In the timestamp generation process 216, the processor 21 generates a timestamp TS indicating the time at which the sensor signal SS was acquired from the on-board sensor 14. The generated timestamp TS is stored in the memory 22 in association with the sensor signal SS.

[0076] In the sensor signal processing 217, the processor 21 generates sensor information SI from the sensor signal SS. The sensor information SI includes, for example, a LiDAR image obtained from a LiDAR sensor signal, and distance information and direction information obtained from a millimeter-wave radar or ultrasonic sensor sensor signal. The generated sensor information SI is associated with a timestamp TS associated with the sensor signal SS and stored in the memory 22.

[0077] In the third data transmission process 218, the processor 21 reads out the third data DT3 including the sensor information SI and the timestamp TS from the memory 22. Then, the processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0078] The processor 41 of the remote monitoring server 40 according to the second embodiment executes a third data reception process 416 in addition to the processes executed in the first embodiment. The processor 41 also executes a simultaneous data extraction process 417 and a superimposed display process 418, which are modified versions of the processes executed in the first embodiment. As with the other processes, these processes are executed by the processor 41 when a program stored in the memory 42 is executed in the processor 41.

[0079] In the third data reception process 416, the processor 41 receives the third data DT3 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received third data DT3 in the third queue 42c.

[0080] In the simultaneous data extraction process 417, the processor 41 extracts, in chronological order, the compressed image CIMG, the object detection result ODR, and the sensor information SI having the same time stamp TS from the respective queues 42a, 42b, and 42c.

[0081] In the superimposed display process 418, the processor 41 superimposes the restored image DIMG, the object detection result ODR, and the sensor information SI and displays them on the monitoring screen 32a of the display device 32. Because the timestamps TS of the restored image DIMG, the object detection result ODR, and the sensor information SI are the same, there is no time lag between the image displayed on the display device 32, the object detection result, and the sensor information. [Explanation of symbols]

[0082] 2. Remote monitoring system 8. Communication Networks 10 vehicles 12 Camera 14 In-vehicle sensors 16. Communications equipment 20. In-vehicle information processing device 21 processors 22 Memory 30 Remote Monitoring Center 32a Monitoring screen 36 Communication equipment 40 Remote monitoring server 41 processors 42 memory 50 Remote Operator

Claims

1. A remote monitoring system for remotely monitoring a vehicle, comprising: a camera mounted on the vehicle; an in-vehicle information processing device mounted on the vehicle and connected to the camera; a remote monitoring server connected to the in-vehicle information processing device via a communication network including mobile communication, The in-vehicle information processing device includes: an object detection process for detecting an object ahead of the vehicle based only on the uncompressed image acquired by the camera; a compression process for compressing the image to generate a compressed image; a first data transmission process of transmitting first data including the compressed image and an image acquisition time of the image before compression; a second data transmission process of transmitting second data including the object detection result and the image acquisition time of the image used for the object detection; The remote monitoring server a first data receiving process for receiving the first data and storing it in a memory; a second data receiving process for receiving the second data and storing it in the memory; an extraction process of extracting, in time series, the compressed images and the object detection results whose image acquisition times are the same from the plurality of first data and the plurality of second data stored in the memory; and performing a display process of superimposing a restored image obtained by restoring the extracted compressed image and the extracted object detection result on a monitoring screen. A remote monitoring system comprising:

2. 2. The remote monitoring system according to claim 1, The in-vehicle information processing device includes: further executing a third data transmission process of transmitting third data including sensor information obtained from a sensor signal of an on-vehicle sensor and a signal acquisition time of the sensor signal; The remote monitoring server further executing a third data receiving process of receiving the third data and storing it in the memory; In the extraction process, the compressed image and the object detection result, which are acquired at the same time as the image acquisition time, are extracted in chronological order from the memory, and the sensor information, which is acquired at the same time as the image acquisition time, is extracted in chronological order; In the display process, the extracted sensor information is superimposed on the restored image together with the extracted object detection result and displayed on the monitoring screen. A remote monitoring system comprising:

3. A remote monitoring method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network including mobile communications, comprising: The in-vehicle information processing device includes: Detecting an object ahead of the vehicle based only on an uncompressed image acquired by a camera mounted on the vehicle; compressing the image to generate a compressed image; transmitting first data including the compressed image and an image acquisition time of the image before compression; transmitting second data including the object detection result and the image acquisition time of the image used for the object detection; The remote monitoring server receiving the first data and storing it in a memory; receiving the second data and storing it in the memory; extracting, in time series, the compressed images and the object detection results whose image acquisition times are the same from the plurality of first data and the plurality of second data stored in the memory; A restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on a monitoring screen. A remote monitoring method comprising:

4. 4. The remote monitoring method according to claim 3, The in-vehicle information processing device includes: transmitting third data including sensor information obtained from a sensor signal of an on-vehicle sensor and a signal acquisition time of the sensor signal; The remote monitoring server receiving the third data and storing it in the memory; extracting, in time series, from the memory the compressed image and the object detection result, the image acquisition times of which are the same, and extracting, in time series, the sensor information, the signal acquisition time of which is the same as the image acquisition time; The extracted sensor information is superimposed on the restored image together with the extracted object detection result and displayed on the monitoring screen. A remote monitoring method comprising:

5. A remote monitoring server connected to an in-vehicle information processing device via a communication network including mobile communication, one or more memories storing one or more programs; one or more processors coupled to the one or more memories; The one or more processors, when executing the one or more programs, receiving, from the in-vehicle information processing device, first data including a compressed image generated by compressing an image captured by an in-vehicle camera and an image capture time of the image before compression, and storing the first data in the one or more memories; receiving second data from the in-vehicle information processing device, the second data including an object detection result based only on the image before compression and the image acquisition time of the image used for the object detection, and storing the second data in the one or more memories; extracting, in time series, the compressed images and the object detection results whose image acquisition times are the same from the plurality of first data and the plurality of second data stored in the one or more memories; A restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on a monitoring screen. A remote monitoring server characterized by:

Images