Remote monitoring system, remote monitoring method, remote monitoring server, and in-vehicle information processing device.

Abstract

To provide a remote monitoring technique that can improve the accuracy of object detection results to be displayed on a monitoring screen while reducing communication costs.SOLUTION: An in-vehicle information processing device 20 compresses an image IMG from a camera 12, and transmits first data including the compressed image CIMG and a timestamp TS indicating an acquisition time of the image IMG. The in-vehicle information processing device 20 also detects an object in front of a vehicle from the image IMG by using an image analysis technique, and transmits second data including an object detection result ODR including the type and position of the detected object and the timestamp TS indicating the acquisition time of the image IMG. A remote monitoring server 40 receives the first data and the second data and stores them in a memory, extracts the compressed image CIMG and the object detection result ODR having the same timestamp TS from the memory in chronological order, and displays a restored image DIMG obtained by restoring the compressed image CIMG and the object detection result ODR while the restored image DIMG and the object detection result ODR are superimposed on each other on a monitoring screen.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present disclosure relates to a remote monitoring system, a remote monitoring method, a remote monitoring server, and an in-vehicle information processing device suitable for using a vehicle for remote monitoring.

Background Art

[0002] Patent Document 1 discloses a technique related to a remote operation device for remotely operating an autonomous driving vehicle that has become unable to perform autonomous driving. When displaying an image captured by a camera mounted on the autonomous driving vehicle on a display unit, the remote operation device according to this conventional technique analyzes the captured image and highlights a specific object in the image.

[0003] The image captured by the camera is compressed and then transmitted from the autonomous driving vehicle to the remote operation device. Although compression of the image is essential for reducing communication costs, the resolution of the image decreases due to compression. Therefore, in the above conventional technique, the image cannot be analyzed accurately, and there is a risk that the highlighting of a specific object may not be appropriately performed.

[0004] In addition to the above Patent Document 1, the following Patent Document 2 and Patent Document 3 can be exemplified as prior art documents representing the technical level of the technical field to which the present application belongs.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] This disclosure has been made in view of the issues described above. The first objective of this disclosure is to provide a remote monitoring system and a remote monitoring method that can improve the accuracy of object detection results displayed on a monitoring screen while reducing communication costs.

[0007] Furthermore, a second objective of this disclosure is to provide a remote monitoring server that can improve the accuracy of object detection results displayed on the monitoring screen while reducing communication costs with the in-vehicle information processing device.

[0008] Furthermore, a third objective of this disclosure is to provide an in-vehicle information processing device that can improve the accuracy of object detection results displayed on the monitoring screen at the remote monitoring server while reducing communication costs with the remote monitoring server. [Means for solving the problem]

[0009] The remote monitoring system described herein comprises a camera mounted on a vehicle, an in-vehicle information processing device mounted on the vehicle and connected to the camera, and a remote monitoring server connected to the in-vehicle information processing device via a communication network. The in-vehicle information processing device performs the following object detection processing, compression processing, first data transmission processing, and second data transmission processing. The remote monitoring server performs the following first data reception processing, second data reception processing, extraction processing, and display processing.

[0010] In the object detection process by the in-vehicle information processing system, objects in front of the vehicle are detected from images acquired by the camera. In the compression process, images are compressed to generate a compressed image. In the first data transmission process, first data, including the acquisition time of the compressed image and the uncompressed image, is sent to the remote monitoring server. In the second data transmission process, second data, including the object detection result and the acquisition time of the image used for object detection, is sent to the remote monitoring server.

[0011] In the first data reception process by the remote monitoring server, the first data transmitted from the in-vehicle information processing unit is received and stored in memory. In the second data reception process, the second data transmitted from the in-vehicle information processing unit is received and stored in memory. In the extraction process, compressed images and object detection results acquired at the same time are extracted from memory in chronological order. In the display process, the restored image obtained by restoring the extracted compressed image and the extracted object detection results are superimposed and displayed on the monitoring screen.

[0012] In this remote monitoring system, the in-vehicle information processing device may further perform a third data transmission process, and the remote monitoring server may further perform a third data reception process. The third data transmission process involves transmitting third data, which includes sensor information obtained from the sensor signal of the in-vehicle sensor and the signal acquisition time of the sensor signal. The third data reception process receives the third data transmitted from the in-vehicle information processing device and stores it in memory. In this case, the extraction process by the remote monitoring server involves extracting compressed images and object detection results in chronological order from memory, where the image acquisition time is the same as the image acquisition time, and extracting sensor information in chronological order where the signal acquisition time is the same as the image acquisition time. Furthermore, the display process involves overlaying the extracted sensor information along with the extracted object detection results onto the reconstructed image and displaying it on the monitoring screen.

[0013] The remote monitoring method described herein is a method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network. This remote monitoring method causes the in-vehicle information processing device to detect an object in front of the vehicle from an image acquired by a camera mounted on the vehicle, compress the image to generate a compressed image, transmit first data including the image acquisition time of the compressed image and the image before compression, and transmit second data including the object detection result and the image acquisition time of the image used for object detection. Furthermore, this remote monitoring method causes the remote monitoring server to receive the first data and store it in memory, receive the second data and store it in memory, extract compressed images and object detection results with the same image acquisition time from memory in chronological order, and display the restored image obtained by restoring the extracted compressed image and the extracted object detection result superimposed on the monitoring screen.

[0014] This remote monitoring method may also involve the in-vehicle information processing device transmitting third data, which includes sensor information obtained from the sensor signals of the in-vehicle sensors and the signal acquisition time of the sensor signals. This remote monitoring method may also involve the remote monitoring server receiving the third data, storing it in memory, extracting compressed images and object detection results from memory in chronological order where the image acquisition time is the same, and extracting sensor information in chronological order where the signal acquisition time is the same as the image acquisition time, and then overlaying the extracted sensor information along with the object detection results onto the reconstructed image and displaying it on the monitoring screen.

[0015] The remote monitoring server relating to this disclosure is a server connected to an in-vehicle information processing device via a communication network. This remote monitoring server comprises one or more memories storing one or more programs, and one or more processors coupled to one or more memories. When one or more programs are executed, one or more processors perform the following processes.

[0016] In the processing performed by one or more processors of this remote monitoring server, first data including the compressed image generated by compressing the image acquired by the in-vehicle camera and the image acquisition time of the uncompressed image is received from the in-vehicle information processing device and stored in one or more memories. Second data including the object detection result based on the image and the image acquisition time of the image used for object detection is also received from the in-vehicle information processing device and stored in one or more memories. Furthermore, compressed images and object detection results with the same image acquisition time are extracted in chronological order from one or more memories. Finally, the extracted compressed image is restored and the extracted object detection results are superimposed and displayed on the monitoring screen.

[0017] The in-vehicle information processing device described herein is a device connected to a remote monitoring server via a communication network. The in-vehicle information processing device comprises one or more memories storing one or more programs, and one or more processors coupled to one or more memories. When one or more programs are executed, the one or more processors perform the following processes.

[0018] In the processing performed by one or more processors of this in-vehicle information processing device, objects in front of the vehicle are detected from images acquired by the in-vehicle camera. The images are also compressed to generate a compressed image. Furthermore, first data, including the acquisition time of the compressed image and the uncompressed image, is transmitted to a remote monitoring server. Finally, second data, including the object detection result and the acquisition time of the image used for object detection, is transmitted to the remote monitoring server. [Effects of the Invention]

[0019] According to this disclosure, since object detection is performed from the image before compression, it is possible to prevent the decrease in detection accuracy that occurs when detecting objects from compressed images. This makes it possible to improve the accuracy of object detection results displayed on the monitoring screen while reducing communication costs with the in-vehicle information processing device. [Brief explanation of the drawing]

[0020] [Figure 1] This is a diagram schematically showing the configuration of a remote monitoring system according to an embodiment of the present disclosure. [Figure 2] This is a diagram showing an example of an image before compression for explaining the problems of the remote monitoring system. [Figure 3] This is a diagram showing an example of an object detection result from an image before compression for explaining the problems of the remote monitoring system. [Figure 4] This is a diagram showing an example of an image after compression for explaining the problems of the remote monitoring system. [Figure 5] This is a diagram showing an example of an object detection result from an image after compression for explaining the problems of the remote monitoring system. [Figure 6] This is a diagram for explaining an overview of the features of a remote monitoring system according to an embodiment of the present disclosure. [Figure 7] This is a block diagram showing the configuration of a vehicle according to the first embodiment of the present disclosure. [Figure 8] This is a block diagram showing the configuration of a remote monitoring center according to the first embodiment of the present disclosure. [Figure 9] This is a block diagram showing the processes executed by the respective processors of an in-vehicle information processing device and a remote monitoring server according to the first embodiment of the present disclosure. [Figure 10] This is a block diagram showing the configuration of a vehicle according to the second embodiment of the present disclosure. [Figure 11] This is a block diagram showing the configuration of a remote monitoring center according to the second embodiment of the present disclosure. [Figure 12] This is a block diagram showing the processes executed by the respective processors of an in-vehicle information processing device and a remote monitoring server according to the second embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0021] Embodiments of this disclosure will be described below with reference to the drawings. However, in the embodiments described below, when numbers such as the number of elements, quantities, amounts, or ranges are mentioned, the concept of this disclosure is not limited to the mentioned numbers unless they are specifically stated or clearly defined in principle. Furthermore, structures and the like described in the embodiments described below are not necessarily essential to the concept of this disclosure unless they are specifically stated or clearly defined in principle.

[0022] 1. Outline configuration of the remote monitoring system Figure 1 is a schematic diagram showing the configuration of a remote monitoring system according to an embodiment of the present disclosure. The remote monitoring system 2 is a system that remotely monitors a vehicle 10 traveling on a road by a remote operator 50. Remote monitoring in the present disclosure includes monitoring for remote driving of the vehicle 10, monitoring for remotely assisting the autonomously driving vehicle 10 as needed, and monitoring for checking the surrounding environment of the autonomously driving vehicle 10.

[0023] The vehicle 10 to be remotely monitored is equipped with a camera 12. The camera 12 may be a stereo camera or a monocular camera. The shooting range IA of the camera 12 covers at least the area in front of the vehicle 10 in the direction of travel. For example, when the vehicle 10 is traveling in lane DL, the area in front of lane DL and both its left and right sides are included in the shooting range IA of the camera 12.

[0024] Vehicle 10 is equipped with an in-vehicle information processing device 20. Images captured by camera 12 are input to the in-vehicle information processing device 20. The in-vehicle information processing device 20 is connected to a remote monitoring server 40 of the remote monitoring center 30 via a communication network 8. The communication network 8 includes mobile communication such as 4G and 5G. Images captured by camera 12 are sent from the in-vehicle information processing device 20 to the remote monitoring server 40 via the communication network 8.

[0025] In the remote monitoring center 30, images from the camera 12 sent from the in-vehicle information processing device 20 to the remote monitoring server 40 are displayed on the monitoring screen 32a of the display device 32. For example, the display device 32 can be an LCD, an organic EL display, a head-mounted display, or a touch panel. The remote operator 50 can monitor the area in front of the vehicle 10 by looking at the images from the camera 12 displayed on the monitoring screen 32a of the display device 32. For example, if an obstacle OJ exists within the shooting range IA of the camera 12, the remote operator 50 can recognize the presence of the obstacle OJ on the monitoring screen 32a of the display device 32.

[0026] The remote monitoring system 2 has a function to assist the remote operator 50 in recognizing obstacles OJ. Obstacles OJ captured in the image from camera 12 can be automatically detected using image analysis techniques such as pattern matching and deep learning. By identifying obstacles OJ captured in the image using this object detection function, the remote operator 50 can be informed of the location of the obstacles OJ on the monitoring screen 32a.

[0027] On the other hand, the remote monitoring system 2 is required to reduce communication costs. Images captured by camera 12 are encoded in the in-vehicle information processing device 20, and the encoded image data is transmitted to the remote monitoring server 40. In this encoding process, the images captured by camera 12 are compressed using a predetermined compression method. The image data transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is decoded by the remote monitoring server 40. The display device 32 displays the image restored by the decoding performed by the remote monitoring server 40.

[0028] There are no limitations on the image transmission format or the image compression method. If the image transmission format is a video stream format, for example, H.264, H.265, VP8, VP9, ​​VP10, and AV1 can be used as image compression methods. If the image transmission format is a format in which each image is transmitted individually, for example, JPEG, Motion JPEG, and PNG can be used as image compression methods. Preferably, H.264, VP8, or VP9 are used. Motion JPEG, VP10, and AV1 are also suitable compression formats for the remote monitoring system 2.

[0029] 2. Challenges of remote monitoring systems As described above, the remote monitoring system 2 is required to assist the remote operator 50 by detecting objects from images captured by the camera 12, and to reduce communication costs by compressing the images captured by the camera 12. However, it is not easy to satisfy both of these requirements simultaneously.

[0030] Here, four images are shown. Figure 2 shows an example of the original image captured by camera 12, i.e., the image before compression. Figure 3 shows an example of the object detection result from the original image shown in Figure 2. Figure 4 shows an example of the compressed image from the original image shown in Figure 2, i.e., the image after compression. Figure 5 shows an example of the object detection result from the compressed image shown in Figure 4. However, although these images are originally color images, they have been binarized in the figures for the purposes of the application document.

[0031] The image displayed on the monitoring screen 32a of the display device 32 is the compressed image shown in Figure 4. However, the compressed image has a lower resolution than the original image. This is evident from the comparison between the image shown in Figure 2 and the image shown in Figure 4, as well as from the binarized image. Therefore, as shown in Figure 3, two people and two vehicles are detected in the high-resolution original image, whereas as shown in Figure 5, the two vehicles are not detected in the compressed image.

[0032] As shown in the example above, objects that should be detected may not be detected in the compressed image. Also, although not shown in the example above, non-existent objects may be mistakenly detected in the compressed image. In other words, the image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 needs to be compressed, but objects in front of the vehicle 10 cannot be detected with high accuracy from the compressed image.

[0033] 3. Overview of the features of the remote monitoring system In remote monitoring system 2, the above issues are resolved as follows. Figure 6 is a diagram illustrating the overview of the features of remote monitoring system 2.

[0034] Figure 6 shows the flow of information in the remote monitoring system 2. In the remote monitoring system 2, the in-vehicle information processing device 20 obtains a compressed image from the original image before compression. In parallel with this, the in-vehicle information processing device 20 also obtains object detection results from the original image before compression. The separately obtained compressed image and object detection results are transmitted separately from the in-vehicle information processing device 20 to the remote monitoring server 40 via the communication network 8. In other words, in the remote monitoring system 2, the process of detecting objects from the image of the camera 12 is performed in the in-vehicle information processing device 20 before the image is transmitted to the remote monitoring server 40, and moreover, it is performed on the original image before compression, not on the compressed image.

[0035] The compressed image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is restored by the remote monitoring server 40. The object detection results, transmitted separately from the compressed image from the in-vehicle information processing device 20, are superimposed on the restored image by the remote monitoring server 40. This results in a superimposed display screen in which the object detection results are superimposed on the restored image. To illustrate with a specific example, the monitoring screen 32a of the display device 32 displays the compressed image shown in Figure 4, and the object detection results shown in Figure 3 are superimposed on it.

[0036] With the remote monitoring system 2 having the above features, the in-vehicle information processing device 20 performs object detection on the original image before compression, making it possible to detect objects in front of the vehicle 10 with high accuracy. Then, by displaying the object detection results together with the compressed image on the monitoring screen 32a of the display device 32, remote monitoring by the remote operator 50 can be assisted.

[0037] Although the compression process of the original image and the object detection process from the original image are performed by the in-vehicle information processing device 20, they are executed separately. Therefore, even if the original image is the same, there may be a time difference between when the compressed image is obtained and when the object detection results are obtained. If the display on the display device 32 is displayed with this time difference, it may cause misidentification by the remote operator 50.

[0038] The following chapters will describe the detailed configuration and functions of the remote monitoring system 2. As will be explained there, the remote monitoring system 2 is technically designed to prevent any time lag between the image displayed on the display device 32 and the object detection results.

[0039] 4. Vehicle configuration according to the first embodiment Figure 7 is a block diagram showing the configuration of a vehicle 10 equipped with an in-vehicle information processing device 20 according to the first embodiment. The vehicle 10 includes a camera 12, an in-vehicle sensor 14, a communication device 16, a driving device 18, and an in-vehicle information processing device 20. Details of the camera 12 equipped in the vehicle 10 are as described above.

[0040] The on-board sensor 14 includes a state sensor that acquires information about the motion of the vehicle 10. The state sensor includes, for example, at least one of a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The on-board sensor 14 also includes a position sensor that detects the position and orientation of the vehicle 10. A GPS (Global Positioning System) sensor is an example of a position sensor. Furthermore, the on-board sensor 14 may include recognition sensors other than the camera 12. The recognition sensors recognize (detect) the surrounding conditions of the vehicle 10. Examples of recognition sensors include LiDAR (Laser Imaging Detection and Ranging), millimeter-wave radar, and ultrasonic sensors.

[0041] The communication device 16 communicates with the outside of the vehicle 10. The communication device 16 communicates with the remote monitoring server 40 via the communication network 8.

[0042] The running gear 18 includes a steering gear for steering the vehicle 10, a drive gear for driving the vehicle 10, and a braking gear for braking the vehicle 10. The steering gear includes, for example, a power steering system, a steer-by-wire steering system, and a rear-wheel steering system. The drive gear includes, for example, an engine, an EV system, and a hybrid system. The braking gear includes, for example, hydraulic brakes and regenerative braking.

[0043] The in-vehicle information processing device 20 is an ECU (Electronic Control Unit) mounted on the vehicle 10, or a collection of multiple ECUs. The in-vehicle information processing device 20 includes one or more processors 21 (hereinafter simply referred to as processor 21) and one or more memories 22 (hereinafter simply referred to as memory 22) coupled to the processor 21. Memory 22 includes main memory and auxiliary memory. Memory 22 stores programs that can be executed by the processor 21 and various information related thereto. By executing the programs, the processor 21 performs various processes. The programs can be stored in main memory or in a computer-readable recording medium, which is an auxiliary memory.

[0044] Memory 22 has at least three storage areas 22a, 22b, and 22c. The processor 21 stores the pre-processing data DT0 in the first storage area 22a of memory 22. The pre-processing data DT0 includes an image IMG captured by camera 12 and a timestamp TS indicating the time the image IMG was acquired. The processor 21 reads the pre-processing data DT0 and performs compression processing. The processor 21 also reads the pre-processing data DT0 and performs object detection processing.

[0045] The processor 21 stores the first processing data (hereinafter referred to as the first data) DT1 in the second storage area 22b of the memory 22. The first data DT1 includes the compressed image CIMG obtained by the image IMG compression process and the timestamp TS associated with the uncompressed image IMG. The processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0046] The processor 21 stores the second processing data (hereinafter referred to as the second data) DT2 in the third storage area 22c of the memory 22. The second data DT2 includes the object detection result ODR obtained by the object detection processing on the image IMG, and the timestamp TS associated with the image IMG used for object detection. The processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0047] 5. Configuration of the remote monitoring center according to the first embodiment Figure 8 is a block diagram showing the configuration of a remote monitoring center 30 where a remote monitoring server 40 according to the first embodiment is located. In addition to the remote monitoring server 40 and the display device 32, the remote monitoring center 30 is also equipped with an input device 34 and a communication device 36. Similar to the display device 32, the input device 34 and the communication device 36 are connected to the remote monitoring server 40.

[0048] The input device 34 is an interface for receiving input from the remote operator 50. Examples of input devices 34 include a touch panel, a keyboard, and a mouse. Furthermore, if the remote monitoring is for remote operation, the input device 34 includes driving control members for the remote operator 50 to operate the vehicle 10 (steering, accelerating, and decelerating).

[0049] The communication device 36 communicates with the outside of the vehicle 10. The communication device 36 communicates with the in-vehicle information processing device 20 via the communication network 8.

[0050] The remote monitoring server 40 is a computer installed in the remote monitoring center 30, or a collection of multiple computers. The remote monitoring server 40 includes one or more processors 41 (hereinafter simply referred to as processor 41) and one or more memories 42 (hereinafter simply referred to as memory 42) coupled to the processor 41. The memory 42 includes main memory and auxiliary memory. The memory 42 stores programs that can be executed by the processor 41 and various information related thereto. By executing programs, the processor 41 performs various processes. Programs can be stored in main memory or in computer-readable recording media, which are auxiliary memory.

[0051] Memory 42 is configured with at least two queues 42a and 42b. The first queue 42a stores multiple first data DT1s. The processor 41 places the latest first data DT1 into the first queue 42a and retrieves the oldest first data DT1 from the first queue 42a. The first queue 42a stores first data DT1s from the i-th to the i+j-th in the timestamp assignment order.

[0052] Multiple second data DT2s are stored in the second queue 42b. The processor 41 places the latest second data DT2 into the second queue 42b and retrieves the oldest second data DT2 from the second queue 42b. The second queue 42b stores second data DT2s from the i-th to the i+j-th in the timestamp assignment order TS. The processor 41 retrieves first data DT1 and second data DT2s with the same timestamp TS from the first queue 42a and the second queue 42b, respectively.

[0053] 6. Processing performed by the remote monitoring system according to the first embodiment Figure 9 is a block diagram showing the processing performed by the remote monitoring system 2 according to the first embodiment, more specifically, the processing performed by the processors 21 and 41 of the in-vehicle information processing device 20 and the remote monitoring server 40, respectively. The processing flow shown in this block diagram also represents the remote monitoring method according to the first embodiment of this disclosure.

[0054] The processor 21 of the in-vehicle information processing device 20 executes a timestamp generation process 211, an image compression process 212, an object detection process 213, a first data transmission process 214, and a second data transmission process 215. These processes are executed by the processor 21 when a program stored in memory 22 is executed by the processor 21.

[0055] In the timestamp generation process 211, the processor 21 generates a timestamp TS indicating the time when the image IMG was acquired by the camera 12. The generated timestamp TS is associated with the image IMG and stored in the memory 22.

[0056] In the image compression process 212, the processor 21 reads the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 applies a predetermined compression process to the image IMG to generate a compressed image CIMG. The generated compressed image CIMG is stored in the memory 22, linked to the timestamp TS.

[0057] In the object detection process 213, the processor 21 reads the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 uses image analysis techniques such as pattern matching and deep learning to detect objects in front of the vehicle 10 from the image IMG. The object detection result ODR is stored in the memory 22, linked to the timestamp TS.

[0058] In the first data transmission process 214, the processor 21 reads the first data DT1, which includes the compressed image CIMG and the timestamp TS, from the memory 22. Then, the processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0059] In the second data transmission process 215, the processor 21 reads the second data DT2, which includes the object detection result ODR and the timestamp TS, from the memory 22. Then, the processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0060] The processor 41 of the remote monitoring server 40 executes a first data reception process 411, a second data reception process 412, a simultaneous data extraction process 413, a compressed image restoration process 414, and a superimposed display process 415. These processes are executed by the processor 41 when a program stored in memory 42 is executed by the processor 41.

[0061] In the first data reception process 411, the processor 41 receives the first data DT1 from the in-vehicle information processing device 20 using the communication device 36. The processor 41 then stores the received first data DT1 in the first queue 42a.

[0062] In the second data reception process 412, the processor 41 receives the second data DT2 from the in-vehicle information processing device 20 using the communication device 36. The processor 41 then stores the received second data DT2 in the second queue 42b.

[0063] In the simultaneous data extraction process 413, the processor 41 extracts compressed image CIMG and object detection result ODR from their respective queues 42a and 42b in chronological order, provided that their timestamps TS are the same.

[0064] In the compressed image restoration process 414, the processor 41 restores the compressed image CIMG extracted together with the object detection result ODR using a restoration method corresponding to the compression method, and generates a restored image DIMG.

[0065] In the superimposed display process 415, the processor 41 superimposes the reconstructed image DIMG and the object detection result ODR and displays them on the monitoring screen 32a of the display device 32. Since the timestamp TS of the reconstructed image DIMG and the object detection result ODR are the same, there is no time difference between the image displayed on the display device 32 and the object detection result.

[0066] 7. Vehicle configuration according to the second embodiment Figure 10 is a block diagram showing the configuration of a vehicle 10 equipped with an in-vehicle information processing device 20 according to the second embodiment. In Figure 10, elements identical to those in the vehicle 10 according to the first embodiment are denoted by the same reference numerals. Descriptions of the configurations and functions already described will be omitted or simplified.

[0067] The information obtained from the sensor signals of the on-board sensor 14 (hereinafter referred to as sensor information) includes information that is useful to present to the remote operator 50. For example, LiDAR images obtained by imaging the LiDAR sensor signal, and information on the distance and direction of objects obtained from the sensor signals of millimeter-wave radar and ultrasonic sensors are important information for understanding the driving environment of the vehicle 10. In the second embodiment, the on-board information processing device 20 performs processing to obtain sensor information from the sensor signals. In the second embodiment, the sensor information is transmitted from the on-board information processing device 20 to the remote monitoring server 40. The remote monitoring server 40 overlays the sensor information on the reconstructed image along with the object detection result and displays it on the display device 32.

[0068] In the second embodiment, at least four storage areas 22a, 22b, 22c, and 22d are allocated in the memory 22. Similar to the first embodiment, the first storage area 22a stores pre-processing data DT0, which includes an image IMG captured by the camera 12 and a timestamp TS indicating the acquisition time of the image IMG. The second storage area 22a stores first data DT1, which includes a compressed image CIMG obtained by compressing the image IMG and a timestamp TS associated with the uncompressed image IMG. The third storage area 22c stores second data DT2, which includes an object detection result ODR obtained by object detection processing on the image IMG and a timestamp TS associated with the image IMG used for object detection.

[0069] The processor 21 stores the third processing data (hereinafter referred to as the third data) DT3 in the fourth storage area 22d of the memory 22. The third data DT3 includes sensor information SI obtained from the sensor signal of the on-board sensor 14 and a timestamp TS indicating the time the sensor signal was acquired. The processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0070] 8. Configuration of the remote monitoring center according to the second embodiment Figure 11 is a block diagram showing the configuration of a remote monitoring center 30 where the remote monitoring server 40 according to the second embodiment is located. In Figure 10, elements identical to those of the remote monitoring center 30 according to the first embodiment are denoted by the same reference numerals. The descriptions of the configuration and functions already described will be omitted or simplified.

[0071] In the second embodiment, at least three queues 42a, 42b, and 42c are configured in the memory 42. Similar to the first embodiment, the first queue 42a stores the first data DT1 from the i-th to the i+j-th in the timestamp assignment order. The second queue 42b stores the second data DT2 from the i-th to the i+j-th in the timestamp assignment order.

[0072] In the second embodiment, the newly configured third queue 42b stores multiple third data DT3s. The processor 41 places the latest third data DT3 into the third queue 42c and retrieves the oldest third data DT3 from the third queue 42c. The third queue 42c stores the third data DT3s from the i-th to the i+j-th in the timestamp assignment order TS. The processor 41 retrieves the first data DT1, second data DT2, and third data DT3 with the same timestamp TS from the first queue 42a, second queue 42b, and third queue 42c, respectively.

[0073] 9. Processing performed by the remote monitoring system according to the second embodiment Figure 12 is a block diagram showing the processes executed by the remote monitoring system 2 according to the second embodiment, more specifically, the processes executed by the processors 21 and 41 of the in-vehicle information processing device 20 and the remote monitoring server 40, respectively. The processing flow shown in this block diagram also represents the remote monitoring method according to the second embodiment of this disclosure. In Figure 10, the same reference numerals are used for processes that are the same as those executed in the first embodiment. Descriptions of processes that have already been described will be omitted or simplified.

[0074] In the second embodiment, the processor 21 of the in-vehicle information processing device 20 performs, in addition to the processing performed in the first embodiment, a timestamp generation process 216, a sensor signal processing 217, and a third data transmission process 218. Like the other processes, these processes are executed by the processor 21 when a program stored in the memory 22 is executed by the processor 21.

[0075] In the timestamp generation process 216, the processor 21 generates a timestamp TS indicating the time when the sensor signal SS was acquired from the on-board sensor 14. The generated timestamp TS is linked to the sensor signal SS and stored in the memory 22.

[0076] In sensor signal processing 217, the processor 21 generates sensor information SI from the sensor signal SS. Sensor information SI includes, for example, LiDAR images obtained from the LiDAR sensor signal, and distance and direction information obtained from the sensor signals of millimeter-wave radar and ultrasonic sensors. The generated sensor information SI is stored in memory 22, linked to a timestamp TS associated with the sensor signal SS.

[0077] In the third data transmission process 218, the processor 21 reads the third data DT3, which includes sensor information SI and timestamp TS, from the memory 22. Then, the processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0078] In the second embodiment, the processor 41 of the remote monitoring server 40 executes a third data reception process 416 in addition to the processes executed in the first embodiment. The processor 41 also executes a simultaneous data extraction process 417 and a superimposed display process 418, which are modified versions of the processes executed in the first embodiment. As with other processes, these processes are executed by the processor 41 when a program stored in memory 42 is executed by the processor 41.

[0079] In the third data reception process 416, the processor 41 receives the third data DT3 from the in-vehicle information processing device 20 using the communication device 36. The processor 41 then stores the received third data DT3 in the third queue 42c.

[0080] In the simultaneous data extraction process 417, the processor 41 extracts compressed image CIMG, object detection result ODR, and sensor information SI from their respective queues 42a, 42b, and 42c in chronological order, provided that their timestamps TS are the same.

[0081] In the superimposed display process 418, the processor 41 superimposes the reconstructed image DIMG, the object detection result ODR, and the sensor information SI and displays them on the monitoring screen 32a of the display device 32. Since the timestamp TS of the reconstructed image DIMG, the object detection result ODR, and the sensor information SI are the same, there is no time difference between the image displayed on the display device 32 and the object detection result and sensor information. [Explanation of Symbols]

[0082] 2 Remote monitoring system 8. Communication Network 10 vehicles 12 cameras 14. In-vehicle sensors 16. Communication equipment 20 In-vehicle information processing system 21 processors 22 memory 30 Remote Monitoring Center 32a Monitoring screen 36 Communication equipment 40 Remote monitoring server 41 processors 42 memory 50 Remote Operators

Claims

1. A remote monitoring system for remotely monitoring vehicles, The camera mounted on the aforementioned vehicle, An in-vehicle information processing device mounted on the aforementioned vehicle and connected to the aforementioned camera, The vehicle includes a remote monitoring server connected to the in-vehicle information processing device via a communication network, The in-vehicle information processing device is Object detection processing that uses image analysis technology to detect an object in front of the vehicle from an image acquired by the aforementioned camera, A compression process that compresses the aforementioned image to generate a compressed image, A first data transmission process that transmits first data including the compressed image and the image acquisition time of the image before compression, The system performs a second data transmission process which transmits a second data set including an object detection result that includes the type and location of the detected object, and the image acquisition time of the image used for object detection. The aforementioned remote monitoring server is The first data reception process receives the first data and stores it in memory, A second data reception process that receives the second data and stores it in the memory, An extraction process that extracts the compressed image and the object detection result from the memory in chronological order, where the image acquisition time is the same. The system performs a display process that superimposes the restored image obtained by restoring the extracted compressed image and the extracted object detection results onto a monitoring screen and displays them together. A remote monitoring system characterized by the following features.

2. A remote monitoring method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network, The in-vehicle information processing device, Using image analysis technology, an object in front of the vehicle is detected from an image acquired by a camera mounted on the vehicle. The aforementioned image is compressed to generate a compressed image, The system transmits first data including the compressed image and the image acquisition time of the image before compression. The system transmits a second set of data, which includes the object detection result, including the type and location of the detected object, and the image acquisition time of the image used for object detection. The aforementioned remote monitoring server, The first data is received and stored in memory, The second data is received and stored in the memory, The compressed image and the object detection result, whose image acquisition times are the same, are extracted from the memory in chronological order. The extracted compressed image is restored to form a reconstructed image, and the extracted object detection results are superimposed and displayed on the monitoring screen. A remote monitoring method characterized by the following features.

3. A remote monitoring server connected to an in-vehicle information processing device via a communication network, One or more memory locations storing one or more programs, The system comprises one or more processors coupled with the one or more memories, The one or more processors, when executing the one or more programs, The system receives first data from the in-vehicle information processing device, which includes a compressed image generated by compressing an image acquired by an in-vehicle camera and the image acquisition time of the image before compression, and stores it in one or more of the memories. The in-vehicle information processing device receives second data from the in-vehicle information processing device, which includes the object detection result including the type and position of an object in front of the vehicle detected from the image using image analysis technology, and the image acquisition time of the image used for object detection, and stores it in one or more of the memories. From one or more of the aforementioned memories, the compressed images and object detection results, whose image acquisition times are the same, are extracted in chronological order. The extracted compressed image is restored to form a reconstructed image, and the extracted object detection results are superimposed and displayed on the monitoring screen. A remote monitoring server characterized by the following features.

4. An in-vehicle information processing device connected to a remote monitoring server via a communication network, One or more memory locations storing one or more programs, The system comprises one or more processors coupled with the one or more memories, The one or more processors, when executing the one or more programs, Using image analysis technology, objects in front of the vehicle are detected from images acquired by the vehicle's onboard camera. The aforementioned image is compressed to generate a compressed image, The first data, including the compressed image and the image acquisition time of the image before compression, is transmitted to the remote monitoring server. Second data, including the object detection result (including the type and location of the detected object) and the image acquisition time of the image used for object detection, is transmitted to the remote monitoring server. An in-vehicle information processing device characterized by the following:

Images