Remote monitoring system, remote monitoring method, remote monitoring server, and in-vehicle information processing device

Abstract

To provide a remote monitoring technique that can improve the accuracy of object detection results to be displayed on a monitoring screen while reducing communication costs.SOLUTION: An in-vehicle information processing device 20 compresses an image IMG from a camera 12, and transmits first data including the compressed image CIMG and a timestamp TS indicating an acquisition time of the image IMG. The in-vehicle information processing device 20 also detects an object in front of a vehicle from the image IMG by using an image analysis technique, and transmits second data including an object detection result ODR including the type and position of the detected object and the timestamp TS indicating the acquisition time of the image IMG. A remote monitoring server 40 receives the first data and the second data and stores them in a memory, extracts the compressed image CIMG and the object detection result ODR having the same timestamp TS from the memory in chronological order, and displays a restored image DIMG obtained by restoring the compressed image CIMG and the object detection result ODR while the restored image DIMG and the object detection result ODR are superimposed on each other on a monitoring screen.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a remote monitoring system, a remote monitoring method, a remote monitoring server, and an in-vehicle information processing device that are suitable for use in remote monitoring of a vehicle. [Background technology]

[0002] Patent Document 1 discloses a technology related to a remote control device that remotely controls an autonomous vehicle that is no longer capable of autonomous driving. When displaying an image captured by a camera mounted on the autonomous vehicle on a display unit, the remote control device according to this conventional technology analyzes the captured image and highlights specific objects in the image.

[0003] Images captured by the camera are compressed before being transmitted from the autonomous vehicle to the remote control device. Image compression is essential to reduce communication costs, but the resolution of the image decreases as a result of compression. For this reason, the above-mentioned conventional technology is unable to analyze images with high accuracy, and there is a risk that specific objects may not be highlighted appropriately.

[0004] In addition to the above Patent Document 1, the following Patent Documents 2 and 3 can be mentioned as examples of prior art documents that show the technical level of the technical field to which the present application pertains. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-161039 A [Patent Document 2] JP 2018-56838 A [Patent Document 3] JP 2011-96063 A Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in consideration of the above-mentioned problems. A first object of the present disclosure is to provide a remote monitoring system and a remote monitoring method that can improve the accuracy of object detection results displayed on a monitoring screen while reducing communication costs.

[0007] A second objective of the present disclosure is to provide a remote monitoring server that can improve the accuracy of object detection results displayed on the monitoring screen while reducing communication costs between the server and an in-vehicle information processing device.

[0008] Furthermore, a third objective of the present disclosure is to provide an in-vehicle information processing device that can improve the accuracy of object detection results displayed on the monitoring screen of the remote monitoring server while reducing communication costs between the remote monitoring server and the in-vehicle information processing device. [Means for solving the problem]

[0009] The remote monitoring system according to the present disclosure includes a camera mounted on a vehicle, an in-vehicle information processing device mounted on the vehicle and connected to the camera, and a remote monitoring server connected to the in-vehicle information processing device via a communication network. The in-vehicle information processing device executes the following object detection process, compression process, first data transmission process, and second data transmission process. The remote monitoring server executes the following first data reception process, second data reception process, extraction process, and display process.

[0010] In the object detection process by the in-vehicle information processing device, an object in front of the vehicle is detected from an image captured by a camera. In the compression process, an image is compressed to generate a compressed image. In the first data transmission process, first data including the compressed image and the image capture time of the image before compression is transmitted to a remote monitoring server. In the second data transmission process, second data including the object detection result and the image capture time of the image used for object detection is transmitted to a remote monitoring server.

[0011] In the first data reception process by the remote monitoring server, the first data transmitted from the in-vehicle information processing device is received and stored in the memory. In the second data reception process, the second data transmitted from the in-vehicle information processing device is received and stored in the memory. In the extraction process, compressed images and object detection results having the same image acquisition time are extracted in chronological order from the memory. In the display process, a restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on the monitoring screen.

[0012] In this remote monitoring system, the in-vehicle information processing device may further execute a third data transmission process, and the remote monitoring server may further execute a third data reception process. In the third data transmission process, third data including sensor information obtained from a sensor signal of an in-vehicle sensor and a signal acquisition time of the sensor signal is transmitted. In the third data reception process, the third data transmitted from the in-vehicle information processing device is received and stored in a memory. In this case, in the extraction process by the remote monitoring server, a compressed image and an object detection result having the same image acquisition time are extracted in chronological order from the memory, and sensor information having the same signal acquisition time as the image acquisition time is extracted in chronological order. In addition, in the display process, the extracted sensor information is superimposed on the restored image together with the extracted object detection result and displayed on the monitoring screen.

[0013] The remote monitoring method according to the present disclosure is a method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network. The remote monitoring method causes the in-vehicle information processing device to detect an object in front of the vehicle from an image captured by a camera mounted on the vehicle, compress the image to generate a compressed image, transmit first data including the compressed image and the image capture time of the image before compression, and transmit second data including the object detection result and the image capture time of the image used for the object detection. The remote monitoring method also causes the remote monitoring server to receive the first data and store it in a memory, receive the second data and store it in a memory, extract the compressed image and the object detection result having the same image capture time from the memory in chronological order, and display a restored image obtained by restoring the extracted compressed image and the extracted object detection result superimposed on the monitoring screen.

[0014] The remote monitoring method may cause the in-vehicle information processing device to transmit third data including sensor information obtained from a sensor signal of the in-vehicle sensor and a signal acquisition time of the sensor signal. The remote monitoring method may cause the remote monitoring server to receive the third data, store it in a memory, extract from the memory in chronological order compressed images and object detection results having the same image acquisition time, and extract in chronological order sensor information having the same signal acquisition time as the image acquisition time, and display the extracted sensor information together with the object detection result on a monitoring screen by superimposing them on a restored image.

[0015] The remote monitoring server according to the present disclosure is a server connected to an in-vehicle information processing device via a communication network. The remote monitoring server includes one or more memories storing one or more programs, and one or more processors coupled to the one or more memories. The one or more processors execute the following processes when the one or more programs are executed.

[0016] In the process executed by one or more processors of the remote monitoring server, first data including a compressed image generated by compressing an image captured by an in-vehicle camera and an image acquisition time of the image before compression is received from the in-vehicle information processing device and stored in one or more memories. Also, second data including an object detection result based on the image and an image acquisition time of an image used for object detection is received from the in-vehicle information processing device and stored in one or more memories. Furthermore, compressed images and object detection results having the same image acquisition time are extracted in chronological order from the one or more memories. Furthermore, a restored image obtained by restoring the extracted compressed image and the extracted object detection result are displayed on a monitoring screen in a superimposed manner.

[0017] The in-vehicle information processing device according to the present disclosure is a device connected to a remote monitoring server via a communication network. The in-vehicle information processing device includes one or more memories storing one or more programs, and one or more processors coupled to the one or more memories. The one or more processors execute the following processes when the one or more programs are executed.

[0018] In the process executed by one or more processors of the in-vehicle information processing device, an object ahead of the vehicle is detected from an image captured by an in-vehicle camera. The image is compressed to generate a compressed image. First data including the compressed image and the image capture time of the image before compression is transmitted to a remote monitoring server. Second data including the object detection result and the image capture time of the image used for the object detection is transmitted to the remote monitoring server. Effect of the Invention

[0019] According to the present disclosure, since object detection is performed from an image before compression, it is possible to prevent a decrease in detection accuracy that occurs when object detection is performed from a compressed image. This makes it possible to improve the accuracy of the object detection results displayed on the monitoring screen while reducing the communication cost with the in-vehicle information processing device. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating a schematic configuration of a remote monitoring system according to an embodiment of the present disclosure. [Diagram 2] FIG. 11 is a diagram showing an example of an image before compression for explaining a problem with the remote monitoring system. [Diagram 3] FIG. 11 is a diagram showing an example of an object detection result from an image before compression, for explaining a problem with the remote monitoring system. [Figure 4] FIG. 13 is a diagram showing an example of a compressed image for explaining a problem with the remote monitoring system. [Diagram 5] FIG. 11 is a diagram showing an example of an object detection result from a compressed image for explaining a problem with the remote monitoring system. [Figure 6] FIG. 1 is a diagram illustrating an overview of features of a remote monitoring system according to an embodiment of the present disclosure. [Figure 7] 1 is a block diagram showing a configuration of a vehicle according to a first embodiment of the present disclosure. [Figure 8] 1 is a block diagram showing a configuration of a remote monitoring center according to a first embodiment of the present disclosure. [Figure 9] 2 is a block diagram showing processes executed by each processor of the in-vehicle information processing device and the remote monitoring server according to the first embodiment of the present disclosure. FIG. [Figure 10] FIG. 4 is a block diagram showing a configuration of a vehicle according to a second embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram showing a configuration of a remote monitoring center according to a second embodiment of the present disclosure. [Figure 12] 13 is a block diagram showing processes executed by each processor of an in-vehicle information processing device and a remote monitoring server according to a second embodiment of the present disclosure. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. However, when the number, quantity, amount, range, etc. of each element is mentioned in the embodiments shown below, the idea of ​​the present disclosure is not limited to the mentioned number unless it is specifically stated or clearly specified in principle. Furthermore, the structures etc. described in the embodiments shown below are not necessarily essential to the idea of ​​the present disclosure unless it is specifically stated or clearly specified in principle.

[0022] 1. Overview of remote monitoring system 1 is a diagram illustrating a schematic configuration of a remote monitoring system according to an embodiment of the present disclosure. The remote monitoring system 2 is a system in which a vehicle 10 traveling on a road is remotely monitored by a remote operator 50. The remote monitoring in the present disclosure includes monitoring for remotely driving the vehicle 10, monitoring for remotely supporting the autonomously traveling vehicle 10 according to the situation, and monitoring for checking the surrounding environment of the autonomously traveling vehicle 10.

[0023] A vehicle 10 to be remotely monitored is equipped with a camera 12. The camera 12 may be a stereo camera or a monocular camera. The capturing range IA of the camera 12 covers at least the area ahead in the traveling direction of the vehicle 10. For example, when the vehicle 10 is traveling in a driving lane DL, the area ahead of the driving lane DL and both the left and right sides of the driving lane DL are included in the capturing range IA of the camera 12.

[0024] The vehicle 10 includes an on-board information processing device 20. An image captured by a camera 12 is input to the on-board information processing device 20. The on-board information processing device 20 is connected to a remote monitoring server 40 of a remote monitoring center 30 via a communication network 8. The communication network 8 includes mobile communications such as 4G and 5G. The image captured by the camera 12 is sent from the on-board information processing device 20 to the remote monitoring server 40 via the communication network 8.

[0025] In the remote monitoring center 30, the image of the camera 12 transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is displayed on the monitoring screen 32a of the display device 32. As the display device 32, for example, a liquid crystal display, an organic EL display, a head mounted display, or a touch panel can be used. The remote operator 50 can monitor the area ahead of the vehicle 10 by the image of the camera 12 displayed on the monitoring screen 32a of the display device 32. For example, when an obstacle OJ is present within the shooting range IA of the camera 12, the remote operator 50 can recognize the presence of the obstacle OJ on the monitoring screen 32a of the display device 32.

[0026] The remote monitoring system 2 has a function of assisting the remote operator 50 in recognizing an obstacle OJ. The obstacle OJ shown in the image of the camera 12 can be automatically detected by using image analysis techniques such as pattern matching and deep learning. By identifying the obstacle OJ shown in the image using this object detection function, it is possible to inform the remote operator 50 of where the obstacle OJ is located on the monitoring screen 32a.

[0027] On the other hand, a reduction in communication costs is required for the remote monitoring system 2. The image captured by the camera 12 is encoded in the in-vehicle information processing device 20, and the encoded image data is transmitted to the remote monitoring server 40. In this encoding, the image captured by the camera 12 is compressed by a predetermined compression method. The image data transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is decoded in the remote monitoring server 40. The image restored by the decoding by the remote monitoring server 40 is displayed on the display device 32.

[0028] There is no limitation on the image transmission format, and there is no limitation on the image compression format. When the image transmission format is a video stream format, for example, H.264, H.265, VP8, VP9, ​​VP10, and AV1 can be used as the image compression format. When the image transmission format is a format in which images are transmitted one by one, for example, JPEG, Motion JPEG, and PNG can be used as the image compression format. Preferably, H.264, VP8, or VP9 is used. Motion JPEG, VP10, and AV1 are also suitable compression formats for the remote monitoring system 2.

[0029] 2. Issues with remote monitoring systems As described above, the remote monitoring system 2 is required to assist the remote operator 50 by detecting objects from images captured by the camera 12, and to reduce communication costs by compressing the images captured by the camera 12. However, it is not easy to simultaneously satisfy these two requirements.

[0030] Here, four images are shown. FIG. 2 is a diagram showing an example of an original image captured by the camera 12, i.e., an image before compression. FIG. 3 is a diagram showing an example of an object detection result from the original image shown in FIG. 2. FIG. 4 is a diagram showing an example of an image obtained by compressing the original image shown in FIG. 2, i.e., an image after compression. FIG. 5 is a diagram showing an example of an object detection result from the image after compression shown in FIG. 4. However, although these images are originally color images, they are binarized in the figures for convenience as application documents.

[0031] The image displayed on the monitoring screen 32a of the display device 32 is the compressed image shown in Fig. 4. However, the resolution of the compressed image is lower than that of the original image. This is also evident from the binarized image, as can be seen from a comparison between the image shown in Fig. 2 and the image shown in Fig. 4. For this reason, while two people and two vehicles are detected from the original image with high resolution as shown in Fig. 3, the two vehicles cannot be detected from the compressed image as shown in Fig. 5.

[0032] As in the above example, there are cases where an object that should be detected cannot be detected from the compressed image. Also, although not shown in the above example, there are cases where a non-existent object is erroneously detected from the compressed image. In other words, although the image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 needs to be compressed, an object in front of the vehicle 10 cannot be detected with high accuracy from the compressed image.

[0033] 3. Overview of features of remote monitoring system In the remote monitoring system 2, the above problems are solved as follows: FIG.

[0034] 6 shows the flow of information in the remote monitoring system 2. In the remote monitoring system 2, a compressed image is obtained from an original image before compression in the in-vehicle information processing device 20. In parallel with this, an object detection result is obtained from the original image before compression in the in-vehicle information processing device 20. The separately obtained compressed image and the object detection result are separately transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 via the communication network 8. That is, in the remote monitoring system 2, a process of detecting an object from an image of the camera 12 is performed in the in-vehicle information processing device 20 before transmitting the image to the remote monitoring server 40, and moreover, the process is performed on the original image before compression, not on the compressed image.

[0035] The compressed image transmitted from the in-vehicle information processing device 20 to the remote monitoring server 40 is restored in the remote monitoring server 40. The object detection result transmitted from the in-vehicle information processing device 20 separately from the compressed image is superimposed on the restored image by the remote monitoring server 40. As a result, a superimposed display screen is obtained in which the object detection result is superimposed on the restored image. To explain this by taking a specific example, the compressed image shown in Fig. 4 is displayed on the monitoring screen 32a of the display device 32, and the object detection result shown in Fig. 3 is superimposed thereon.

[0036] According to the remote monitoring system 2 having the above-mentioned characteristics, it becomes possible to detect objects ahead of the vehicle 10 with high accuracy by performing object detection on the original image before compression using the in-vehicle information processing device 20. Then, by displaying the object detection result together with the compressed image on the monitoring screen 32a of the display device 32, it is possible to assist the remote operator 50 in remote monitoring.

[0037] Although the compression process of the original image and the object detection process from the original image are performed by the in-vehicle information processing device 20, they are executed separately. Therefore, even if the original image is the same, there may be a difference between the time when the compressed image is obtained and the time when the object detection result is obtained. If the display device 32 displays the image with this time difference, it may cause a misunderstanding by the remote operator 50.

[0038] The following chapters will explain in detail the configuration and functions of the remote monitoring system 2. As will be explained there, the remote monitoring system 2 is technically designed so as to prevent a time lag from occurring between the image displayed on the display device 32 and the object detection result.

[0039] 4. Configuration of the Vehicle According to the First Embodiment 7 is a block diagram showing the configuration of a vehicle 10 equipped with an on-board information processing device 20 according to the first embodiment. The vehicle 10 is equipped with a camera 12, an on-board sensor 14, a communication device 16, a driving device 18, and the on-board information processing device 20. The details of the camera 12 equipped in the vehicle 10 are as described above.

[0040] The on-board sensor 14 includes a status sensor that acquires information regarding the motion of the vehicle 10. The status sensor includes, for example, at least one of a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. The on-board sensor 14 also includes a position sensor that detects the position and orientation of the vehicle 10. An example of the position sensor is a Global Positioning System (GPS) sensor. Furthermore, the on-board sensor 14 may include a recognition sensor other than the camera 12. The recognition sensor recognizes (detects) the situation around the vehicle 10. An example of the recognition sensor is a laser imaging detection and ranging (LiDAR), a millimeter wave radar, and an ultrasonic sensor.

[0041] The communication device 16 communicates with the outside of the vehicle 10. The communication device 16 communicates with the remote monitoring server 40 via the communication network 8.

[0042] The traveling device 18 includes a steering device for steering the vehicle 10, a drive device for driving the vehicle 10, and a braking device for braking the vehicle 10. The steering device includes, for example, a power steering system, a steer-by-wire steering system, and a rear-wheel steering system. The drive device includes, for example, an engine, an EV system, and a hybrid system. The braking device includes, for example, a hydraulic brake and a regenerative brake.

[0043] The in-vehicle information processing device 20 is an ECU (Electronic Control Unit) mounted on the vehicle 10, or a collection of multiple ECUs. The in-vehicle information processing device 20 includes one or multiple processors 21 (hereinafter simply referred to as the processor 21) and one or multiple memories 22 (hereinafter simply referred to as the memory 22) coupled to the processor 21. The memory 22 includes a main storage device and an auxiliary storage device. The memory 22 stores programs executable by the processor 21 and various pieces of information related to the programs. The processor 21 executes the programs to realize various processes by the processor 21. The programs can be stored in the main storage device, or can be stored in a computer-readable recording medium that is an auxiliary storage device.

[0044] At least three storage areas 22a, 22b, and 22c are secured in the memory 22. The processor 21 stores the unprocessed data DT0 in the first storage area 22a of the memory 22. The unprocessed data DT0 includes an image IMG captured by the camera 12 and a timestamp TS indicating the acquisition time of the image IMG. The processor 21 reads the unprocessed data DT0 and performs a compression process. The processor 21 also reads the unprocessed data DT0 and performs an object detection process.

[0045] The processor 21 stores first processing data (hereinafter referred to as first data) DT1 in the second storage area 22b of the memory 22. The first data DT1 includes a compressed image CIMG obtained by a compression process of the image IMG and a time stamp TS associated with the uncompressed image IMG. The processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0046] The processor 21 stores second processing data (hereinafter referred to as second data) DT2 in the third storage area 22c of the memory 22. The second data DT2 includes an object detection result ODR obtained by the object detection process on the image IMG and a time stamp TS associated with the image IMG used for the object detection. The processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0047] 5. Configuration of the remote monitoring center according to the first embodiment 8 is a block diagram showing the configuration of a remote monitoring center 30 in which a remote monitoring server 40 according to the first embodiment is arranged. In addition to the remote monitoring server 40 and a display device 32, the remote monitoring center 30 is provided with an input device 34 and a communication device 36. Like the display device 32, the input device 34 and the communication device 36 are connected to the remote monitoring server 40.

[0048] The input device 34 is an interface for receiving input from the remote operator 50. Examples of the input device 34 include a touch panel, a keyboard, and a mouse. When the remote monitoring is monitoring for remote driving, the input device 34 includes a driving operation member for the remote operator 50 to drive (steer, accelerate, and decelerate) the vehicle 10.

[0049] The communication device 36 communicates with the outside of the vehicle 10. The communication device 36 communicates with the in-vehicle information processing device 20 via the communication network 8.

[0050] The remote monitoring server 40 is a computer installed in the remote monitoring center 30, or a group of multiple computers. The remote monitoring server 40 includes one or more processors 41 (hereinafter simply referred to as the processor 41) and one or more memories 42 (hereinafter simply referred to as the memory 42) coupled to the processor 41. The memory 42 includes a main storage device and an auxiliary storage device. The memory 42 stores programs executable by the processor 41 and various information related to the programs. The processor 41 executes the programs to realize various processes by the processor 41. The programs can be stored in the main storage device, or can be stored in a computer-readable recording medium that is an auxiliary storage device.

[0051] At least two queues 42a and 42b are configured in the memory 42. A plurality of first data DT1 are accumulated in the first queue 42a. The processor 41 puts the latest first data DT1 into the first queue 42a and takes out the oldest first data DT1 at the top from the first queue 42a. The first queue 42a stores the i-th to (i+j)-th first data DT1 in the order in which the time stamps TS are assigned.

[0052] A plurality of second data DT2 are accumulated in the second queue 42b. The processor 41 puts the latest second data DT2 into the second queue 42b and removes the oldest second data DT2 at the top from the second queue 42b. The second queue 42b stores the second data DT2 from the i-th to the i+j-th in the order in which the timestamps TS were assigned. The processor 41 removes the first data DT1 and the second data DT2 having the same time stamps TS from the first queue 42a and the second queue 42b, respectively.

[0053] 6. Processing Executed in the Remote Monitoring System According to the First Embodiment 9 is a block diagram showing the processing executed in the remote monitoring system 2 according to the first embodiment, more specifically, the processing executed by each of the processors 21, 41 of the in-vehicle information processing device 20 and the remote monitoring server 40. The flow of processing shown in this block diagram also represents the remote monitoring method according to the first embodiment of the present disclosure.

[0054] The processor 21 of the in-vehicle information processing device 20 executes a timestamp generation process 211, an image compression process 212, an object detection process 213, a first data transmission process 214, and a second data transmission process 215. These processes are executed by the processor 21 when a program stored in the memory 22 is executed in the processor 21.

[0055] In the time stamp generation process 211, the processor 21 generates a time stamp TS indicating the time when the image IMG was captured by the camera 12. The generated time stamp TS is stored in the memory 22 in association with the image IMG.

[0056] In the image compression process 212, the processor 21 reads the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 performs a predetermined compression process on the image IMG to generate a compressed image CIMG. The generated compressed image CIMG is stored in the memory 22 in association with the time stamp TS.

[0057] In the object detection process 213, the processor 21 reads out the image IMG captured by the camera 12 from the memory 22. Then, the processor 21 detects an object in front of the vehicle 10 from the image IMG using an image analysis technique such as pattern matching or deep learning. The object detection result ODR is stored in the memory 22 in association with the time stamp TS.

[0058] In the first data transmission process 214, the processor 21 reads out the first data DT1 including the compressed image CIMG and the time stamp TS from the memory 22. Then, the processor 21 transmits the first data DT1 to the remote monitoring server 40 using the communication device 16.

[0059] In the second data transmission process 215, the processor 21 reads out the second data DT2 including the object detection result ODR and the time stamp TS from the memory 22. Then, the processor 21 transmits the second data DT2 to the remote monitoring server 40 using the communication device 16.

[0060] The processor 41 of the remote monitoring server 40 executes a first data reception process 411, a second data reception process 412, a simultaneous data extraction process 413, a compressed image restoration process 414, and a superimposed display process 415. These processes are executed by the processor 41 when a program stored in the memory 42 is executed in the processor 41.

[0061] In the first data reception process 411, the processor 41 receives the first data DT1 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received first data DT1 in the first queue 42a.

[0062] In the second data reception process 412, the processor 41 receives the second data DT2 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received second data DT2 in the second queue 42b.

[0063] In the simultaneous data extraction process 413, the processor 41 extracts, in chronological order, compressed images CIMG and object detection results ODR whose time stamps TS are the same, from the respective queues 42a and 42b.

[0064] In the compressed image restoration process 414, the processor 41 restores the compressed image CIMG extracted together with the object detection result ODR using a restoration method corresponding to the compression method, to generate a restored image DIMG.

[0065] In the superimposed display process 415, the processor 41 superimposes the restored image DIMG and the object detection result ODR and displays them on the monitoring screen 32a of the display device 32. Since the time stamps TS of the restored image DIMG and the object detection result ODR are the same, no time lag occurs between the image displayed on the display device 32 and the object detection result.

[0066] 7. Configuration of vehicle according to second embodiment Fig. 10 is a block diagram showing the configuration of a vehicle 10 equipped with an in-vehicle information processing device 20 according to the second embodiment. In Fig. 10, the same elements as those of the vehicle 10 according to the first embodiment are denoted by the same reference numerals. Descriptions of the configuration and functions already described will be omitted or simplified.

[0067] The information obtained from the sensor signals of the on-board sensors 14 (hereinafter referred to as sensor information) includes information that is useful to present to the remote operator 50. For example, a LiDAR image obtained by imaging a LiDAR sensor signal, and information on the distance and direction of an object obtained from a sensor signal of a millimeter wave radar or an ultrasonic sensor are important information for knowing the driving environment of the vehicle 10. In the second embodiment, a process for obtaining the sensor information from the sensor signal is performed in the on-board information processing device 20. In the second embodiment, the sensor information is transmitted from the on-board information processing device 20 to the remote monitoring server 40. The remote monitoring server 40 superimposes the sensor information on the restored image together with the object detection result and displays it on the display device 32.

[0068] In the second embodiment, at least four storage areas 22a, 22b, 22c, and 22d are secured in the memory 22. As in the first embodiment, the first storage area 22a stores pre-processing data DT0 including an image IMG captured by the camera 12 and a timestamp TS indicating the acquisition time of the image IMG. The second storage area 22a stores first data DT1 including a compressed image CIMG obtained by compressing the image IMG and a timestamp TS linked to the pre-compressed image IMG. The third storage area 22c stores second data DT2 including an object detection result ODR obtained by object detection processing on the image IMG and a timestamp TS linked to the image IMG used for object detection.

[0069] The processor 21 stores third processing data (hereinafter referred to as third data) DT3 in a fourth storage area 22d of the memory 22. The third data DT3 includes sensor information SI obtained from a sensor signal of the in-vehicle sensor 14 and a timestamp TS indicating the acquisition time of the sensor signal. The processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0070] 8. Configuration of remote monitoring center according to the second embodiment Fig. 11 is a block diagram showing the configuration of a remote monitoring center 30 in which a remote monitoring server 40 according to the second embodiment is arranged. In Fig. 10, the same elements as those in the remote monitoring center 30 according to the first embodiment are denoted by the same reference numerals. Descriptions of the configuration and functions already described will be omitted or simplified.

[0071] In the second embodiment, at least three queues 42a, 42b, and 42c are configured in the memory 42. As in the first embodiment, the first queue 42a stores the first data DT1 from the i-th to the i+j-th in the order in which the timestamps TS are assigned. The second queue 42b stores the second data DT2 from the i-th to the i+j-th in the order in which the timestamps TS are assigned.

[0072] A plurality of third data DT3 are stored in the third queue 42b newly configured in the second embodiment. The processor 41 puts the latest third data DT3 into the third queue 42c and takes out the oldest third data DT3 at the top from the third queue 42c. The third queue 42c stores the i-th to (i+j-th) third data DT3 in the order in which the timestamps TS were added. The processor 41 takes out the first data DT1, the second data DT2, and the third data DT3, which have the same time stamps TS, from the first queue 42a, the second queue 42b, and the third queue 42c, respectively.

[0073] 9. Processing executed in the remote monitoring system according to the second embodiment Fig. 12 is a block diagram showing the processing executed in the remote monitoring system 2 according to the second embodiment, more specifically, the processing executed by each of the processors 21, 41 of the in-vehicle information processing device 20 and the remote monitoring server 40. The flow of processing shown in this block diagram also represents the remote monitoring method according to the second embodiment of the present disclosure. In Fig. 10, the same processes as those executed in the first embodiment are denoted by the same reference numerals. The description of the processes already described will be omitted or simplified.

[0074] The processor 21 of the in-vehicle information processing device 20 according to the second embodiment executes a time stamp generation process 216, a sensor signal process 217, and a third data transmission process 218, in addition to the processes executed in the first embodiment. As with the other processes, these processes are processes executed by the processor 21 when a program stored in the memory 22 is executed in the processor 21.

[0075] In the time stamp generation process 216, the processor 21 generates a time stamp TS indicating the time when the sensor signal SS was acquired from the vehicle-mounted sensor 14. The generated time stamp TS is stored in the memory 22 in association with the sensor signal SS.

[0076] In the sensor signal processing 217, the processor 21 generates sensor information SI from the sensor signal SS. The sensor information SI includes, for example, a LiDAR image obtained from a LiDAR sensor signal, and distance information and direction information obtained from a millimeter wave radar or ultrasonic sensor sensor signal. The generated sensor information SI is stored in the memory 22 in association with a time stamp TS associated with the sensor signal SS.

[0077] In the third data transmission process 218, the processor 21 reads out the third data DT3 including the sensor information SI and the time stamp TS from the memory 22. Then, the processor 21 transmits the third data DT3 to the remote monitoring server 40 using the communication device 16.

[0078] The processor 41 of the remote monitoring server 40 according to the second embodiment executes a third data reception process 416 in addition to the processes executed in the first embodiment. The processor 41 also executes a simultaneous data extraction process 417 and a superimposed display process 418, which are modified from the processes executed in the first embodiment. As with the other processes, these processes are executed by the processor 41 when a program stored in the memory 42 is executed in the processor 41.

[0079] In the third data reception process 416, the processor 41 receives the third data DT3 from the in-vehicle information processing device 20 using the communication device 36. Then, the processor 41 stores the received third data DT3 in the third queue 42c.

[0080] In the simultaneous data extraction process 417, the processor 41 extracts, in chronological order, the compressed images CIMG, the object detection results ODR, and the sensor information SI having the same time stamps TS from the respective queues 42a, 42b, and 42c.

[0081] In the superimposed display process 418, the processor 41 superimposes the restored image DIMG, the object detection result ODR, and the sensor information SI and displays them on the monitoring screen 32a of the display device 32. Since the time stamps TS of the restored image DIMG, the object detection result ODR, and the sensor information SI are the same, no time lag occurs between the image displayed on the display device 32, the object detection result, and the sensor information. [Explanation of symbols]

[0082] 2. Remote monitoring system 8. Communication Networks 10 Vehicles 12 Camera 14 In-vehicle sensors 16 Communication Equipment 20. Vehicle-mounted information processing device 21 Processors 22 Memory 30 Remote Monitoring Center 32a Monitoring screen 36 Communication Equipment 40 Remote monitoring server 41 Processors 42 Memory 50 Remote Operator

Claims

1. A remote monitoring system for remotely monitoring a vehicle, comprising: A camera mounted on the vehicle; an in-vehicle information processing device mounted on the vehicle and connected to the camera; a remote monitoring server connected to the in-vehicle information processing device via a communication network, The in-vehicle information processing device includes: An object detection process for detecting an object in front of the vehicle using an image analysis technique from an image acquired by the camera; a compression process for compressing the image to generate a compressed image; a first data transmission process for transmitting first data including the compressed image and an image acquisition time of the image before compression; a second data transmission process for transmitting second data including an object detection result including a type and a position of the detected object and the image acquisition time of the image used for the object detection; The remote monitoring server includes: a first data receiving process for receiving the first data and storing the first data in a memory; a second data receiving process for receiving the second data and storing the second data in the memory; an extraction process of extracting, in time series, the compressed images and the object detection results, the images being acquired at the same time from the memory; and displaying the extracted compressed image and the extracted object detection result superimposed on each other on a monitoring screen. A remote monitoring system comprising:

2. A remote monitoring method for remotely monitoring a vehicle using an in-vehicle information processing device and a remote monitoring server connected via a communication network, comprising: The in-vehicle information processing device includes: Detecting an object in front of the vehicle using an image analysis technique from an image acquired by a camera mounted on the vehicle; compressing the image to generate a compressed image; transmitting first data including the compressed image and an image acquisition time of the image before compression; transmit second data including an object detection result including a type and a position of the detected object and the image acquisition time of the image used for the object detection; The remote monitoring server includes: receiving the first data and storing it in a memory; receiving the second data and storing it in the memory; extracting, from the memory, the compressed images and the object detection results, the images having the same acquisition time, in time series; A restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on a monitoring screen. A remote monitoring method comprising:

3. A remote monitoring server connected to an in-vehicle information processing device via a communication network, one or more memories storing one or more programs; one or more processors coupled to the one or more memories; The one or more processors, when executing the one or more programs, receiving first data from the in-vehicle information processing device, the first data including a compressed image generated by compressing an image captured by an in-vehicle camera and an image capture time of the image before compression, and storing the first data in the one or more memories; receiving from the in-vehicle information processing device second data including an object detection result including a type and a position of an object in front of the vehicle detected from the image using an image analysis technique and the image acquisition time of the image used for the object detection, and storing the second data in the one or more memories; extracting, in time series, the compressed images and the object detection results whose image acquisition times are the same from the one or more memories; A restored image obtained by restoring the extracted compressed image and the extracted object detection result are superimposed and displayed on a monitoring screen. A remote monitoring server comprising:

4. An in-vehicle information processing device connected to a remote monitoring server via a communication network, one or more memories storing one or more programs; one or more processors coupled to the one or more memories; The one or more processors, when executing the one or more programs, The system uses image analysis technology to detect objects in front of the vehicle from images captured by the vehicle's onboard camera, compressing the image to generate a compressed image; transmitting first data including the compressed image and an image acquisition time of the image before compression to the remote monitoring server; Transmitting an object detection result including a type and a position of the detected object and second data including the image acquisition time of the image used for object detection to the remote monitoring server.

2. An in-vehicle information processing device comprising:

Images