Information processing method, and information processing device

Abstract

To provide an information processing method capable of reducing waste in network bands, and a video transmission device.SOLUTION: An information processing method acquires first video information to be used for remote monitoring and remote operation of a mobile body from an imaging part for imaging a periphery of the mobile body (S10), and creates second video image from a route of a vehicle to be monitored and a danger degree map (S60) to transmit the second video information to a remote monitoring device (S70) after acquiring obstacle information from a sensing result (S30) and creating the danger degree map (S50).SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a video transmission device, a video transmission method, and a program installed in a vehicle. [Background technology]

[0002] There is a vehicle control system in which a vehicle without a driver or a vehicle that is not operated by a driver is indirectly driven or operated by an operator in a remote location by using wireless communication such as a wireless LAN (Local Area Network) or a mobile phone line.

[0003] In such a vehicle control system, the sensing results obtained by sensors mounted on the vehicle (e.g., millimeter wave radar, laser radar, and cameras) observing the area around the vehicle are transmitted from the vehicle to an operator via a communication network, and control information regarding the vehicle's driving is transmitted from the operator to the vehicle, allowing the operator to remotely control the vehicle.

[0004] Patent Document 1 discloses a remote operation system (vehicle control system) for a semi-autonomous unmanned vehicle (operated vehicle). The remote operation system in Patent Document 1 includes a semi-autonomous vehicle that acquires distance measurement data within the driving area of ​​the vehicle, performs autonomous driving based on the distance measurement data, and is remotely operated by an operator at a remote location using operation information (control information) transmitted from a remote operation device. Images captured by a camera on the vehicle are displayed on a display provided in the remote operation system, and the operator can remotely operate the semi-autonomous vehicle by checking the images displayed on the display. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5366703 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, although an operator can check the image captured by the vehicle camera, no consideration is given to the waste of the network bandwidth between the vehicle and the remote monitoring system, which may result in the network bandwidth being wasted and insufficient.

[0007] Therefore, an object of the present disclosure is to provide an information processing method and an information processing device that can reduce waste of network bandwidth. [Means for solving the problem]

[0008] An information processing method according to one aspect of the present disclosure acquires first video information used for at least one of remote monitoring and remote control of a moving object from an imaging unit that images an image of a surrounding area of ​​the moving object.

[0009] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires first video information used for at least one of remote monitoring and remote control of a moving object from an imaging unit that images an image of the surroundings of the moving object.

[0010] These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, or the recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet. Effect of the Invention

[0011] According to an information processing method and an information processing device according to an aspect of the present disclosure, it is possible to reduce waste of network bandwidth. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram showing a schematic configuration of a vehicle control system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of the remote monitoring system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing a functional configuration of the monitored vehicle according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the operation of the video transmission device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing a method of calculating a dynamic danger map for a stationary obstacle according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing a method of calculating a risk map taking into consideration a blind spot area for an obstacle as seen from a monitored vehicle according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing a method of calculating a dynamic danger map for a moving obstacle according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing a static risk map according to the first embodiment. [Figure 9A] FIG. 9A is an image diagram showing a case where a travel route and a danger area overlap in accordance with the first embodiment. [Figure 9B] FIG. 9B is an image diagram showing a case where the travel route and the danger area do not overlap in accordance with the first embodiment. [Figure 10] FIG. 10 is a diagram showing a functional configuration of a monitored vehicle according to the second embodiment. [Figure 11] FIG. 11 is a sequence diagram showing an operation of calculating a packet loss rate between a monitored vehicle and a remote monitoring device according to the second embodiment. [Figure 12A] FIG. 12A is a sequence diagram showing an example of an operation for acquiring a delay time of a network according to the second embodiment. [Figure 12B] FIG. 12B is a sequence diagram showing another example of the operation of acquiring the delay time of the NW according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing the operation of the video transmission device according to the second embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the video transmission device according to the first modification of the second embodiment. [Figure 15] FIG. 15 is a flowchart showing the operation of the video transmission device according to the second modification of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A video transmission device according to one embodiment of the present disclosure is a video transmission device mounted on a vehicle, and includes a first acquisition unit that acquires first video information from an imaging unit that images the surroundings of the vehicle, a danger map creation unit that creates a danger map indicating danger areas around the vehicle, a video creation unit that creates second video information having a reduced amount of data compared to the first video information based on the danger map, and a communication unit that transmits the second video information to a remote monitoring device for remotely monitoring the vehicle.

[0014] This allows the video transmission device to reduce the amount of data of the second video information to be transmitted to the remote monitoring device in accordance with the danger area around the vehicle, thereby reducing waste of network bandwidth between the vehicle and the remote monitoring device.

[0015] Also, for example, the image creation unit creates the second video information by at least one of degrading the image quality of a first area of ​​the first video information corresponding to an area other than the danger area indicated by the danger map, compared to the image of a second area corresponding to the danger area, and cutting out the image of the second area from the first video information.

[0016] This allows the operator to view a clear image of the area corresponding to the danger area, and therefore the video transmission device can effectively notify the operator of dangers while driving, while reducing waste of network bandwidth.

[0017] Also, for example, the communication unit may further include a judgment unit that judges whether or not the communication network between the vehicle and the remote monitoring device is congested, and when the judgment unit judges that the communication network is congested, the communication unit transmits the second video information to the remote monitoring device, and when the judgment unit judges that the communication network is not congested, the communication unit transmits the first video information to the remote monitoring device.

[0018] As a result, when the communication network is congested, the video transmission device transmits the second video information having a reduced amount of data compared to the first video information. In other words, even when the communication network is congested, the operator can quickly view the video based on the second video information. Therefore, the video transmission device can notify the operator of dangers while driving in a shorter time than when the first video information is transmitted, while reducing waste of network bandwidth.

[0019] Also, for example, when the determination unit determines that the communications network is congested, the communication unit transmits the second video information at a transmission rate lower than when the determination unit determines that the communications network is not congested, and when the determination unit determines that the communications network is congested, the video creation unit creates the second video information from the first video information in accordance with the transmission rate.

[0020] As a result, the video transmission device transmits the second video information created according to the transmission rate to the remote monitoring device. For example, when the transmission rate is set to a low value, the amount of data of the second video information can be further reduced, thereby shortening the time required to transmit the second video information. Thus, the video transmission device can notify the operator of dangers while driving in a shorter time than when the first video information is transmitted, while reducing waste of network bandwidth.

[0021] Also, for example, the remote monitoring device is connected to an operation input device into which operations for remotely operating the vehicle are input, and when the vehicle is being remotely operated from the operation input device, the image creation unit creates the second image information further using the driving route determined by the remote operation.

[0022] In this way, the second video information is created according to the driving route by remote control, so that the second video information suitable for the driving route can be created. By visually checking the video based on the second video information according to the driving route, the operator can determine with a clearer image whether or not there is a danger caused by the driving by remote control.

[0023] Also, for example, when the vehicle is traveling in an autonomous driving mode, the image creation unit creates the second image information by further using a target driving route along which the vehicle will travel.

[0024] In this way, the second video information is created according to the driving route by the autonomous driving, so that the second video information suitable for the driving route can be created. By visually checking the video based on the second video information according to the driving route, the operator can determine with a clearer image whether or not there is a danger caused by the autonomous driving.

[0025] Also, for example, the image creation unit further creates the second video information by at least one of degrading the image quality of a third area of ​​the first video information, which is an image captured in a direction other than the traveling direction on the driving route, compared to the image of a fourth area, which is an image captured in the traveling direction, and cutting out the image of the fourth area from the first video information.

[0026] This allows the video transmission device to transmit the video of the area corresponding to the danger area and the video of the traveling direction to the remote monitoring device with good image quality. In other words, the operator can visually confirm the traveling direction of the traveling route with a clearer image. Therefore, the video transmission device can effectively inform the operator of dangers during traveling by remote control or automatic driving while reducing waste of network bandwidth.

[0027] Also, for example, the vehicle may further include a second acquisition unit that acquires sensing results from a sensor unit that senses the surroundings of the vehicle, and an object detection unit that creates obstacle information indicating the position and speed of an obstacle based on the sensing results and the first video information, and the danger map creation unit creates the danger map based on the obstacle information.

[0028] This allows the video transmission device to create a danger map using information obtained from the imaging unit and sensor unit equipped in the vehicle.

[0029] Also, for example, the danger map creation unit creates the danger map indicating the degree of danger around the vehicle based on at least one of the first video information and the obstacle information.

[0030] This allows the danger map creation unit to arbitrarily set the danger area. For example, the danger map creation unit can set an area having a danger level equal to or higher than a predetermined value as the danger area. This allows the danger area to be set more accurately.

[0031] Furthermore, for example, when the determination unit determines that the communication network is congested, the determination unit further determines whether the degree of congestion of the communication network is below a threshold, and when the determination unit determines that the degree of congestion is below the threshold, the communication unit transmits the second video information, and when the determination unit determines that the degree of congestion is greater than the threshold, the communication unit transmits at least one of the sensing result and the obstacle information.

[0032] This allows the information to be transmitted to be switched depending on the congestion level of the network, and therefore the video transmission device can transmit information about the surroundings of the vehicle at a more appropriate transmission rate depending on the congestion level of the network.

[0033] Also, for example, the danger map creation unit creates the danger map from traffic information about the surroundings of the vehicle.

[0034] This makes it possible to create a danger map using traffic information (for example, traffic accident information) obtained from outside the vehicle.

[0035] Also, for example, when the danger area does not exist in the danger map, the image creation unit creates the second image information by uniformly degrading the image quality of the first image indicated by the first image information.

[0036] This further reduces the amount of data of the second video information when no dangerous area exists, thereby enabling the video transmission device to further reduce waste of network bandwidth.

[0037] Also, for example, when the danger area does not exist in the danger map, the image creating unit stops creating the second image information.

[0038] As a result, when no dangerous area exists, the network bandwidth is not used, and the video transmission device can further reduce waste of the network bandwidth.

[0039] Moreover, a video transmission device according to one aspect of the present disclosure is a video transmission device mounted on a vehicle, and includes a first acquisition unit that acquires first video information from an imaging unit that images the surroundings of the vehicle, a second acquisition unit that acquires sensing results from a sensor unit that senses the surroundings of the vehicle, an object detection unit that creates obstacle information indicating the position and speed of an obstacle based on the sensing results and the first video information, a danger map creation unit that creates a danger map that indicates danger areas around the vehicle, a video creation unit that creates second video information based on the danger map, a communication unit that transmits monitoring information to be sent to a remote monitoring device for remotely monitoring the vehicle, and a judgment unit that judges whether a communication network between the vehicle and the remote monitoring device is congested, and the communication unit transmits the monitoring information including at least one of the sensing result, the obstacle information, and the second video information selected based on the judgment result of the judgment unit.

[0040] This allows information based on the determination result to be transmitted to the remote monitoring device. For example, when the communication network is congested, information with a smaller amount of data (e.g., sensing results, obstacle information, etc.) is selected as monitoring information, making it possible to transmit information about the surroundings of the vehicle at a lower transmission rate.

[0041] Also, for example, when the determination unit determines that the communication network is congested, the communication unit transmits at least one of the sensing result and the obstacle information to the remote monitoring device, and when the determination unit determines that the communication network is not congested, the communication unit transmits the second video information to the remote monitoring device.

[0042] This makes it possible to transmit information about the surroundings of the vehicle at a lower transmission rate by limiting the information to be transmitted to at least one of the sensing information and the obstacle information when the communication network is congested.

[0043] A video transmission method according to an aspect of the present disclosure is a video transmission method for a video transmission device mounted on a vehicle, and includes the steps of acquiring first video information from an imaging unit that images the surroundings of the vehicle, creating a danger map showing danger areas around the vehicle, creating second video information with a reduced amount of data from the first video information based on the danger map, and transmitting the second video information to a remote monitoring device for remotely monitoring the vehicle. A program according to an aspect of the present disclosure is a program for causing a computer to execute the video transmission method.

[0044] This provides the same effect as the video transmission device.

[0045] Hereinafter, the embodiment will be specifically described with reference to the drawings.

[0046] Note that each of the embodiments described below is a comprehensive or specific example. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components.

[0047] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations may be omitted or simplified.

[0048] In addition, in this specification, terms indicating relationships between elements, such as "equal," as well as numerical values ​​and numerical ranges, are not expressions that only indicate the strict meaning, but are expressions that include a substantially equivalent range, for example, a difference of about a few percent.

[0049] (Embodiment 1) Hereinafter, a video transmission device and the like according to the present embodiment will be described with reference to FIGS. 1 to 9B.

[0050] [1-1. Vehicle control system configuration] First, the configuration of a vehicle control system including a monitored vehicle equipped with a video transmission device according to the present embodiment will be described with reference to Figures 1 to 3. Figure 1 is a diagram showing a schematic configuration of a vehicle control system 10 according to the present embodiment.

[0051] As shown in FIG. 1, the vehicle control system 10 is a system that communicatively connects a monitored vehicle 200 and a remote monitoring system 100 (specifically, a remote monitoring device 130) via a wireless base station 310 such as a wireless LAN or a communication terminal, and a network 300. The wireless base station 310 and the network 300 are an example of a communication network. The monitored vehicle 200 is an example of a vehicle that is at least remotely monitored by an operator H. The monitored vehicle 200 may also be a vehicle that is remotely monitored and remotely operated by an operator H.

[0052] In this specification, the vehicle is, for example, an autonomous vehicle that controls the driving of the vehicle without the need for driver operation, but may be a vehicle that can be driven by switching between autonomous driving and manual driving. In addition, the vehicle includes not only what is generally called a vehicle, such as an automobile, a train, or a bus, but also ships such as a ferry and aircraft such as an airplane.

[0053] The remote monitoring system 100 will be described in further detail with reference to Fig. 2. Fig. 2 is a diagram showing the functional configuration of the remote monitoring system 100 according to the present embodiment.

[0054] As shown in FIGS. 1 and 2, the remote monitoring system 100 includes a display device 110, an operation input device 120, and a remote monitoring device .

[0055] The display device 110 is a monitor connected to the remote monitoring device 130 and displays an image related to the monitored vehicle 200. The display device 110 displays an image captured by an imaging unit (see imaging unit 210 shown in FIG. 3) equipped in the monitored vehicle 200. The display device 110 may also display the state of the monitored vehicle 200 and obstacles around the monitored vehicle 200 to the operator H, thereby allowing the operator H to recognize the state of the monitored vehicle 200 and the obstacles. Note that the image includes a moving image and a still image. Also, the obstacle is another vehicle, a person, etc. other than the monitored vehicle 200, and mainly means a moving object that becomes an obstacle when the monitored vehicle 200 travels. Note that the obstacle may be real estate fixed to the ground.

[0056] Furthermore, a plurality of display devices 110 may be connected to the remote monitoring device 130 .

[0057] The operation input device 120 is a device connected to the remote monitoring device 130 and receives remote operation input from the operator H. The operation input device 120 is, for example, a steering wheel, foot pedals (e.g., an accelerator pedal and a brake pedal), etc., and is a device for operating the monitored vehicle 200. The operation input device 120 outputs input vehicle operation information to the remote monitoring device 130. Note that, when the remote monitoring system 100 does not remotely operate the monitored vehicle 200, it does not have to be equipped with the operation input device 120 for remotely operating the monitored vehicle 200.

[0058] The remote monitoring device 130 is a device that allows an operator H at a remote location to remotely monitor the monitored vehicle 200 via a communication network. As shown in Fig. 2, the remote monitoring device 130 has a control unit 131, a communication unit 132, and a storage unit 133. In this embodiment, the remote monitoring device 130 is connected to the operation input device 120, and also functions as a remote operation device for remotely operating the monitored vehicle 200.

[0059] The control unit 131 is a control device that controls various components of the remote monitoring device 130. In this embodiment, the control unit 131 creates an image required for the operator H to monitor the monitored vehicle 200 based on the image information of the surroundings of the monitored vehicle 200 received via the communication unit 132, and outputs the created image to the display device 110. At this time, if obstacle information including the current position of an obstacle is received from the monitored vehicle 200, the image may be created using the received obstacle information. For example, the color of an obstacle close to the monitored vehicle 200 shown in the image may be changed, or the obstacle close to the monitored vehicle 200 shown in the image may be displayed by blinking. In addition, when the control unit 131 acquires vehicle operation information from the operation input device 120, the control unit 131 transmits vehicle control information (hereinafter also referred to as control information) based on the acquired vehicle operation information to the monitored vehicle 200 via the communication network.

[0060] The control unit 131 may further have a real-time clock function for measuring the current date and time. Alternatively, the control unit 131 may use a time determined based on a GPS signal, which is a signal from a GPS satellite received via a GPS (Global Positioning System) sensor (not shown), as the GPS time, which is an accurate time. The control unit 131 receives the GPS signal at a predetermined time interval.

[0061] The communication unit 132 is a wireless communication module for wirelessly communicating with the monitored vehicle 200 via a communication network. The communication unit 132 receives image information based on an image captured by the imaging unit 210 of the monitored vehicle 200 via the communication network. The communication unit 132 may also receive information for calculating a delay time of the communication network (specifically, an RTT (Round Trip Time) measurement packet) and vehicle information of the monitored vehicle 200. The communication unit 132 transmits control information for controlling the traveling of the monitored vehicle 200 to the monitored vehicle 200 via the communication network under the control of the control unit 131. The vehicle information is information related to the traveling of the monitored vehicle 200 that is possessed by the monitored vehicle 200 itself. The vehicle information may include the speed and current position of the monitored vehicle 200 itself. The vehicle information may also include information related to a traveling route that is a target for the monitored vehicle 200 to travel when the monitored vehicle 200 is traveling in an autonomous driving mode.

[0062] The storage unit 133 is a storage device that stores a control program executed by the control unit 131. The storage unit 133 may also store video information acquired via the communication unit 132. The storage unit 133 is realized by, for example, a semiconductor memory.

[0063] Although not shown, the remote monitoring system 100 may be connected to the remote monitoring device 130 and may include a sound output device (e.g., a speaker) that outputs a warning sound to the operator H to alert him / her to an obstacle, thereby making the operator H aware of an approaching danger. This allows the operator H to know that the monitored vehicle 200 is in a situation where it is necessary to make an emergency stop. The remote monitoring system 100 may also be connected to the remote monitoring device 130 and may include an emergency stop device for making an emergency stop of the monitored vehicle 200. The emergency stop device is realized, for example, by an emergency stop button. The emergency stop device outputs emergency stop information input by the operator H to the remote monitoring device 130. In this case, the control information may include emergency stop control information based on the emergency stop information.

[0064] Next, the monitored vehicle 200 will be described in more detail with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the monitored vehicle 200 according to this embodiment.

[0065] As shown in Fig. 3, the monitored vehicle 200 includes an imaging unit 210, a sensor unit 220, and a video transmission device 230. The monitored vehicle 200 is a vehicle capable of autonomous driving based on driving data held in advance by the vehicle and driving data created based on information detected by the sensor unit 220, etc., and is a vehicle capable of automatic driving that is remotely monitored by an operator H located at a remote location under predetermined conditions. In this embodiment, the monitored vehicle 200 is a vehicle that is remotely monitored and remotely operated by an operator H located at a remote location under predetermined conditions. The imaging unit 210, the sensor unit 220, and the video transmission device 230 constitute an in-vehicle system equipped with functions necessary for automatic driving.

[0066] The imaging unit 210 is a camera that captures an image of the surroundings of the monitored vehicle 200. The imaging unit 210 is provided, for example, at a position where it is possible to capture images in the forward, backward, left and right directions from the monitored vehicle 200. That is, the imaging unit 210 is provided on the monitored vehicle 200 so as to be able to capture images of the surroundings of the monitored vehicle 200. The imaging unit 210 outputs first image information based on a first image obtained by capturing an image of the surroundings of the monitored vehicle 200 to the first acquisition unit 231. The imaging unit 210 may be composed of, for example, a plurality of cameras. Note that the imaging unit 210 does not include a camera provided to record the traveling status of the monitored vehicle 200. The imaging unit 210 is provided to capture an image to be transmitted to the remote monitoring device 130 for remotely monitoring the monitored vehicle 200. In other words, the imaging unit 210 is provided to capture an image to be confirmed by the operator H for remote monitoring.

[0067] The sensor unit 220 is a device that detects the situation around the monitored vehicle 200. The sensor unit 220 is provided, for example, so as to be able to detect the situation in the forward, backward, left and right directions from the monitored vehicle 200. The sensor unit 220 outputs the sensing result (sensing information) obtained by sensing the surroundings of the monitored vehicle 200 to the second acquisition unit 232. The sensor unit 220 is realized by a LIDAR (Light Detection and Ranging), a radar (e.g., a millimeter wave radar), an ultrasonic sensor, or a combination of these. The sensing result is information about obstacles around the monitored vehicle 200.

[0068] A plurality of imaging units 210 and a plurality of sensor units 220 may be installed in the monitored vehicle 200. That is, the monitored vehicle 200 may include one or more imaging units 210 and one or more sensor units 220.

[0069] The video transmission device 230 is a device that creates and transmits second video information to be transmitted to the monitored vehicle 200 from the first video information according to at least one of the first video information acquired from the imaging unit 210 and the sensing result acquired from the sensor unit 220. The video transmission device 230 acquires information related to the traveling of the monitored vehicle 200 from various sensors including the imaging unit 210 and the sensor unit 220 mounted on the monitored vehicle 200 via an in-vehicle network such as a CAN (Control Area Network). The video transmission device 230 acquires first video information of the surroundings of the monitored vehicle 200 from the imaging unit 210, and acquires the sensing result of the surroundings of the monitored vehicle 200 from the sensor unit 220. The video transmission device 230 is an example of an information processing device.

[0070] The various sensors may include a speed sensor that detects the speed of the monitored vehicle 200, and a GPS sensor that detects the current position of the monitored vehicle 200. The video transmission device 230 may acquire the speed of the monitored vehicle 200 from the speed sensor, and the current position of the monitored vehicle 200 from the GPS sensor. In other words, the information regarding the travel of the monitored vehicle 200 may include at least one of the speed and the current position of the monitored vehicle 200. The information regarding the travel of the monitored vehicle 200 is an example of the vehicle information of the monitored vehicle 200. The various sensors may include a steering angle sensor that detects the steering angle of the monitored vehicle 200, a brake sensor that detects the degree of braking, an accelerator sensor that detects the degree of acceleration (hereinafter also referred to as the accelerator opening degree), and a turn signal sensor that detects the indication direction of the turn signal.

[0071] As shown in FIG. 3, the video transmission device 230 includes a first acquisition unit 231, a second acquisition unit 232, an object detection unit 233, a risk map creation unit 234, a control unit 235, a memory unit 236, and a communication unit 237.

[0072] The first acquisition unit 231 is a communication interface that acquires the first video information from the imaging unit 210. When the first acquisition unit 231 acquires the first video information from the imaging unit 210, the first acquisition unit 231 outputs the acquired first video information to the object detection unit 233 and the control unit 235.

[0073] The second acquisition unit 232 is a communication interface that acquires the sensing result from the sensor unit 220. When the second acquisition unit 232 acquires the sensing result from the sensor unit 220, the second acquisition unit 232 outputs the acquired sensing result to the object detection unit 233.

[0074] The object detection unit 233 detects an obstacle around the monitored vehicle 200 from at least one of the first video information acquired from the imaging unit 210 and the sensing result acquired from the sensor unit 220, and creates obstacle information including at least one of the current position, size, and traveling direction of the obstacle. Then, the object detection unit 233 outputs the obstacle information to the risk map creation unit 234. Note that the obstacle information may further include information regarding the acceleration, speed, and type of the obstacle. The type of the obstacle is, for example, a type that distinguishes between pedestrians, motorcycles, automobiles, etc. The current position of the obstacle indicates the position of the obstacle at the time when the sensor unit 220 senses the obstacle.

[0075] The risk map creating unit 234 creates a risk map showing a risk area in the traveling of the vehicle based on the obstacle information acquired from the object detection unit 233. Then, the risk map creating unit 234 outputs the created risk map to the control unit 235. In this embodiment, an example will be described in which the risk map creating unit 234 creates a risk map that displays the risk area around the monitored vehicle 200 in a identifiable manner according to the risk level, but it may only indicate whether or not there is a risk area. In other words, the risk map created by the risk map creating unit 234 does not need to display according to the risk level. The risk map creating unit 234 is an example of a risk map creating unit, and the risk map is an example of a risk map. Also, the risk area is an area having a risk level equal to or higher than a predetermined value.

[0076] The control unit 235 is a control device that controls various components of the monitored vehicle 200. In this embodiment, the control unit 235 creates second video information to be transmitted to the remote monitoring device 130 from the first video information captured by the imaging unit 210 based on the risk map created by the risk map creation unit 234. The data amount of the second video information is equal to or less than the data amount of the first video information. The control unit 235 functions as a video creation unit that creates second video information with a data amount reduced from that of the first video information. The control unit 235 creates second video information for allowing the operator H to visually recognize the video of the surroundings of the monitored vehicle 200 via the remote monitoring system 100. The first video information may be, for example, video information corresponding to one frame of video or video information corresponding to multiple frames of video. The second video indicated by the second video information is, for example, a video with at least a part of the first video indicated by the first video information having a deteriorated image quality, or a video with at least a part of the first video indicated by the first video information cut out.

[0077] In addition, the first video information of the surroundings of the monitored vehicle 200 may be created, for example, by creating each of the front, rear, left and right directions of the monitored vehicle 200 as separate first video information, or may be created by combining the front, rear, left and right directions of the monitored vehicle 200 together as a single piece of first video information.

[0078] The control unit 235 also outputs control information acquired from the remote monitoring system 100 via the communication unit 237 to a driving control unit (not shown) that controls the driving of the vehicle itself. The driving control unit has a speed control unit (e.g., an ECU (engine control unit)) that controls the speed by operating the accelerator, brake, and shift lever of the monitored vehicle 200, and a steering control unit that controls the traveling direction of the monitored vehicle 200 by operating the steering wheel of the monitored vehicle 200.

[0079] The control unit 235 may have a real-time clock function for measuring the current date and time. Alternatively, the control unit 235 may use a time determined based on a GPS signal, which is a signal from a GPS satellite acquired via a GPS sensor (not shown), as the accurate GPS time. Alternatively, the monitored vehicle 200 may be equipped with a radio-controlled clock, and the control unit 235 may acquire the current time from the radio-controlled clock.

[0080] The memory unit 236 is a storage device that stores a control program executed by a processing unit such as the control unit 235. The memory unit 236 also stores, for example, map data for automatic driving, route information of the monitored vehicle 200, and a static risk map. The route information is, for example, data representing the position and route of the company in the map data. The static risk map is a map showing a risk area created based on traffic information around the monitored vehicle 200, and will be described in detail later. The traffic information includes traffic accident information. The traffic information may further include event information (for example, information on the commuting and returning home times of nearby schools, information on outdoor events, and traffic regulation information). The traffic information may be information associated with event information that may change from moment to moment or in real time. The memory unit 236 is realized, for example, by a semiconductor memory or the like.

[0081] The communication unit 237 is a wireless communication module for wirelessly communicating with the remote monitoring device 130 for remotely monitoring the monitored vehicle 200 via the wireless base station 310 and the network 300. Under the control of the control unit 235, the communication unit 237 transmits second video information based on the first video information captured by the imaging unit 210 to the remote monitoring device 130 via the wireless base station 310 and the network 300. In addition, the communication unit 237 receives control information related to the traveling of the monitored vehicle 200 from the remote monitoring device 130 via the wireless base station 310 and the network 300.

[0082] [1-2. Behavior of monitored vehicle] Next, the operation of the video transmission device 230 provided in the monitored vehicle 200 will be described with reference to Fig. 4 to Fig. 9B. Fig. 4 is a flowchart showing the operation of the video transmission device 230 according to this embodiment.

[0083] 4, the video transmission device 230 acquires first video information based on a first video captured around the monitored vehicle 200 from the imaging unit 210 (S10). Specifically, the video transmission device 230 acquires the first video information from the imaging unit 210 via the first acquisition unit 231. The first acquisition unit 231 outputs the acquired first video information to the object detection unit 233 and the control unit 235.

[0084] Furthermore, the video transmission device 230 acquires a sensing result obtained by sensing the surroundings of the monitored vehicle 200 from the sensor unit 220 (S20). Specifically, the video transmission device 230 acquires the sensing result from the sensor unit 220 via the second acquisition unit 232. The second acquisition unit 232 outputs the acquired sensing result to the object detection unit 233.

[0085] The object detection unit 233 creates obstacle information from the first video information and the sensing result (S30). The obstacle information includes the current position, size, and traveling direction of an obstacle that is an obstacle to the travel of the monitored vehicle 200. The object detection unit 233 outputs the created obstacle information to the danger level map creation unit 234.

[0086] The risk map creating unit 234 creates a dynamic risk map from, for example, the obstacle information (S40). Here, the creation of the dynamic risk map in the risk map creating unit 234 will be described with reference to FIG. 5. FIG. 5 is a diagram showing a calculation method of a dynamic risk map for a stationary obstacle 400 according to this embodiment. Note that FIG. 5 shows an example of a calculation method of a risk map in which an area indicating a risk is displayed on a two-dimensional grid map. Also, the risk map shown in FIG. 5 shows, for example, a risk map in front of the monitored vehicle 200.

[0087] As shown in (a) of Fig. 5, the risk level map creation unit 234 superimposes an obstacle 400 on a two-dimensional grid map based on the current position, size, and type of the obstacle 400 included in the obstacle information. Fig. 5 (a) shows an example in which the obstacle is a truck.

[0088] Then, as shown in (b) of FIG. 5, the risk map creation unit 234 assigns a risk level to a grid that at least partially overlaps with the obstacle 400 on the grid map. (b) of FIG. 5 shows an example in which a risk level is assigned to an area R1. For example, the risk map creation unit 234 may assign an equal risk level to each grid included in the area R1, or may assign a different risk level to each grid. For example, the risk levels may be assigned such that the larger the area of ​​a grid that overlaps with the obstacle 400, the higher the risk level of the grid. Note that the risk map creation unit 234 assigns one risk level to one grid.

[0089] Then, as shown in (c) of FIG. 5, the risk map creation unit 234 also assigns risk levels to grids (areas R2 and R3 shown in (c) of FIG. 5) surrounding the grid to which a risk level has been assigned (the grid included in area R1 shown in (b) of FIG. 5). Area R2 is, for example, an area surrounding area R1. Area R3 is, for example, an area surrounding area R2.

[0090] The risk map creation unit 234 assigns a lower risk level to the region R2 than to the region R1, and assigns a lower risk level to the region R3 than to the region R2. In other words, the risk map creation unit 234 assigns a risk level to each grid so that the risk level gradually decreases as it moves away from the region R1. In this way, a dynamic risk map M1 shown in FIG. 5(c) is created. Note that in FIG. 5(c), regions with high risk levels are shown darker, and regions with lower risk levels are shown lighter. For example, the regions R1 to R3 in the dynamic risk map M1 shown in FIG. 5(c) are examples of risk regions.

[0091] As described above, when the obstacle 400 is stationary, the grid that overlaps with the obstacle 400 on the grid map (that is, the grid corresponding to the position where the obstacle 400 currently exists) is assigned a high risk level.

[0092] The risk map creating unit 234 is not limited to creating a risk map that displays areas for each risk level as shown in Fig. 5. The risk map creating unit 234 may create a map that indicates dangerous areas (e.g., area R1) and other areas as shown in Fig. 5(b). The risk map creating unit 234 may create a risk map by assuming that a grid overlapping with an obstacle 400 is a dangerous area, for example. In this case, the grid overlapping with the obstacle 400 (e.g., area R1 shown in Fig. 5(b)) is the dangerous area.

[0093] The risk map creating unit 234 may create a risk map by considering a blind spot area caused by the obstacle 400 as seen from the monitored vehicle 200. A risk map taking into consideration a blind spot area will be described with reference to Fig. 6. Fig. 6 is a diagram showing a method of calculating a risk map taking into consideration a blind spot area R4 for the obstacle 400 as seen from the monitored vehicle 200 according to this embodiment. In Fig. 6, the position of the host vehicle is also displayed on the grid map.

[0094] 6(a), when an obstacle 400 is present ahead of the monitored vehicle 200, the rear side of the obstacle 400 as viewed from the monitored vehicle 200 becomes the blind spot of the monitored vehicle 200, and the imaging unit 210 and the sensor unit 220 cannot observe the blind spot. Therefore, the risk map creation unit 234 creates a risk map by predicting the blind spot area, for example, by the following method, and assigning a risk level to the blind spot area as well.

[0095] The risk map creation unit 234 predicts the blind spot area by calculating the blind spot area based on the positional relationship between the obstacle 400 and the vehicle (the monitored vehicle 200). For example, the risk map creation unit 234 predicts the hatched area shown in FIG. 6A as the blind spot area R4. Specifically, based on the positional relationship between the obstacle 400 and the vehicle, the risk map creation unit 234 predicts the range surrounded by a straight line connecting the vehicle and one end of the obstacle 400 corresponding to one end of the blind spot area R4 (the upper left end of the obstacle 400 shown in FIG. 6A) and a straight line connecting the vehicle and the other end of the obstacle 400 corresponding to the other end of the blind spot area R4 (the lower right end of the obstacle 400 shown in FIG. 6A) as the blind spot area R4. Note that the method of predicting the blind spot area R4 by the risk map creation unit 234 may be a method other than the above. The blind spot area R4 is an unobservable area that the imaging unit 210 and the sensor unit 220 cannot observe.

[0096] As shown in (b) of FIG. 6, when the blind spot area R4 shown in (a) of FIG. 6 exists, the risk map creation unit 234 assigns a risk level to the grid corresponding to the blind spot area R4 on the grid map. The risk map creation unit 234 uniquely assigns a risk level to the grid at least partially within the blind spot area R4. For example, the risk map creation unit 234 may assign a lower risk level to the grid within the blind spot area R4 than the risk level assigned to the area R1 at least partially overlapping with the obstacle 400. In this way, a dynamic risk map M2 including areas R1 and R5 shown in (b) of FIG. 6 is created. The area R5 is an area corresponding to the blind spot area R4. For example, the areas R1 and R5 in the dynamic risk map M2 are examples of risk areas.

[0097] In the above, the creation of a risk map when the obstacle 400 is stationary has been described, but the creation of a risk map when the obstacle 400 is moving will be described with reference to Fig. 7. Fig. 7 is a diagram showing a method of calculating a dynamic risk map for a moving obstacle 400 according to this embodiment. In Fig. 7, the risk map is created by further using the traveling direction of the obstacle 400 included in the obstacle information.

[0098] 7(a), the risk level map creating unit 234 superimposes the obstacle 400 on a two-dimensional grid map based on the current position, size, and type of the obstacle 400 included in the obstacle information. Also, the arrow shown in FIG. 7(a) indicates the moving direction of the obstacle 400 based on the traveling direction included in the obstacle information.

[0099] Then, as shown in (b) of FIG. 7, the risk map creating unit 234 assigns risk levels to the grids in consideration of the movement of the obstacle 400. For example, the risk map creating unit 234 assigns risk levels to grids that at least partially overlap with the obstacle 400 on the grid map and to grids in the direction in which the obstacle 400 moves. (b) of FIG. 7 shows an example in which risk levels are assigned to the region R6. For example, the risk map creating unit 234 may assign equal risk levels to each grid included in the region R6, or may assign different risk levels to the grid in which the obstacle 400 is currently located and the grid in the traveling direction of the obstacle 400. For example, the grid in which the obstacle 400 is currently located may be assigned a risk level higher than that of the grid in the traveling direction of the obstacle 400. For example, the range of the region R6 may be determined according to the distance the obstacle 400 moves in a predetermined period (for example, several seconds). The region R6 may be formed by the current position of the obstacle 400, the position of the obstacle 400 after a predetermined period of time has elapsed (calculated position), and a region sandwiched between the current position and the calculated position. Note that the distance traveled by the obstacle 400 may be calculated using the speed of the obstacle 400 acquired from the sensing result of the sensor unit 220.

[0100] Then, as shown in FIG. 7(c), the risk level map creation unit 234 assigns risk levels to grids (areas R7 and R8 shown in FIG. 7(c)) surrounding the grid (the grid included in area R6 shown in FIG. 7(b)) to which the risk level is assigned. The risk level map creation unit 234 assigns a lower risk level to area R7 than area R6, and assigns a lower risk level to area R8 than area R7. In other words, the risk level map creation unit 234 assigns risk levels to the grids such that the risk level gradually decreases with increasing distance from area R6. In this way, a dynamic risk level map M3 shown in FIG. 7(c) is created. Note that, for example, areas R6 to R8 in the dynamic risk level map M3 are examples of risk areas.

[0101] As described above, when the obstacle 400 is moving, the grids that overlap the obstacle 400 on the grid map and the grids in the direction of the movement are assigned a high risk level.

[0102] The risk map creating unit 234 may create a dynamic risk map in consideration of blind spot areas when the obstacle 400 is moving, similar to the dynamic risk map in the case where the obstacle 400 is stationary. In this case, the relative positional relationship between the monitored vehicle 200 and the obstacle 400 changes as the obstacle 400 moves, so the blind spot areas change from moment to moment.

[0103] Referring again to FIG. 4, the risk map creation unit 234 creates a risk map for the monitored vehicle 200 based on the dynamic risk map (e.g., dynamic risk maps M1-M3) and the static risk map (S50). Here, the static risk map will be described with reference to FIG. 8. FIG. 8 is a diagram showing a static risk map M4 according to this embodiment. The static risk map M4 is created without using obstacle information from the object detection unit 233. In other words, the static risk map M4 is created without using the first video information and the sensing result.

[0104] 8, the static risk level map M4 is a map created based on traffic information around the monitored vehicle 200. The static risk level map M4 may be created based on traffic accident information as the traffic information, for example. The static risk level map M4 is stored in the storage unit 236, for example. The static risk level map M4 may be created in advance by the risk level map creation unit 234 based on traffic information, for example, or may be received from the outside via the communication unit 237.

[0105] In the static risk map M4, for example, a risk level is assigned to a grid corresponding to a location where a traffic accident occurred. In FIG. 8, area R9 indicates the location where the traffic incident occurred. In the static risk map M4, risk levels are also assigned to grids (areas R10 and R11 shown in FIG. 8) surrounding a grid (a grid included in area R9 shown in FIG. 8) to which a risk level is assigned (for example, a grid included in area R9 shown in FIG. 8). For example, area R10 is assigned a lower risk level than area R9, and area R11 is assigned a lower risk level than area R10. In other words, in the static risk map M4, risk levels are assigned to grids such that the risk level gradually decreases as the grid moves away from area R9.

[0106] Region R12 is a region surrounding region R11, and is assigned a lower risk level than region R11. Note that no risk level may be assigned to region R12. In other words, the risk level of each grid included in region R12 may be zero. A risk level based on region R9 may not be assigned to grids located a predetermined distance away from region R9, or grids located a predetermined number of grids away from region R9. Note that this may be applied to dynamic risk maps (for example, dynamic risk maps M1 to M3). Regions R9 to R11 are examples of risk regions.

[0107] In addition, in the static risk level map M4, for example, when the traffic information is traffic accident information, a higher risk level may be assigned as the number of traffic accidents increases or the damage caused by the traffic accidents increases.

[0108] The risk map creation unit 234 creates a risk map for the monitored vehicle 200 by calculating the risk level assigned to each grid of the dynamic risk map and the risk level assigned to each grid of the static risk map M4. The calculation may include at least one of addition, subtraction, multiplication, and division. In this embodiment, the risk map for the monitored vehicle 200 is created by adding the risk levels. The risk map creation unit 234 adds up the risk levels of corresponding grids in the dynamic risk map and the static risk map to calculate one risk level for the grid. The risk map creation unit 234 may take the average value of the risk levels of corresponding grids in the dynamic risk map and the static risk map as the risk level for the grid in the risk map. The calculation may also be to select one risk level from a plurality of risk levels. The risk map creating section 234 may determine, from among the risk levels of corresponding grids in the dynamic risk map and the static risk map, the higher risk level as the risk level of the grid in the risk map.

[0109] The risk map creating unit 234 outputs the created risk map to the control unit 235.

[0110] The control unit 235 creates second video information from the first video information based on the travel route of the monitored vehicle 200 and the risk map (S60). A case where the travel route of the monitored vehicle 200 overlaps with a risk area of ​​the risk map, that is, where the travel route of the monitored vehicle 200 passes through a risk area, will be described with reference to Fig. 9A. Fig. 9A is an image diagram showing a case where the travel route according to this embodiment overlaps with a risk area (area R13 shown in Fig. 9A). In other words, Fig. 9A is an image diagram in which the risk map and the travel route are superimposed.

[0111] 9A shows an example in which monitored vehicle 200 is traveling on road 502 in the direction of the arrow, and an obstacle 400 is present on road 501. Area R13 shows a danger area for obstacle 400. Area R13 is an area having a danger level equal to or higher than a predetermined value in a danger level map created by calculating a dynamic danger level map and a static danger level map, for example. The position where monitored vehicle 200 is shown in FIG. 9A is the current position of monitored vehicle 200. Also, the arrow in FIG. 9A shows the travel route of monitored vehicle 200.

[0112] As shown in FIG. 9A, the travel route of the monitored vehicle 200 overlaps with the area R13 at the intersection where the roads 501 and 502 intersect. That is, a dangerous area exists on the travel route of the monitored vehicle 200. In this case, it is preferable that the image of the travel route (in the example of FIG. 9A, the front of the monitored vehicle 200) displayed on the display device 110 is clear so that the operator H can make an accurate judgment or perform remote monitoring. Therefore, it is desirable that the monitored vehicle 200 transmits the image of the travel route to the remote monitoring device 130 while suppressing waste of the network bandwidth and deteriorating the image quality as much as possible.

[0113] For example, when a danger area in the danger map is on the route of the monitored vehicle 200, the control unit 235 creates second video information having a smaller amount of data from the first video information than the first video information. For example, the control unit 235 creates the second video information by at least one of degrading the image quality of the image of the first area corresponding to an area other than the danger area shown in the danger map among the first video information shown in the first video information, compared to the image of the second area corresponding to the danger area, and cutting out the image of the second area from the first video information shown in the first video information. Note that the image quality of the second area does not need to be degraded. In other words, the control unit 235 may create the second video information by degrading the image quality of only the image of the first area among the first video information shown in the first video information.

[0114] When there is a danger area in front of the monitored vehicle 200, the control unit 235 creates second image information by, for example, at least one of degrading the image quality of the rear and left and right directions (an example of a first area) of the images of the monitored vehicle 200 captured by the imaging unit 210 compared to the image of the front direction (an example of a second area), and cutting out the image of the front from the images of the front, rear, left and right directions of the monitored vehicle 200 captured by the imaging unit 210.

[0115] The control unit 235 may acquire the travel route of the monitored vehicle 200 based on the route information stored in the storage unit 236. Furthermore, when the control unit 235 is remotely operated by the operation input device 120 via the remote monitoring device 130, the control unit 235 may acquire the remotely operated travel route from the control information received from the remote monitoring device 130. In other words, the control unit 235 may create the second video information using the travel route of the monitored vehicle 200 based on the control information received from the operation input device 120 via the remote monitoring device 130.

[0116] In this case, the control unit 235 may change the second video information to be created depending on the received control information, i.e., the operation contents from the operator H. The control unit 235 may further create the second video information by at least one of degrading the image quality of an image of a third region, which is an image of a direction other than the traveling direction on the travel route based on the control information, of the first video information, compared with an image of a fourth region, which is an image of the traveling direction, and cutting out the image of the fourth region from the first video information.

[0117] A case will be described as an example in which there is a danger area ahead of the monitored vehicle 200, and the monitored vehicle 200 changes lanes to the right lane in response to the control information. In this case, the control unit 235 may create the second video information so that the image quality of the rear and left and right directions (an example of a first area) is degraded more than that of the front direction (an example of a second area) based on the danger map and the driving route according to the control information, but the image quality of the right direction image (an example of a third area) among the rear and left and right directions is less degraded than that of the rear and left direction image (an example of a fourth area). The control unit 235 may create the second video information so that the images of the second and third areas have the same image quality, for example. The control unit 235 may create the second video information without degrading the image quality of the images of the second and third areas.

[0118] Furthermore, when the monitored vehicle 200a is traveling in an autonomous driving mode, the control unit 235 may create the second video information by using a driving route that is a target route along which the monitored vehicle 200a will travel. In this case, the control unit 235 may change the second video information to be created depending on the driving route of the autonomous driving mode. The method of creating the second video information is the same as the method of creating the second video information according to the operation contents from the operator H.

[0119] Then, the control unit 235 transmits the created second video information to the remote monitoring device 130 via the communication unit 237 (S70). If the second video information is created by cutting out images of the second area and the third area from the first image, the second video information based on the image of the area where the risk level is equal to or higher than a predetermined value is transmitted to the remote monitoring device 130.

[0120] Next, a case where the travel route of the monitored vehicle 200 does not overlap with the danger area of ​​the danger level map, that is, the monitored vehicle 200 does not pass through the danger area, will be described with reference to Fig. 9B. Fig. 9B is an image diagram showing a case where the travel route and the danger area do not overlap according to this embodiment. In Fig. 9B, the monitored vehicle 200 travels on the road 502 in the direction of the arrow.

[0121] As shown in FIG. 9B, there is no danger area on the travel route of the monitored vehicle 200. When there is no danger area in the danger map, the control unit 235 may create the second video information by uniformly degrading the image quality of the first video indicated by the first video information. Furthermore, when there is no danger area in the danger map, the control unit 235 may stop creating the second video information. That is, when there is no danger area in the danger map, the video transmission device 230 may stop transmitting the second video information to the remote monitoring device 130. Furthermore, when there is no danger area in the danger map, the video transmission device 230 may transmit at least one of the sensing result and the obstacle information to the remote monitoring device 130.

[0122] The control unit 235 can reduce waste of network bandwidth between the monitored vehicle 200 and the remote monitoring device 130 by adjusting the amount of video data to be transmitted to the remote monitoring device 130 depending on the level of danger around the monitored vehicle 200.

[0123] The method of calculating the risk in the dynamic risk map and the static risk map is not limited to the above, and any method may be used as long as it is possible to calculate the degree of risk of a collision between the monitored vehicle 200 and the obstacle 400. The method of creating the risk map is not limited to the above, and any method may be used as long as it is possible to display a risk area around the monitored vehicle 200. The risk map may be displayed, for example, as a three-dimensional grid map, or may be created by superimposing a risk area on the first image captured by the imaging unit 210, or may be created by superimposing a risk area on map information stored in the storage unit 236 or acquired from an external source.

[0124] In the above, an example in which the risk map is created from the dynamic risk map and the static risk map has been described, but the present invention is not limited thereto. The risk map may be created using at least one of the dynamic risk map and the static risk map. The dynamic risk map may be used as the risk map, or the static risk map M4 may be used as the risk map. When the static risk map M4 is used as the risk map, the risk map creation unit 234 reading out the static risk map M4 created using the traffic information stored in the storage unit 236 is included in the risk map creation unit 234 creating the risk map from the traffic information.

[0125] When the control unit 235 receives control information including emergency stop control information from the remote monitoring device 130, the control unit 235 may transmit the first video captured by the imaging unit 210 to the remote monitoring device 130 without degrading the image quality of the first video.

[0126] As described above, the video transmission device 230 is a video transmission device mounted on the monitored vehicle 200 (an example of a vehicle), and includes a first acquisition unit 231 that acquires first video information from the imaging unit 210 that images the surroundings of the monitored vehicle 200, a risk map creation unit 234 (an example of a risk map creation unit) that creates a risk map showing danger areas around the monitored vehicle 200, a control unit 235 (an example of a video creation unit) that creates second video information having a reduced amount of data compared to the first video information based on the risk map, and a communication unit 237 that transmits the second video information to the remote monitoring device 130 for remotely monitoring the monitored vehicle 200.

[0127] As a result, the video transmission device 230 creates second video information to be transmitted to the remote monitoring device 130 according to the danger area around the monitored vehicle 200. The video transmission device 230 can reduce the amount of data, for example, by degrading the image quality of the image of the area other than the danger area to create the second video information. In other words, by transmitting the second video information, the video transmission device 230 can reduce the amount of network bandwidth used for transmission to the remote monitoring device 130 compared to the case of transmitting the first video information. Therefore, the video transmission device 230 can reduce waste of the network bandwidth between the monitored vehicle 200 and the remote monitoring device 130.

[0128] (Embodiment 2) Hereinafter, the video transmission device and the like according to the present embodiment will be described with reference to FIGS.

[0129] [2-1. Configuration of monitored vehicles] First, the configuration of a monitored vehicle equipped with a video transmission device according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing the functional configuration of a monitored vehicle 200a according to this embodiment.

[0130] 10, the monitored vehicle 200a includes an imaging unit 210, a sensor unit 220, and a video transmission device 230a. The video transmission device 230a further includes a NW (Network) state monitoring unit 238 in addition to the components of the video transmission device 230 according to the first embodiment. In this embodiment, the following description will focus on the difference from the first embodiment, that is, the NW state monitoring unit 238 included in the video transmission device 230a.

[0131] The NW status monitoring unit 238 monitors the status of the communication network between the monitored vehicle 200a and the remote monitoring device 130. Specifically, the NW status monitoring unit 238 judges whether the communication network is congested or not. The NW status monitoring unit 238 is an example of a judgment unit. This embodiment is characterized in that the images created by the control unit 235 according to the judgment result of the NW status monitoring unit 238 are different.

[0132] The NW status monitoring unit 238 judges whether the communication network is congested using, for example, at least one of the packet loss rate and the delay time of the communication network. The packet loss rate is the rate of packet loss that occurs suddenly due to the influence of noise or the like. The delay time is the time required for information transmission between the monitored vehicle 200a and the remote monitoring device 130. If the packet loss rate is high or the delay time is long, the time required from when the monitored vehicle 200a transmits the second video information until the operator H visually recognizes the second video based on the second video information becomes long. As a result, the distance traveled by the monitored vehicle 200a from when the monitored vehicle 200a transmits the second video information to the remote monitoring device 130 until the operator H receives the control information increases, increasing the risk of an accident. Therefore, the video transmission device 230a needs to transmit the second video information to the remote monitoring device 130 in a short time even if the communication network is congested. Therefore, in this embodiment, when the communication network is congested, the video transmission device 230a transmits, to the remote monitoring device 130, second video information having a reduced data amount compared to the first video information corresponding to the first video captured by the imaging unit 210. The process of the video transmission device 230a will be described later.

[0133] Here, the packet loss rate and delay time in a communication network will be described with reference to Fig. 11 to Fig. 12B. First, the packet loss rate in a communication network will be described with reference to Fig. 11. Fig. 11 is a sequence diagram showing the operation of calculating the packet loss rate between the monitored vehicle 200a and the remote monitoring device 130 according to this embodiment.

[0134] As shown in FIG. 11, the monitored vehicle 200a transmits an RTT measurement packet to the remote monitoring device 130 for each first period in order to calculate a packet loss rate between the monitored vehicle 200a and the remote monitoring device 130, i.e., in the communication network. For example, the NW status monitoring unit 238 transmits the RTT measurement packet to the remote monitoring device 130 via the control unit 235 and the communication unit 237. A different sequence number is assigned to each RTT measurement packet. FIG. 11 shows an example in which RTT measurement packets with sequence numbers seq1 to seq10 are transmitted. Furthermore, when the remote monitoring device 130 receives an RTT measurement packet from the monitored vehicle 200a, it transmits an RTT measurement response packet with the sequence number included in the RTT measurement packet assigned to it to the monitored vehicle 200a. The NW status monitoring unit 238 determines that a packet loss has occurred if, after transmitting an RTT measurement packet with a sequence number (e.g., seq1) to the remote monitoring device 130, the NW status monitoring unit 238 does not receive an RTT measurement response packet including the sequence number (e.g., seq1) within a second period. The second period may be, for example, shorter than the first period. The first period is, for example, one second.

[0135] The NW status monitoring unit 238 calculates the packet loss rate from the reception status of the RTT measurement response packet in a predetermined period. Specifically, the NW status monitoring unit 238 calculates the packet loss rate based on the number of times that the RTT measurement packet was transmitted in a predetermined period (hereinafter also referred to as the number of times that the packet was transmitted) and the number of times that the RTT measurement response packet corresponding to the RTT measurement packet was not received within a second period after the RTT measurement packet was transmitted (hereinafter also referred to as the number of times that the packet was not received). More specifically, the NW status monitoring unit 238 calculates the packet loss rate by dividing the number of times that the packet was not received by the number of times that the packet was transmitted. The packet loss rate is expressed, for example, as a percentage.

[0136] In the example of FIG. 11, the number of packet transmissions is 10, and the number of packet failures is 1 (specifically, the RTT measurement response packet seq5), so the packet loss rate is 10%.

[0137] The NW status monitoring unit 238 determines that the communication network is congested when the packet loss rate is equal to or greater than a predetermined value. For example, the NW status monitoring unit 238 may determine that the communication network is congested when the packet loss rate is equal to or greater than 30%.

[0138] Next, the delay time in the communication network will be described with reference to FIG. 12A and FIG. 12B. FIG. 12A and FIG. 12B show a case where a round trip delay time is calculated as an example of the delay time. FIG. 12A is a sequence diagram showing an example of an operation for acquiring the delay time of the NW (communication network) according to the present embodiment. FIG. 12A explains acquisition of the delay time when the monitored vehicle 200a and the remote monitoring device 130 are synchronized in time. The time synchronization means, for example, when the monitored vehicle 200a and the remote monitoring device 130 use GPS time as the time or when they have a radio clock. In the following, the monitored vehicle 200a and the remote monitoring device 130 are described as having a GPS sensor (not shown). Note that the GPS time means time information included in the radio wave received by the GPS sensor from a satellite.

[0139] 12A, first, the monitored vehicle 200a assigns a timestamp (timestamp_1) to the RTT measurement packet (S110). For example, the NW status monitoring unit 238 performs the process of step S110. The NW status monitoring unit 238 acquires the time at which the RTT measurement packet is to be transmitted as timestamp_1 from the GPS sensor, and writes it in the RTT measurement packet. Then, the NW status monitoring unit 238 transmits the RTT measurement packet to the remote monitoring device 130 via the communication unit 237 (S120).

[0140] The remote monitoring device 130 receives the RTT measurement packet (S130). The control unit 131 acquires a timestamp (timestamp_2) at the time of receiving the RTT measurement packet based on the time information received by the GPS sensor (S140). The control unit 131 also assigns timestamps (timestamp_1 to timestamp_3) to an RTT measurement response packet, which is a response corresponding to the RTT measurement packet (S150). The control unit 131 acquires the time at which the RTT measurement response packet is transmitted from the GPS sensor as timestamp_3, and writes it in the RTT measurement response packet. The control unit 131 then transmits the RTT measurement response packet to the monitored vehicle 200a via the communication unit 132 (S160).

[0141] The NW status monitoring unit 238 receives the RTT measurement response packet via the communication unit 237 (S170). The NW status monitoring unit 238 acquires a timestamp (timestamp_4) at the time of receiving the RTT measurement response packet (S180). The NW status monitoring unit 238 acquires timestamp_4 based on time information received by the GPS sensor. Then, the NW status monitoring unit 238 calculates a NW round trip delay time using the timestamps (timestamp_1 to timestamp_4) (S190). The NW status monitoring unit 238 calculates the NW round trip delay time, for example, by subtracting the difference between timestamp_2 and timestamp_3 from the difference between timestamp_1 and timestamp_4.

[0142] Next, the acquisition of the delay time when the monitored vehicle 200a and the remote monitoring device 130 are not synchronized in time will be described with reference to Fig. 12B. Fig. 12B is a sequence diagram showing another example of the operation of acquiring the NW delay time according to the present embodiment. Note that steps S210 to S230 shown in Fig. 12B are similar to steps S110 to S130 shown in Fig. 12A, and therefore description thereof will be omitted.

[0143] 12B, the monitored vehicle 200a assigns timestamp_1 and duration time to the RTT measurement response packet (S240). The duration time is the time required for the remote monitoring device 130 to process from receiving the RTT measurement packet to transmitting the RTT measurement response packet, and is calculated, for example, by the real-time clock function of the control unit 131. Then, the control unit 131 transmits the RTT measurement response packet to the monitored vehicle 200a via the communication unit 132 (S250).

[0144] The NW status monitoring unit 238 of the monitored vehicle 200a receives the RTT measurement response packet via the communication unit 237 (S260). The NW status monitoring unit 238 acquires a timestamp (timestamp_4) at the time of receiving the RTT measurement response packet (S270). Then, the NW status monitoring unit 238 calculates the NW round trip delay time using the timestamps (timestamp_1 and timestamp_4) and the duration time (S280). The NW status monitoring unit 238 calculates the NW round trip delay time, for example, by subtracting the duration time from the difference between timestamp_1 and timestamp_4.

[0145] The NW status monitoring unit 238 determines that the communication network is congested when the delay time is equal to or greater than a predetermined value. For example, the NW status monitoring unit 238 may determine that the communication network is congested when the delay time is equal to or greater than 500 ms.

[0146] In the above, an example is shown in which the round trip delay time is calculated by subtracting the processing time in the monitored vehicle 200a (for example, the difference between timestamp_2 and timestamp_3 shown in FIG. 12A, or the duration time shown in FIG. 12B), but the present invention is not limited to this. The round trip delay time may be calculated, for example, from the difference between timestamp_1 and timestamp_4.

[0147] In the above, an example was described in which the delay time is a round-trip delay time, but it may also be a one-way delay time (for example, the time from when the RTT measurement packet is transmitted by the monitored vehicle 200a to when the RTT measurement packet is received by the remote monitoring device 130).

[0148] The NW status monitoring unit 238 may not directly write the transmission time in the packet, but may associate the transmission time with the sequence number of the packet and store the packet in the storage unit 236. Then, in step S190 or S280, the NW status monitoring unit 238 may read the transmission time from the storage unit 236 and calculate the round trip delay time by using the read transmission time and the reception time of the RTT measurement response packet.

[0149] The calculation of the state of the communication network including at least one of the packet loss rate and the delay time is repeatedly performed at a predetermined time interval. The NW state monitoring unit 238 may store the calculated state of the communication network in the storage unit 236, for example.

[0150] The NW status monitoring unit 238 may acquire at least one of the packet loss rate and the delay time by receiving information about the state of the communication network between the monitored vehicle 200a and the remote monitoring device 130 from another device via the communication unit 237. The NW status monitoring unit 238 acquires the packet loss rate by receiving the packet loss rate from the remote monitoring device 130, for example.

[0151] [2-2. Behavior of monitored vehicle] Next, the operation of the video transmission device 230a will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the operation of the video transmission device 230a according to this embodiment. Note that steps S310 to S350 in Fig. 13 are similar to steps S10 to S50 shown in Fig. 4 of the first embodiment, and therefore description thereof will be omitted.

[0152] The NW status monitoring unit 238 acquires the status of the NW (communication network) (S360). The NW status monitoring unit 238 acquires at least one of the packet loss rate and the delay time of the NW as the status of the NW. The NW status monitoring unit 238 may acquire at least one of the packet loss rate and the delay time of the NW by calculating at least one of the packet loss rate and the delay time of the NW by the operations shown in Figs. 11 to 12B. The NW status monitoring unit 238 may also acquire at least one of the packet loss rate and the delay time of the NW by reading at least one of the packet loss rate and the delay time of the NW from the storage unit 236.

[0153] The NW status monitoring unit 238 may calculate the packet loss rate from the most recent packet transmission count and the most recent packet non-reception count at the time when the first video information is acquired in step S310 or the sensing result is acquired in step S320. The NW status monitoring unit 238 may calculate the packet loss rate from the most recent packet transmission count and the most recent packet non-reception count during a period going back a third period from the time when the first video information or the sensing result is acquired. Alternatively, the NW status monitoring unit 238 may calculate the packet loss rate from the most recent packet non-reception count during a predetermined number of packet transmissions (e.g., 10 times) going back from the time when the first video information is acquired in step S310 or the time when the sensing result is acquired in step S320.

[0154] The NW status monitoring unit 238 may calculate the delay time from the reception status of the latest RTT measurement response packet at the time when the first video information is acquired in step S310 or the time when the sensing result is acquired in step S320. The NW status monitoring unit 238 may calculate the delay time from a plurality of RTT measurement response packets in a period from the time when the first video information or the sensing result is acquired to a time going back a third period. The NW status monitoring unit 238 may calculate the delay time of the NW using the longest delay time among a plurality of delay times calculated using a plurality of RTT measurement response packets, or may calculate the delay time of the NW using an average value of the plurality of delay times.

[0155] The NW state monitoring unit 238 judges whether the NW is congested or not from the acquired NW state (S370). For example, the NW state monitoring unit 238 judges that the NW is congested when at least one of the packet loss rate and the delay time is equal to or greater than a predetermined value.

[0156] When the NW status monitoring unit 238 determines that the NW is congested (Yes in S370), the control unit 235 adjusts the transmission rate (bit rate) for transmitting the video information to the remote monitoring device 130 (S380). Specifically, when the NW status monitoring unit 238 determines that the NW is congested, the control unit 235 adjusts the transmission rate to be lower. For example, the control unit 235 changes the current first transmission rate to a second transmission rate that is lower than the first transmission rate. In addition, the control unit 235 may change the adjustment amount of the transmission rate according to the degree of congestion of the NW. The control unit 235 may adjust the second transmission rate to be lower as the degree of congestion of the NW increases, that is, as at least one of the packet loss rate and the delay time increases. In addition, the storage unit 236 may store a table indicating the relationship between the degree of congestion of the NW and the second transmission rate, and the control unit 235 may adjust the second transmission rate using the table.

[0157] Then, the control unit 235 creates second video information with a reduced data amount according to the second transmission rate adjusted in step S380 (S390). The control unit 235 switches the data amount of the video information to be transmitted to the remote monitoring device 130 according to the second transmission rate. The control unit 235 creates the second video information so that the data amount (hereinafter, also referred to as the adjusted data amount) is the amount that can be transmitted within a predetermined time at the adjusted second transmission rate, for example. The control unit 235 creates the second video information with an adjusted data amount or less by at least one of degrading the image quality of the image of the first area corresponding to an area other than the dangerous area shown in the danger map in the first video information, compared to the image of the second area corresponding to the dangerous area, and cutting out the image of the second area from the first video information. Note that the image quality of the image of the second area does not need to be degraded. In other words, the control unit 235 may create the second video information by degrading the image quality of only the image of the first area in the first video shown by the first video information.

[0158] Then, the control unit 235 transmits the second video information created in step S390 to the remote monitoring device 130 (S400). The control unit 235 transmits the second video information to the remote monitoring device 130 via the communication unit 237 at the second transmission rate.

[0159] Furthermore, when the NW status monitoring unit 238 determines that the NW is not congested (No in S370), the control unit 235 creates second video information based on the first video information. The second video information created when the result is No in step S370 has a larger data amount than the second video information created when the result is Yes in step S370. For example, the second video information may be created without reducing the data amount from the first video information. In other words, the control unit 235 may use the first video information as the second video information to be transmitted to the remote monitoring device 130 (S410).

[0160] Then, the control unit 235 transmits the second video information created in step S410 to the remote monitoring device 130 (S400). The control unit 235 transmits the second video information to the remote monitoring device 130 at the first transmission rate via the communication unit 237. When the NW is not congested, the control unit 235 may transmit the first video information to the remote monitoring device 130 as the second video information without reducing the data amount. For example, when the NW status monitoring unit 238 determines that the NW is congested, the communication unit 237 may transmit the second video information to the remote monitoring device 130, and when the NW status monitoring unit 238 determines that the NW is not congested, the communication unit 237 may transmit the first video information to the remote monitoring device 130.

[0161] As described above, the video transmission device 230a can suppress waste of network bandwidth while suppressing deterioration in real-timeness of the second video information by changing the amount of data of the second video information to be transmitted to the remote monitoring device 130 depending on the danger area shown on the danger map and the state of the network.

[0162] In the above description, the control unit 235 adjusts the transmission rate when the NW status monitoring unit 238 determines that the NW is congested, but the present invention is not limited to this. When the NW status monitoring unit 238 determines that the NW is congested, the control unit 235 may create second video information in which the amount of data is reduced from the first video information without adjusting the transmission rate.

[0163] As described above, the video transmission device 230a further includes a NW status monitoring unit 238 (an example of a determination unit) that determines whether or not the communication network between the monitored vehicle 200 (an example of a vehicle) and the remote monitoring device 130 is congested. If the NW status monitoring unit 238 determines that the communication network is congested, the communication unit 237 transmits the second video information to the remote monitoring device 130, and if the NW status monitoring unit 238 determines that the communication network is not congested, the communication unit 237 transmits the first video information to the remote monitoring device 130.

[0164] As a result, when the communication network is congested, the video transmission device 230a transmits the second video information having a reduced amount of data compared to the first video information. In other words, by transmitting the second video information when the communication network is congested, the video transmission device 230a can reduce the amount of network bandwidth used for transmission to the remote monitoring device 130 compared to when transmitting the first video information. The operator H can quickly view the video based on the second video information even when the communication network is congested. Thus, the video transmission device 230a can notify the operator H of dangers while driving in a shorter time compared to when transmitting the first video information, while reducing waste of network bandwidth.

[0165] (First modification of the second embodiment) Hereinafter, the video transmission device and the like according to this modification will be described with reference to FIG. 14. The configuration of the monitored vehicle according to this modification is the same as that of the monitored vehicle 200a according to the second embodiment, and therefore a description thereof will be omitted. In this modification, the object detection unit 233 outputs the generated obstacle information to the danger level map 234 and the control unit 235. The operation of such a video transmission device 230a will be described below. FIG. 14 is a flowchart showing the operation of the video transmission device 230a according to this modification. Note that steps S510, S520, and S540 to S570 shown in FIG. 14 are the same processes as steps S310, S320, and S340 to S370 shown in FIG. 13, respectively, and therefore a description thereof will be omitted.

[0166] 14, the object detection unit 233 creates obstacle information from the sensing result, thereby acquiring the obstacle information (S530). The object detection unit 233 creates the obstacle information, for example, from the first video information and the sensing result. The obstacle information includes at least one of the current position, size, and traveling direction of an obstacle that is an obstacle to the travel of the monitored vehicle 200a. The object detection unit 233 outputs the created obstacle information to the danger level map creation unit 234 and the control unit 235.

[0167] Then, when the NW status monitoring unit 238 determines that the NW is congested (Yes in S570), the control unit 235 transmits at least one of the sensing result and the obstacle information to the remote monitoring device 130 (S580). In other words, when the NW status monitoring unit 238 determines that the NW is congested, the control unit 235 prohibits the transmission of video information. Here, the sensor unit 220 is a sensor other than the imaging unit 210, such as a radar, an ultrasonic sensor, or a LiDAR. Note that, hereinafter, at least one of the sensing result and the obstacle information will also be referred to as non-video information.

[0168] In this way, when there is a limit to the communication bandwidth, the control unit 235 according to this modification transmits, for example, only the non-video information out of the video information captured by the imaging unit 210 and the non-video information to the remote monitoring device 130.

[0169] Furthermore, when the communication band is limited and sensing results are to be transmitted, the control unit 235 may further prioritize sensing results acquired by sensors with a smaller amount of information and transmit them to the remote monitoring device 130. For example, when the communication band is limited and sensing results are to be transmitted, the control unit 235 may further transmit only sensing results acquired by sensors with an amount of information less than a predetermined amount of information to the remote monitoring device 130. The predetermined amount of information may be set in advance, or when the communication band is limited in stages, it may be set according to the stage.

[0170] Note that a congested NW may mean, for example, that the communication band is so limited that the image captured by the imaging unit 210 is distorted.

[0171] Furthermore, when the NW status monitoring unit 238 determines that the NW is not congested (No in S570), the control unit 235 transmits the video information to the remote monitoring device 130 (S590). The video information may be the first video information acquired in step S510, or the second video information generated based on the first video information. The second video information is, for example, video information generated by reducing the amount of data from the first video information. The second video information may be, for example, a video captured in the traveling direction of the monitored vehicle 200a. The video information may also include, as additional information, at least one of obstacle information detected by the object detection unit 233 and the sensing result (sensor information) of the sensor unit 220. For example, when the video information includes a video captured in the traveling direction of the monitored vehicle 200a, the video information may include, as additional information, obstacle information or sensing results in a direction other than the traveling direction (for example, rearward, left, right, etc. of the monitored vehicle 200a).

[0172] As described above, the video transmission device 230a of this modified example includes a first acquisition unit 231 that acquires first video information from an imaging unit 210 that images the surroundings of the monitored vehicle 200a (an example of a vehicle), a second acquisition unit 232 that acquires sensing results from a sensor unit 220 that senses the surroundings of the monitored vehicle 200a, an object detection unit 233 that creates obstacle information indicating the position and speed of an obstacle based on the sensing results and the first video information, a risk map creation unit 234 that creates a risk map that indicates risk areas around the monitored vehicle 200a, a control unit 235 (an example of a video creation unit) that creates second video information based on the risk map, a communication unit 237 that transmits monitoring information to be sent to a remote monitoring device 130 for remotely monitoring the monitored vehicle 200a, and a NW status monitoring unit 238 (an example of a determination unit) that determines whether the communication network between the monitored vehicle 200a and the remote monitoring device 130 is congested. Then, the communication unit 237 transmits monitoring information selected based on the determination result of the NW state monitoring unit 238, the monitoring information including at least one of the sensing result, the obstacle information, and the second video information.

[0173] In addition, for example, when the NW status monitoring unit 238 determines that the communication network is congested, the communication unit 237 transmits at least one of the sensing result and obstacle information to the remote monitoring device 130, and when the NW status monitoring unit 238 determines that the communication network is not congested, the communication unit 237 transmits second video information to the remote monitoring device 130.

[0174] In this modification, the object detection unit 233 outputs obstacle information to the risk level map creation unit 234 and the control unit 235. Then, when the NW state monitoring unit 238 determines that the communication network between the monitored vehicle 200a and the remote monitoring device 130 is not congested (No in S570), the control unit 235 transmits at least the video information of the video information and non-video information to the remote monitoring device 130 via the communication unit 237. Note that in step S580, in addition to the video information, non-video information (for example, at least one of the obstacle information and the sensor information) may be transmitted.

[0175] Furthermore, when the NW status monitoring unit 238 determines that the communication network between the monitored vehicle 200a and the remote monitoring device 130 is congested (Yes in S570), the control unit 235 transmits information with a smaller transmission amount than the information (e.g., video information) transmitted in step S590. For example, of the video information and non-video information, the control unit 235 transmits only the non-video information to the remote monitoring device 130 via the communication unit 237 (S580).

[0176] In this way, by limiting the information transmitted by the remote monitoring device 130 to information about obstacles detected by the sensor unit 220, etc., when the communication bandwidth is limited, information about the surroundings of the monitored vehicle 200a can be transmitted to the remote monitoring device 130 at a lower transmission rate.

[0177] Furthermore, when the sensor unit 220 is configured with multiple types of sensors, the control unit 235 acquires the sensing results of each of the multiple types of sensors. Then, the control unit 235 may transmit only the sensing results detected by a predetermined sensor among the sensing results of each of the multiple types of sensors to the remote monitoring device 130. The predetermined sensor is a sensor that has an amount of information less than a predetermined amount of information. The predetermined sensor is, for example, a radar other than LiDAR, an ultrasonic sensor, etc., but is not limited thereto.

[0178] (Modification 2 of the second embodiment) Hereinafter, the video transmission device and the like according to this modified example will be described with reference to Fig. 15. The configuration of the monitored vehicle equipped with the video transmission device according to this modified example is the same as that of modified example 1, and therefore a description thereof will be omitted. Fig. 15 is a flowchart showing the operation of the video transmission device 230a according to this modified example. The processing of steps S610 to S670 shown in Fig. 15 is the same as that of steps S510 to S570 shown in Fig. 14, and therefore a description thereof will be omitted.

[0179] When the NW status monitoring unit 238 determines that the NW is not congested (No in S670), the control unit 235 creates second video information to be transmitted to the remote monitoring device 130. The second video information created when the result is No in step S670 has a larger amount of data than the second video information created when the results are Yes in steps S670 and S680. The control unit 235 creates the second video information based on the first video information. The second video information may be, for example, information created without reducing the amount of data from the first video information. That is, the control unit 235 may treat the first video information as the second video information to be transmitted to the remote monitoring device 130 (S730). The control unit 235 may also create the second video information by, for example, reducing the amount of data from the first video information.

[0180] Furthermore, when the NW status monitoring unit 238 determines that the NW is congested (Yes in S670), it further determines whether or not the congestion degree of the NW is equal to or lower than a threshold (S680). The NW status monitoring unit 238 determines whether or not the congestion degree of the NW is equal to or lower than a threshold based on, for example, at least one of a packet loss rate and a delay time of the NW. The NW status monitoring unit 238 may determine that the congestion degree of the NW is equal to or lower than a threshold when, for example, at least one of the packet loss rate and the delay time is equal to or lower than a threshold.

[0181] If the NW status monitoring unit 238 determines that the congestion degree of the NW is equal to or lower than the threshold (Yes in S680), the control unit 235 proceeds to step S690. The processes in steps S690 to S710 are similar to steps S380 to S400 shown in FIG. 13, and therefore the description will be omitted.

[0182] Furthermore, if the NW status monitoring unit 238 determines that the congestion degree of the NW is greater than the threshold (No in S680), the control unit 235 proceeds to step S720. The process of step S720 is similar to step S580 shown in FIG. 14, and therefore the description will be omitted.

[0183] As described above, the video transmission device 230a of this modified example includes a first acquisition unit 231 that acquires first video information from an imaging unit 210 that images the surroundings of the monitored vehicle 200a (an example of a vehicle), a risk map creation unit 234 that creates a risk map showing risk areas around the monitored vehicle 200a, a control unit 235 (an example of a video creation unit) that creates second video information having a reduced amount of data compared to the first video information based on the risk map, a communication unit 237 that transmits the second video information to a remote monitoring device 130 for remotely monitoring the monitored vehicle 200a, and a NW status monitoring unit 238 (an example of a determination unit) that determines whether the communication network between the monitored vehicle 200a and the remote monitoring device 130 is congested or not.

[0184] When the NW state monitoring unit 238 determines that the communication network is congested, the NW state monitoring unit 238 further determines whether the congestion degree of the communication network is equal to or lower than a threshold. When the NW state monitoring unit 238 determines that the congestion degree is equal to or lower than a threshold, the communication unit 237 transmits the second video information, and when the NW state monitoring unit 238 determines that the congestion degree is greater than the threshold, the communication unit 237 transmits at least one of the sensing result and the obstacle information.

[0185] In this modification, the object detection unit 233 outputs obstacle information to the risk map creation unit 234 and the control unit 235. Then, when the NW state monitoring unit 238 determines that the communication network between the monitored vehicle 200a and the remote monitoring device 130 is not congested (No in S670), the control unit 235 creates second video information (S730) and transmits it to the remote monitoring device 130 (S710). In other words, when the NW state monitoring unit 238 determines that the communication network is not congested, the communication unit 237 transmits the second video information. The second video information at this time may be, for example, the first video information.

[0186] Furthermore, when the NW status monitoring unit 238 determines that the communication network between the monitored vehicle 200a and the remote monitoring device 130 is congested and the degree of congestion of the communication network is equal to or lower than the threshold (Yes in S670 and S680), the control unit 235 creates second video information (S700) and transmits it to the remote monitoring device 130 (S710). In other words, when the NW status monitoring unit 238 determines that the degree of congestion of the communication network is equal to or lower than the threshold, the communication unit 237 transmits the second video information. The second video information at this time is video information with a smaller data amount than the second video information created when the result in step S670 is No.

[0187] Furthermore, when the NW state monitoring unit 238 determines that the communication network between the monitored vehicle 200a and the remote monitoring device 130 is congested and the degree of congestion of the NW is greater than the threshold (Yes in S670 and No in S680), the control unit 235 transmits at least one of the sensing result and the obstacle information to the remote monitoring device 130 (S720). In other words, when the NW state monitoring unit 238 determines that the degree of congestion of the communication network is greater than the threshold, the communication unit 237 transmits at least one of the sensing result and the obstacle information. It can also be said that the control unit 235 prohibits the transmission of the video information when the results are Yes in step S670 and No in step S680. Note that each of the sensing result and the obstacle information is information having a smaller data amount than the second video information created in step S700.

[0188] In this manner, the control unit 235 according to this modification switches the information to be transmitted depending on the congestion level of the network. Therefore, the video transmission device 230a can transmit the information about the surroundings of the monitored vehicle 200a at a more appropriate transmission rate depending on the congestion level of the network.

[0189] In step S720, only one of the sensing result and the obstacle information may be transmitted.

[0190] (Other embodiments) The present disclosure has been described above based on the first and second embodiments (hereinafter, each embodiment will also be described), but the present disclosure is not limited to the above-described embodiments.

[0191] For example, in each of the above-described embodiments, the control unit creates the second video information according to the danger area when the travel route and the danger area overlap, but this is not limited to this. When the danger map includes a danger area, the control unit may create the second video information according to the danger area. This makes it possible to suppress waste of network bandwidth while allowing the operator to clearly view the video corresponding to the danger area around the monitored vehicle.

[0192] In addition, in each of the above embodiments, an example has been described in which the image quality of the rear and left and right directions (an example of a first region) among the images captured in the front, rear, left and right directions is degraded more than that of the image in the front (an example of a second region), but this is not limited thereto. The second region may be, for example, an area of ​​the image captured in the front that corresponds to a dangerous area in the image. In other words, the second region may be a part of the image captured in the front. This makes it possible to increase the area of ​​the first video information in which the image quality is degraded, thereby further reducing waste of network bandwidth.

[0193] In addition, in each of the above embodiments, when the first video information includes a plurality of frames of video information, the control unit of the video transmission device may create the second video information by extracting one or more frames from the plurality of frames. In other words, the control unit may reduce the amount of data by reducing the number of frames constituting the video to be transmitted to the remote monitoring device.

[0194] Furthermore, the control unit of the video transmission device in each of the above embodiments may include obstacle information in the second video information and transmit it to the remote monitoring device. This allows the remote monitoring device to, for example, change the color of an obstacle that is close to the monitored vehicle shown in the video displayed on the display device, or to blink an obstacle that is close to the monitored vehicle shown in the video. Thus, the video transmission device can effectively notify the operator of the danger of an obstacle while reducing waste of network bandwidth.

[0195] In addition, the control unit included in the video transmission device in each of the above embodiments may transmit vehicle information of the monitored vehicle to the remote monitoring device via the communication unit. For example, the control unit may transmit the vehicle information to the remote monitoring device when a danger area exists in the danger map, and may stop transmitting the vehicle information to the remote monitoring device when no danger area exists.

[0196] In addition, in each of the above embodiments, the imaging unit captures images of the surroundings of the monitored vehicle, but the imaging unit may also capture images of the interior of the vehicle. For example, when the video transmission device receives an instruction from an operator to transmit images of the interior of the vehicle, the video transmission device may transmit third video information based on the images of the interior of the vehicle captured by the imaging unit to the remote monitoring device. Even in this case, the video transmission device may transmit third video information with a reduced data amount depending on the state of the NW shown in the second embodiment.

[0197] In addition, the communication method between the devices provided in the monitored vehicle in each of the above embodiments is not particularly limited. The devices may communicate wirelessly or may communicate via wired communication. The devices may communicate via a combination of wireless and wired communication.

[0198] Furthermore, each processing unit of the remote monitoring device and the monitored vehicle according to each of the above embodiments (e.g., a control unit, a danger map creation unit, an object detection unit, and a network status monitoring unit) is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or may be integrated into a single chip to include some or all of them.

[0199] The integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. A field programmable gate array (FPGA) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of the circuit cells inside the LSI may also be used.

[0200] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0201] Furthermore, all the numbers used above are merely examples for specifically explaining the present disclosure, and each embodiment of the present disclosure is not limited to the exemplified numbers.

[0202] In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to another functional block. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0203] In addition, the order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. In addition, some of the steps may be performed simultaneously (in parallel) with other steps.

[0204] In addition, the present disclosure also includes forms obtained by applying various modifications to the above-described embodiments, etc. that a person skilled in the art may think of, and forms realized by arbitrarily combining the components and functions of the embodiments, etc., within the scope that does not deviate from the spirit of the present disclosure. [Industrial Applicability]

[0205] An information processing method according to one aspect of the present disclosure is effective for a remote monitoring device that enables an operator at a remote location to remotely monitor a vehicle via a communication network. [Explanation of symbols]

[0206] 10 Vehicle Control System 100 Remote Monitoring System 110 Display device 120 Operation input device 130 Remote monitoring device 131 Control Unit 132 Communications Department 133 Storage section 200, 200a Monitored vehicle (vehicle) 210 Imaging unit 220 Sensor section 230, 230a Video transmission device 231 First Acquisition Department 232 Second Acquisition Department 233 Object Detection Unit 234 Danger Map Creation Department (Danger Map Creation Department) 235 Control Unit (Video Creation Unit) 236 Memory section 237 Communications Department 238 NW status monitoring unit (judgment unit) 300 Network 310 Radio Base Station 400 Obstacles 501, 502 road

Claims

1. acquiring first video information used for at least one of remote monitoring and remote operation of the mobile object from an imaging unit that images an image of the surroundings of the mobile object; Information processing methods.

2. obtaining second image information having a quality different from that of the first image represented by the first image information; The information processing method according to claim 1 .

3. receiving an image of the surroundings of the mobile object from the mobile object via a communication network; The information processing method according to claim 2 .

4. selecting, based on a quality of the communication network, one of the first video information and the second video information to be used for the remote monitoring; The information processing method according to claim 3.

5. generating information for notifying an operator of dangers during driving, the information being used for the remote monitoring and the remote operation, based on the first video information and map information; The information processing method according to any one of claims 1 to 4.

6. generating information for notifying an operator of dangers during driving, the information being used for the remote monitoring and the remote operation, based on a sensing result from a sensor unit that senses the surroundings of the moving body and the first video information; The information processing method according to any one of claims 1 to 4.

7. The information for notifying the operator of a danger during driving is generated using obstacle information based on at least one of the sensing result and the first image information, and the obstacle information is used for the remote monitoring and the remote operation; The obstacle information includes at least one of a position, an attribute, and a traveling direction of the obstacle. The information processing method according to claim 6.

8. The moving object is a vehicle. The information processing method according to any one of claims 1 to 7.

9. determining whether the communications network is congested; When it is determined that the communication network is congested, the first video information or the second video information is acquired, whichever is lower in quality.

5. The information processing method according to claim 3.

10. an acquisition unit that acquires first video information used for at least one of remote monitoring and remote operation of the moving object from an imaging unit that images the surroundings of the moving object; Information processing device.

Images