Video transmission method, video transmission system, and control device

Abstract

To reduce an amount of video data transmitted from a moving object to an external device.SOLUTION: A video transmission method includes: a process for acquiring a plurality of video data obtained by each of a plurality of cameras mounted on a moving object; a data reduction process for acquiring transmit video data by reducing a data volume of the plurality of video data; and a process for transmitting the transmit video data from the moving object to an external device after the data reduction process. The data reduction process includes at least two of a selection process for excluding at least one of the plurality of video data from the transmit video data in accordance with a scene in which in the moving object is placed, a reduction process for reducing at least one of the plurality of video data, and a connection process for connecting first video data and second video data so that a screen of the second video data is displayed within a screen of the first video data.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a technique for transmitting video data from a mobile object to an external device. [Background technology]

[0002] Patent Document 1 discloses a picture-in-picture display method in a two-way system such as a videophone or web chat via a network.

[0003] Other known techniques relating to video transmission include those disclosed in Patent Documents 2 and 3.

[0004] Non-Patent Document 1 discloses a "super resolution technology" that converts an input low-resolution image into a high-resolution image. In particular, Non-Patent Document 1 discloses SRCNN, which applies deep learning based on a Convolutional Neural Network (CNN) to super-resolution (SR). A model that converts (maps) an input low-resolution image into a high-resolution image is obtained by machine learning. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-150299 A [Patent Document 2] Patent Publication No. 2021-026554 [Patent Document 3] JP 2014-071776 A [Non-patent literature]

[0006] [Non-Patent Document 1] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang, "Image Super-Resolution Using Deep Convolutional Networks", arXiv:1501.00092v3 [cs.CV], July 31, 2015 (https: / / arxiv.org / pdf / 1501.00092.pdf) Summary of the Invention [Problem to be solved by the invention]

[0007] Consider a situation in which video data obtained by a camera mounted on a mobile object is transmitted to an external device. For example, in remote support of a mobile object, the video data obtained by a camera mounted on the mobile object is transmitted to the remote operator and used. However, the amount of video data is relatively large. As the number of cameras increases and the amount of video data (number of streams) transmitted simultaneously increases, the amount of transmitted data increases. The increase in the amount of transmitted data leads to communication delays and increases in communication costs. From the viewpoint of using video data, it is desirable to suppress communication delays and communication costs as much as possible.

[0008] One object of the present disclosure is to provide a technique that can reduce the amount of video data transmitted from a mobile object to an external device. [Means for solving the problem]

[0009] The first aspect relates to a video transmission method for transmitting video data from a mobile object to an external device. The video transmission method is A process of acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on a moving object; A data reduction process for obtaining transmission video data by reducing the data amount of the plurality of video data; A process of transmitting the transmission video data from the mobile object to an external device after the data reduction process; Includes. The data reduction process is A selection process of omitting at least one of the plurality of video data from the transmission video data according to a scene in which the moving object is located; A reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Includes at least two of the following:

[0010] The second aspect relates to a video transmission system for transmitting video data from a mobile object to an external device. The video transmission system includes one or more processors. The one or more processors Acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on a moving object; performing a data reduction process to obtain transmission video data by reducing a data amount of the plurality of video data; After the data reduction process, the transmission video data is transmitted from the mobile unit to an external device. It is configured as follows. The data reduction process is A selection process of omitting at least one of the plurality of video data from the transmission video data according to a scene in which the moving object is located; A reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Includes at least two of the following:

[0011] The third aspect relates to a control device that controls a moving object. The controller comprises one or more processors. The one or more processors Acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on a moving object; performing a data reduction process to obtain transmission video data by reducing a data amount of the plurality of video data; Transmitting the transmission video data to an external device after data reduction processing It is configured as follows. The data reduction process is A selection process of omitting at least one of the plurality of video data from the transmission video data according to a scene in which the moving object is located; A reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Includes at least two of the following: Effect of the Invention

[0012] According to the present disclosure, it is possible to reduce the amount of video data transmitted from a moving object to an external device. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an overview of a video transmission system according to an embodiment of the present disclosure. [Diagram 2] 1 is a conceptual diagram illustrating a remote support system that is an application example of a video transmission system according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a conceptual diagram illustrating an example of a plurality of cameras mounted on a moving object according to an embodiment of the present disclosure. [Figure 4] FIG. 11 is a conceptual diagram for explaining an example of a selection process according to an embodiment of the present disclosure. [Diagram 5] 11A and 11B are conceptual diagrams for explaining another example of the selection process according to the embodiment of the present disclosure. [Figure 6] FIG. 2 is a conceptual diagram illustrating a selection processing unit according to an embodiment of the present disclosure. [Figure 7] 1A to 1C are conceptual diagrams for explaining an example of reduction processing and super-resolution processing according to an embodiment of the present disclosure. [Figure 8] 2 is a conceptual diagram illustrating a reduction processing unit and a super-resolution processing unit according to an embodiment of the present disclosure. [Figure 9]11A and 11B are conceptual diagrams for explaining an example of a merging process according to an embodiment of the present disclosure. [Figure 10] 11A and 11B are conceptual diagrams for explaining an example of a merging process according to an embodiment of the present disclosure. [Figure 11] FIG. 2 is a conceptual diagram illustrating a combination processing unit according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a block diagram for explaining a first example of a data reduction process according to an embodiment of the present disclosure. [Figure 13] FIG. 2 is a block diagram for explaining a first example of a data reduction process according to an embodiment of the present disclosure. [Figure 14] FIG. 11 is a block diagram for explaining a second example of a data reduction process according to an embodiment of the present disclosure. [Figure 15] FIG. 11 is a block diagram for explaining a third example of a data reduction process according to an embodiment of the present disclosure. [Figure 16] FIG. 13 is a block diagram for explaining a fourth example of a data reduction process according to an embodiment of the present disclosure. [Figure 17] FIG. 13 is a block diagram for explaining a fifth example of a data reduction process according to an embodiment of the present disclosure. [Figure 18] 1 is a block diagram illustrating a configuration example of a moving body according to an embodiment of the present disclosure. [Figure 19] 1 is a block diagram illustrating a configuration example of a remote operator terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0015] 1. Overview of the video transmission system 1 is a conceptual diagram showing an overview of a video transmission system 1 according to the present embodiment. The video transmission system 1 includes a mobile object 100 and an external device 200 that exists outside the mobile object 100. The mobile object 100 and the external device 200 are connected to each other via a communication network 10. The mobile object 100 and the external device 200 are capable of communicating with each other via the communication network 10. Typically, the mobile object 100 and the external device 200 perform wireless communication.

[0016] Examples of the moving body 100 include a vehicle, a robot, an airborne object, and the like. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of the robot include a logistics robot, a work robot, and the like. Examples of the airborne object include an airplane, a drone, and the like.

[0017] The external device 200 is, for example, a management server that manages the mobile object 100. As another example, the external device 200 may be a remote operator terminal used for remote support of the mobile object 100. As yet another example, the external device 200 may be a mobile object other than the mobile object 100.

[0018] A camera 110 is mounted on the moving object 100. The camera 110 acquires video data (streaming data) showing the situation around the moving object 100. The moving object 100 transmits the video data acquired by the camera 110 to an external device 200. The external device 200 receives the video data transmitted from the moving object 100 and uses the received video data.

[0019] FIG. 2 shows a remote support system 1A which is an application example of the video transmission system 1. The mobile object 100 is a target of remote support by a remote operator. The external device 200 is a remote operator terminal 200A operated by the remote operator. The mobile object 100 transmits video data obtained by a camera 110 to the remote operator terminal 200A. The remote operator terminal 200A receives the video data transmitted from the mobile object 100 and displays the received video data on a display device 210. The remote operator sees the video data displayed on the display device 210, grasps the situation around the mobile object 100, and remotely supports the operation of the mobile object 100. Examples of remote support by the remote operator include recognition support, judgment support, remote driving, and the like. Instructions by the remote operator are sent from the remote operator terminal 200A to the mobile object 100. The mobile object 100 operates according to the instructions by the remote operator.

[0020] There are cases where multiple cameras 110 are mounted on the moving object 100. For example, in the above-mentioned remote assistance, it is useful to grasp the situation around the moving object 100 using multiple cameras 110.

[0021] FIG. 3 is a conceptual diagram showing an example of a plurality of cameras 110 mounted on a moving body 100. In the example shown in FIG. 3, the moving body 100 is equipped with a front camera C1, a left front camera C2, a right front camera C3, a left side camera C4, a right side camera C5, and a rear camera C6. The front camera C1 acquires video data of the front of the moving body 100. The left front camera C2 acquires video data of the left front of the moving body 100. The right front camera C3 acquires video data of the right front of the moving body 100. The left side camera C4 acquires video data of the left direction and the left rear of the moving body 100. The right side camera C5 acquires video data of the right direction and the right rear of the moving body 100. The rear camera C6 acquires video data of the rear of the moving body 100.

[0022] In this way, when multiple cameras 110 are mounted on the moving object 100, multiple pieces of video data are obtained from each of the multiple cameras 110. The moving object 100 transmits the multiple pieces of video data (streaming data) to the external device 200 simultaneously and in parallel.

[0023] However, the amount of video data is relatively large. As the number of cameras increases and the number of video data (streams) transmitted simultaneously increases, the amount of transmitted data increases. The increase in the amount of transmitted data leads to increased communication delays and communication costs. From the viewpoint of using video data, it is desirable to suppress communication delays and communication costs as much as possible. For example, in the case of remote support of a mobile object 100, communication delays can cause the remote operator to make delayed decisions, awkward remote operation (in some cases, erratic driving), and the like. Therefore, it is desirable to reduce the amount of transmitted data from the mobile object 100 as much as possible without compromising the accuracy of remote support.

[0024] Therefore, this embodiment proposes a technique that can appropriately reduce the amount of video data transmitted from the moving object 100 to the external device 200.

[0025] 2. Data reduction process The moving object 100 acquires a plurality of pieces of video data obtained by each of the plurality of cameras 110. Before transmitting the plurality of pieces of video data to the external device 200, the moving object 100 executes a "data reduction process" to reduce the amount of data of the plurality of pieces of video data. Hereinafter, at least one piece of video data obtained as a result of the data reduction process is referred to as "transmission video data VT." After the data reduction process, the moving object 100 transmits the transmission video data VT to the external device 200.

[0026] Various examples of data reduction processes are described below.

[0027] 2-1. Selection process The "selection process" is a process of omitting at least one of the multiple video data obtained by the multiple cameras 110 from the transmission video data VT. In other words, the selection process selects only the video data with a high priority from the multiple video data, and excludes the video data with a low priority. At this time, the priority is dynamically determined according to the scene in which the moving object 100 is placed. In other words, the selection process omits at least one of the multiple video data from the transmission video data VT according to the scene in which the moving object 100 is placed.

[0028] Fig. 4 is a conceptual diagram for explaining an example of the selection process. In the example shown in Fig. 4, the moving body 100 is planning to make a left turn or is in the middle of making a left turn. In this case, the priority of the video data of the left front, left direction, and left rear is relatively high. On the other hand, the priority of the video data of the right front, right direction, and right rear is relatively low. Therefore, it is conceivable to omit the video data of the right front, right direction, and right rear from the transmission video data VT. The same applies when the bus pulls over to the left to stop at a bus stop.

[0029] Fig. 5 is a conceptual diagram for explaining another example of the selection process. In the example shown in Fig. 5, the moving body 100 is planning to reverse or is currently reverse. In this case, the rear image data has the highest priority. On the other hand, the front, left front, and right front image data have relatively low priority. Therefore, it is considered to omit the front, left front, and right front image data from the transmission image data VT.

[0030] 6 is a conceptual diagram showing the selection processing unit 101 according to this embodiment. The selection processing unit 101 is included in the moving object 100, and executes the selection process.

[0031] For example, the selection processing unit 101 acquires information reflecting the "planned movement direction" of the moving body 100. For example, when the moving body 100 is a vehicle, the planned movement direction is determined based on the steering direction, steering angle, blinker information, gear position, wheel speed, and the like. As another example, the planned movement direction may be determined based on the current position and target movement route of the moving body 100. The selection processing unit 101 dynamically sets the priority of multiple video data based on the planned movement direction of the moving body 100. Specifically, the selection processing unit 101 sets the priority of video data in a direction closer to the planned movement direction higher than the priority of video data in a direction farther from the planned movement direction. Then, the selection processing unit 101 selects the video data with the high priority and omits the video data with the low priority from the transmission video data VT.

[0032] As another example, a "specific object" appearing in the video data may be taken into consideration. A specific object is an object that is likely to be focused on by a remote operator. For example, a specific object includes at least one of a person, a bicycle, another vehicle, a traffic light, and a sign. The selection processing unit 101 recognizes a specific object in the video data by analyzing images constituting the video data using a known method. Then, the selection processing unit 101 sets the priority of video data in which more specific objects are appearing than the priority of video data in which fewer specific objects are appearing. Then, the selection processing unit 101 selects the video data with the higher priority and omits the video data with the lower priority from the transmission video data VT.

[0033] By the above-described selection process, the amount of transmission video data VT is reduced.

[0034] 2-2.Reduction process The "reduction process" is a process of reducing at least one of the multiple pieces of video data obtained by the multiple cameras 110. More specifically, the reduction process reduces the number of pixels (i.e., the size) of the images that make up the video data. The reduction process can also be said to reduce the resolution of the video data.

[0035] However, when the reduction process is applied, the image quality of the video data is reduced. This is undesirable from the viewpoint of utilization of the video data on the receiving side. Therefore, in this embodiment, a "super-resolution technology" is used to improve the image quality of the video data in the receiving external device 200. The super-resolution technology can convert an input low-resolution image into a high-resolution image. Various methods of the super-resolution technology have been proposed (for example, see Non-Patent Document 1). In this embodiment, the method of the super-resolution technology is not particularly limited.

[0036] FIG. 7 is a conceptual diagram for explaining an example of reduction processing and super-resolution processing. The resolution of the original image is 1080p. By reduction processing, the resolution is reduced to 360p. When the resolution is reduced from 1080p to 360p, the amount of data is reduced to about 1 / 9. By super-resolution processing, the resolution is restored to 1080p.

[0037] 8 is a conceptual diagram showing the reduction processing unit 102 and the super-resolution processing unit 202 according to this embodiment. The reduction processing unit 102 is included in the moving object 100 and executes the reduction processing. The reduction processing reduces the amount of data of the transmission video data VT.

[0038] The super-resolution processor 202 is included in the external device 200 and executes super-resolution processing. More specifically, the super-resolution processor 202 determines whether or not reduction processing has been applied to the video data received from the moving object 100, based on the video data. The reception bit rate of the video data to which reduction processing has been applied is significantly reduced. The reduction width is significantly larger than the fluctuation width of the bit rate that is not related to the reduction processing. Therefore, the super-resolution processor 202 can determine whether or not reduction processing has been applied to the video data, based on the fluctuation of the reception bit rate. If reduction processing has been applied to the received video data, the super-resolution processor 202 applies a super-resolution technique to the video data. This improves the image quality of the video data to which reduction processing has been applied. Since the image quality of the video data is improved, it becomes easier to accurately grasp the situation around the moving object 100.

[0039] 2-3. Joining process The "combining process" is a process of combining two or more of the multiple video data obtained by the multiple cameras 110. For the sake of explanation, consider two pieces of video data, a first video data and a second video data, included in the multiple video data. The first video data is obtained by the main camera, and the second video data is obtained by the sub-camera. The combining process combines (combines) the first video data and the second video data so that the screen of the second video data is displayed within the screen of the first video data.

[0040] 9 and 10 are conceptual diagrams for explaining an example of the joining process, with Fig. 9 showing the video data group before the joining process, and Fig. 10 showing the video data group after the joining process.

[0041] For example, a front camera C1 and a rear camera C6 form a pair. The front camera C1 is a main camera, and the rear camera C6 is a sub-camera. As shown in Fig. 10, a combination process is performed so that a screen of the second image data obtained by the rear camera C6 is displayed on a screen of the first image data obtained by the front camera C1.

[0042] More specifically, a part of the image area of ​​the first video data obtained by the front camera C1 is deleted. Also, the image size of the second video data obtained by the rear camera C6 is adjusted (reduced) to fit the image area deleted from the first video data. Then, the first video data and the second video data are combined (combined) so that the image of the adjusted second video data is inserted into the image area deleted from the first video data. In this way, composite video data in which the first video data and the second video data are combined is generated.

[0043] Similarly, the left front camera C2 and the left side camera C4 form a pair. The left front camera C2 is the main camera, and the left side camera C4 is the sub-camera. In addition, the right front camera C3 and the right side camera C5 form a pair. The right front camera C3 is the main camera, and the right side camera C5 is the sub-camera.

[0044] 9 and 10, the number of video data (the number of streams) is reduced from six to three by the combining process. This contributes to reducing the amount of data. In addition, deleting a part of the image area of ​​the first video data and reducing the image size of the second video data also contribute to reducing the amount of data.

[0045] 11 is a conceptual diagram showing the merging processing unit 103 according to this embodiment. The merging processing unit 103 is included in the moving object 100 and executes the merging process.

[0046] The combining process can also be called “Picture-in-Picture processing.” However, while general Picture-in-Picture processing is performed on the receiving side, the combining process according to the present embodiment is performed on the transmitting side.

[0047] 2-4.Effects As described above, according to this embodiment, the data reduction process is executed in the mobile object 100. That is, before the multiple video data are transmitted from the mobile object 100 to the external device 200, the data amount of the multiple video data is reduced. Since the amount of transmitted data is reduced, communication delays and communication costs are suppressed. Furthermore, communication is stabilized.

[0048] 3. Combination of data reduction processes It is also possible to combine two or more of the above-mentioned selection, reduction, and combination processes. Various examples of combinations will be described below.

[0049] 3-1. First Example FIG. 12 is a block diagram illustrating a first example of the data reduction process.

[0050] The moving object 100 includes a plurality of cameras C1 to Cn (n is an integer equal to or greater than 2), a selection processor 101, a reduction processor 102, a joining processor 103, and an encoder 104. A plurality of video data V1 to Vn are acquired by each of the plurality of cameras C1 to Cn. The selection processor 101 executes a selection process on the plurality of video data V1 to Vn. The reduction processor 102 then executes a reduction process on the video data after the selection process. That is, the reduction processor 102 executes a reduction process on the video data that has not been omitted by the selection process and has been selected. The joining processor 103 then executes a joining process on the video data after the reduction process. As a result, the transmission video data VT with a reduced data amount is obtained. The encoder 104 then encodes the transmission video data VT. The moving object 100 transmits the transmission video data VT to the external device 200.

[0051] The external device 200 includes a decoder 201, a super-resolution processing unit 202, and a display device 210. The external device 200 receives transmission video data VT transmitted from the moving object 100. The decoder 201 decodes the received video data. The super-resolution processing unit 202 applies a super-resolution technique to the received video data to which reduction processing has been applied, thereby improving the image quality. Thereafter, the display device 210 displays the received video data.

[0052] FIG. 13 shows an example of the amount of data reduction. Nine types of video data V1 to V9 are obtained from nine cameras C1 to C9. The resolution of each video data is 1080p. Six types of video data V1 to V6 are selected by a selection process, and the amount of data is reduced to 2 / 3. Then, the video data V1 to V6 are reduced to reduced video data VR1 to VR6 by a reduction process. The resolution is reduced from 1080p to 360p, and the amount of data is reduced to about 1 / 9. Then, the six types of reduced video data VR1 to VR6 are combined to three types of combined video data VA, VB, and VC by a combination process, and the amount of data is reduced to 1 / 2. As a result, the amount of data is reduced by about 97% in total.

[0053] As described above, in the first example, the data reduction process includes the selection process, the reduction process, and the combination process. The selection process, the reduction process, and the combination process are executed in this order. This processing order provides the following effects.

[0054] The reduction process is performed after at least one image data is omitted by the selection process. Since the reduction process is not performed on unnecessary image data, the processing load of the reduction process is reduced.

[0055] Furthermore, the combining process is performed after at least one video data item has been omitted by the selection process. Since the combining process is not performed on unnecessary video data items, the processing load of the combining process is reduced.

[0056] Furthermore, the combining process is performed after the video data is reduced by the reduction process, i.e., after the image size is reduced by the reduction process, the combining process is performed. Therefore, the processing load of the image processing required for the combining process is reduced.

[0057] 3-2. Second Example FIG. 14 is a block diagram for explaining a second example of the data reduction process. Explanations that overlap with the first example will be omitted as appropriate. In the second example, the order of the reduction process and the combination process is reversed compared to the first example. That is, the selection process, the combination process, and the reduction process are performed in this order. The combination processing unit 103 performs the combination process on the video data after the selection process. That is, the combination processing unit 103 performs the combination process on the video data that has been selected without being omitted by the selection process. Next, the reduction processing unit 102 performs the reduction process on the video data after the combination process. Such a processing order provides the following effects.

[0058] The combining process is performed after at least one video data item is omitted by the selection process. Since the combining process is not performed on unnecessary video data items, the processing load of the combining process is reduced.

[0059] Furthermore, the reduction process is performed after at least one image data is omitted by the selection process. Since the reduction process is not performed on unnecessary image data, the processing load of the reduction process is reduced.

[0060] 3-3.Third Example 15 is a block diagram for explaining a third example of the data reduction process. Explanations that overlap with the first and second examples will be omitted as appropriate. In the third example, the data reduction process includes a selection process and a reduction process. The selection process and the reduction process are executed in this order. Such a processing order provides the following effects.

[0061] The reduction process is performed after at least one image data is omitted by the selection process. Since the reduction process is not performed on unnecessary image data, the processing load of the reduction process is reduced.

[0062] 3-4. Fourth Example 16 is a block diagram for explaining a fourth example of the data reduction process. Explanations that overlap with the first and second examples will be omitted as appropriate. In the fourth example, the data reduction process includes a selection process and a combination process. The selection process and the combination process are executed in this order. Such a processing order provides the following effects.

[0063] The combining process is performed after at least one video data item is omitted by the selection process. Since the combining process is not performed on unnecessary video data items, the processing load of the combining process is reduced.

[0064] 3-5. Fifth Example FIG. 17 is a block diagram for explaining a fifth example of the data reduction process. Explanations that overlap with the first and second examples will be omitted as appropriate. In the fifth example, the data reduction process includes a reduction process and a combination process. The reduction process and the combination process are executed in this order. This processing order provides the following effects.

[0065] The combining process is performed after the video data is reduced by the reduction process. In other words, the combining process is performed after the image size is reduced by the reduction process. Therefore, the processing load of the image processing required for the combining process is reduced.

[0066] 4. Example of moving object configuration 18 is a block diagram showing a configuration example of a moving body 100. The moving body 100 includes a camera 110 (multiple cameras C1 to Cn), a sensor group 120, a communication device 130, a traveling device 140, and a control device 150. In this example, the moving body 100 is a moving body equipped with wheels, such as a vehicle or a robot.

[0067] The camera 110 acquires video data showing the situation around the moving object 100 .

[0068] The sensor group 120 includes a status sensor that detects the status of the moving body 100. The status sensors include a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The sensor group 120 also includes a position sensor that detects the position and orientation of the moving body 100. An example of the position sensor is a Global Positioning System (GPS) sensor. Furthermore, the sensor group 120 may include a recognition sensor other than the camera 110. The recognition sensor recognizes (detects) the situation around the moving body 100. An example of the recognition sensor is a laser imaging detection and ranging (LIDAR), a radar, and the like.

[0069] The communication device 130 communicates with an external device outside the mobile object 100. For example, the communication device 130 communicates with an external device 200.

[0070] The traveling device 140 includes a steering device, a driving device, and a braking device. The steering device steers the wheels of the moving body 100. For example, the steering device includes an electric power steering (EPS) device. The driving device is a power source that generates a driving force. Examples of the driving device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0071] The control device 150 controls the moving body 100. The control device 150 includes one or more processors 151 (hereinafter simply referred to as processor 151) and one or more memories 152 (hereinafter simply referred to as memory 152). The processor 151 executes various processes. For example, the processor 151 includes a CPU (Central Processing Unit). The memory 152 stores various information. Examples of the memory 152 include a volatile memory, a non-volatile memory, a HDD (Hard Disk Drive), and an SSD (Solid State Drive). The processor 151 executes a control program 170, which is a computer program, to realize the functions of the control device 150. The control program 170 is stored in the memory 152. Alternatively, the control program 170 may be recorded in a computer-readable recording medium.

[0072] The processor 151 acquires moving body information 160 using the camera 110 and the group of sensors 120. The moving body information 160 includes video data obtained by the camera 110. The moving body information 160 also includes status information indicating the status of the moving body 100 detected by a status sensor. The moving body information 160 also includes position information indicating the position and orientation of the moving body 100 detected by a position sensor. The moving body information 160 also includes object information regarding an object recognized (detected) by a recognition sensor. The object information indicates the relative position and relative speed of the object with respect to the moving body 100.

[0073] The processor 151 also controls the traveling of the moving body 100. The traveling control includes steering control, acceleration control, and deceleration control. The processor 151 executes the traveling control by controlling the traveling device 140. The processor 151 may also execute automatic driving control. When executing the automatic driving control, the processor 151 generates a target trajectory of the moving body 100 based on the moving body information 160. The target trajectory includes a target position and a target speed. Then, the processor 151 executes the traveling control so that the moving body 100 follows the target trajectory.

[0074] Furthermore, the processor 151 communicates with the external device 200 via the communication device 130. For example, the processor 151 transmits at least a portion of the mobile unit information 160 to the external device 200 as necessary.

[0075] In particular, the processor 151 transmits the video data obtained by the camera 110 to the external device 200. At this time, the processor 151 executes the data reduction process described above. Then, the processor 151 transmits the transmission video data VT obtained by the data reduction process to the external device 200.

[0076] When remote assistance of the moving body 100 is performed, the processor 151 receives an operator instruction from the remote operator terminal 200A. Then, the processor 151 executes travel control according to the operator instruction.

[0077] 5. Example of remote operator terminal configuration 19 is a block diagram showing an example of the configuration of a remote operator terminal 200A, which is an example of the external device 200. The remote operator terminal 200A includes a display device 210, an input device 220, a communication device 230, and an information processing device 250.

[0078] The display device 210 displays various information. Examples of the display device 210 include a liquid crystal display, an organic EL display, a head mounted display, and a touch panel.

[0079] The input device 220 is an interface for receiving input from a remote operator. Examples of the input device 220 include a touch panel, a keyboard, a mouse, etc. In addition, when the remote assistance is remote driving, the input device 220 includes a driving operation member for the remote operator to perform driving operations (steering, acceleration, and deceleration).

[0080] The communication device 230 communicates with the outside world. For example, the communication device 230 communicates with the mobile object 100.

[0081] The information processing device 250 performs various information processing. The information processing device 250 includes one or more processors 251 (hereinafter simply referred to as the processor 251) and one or more memories 252 (hereinafter simply referred to as the memory 252). The processor 251 executes various processes. For example, the processor 251 includes a CPU. The memory 252 stores various information. Examples of the memory 252 include a volatile memory, a non-volatile memory, a HDD, and an SSD. The processor 251 executes a control program 270, which is a computer program, to realize the functions of the information processing device 250. The control program 270 is stored in the memory 252. The control program 270 may be recorded in a computer-readable recording medium.

[0082] The processor 251 executes a remote support process for remotely supporting the operation of the moving body 100. The remote support process includes an "information provision process" and an "operator instruction notification process".

[0083] The information provision process is as follows. The processor 251 receives the mobile body information 260 required for remote support from the mobile body 100 via the communication device 230. The mobile body information 260 includes at least a part of the mobile body information 160. In particular, the mobile body information 260 includes the transmission video data VT transmitted from the mobile body 100. The processor 251 performs the above-mentioned super-resolution processing as necessary to improve the image quality of the transmission video data VT. Then, the processor 251 presents the mobile body information 260 to the remote operator by displaying the mobile body information 260 on the display device 210.

[0084] The remote operator views the moving body information 260 displayed on the display device 210 to grasp the situation around the moving body 100 and the state of the moving body 100. The remote operator remotely supports the operation of the moving body 100. Examples of remote support provided by the remote operator include recognition support, decision support, remote driving, and the like. The remote operator inputs operator instructions using the input device 220.

[0085] The operator instruction notification process is as follows: The processor 251 receives an operator instruction input by a remote operator from the input device 220. The processor 251 then transmits the operator instruction to the mobile unit 100 via the communication device 230. [Explanation of symbols]

[0086] 1. Video transmission system 1A Remote Support System 10. Communications Networks 100 Mobile 101 Selection processing unit 102 Reduction processing section 103 Joint processing section 110 Camera 120 Sensors 130 Communication Equipment 140 Running gear 150 Control device 151 processor 152 Memory 160 Mobile Information 170 Control Program 200 External device 200A Remote Operator Terminal 202 Super-resolution processing section 210 Display device C Camera VT Transmission video data

Claims

1. A video transmission method for transmitting video data from a mobile object to an external device, comprising the steps of: A process of acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on the moving object; a data reduction process for acquiring transmission video data by reducing a data amount of the plurality of video data; a process of transmitting the transmission video data from the mobile object to the external device after the data reduction process; Including, The data reduction process includes: a selection process for omitting at least one of the plurality of pieces of video data from the transmission video data in accordance with a number of specific objects included in the plurality of pieces of video data; the selection process includes preferentially omitting from the transmission video data, among the plurality of video data, video data in which the specific object is less visible; The data reduction process further comprises: a reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Contains at least one of Video transmission method.

2. 2. A video transmission method according to claim 1, comprising: The specific object includes at least one of a person, a bicycle, a vehicle, a traffic light, and a sign. Video transmission method.

3. 2. A video transmission method according to claim 1, comprising: the data reduction process includes at least the selection process and the combination process; The selection process is performed before the combination process; The combining process is performed on the video data that is not omitted and is selected by the selection process. Video transmission method.

4. 4. A video transmission method according to claim 3, comprising: The data reduction process further includes the reduction process, The selection process is performed before the reduction process; The reduction process is performed on the video data that is not omitted and is selected by the selection process. Video transmission method.

5. 5. A video transmission method according to claim 4, comprising: The reduction process is performed before the combination process. Video transmission method.

6. 2. A video transmission method according to claim 1, comprising: the data reduction process includes at least the selection process and the reduction process; The selection process is performed before the reduction process; The reduction process is performed on the video data that is not omitted and is selected by the selection process. Video transmission method.

7. 7. A video transmission method according to claim 4, further comprising: Further, the method includes a process of applying a super-resolution technique to the video data to which the reduction process has been applied in the external device. Video transmission method.

8. 3. A video transmission method according to claim 1, further comprising: the mobile object is a target for remote assistance by a remote operator; The external device is a remote operator terminal at the remote operator's side. Video transmission method.

9. A video transmission system for transmitting video data from a mobile object to an external device, comprising: One or more processors; The one or more processors: acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on the moving object, performing a data reduction process to obtain transmission video data by reducing the data amount of the plurality of video data; After the data reduction process, the transmission video data is transmitted from the mobile object to the external device. It is configured as follows: The data reduction process includes: a selection process for omitting at least one of the plurality of pieces of video data from the transmission video data in accordance with a number of specific objects included in the plurality of pieces of video data; the selection process includes preferentially omitting from the transmission video data, among the plurality of video data, video data in which the specific object is less visible; The data reduction process further comprises: a reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Contains at least one of Video transmission system.

10. A control device for controlling a moving object, One or more processors; The one or more processors: acquiring a plurality of pieces of video data obtained by a plurality of cameras mounted on the moving object, performing a data reduction process to obtain transmission video data by reducing the data amount of the plurality of video data; After the data reduction process, the transmission video data is transmitted to an external device. It is configured as follows: The data reduction process includes: a selection process for omitting at least one of the plurality of pieces of video data from the transmission video data in accordance with a number of specific objects included in the plurality of pieces of video data; the selection process includes preferentially omitting from the transmission video data, among the plurality of video data, video data in which the specific object is less visible; The data reduction process further comprises: a reduction process for reducing at least one of the plurality of video data; a combining process for combining the first image data and the second image data so that a screen of the second image data is displayed within a screen of the first image data; Contains at least one of Control device.

Images