Image transmission method, image transmission system, and control device

Abstract

The present disclosure relates to an image transmission method, an image transmission system, and a control device. The amount of image data transmitted from a mobile body to an external device is reduced. The image transmission method includes a process of acquiring a plurality of image data obtained by a plurality of cameras mounted on the mobile body, a data reduction process of reducing the amount of the plurality of image data to obtain transmission image data, and a process of transmitting the transmission image data from the mobile body to the external device after the data reduction process. The data reduction process includes at least two of a selection process of omitting at least one of the plurality of image data from the transmission image data according to a scene in which the mobile body is located, a reduction process of reducing at least one of the plurality of image data, and a combination process of combining a first image data and a second image data in a manner that a screen of the second image data is displayed in a screen of the first image data.

Description

Technical Field

[0001] This disclosure relates to a technique for transmitting image data from a mobile body to an external device. Background Technology

[0002] Patent document 1 discloses a picture-in-picture display method in a two-way system such as video telephony or web chat via the Internet.

[0003] In addition, as a technology related to image transmission, patent documents 2 and 3 are known.

[0004] Non-Patent Document 1 discloses a "super-resolution technique" for converting an input low-resolution image into a high-resolution image. Specifically, Non-Patent Document 1 discloses SRCNN, which applies deep learning based on Convolutional Neural Networks (CNNs) to super-resolution (SR). A model for converting (mapping) an input low-resolution image into a high-resolution image is obtained through machine learning.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2014-150299

[0008] Patent Document 2: Japanese Patent Application Publication No. 2021-026554

[0009] Patent Document 3: Japanese Patent Application Publication No. 2014-071776

[0010] Non-patent literature

[0011] Non-patent literature 1: Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang, "Image Super-Resolution Using Deep Convolutional Networks", arXiv:1501.00092v3[cs.CV], July 31, 2015 (https: / / arxiv.org / pdf / 1501.00092.pdf) Summary of the Invention

[0012] The problem the invention aims to solve

[0013] Consider the scenario where image data obtained from a camera mounted on a mobile device is transmitted to an external device. For example, in remote assistance of a mobile device, image data obtained from a camera mounted on the mobile device is transmitted to a remote operator for use. However, the amount of image data is relatively large. As the number of cameras and the number of image data streams transmitted simultaneously increase, the amount of data transmitted increases. This increase in data transmission leads to increased communication latency and communication costs. From the perspective of utilizing image data, it is desirable to minimize communication latency and communication costs as much as possible.

[0014] One object of this disclosure is to provide a technique that can reduce the amount of image data transmitted from a mobile body to an external device.

[0015] Solution for solving the problem

[0016] The first viewpoint relates to image transmission methods that transmit image data from a mobile body to an external device.

[0017] Image transmission methods include the following processing:

[0018] Processing of multiple image data obtained from multiple cameras mounted on a moving body;

[0019] Data reduction processing of transmitted image data is achieved by reducing the data volume of multiple image data sets; and

[0020] After data reduction processing, the image data is sent from the mobile body to an external device.

[0021] Data reduction processing includes at least two of the following processes:

[0022] In the selection process, at least one image data from a plurality of image data is omitted from the transmitted image data, depending on the scene in which the moving body is located.

[0023] Downsizing processing, wherein at least one image data among a plurality of image data is downsized; and

[0024] The combination process combines the first image data and the second image data in such a way that the second image data is displayed in the frame of the first image data.

[0025] The second viewpoint relates to image transmission systems that transmit image data from a mobile body to an external device.

[0026] The image transmission system has one or more processors.

[0027] One or more processors constitute:

[0028] Acquire multiple image data obtained from multiple cameras mounted on a moving body.

[0029] Perform data reduction processing on the acquired and transmitted image data by reducing the data volume of multiple image data sets.

[0030] After data reduction processing, the image data will be sent from the mobile body to the external device.

[0031] Data reduction processing includes at least two of the following processes:

[0032] In the selection process, at least one image data from a plurality of image data is omitted from the transmitted image data, depending on the scene in which the moving body is located.

[0033] Downsizing processing, wherein at least one image data among a plurality of image data is downsized; and

[0034] The combination process combines the first image data and the second image data in such a way that the second image data is displayed in the frame of the first image data.

[0035] The third point is related to the control device for controlling the moving body.

[0036] The control device has one or more processors.

[0037] One or more processors constitute:

[0038] Acquire multiple image data obtained from multiple cameras mounted on a moving body.

[0039] Perform data reduction processing to obtain the transmitted image data by reducing the data volume of multiple image data sets.

[0040] After data reduction processing, the image data will be sent to an external device.

[0041] Data reduction processing includes at least two of the following processes:

[0042] In the selection process, at least one image data from a plurality of image data is omitted from the transmitted image data, depending on the scene in which the moving body is located.

[0043] Downsizing processing, wherein at least one image data among a plurality of image data is downsized; and

[0044] The combination process combines the first image data and the second image data in such a way that the second image data is displayed in the frame of the first image data.

[0045] The effects of the invention

[0046] According to this disclosure, it is possible to reduce the amount of image data transmitted from a mobile body to an external device. Attached Figure Description

[0047] Figure 1 This is a block diagram illustrating an overview of an image transmission system according to an embodiment of the present disclosure.

[0048] Figure 2 This is a schematic diagram illustrating an application example of the image transmission system according to an embodiment of the present disclosure, namely a remote assistance system.

[0049] Figure 3 This is a schematic diagram illustrating an example of a plurality of cameras mounted on a mobile body according to an embodiment of the present disclosure.

[0050] Figure 4 This is a schematic diagram illustrating an example of the selection process in an embodiment of this disclosure.

[0051] Figure 5 This is a schematic diagram illustrating another example of the selection process for implementing the embodiments of this disclosure.

[0052] Figure 6 This is a schematic diagram illustrating the selection processing unit of an embodiment of the present disclosure.

[0053] Figure 7 This is a schematic diagram illustrating an example of the scaling and super-resolution processing of embodiments of the present disclosure.

[0054] Figure 8 This is a schematic diagram illustrating the downscaling processing unit and the super-resolution processing unit according to embodiments of the present disclosure.

[0055] Figure 9 This is a schematic diagram illustrating an example of the combined processing used to illustrate embodiments of the present disclosure.

[0056] Figure 10 This is a schematic diagram illustrating an example of the combined processing used to illustrate embodiments of the present disclosure.

[0057] Figure 11 This is a schematic diagram illustrating the bonding processing unit of an embodiment of the present disclosure.

[0058] Figure 12 This is a block diagram illustrating a first example of data reduction processing in an embodiment of the present disclosure.

[0059] Figure 13 This is a block diagram illustrating a first example of data reduction processing in an embodiment of the present disclosure.

[0060] Figure 14 This is a block diagram illustrating a second example of data reduction processing in an embodiment of the present disclosure.

[0061] Figure 15 This is a block diagram illustrating a third example of data reduction processing in an embodiment of the present disclosure.

[0062] Figure 16 This is a block diagram illustrating a fourth example of data reduction processing in an embodiment of this disclosure.

[0063] Figure 17 This is a block diagram illustrating the fifth example of data reduction processing in an embodiment of the present disclosure.

[0064] Figure 18 This is a block diagram illustrating a structural example of a movable body according to an embodiment of the present disclosure.

[0065] Figure 19 This is a block diagram illustrating a structural example of a remote operator terminal according to an embodiment of the present disclosure.

[0066] Explanation of reference numerals in the attached figures

[0067] 1. Image transmission system; 1A. Remote assistance system; 10. Communication network; 100. Moving body; 101. Selection processing unit; 102. Reduction processing unit; 103. Combination processing unit; 110. Camera; 120. Sensor group; 130. Communication device; 140. Driving device; 150. Control device; 151. Processor; 152. Memory; 160. Moving body information; 170. Control program; 200. External device; 200A. Remote operator terminal; 202. Super-resolution processing unit; 210. Display device; C. Camera; VT. Transmitting image data. Detailed Implementation

[0068] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0069] 1. Overview of Image Transmission Systems

[0070] Figure 1 This is a schematic diagram showing the general outline of the image transmission system 1 according to this embodiment. The image transmission system 1 includes a mobile body 100 and an external device 200 located outside the mobile body 100. The mobile body 100 and the external device 200 are interconnected via a communication network 10. The mobile body 100 and the external device 200 are capable of communicating with each other via the communication network 10. Typically, the mobile body 100 and the external device 200 communicate wirelessly.

[0071] Examples of mobile objects 100 include vehicles, robots, and flying objects. Vehicles can be either autonomous vehicles or vehicles driven by a human. Examples of robots include logistics robots and operational robots. Examples of flying objects include airplanes and drones.

[0072] External device 200 may be, for example, a management server that manages mobile body 100. As another example, external device 200 may also be a remote operator terminal for remote assistance of mobile body 100. As yet another example, external device 200 may also be a mobile body different from mobile body 100.

[0073] A camera 110 is mounted on the mobile body 100. The camera 110 acquires image data (stream data) representing the surrounding situation of the mobile body 100. The mobile body 100 transmits the image data acquired by the camera 110 to an external device 200. The external device 200 receives the image data transmitted from the mobile body 100 and utilizes the received image data.

[0074] Figure 2 This describes a remote assistance system 1A as an application example of an image transmission system 1. A mobile body 100 is the object of remote assistance by a remote operator. An external device 200 is a remote operator terminal 200A operated by the remote operator. The mobile body 100 transmits image data obtained from a camera 110 to the remote operator terminal 200A. The remote operator terminal 200A receives the image data transmitted from the mobile body 100 and displays the received image data on a display device 210. The remote operator observes the image data displayed on the display device 210, understands the surrounding situation of the mobile body 100, and remotely assists the actions of the mobile body 100. Examples of remote assistance for a remote operator include recognition assistance, judgment assistance, and remote driving. Instructions issued by the remote operator are sent from the remote operator terminal 200A to the mobile body 100. The mobile body 100 performs actions according to the instructions of the remote operator.

[0075] There are also cases where multiple cameras 110 are mounted on the mobile body 100. For example, in the aforementioned remote assistance, it is useful to use multiple cameras 110 to monitor the surroundings of the mobile body 100.

[0076] Figure 3 This is a schematic diagram illustrating an example of multiple cameras 110 mounted on a moving body 100. Figure 3 In the example shown, the moving body 100 includes a front camera C1, a left front camera C2, a right front camera C3, a left front camera C4, a right front camera C5, and a rear camera C6. The front camera C1 acquires image data from the front of the moving body 100. The left front camera C2 acquires image data from the left front of the moving body 100. The right front camera C3 acquires image data from the right front of the moving body 100. The left camera C4 acquires image data from the left and left rear of the moving body 100. The right camera C5 acquires image data from the right and right rear of the moving body 100. The rear camera C6 acquires image data from the rear of the moving body 100.

[0077] Thus, when multiple cameras 110 are mounted on the mobile body 100, multiple image data are obtained from each of the multiple cameras 110. The mobile body 100 then transmits these multiple image data (stream data) to the external device 200 simultaneously and in parallel.

[0078] However, the amount of video data is relatively large. As the number of cameras increases and the number of video streams transmitted simultaneously increases, the amount of data transmitted also increases. This increase in data transmission leads to increased communication latency and costs. From the perspective of utilizing video data, it is desirable to minimize communication latency and costs as much as possible. For example, in the case of remote assistance with mobile unit 100, communication latency causes delays in remote operator judgment and unsmooth remote operation (such as serpentine driving, depending on the situation). Therefore, it is desirable to reduce the amount of data transmitted from mobile unit 100 as much as possible without compromising the accuracy of remote assistance.

[0079] Therefore, this embodiment proposes a technique that can appropriately reduce the amount of image data transmitted from the mobile body 100 to the external device 200.

[0080] 2. Data reduction processing

[0081] Mobile body 100 acquires multiple image data obtained by multiple cameras 110 respectively. Before sending these multiple image data to external device 200, mobile body 100 performs a "data reduction process" to reduce the data volume of the multiple image data. Hereinafter, at least one image data obtained as a result of the data reduction process will be referred to as "transmitted image data VT". After the data reduction process, mobile body 100 sends the transmitted image data VT to external device 200.

[0082] The following examples illustrate various data reduction processes.

[0083] 2-1. Select Process

[0084] "Selection processing" is the process of omitting at least one of the multiple image data obtained from multiple cameras 110 from the transmitted image data VT. In other words, selection processing selects only the image data with higher priority from the multiple image data, excluding the image data with lower priority. At this time, the priority is dynamically determined based on the scene in which the moving body 100 is located. That is, in selection processing, at least one of the multiple image data is omitted from the transmitted image data VT based on the scene in which the moving body 100 is located.

[0085] Figure 4 This is a schematic diagram illustrating an example of selection processing. Figure 4In the example shown, the moving body 100 is scheduled to turn left or is currently making a left turn. In this case, the image data from the left front, left, and left rear have higher priority. On the other hand, the image data from the right front, right, and right rear have lower priority. Therefore, it is considered to omit the image data from the right front, right, and right rear from the transmitted image data VT. The same applies when a bus moves to the left to stop at a bus stop.

[0086] Figure 5 This is a schematic diagram illustrating another example of selection processing. In Figure 5 In the example shown, the moving body 100 is scheduled to move backward or is in the process of moving backward. In this case, the image data from the rear has the highest priority. On the other hand, the image data from the front, left front, and right front has a lower priority. Therefore, it is considered to omit the image data from the front, left front, and right front from the transmitted image data VT.

[0087] Figure 6 This is a schematic diagram showing the selection processing unit 101 of this embodiment. The selection processing unit 101 is included in the moving body 100 and performs selection processing.

[0088] For example, the selection processing unit 101 acquires information reflecting the "predetermined direction of movement" of the moving body 100. For example, if the moving body 100 is a vehicle, the predetermined direction of movement is determined based on steering wheel direction, steering wheel angle, turn signal information, gear position, wheel speed, etc. As another example, the predetermined direction of movement can also be determined based on the current position of the moving body 100 and the target movement path. The selection processing unit 101 dynamically sets the priority of multiple image data based on the predetermined direction of movement of the moving body 100. Specifically, the selection processing unit 101 sets the priority of image data in directions closer to the predetermined direction of movement to be higher than the priority of image data in directions farther from the predetermined direction of movement. Then, the selection processing unit 101 selects the image data with higher priority and omits the image data with lower priority from the transmitted image data VT.

[0089] As another example, "specific objects" reflected in the image data can also be considered. Specific objects are those that are highly likely to be observed by a remote operator. For example, specific objects include at least one of people, bicycles, other vehicles, traffic lights, and signs. The selection processing unit 101 analyzes the images constituting the image data using known methods, thereby identifying specific objects within the image data. Then, the selection processing unit 101 sets the priority of image data reflecting more specific objects to be higher than that of image data reflecting fewer specific objects. Then, the selection processing unit 101 selects the higher-priority image data and omits lower-priority image data from the transmitted image data VT.

[0090] By using the selection and processing described above, the amount of image data (VT) sent can be reduced.

[0091] 2-2. Reduction Processing

[0092] "Reduction processing" is a process of reducing the size of at least one of the multiple image data obtained by multiple cameras 110. More specifically, reduction processing reduces the number of pixels (i.e., size) of the image that constitutes the image data. Reduction processing can also be described as reducing the resolution of the image data.

[0093] However, if downscaling is applied, the image quality of the image data is reduced. This is undesirable from the viewpoint of utilizing the image data on the receiving side. Therefore, in this embodiment, in order to improve the image quality of the image data in the external device 200 on the receiving side, a "super-resolution technique" is used. Super-resolution technique can convert an input low-resolution image into a high-resolution image. Various methods have been proposed as super-resolution techniques (for example, see Non-Patent Document 1). In this embodiment, the method of super-resolution technique is not particularly limited.

[0094] Figure 7 This is a schematic diagram illustrating an example of downscaling and super-resolution processing. The original image has a resolution of 1080p. Downscaling reduces the resolution to 360p. When the resolution is reduced from 1080p to 360p, the data size is approximately 1 / 9. Super-resolution processing restores the resolution to 1080p.

[0095] Figure 8 This is a schematic diagram showing the downsizing processing unit 102 and the super-resolution processing unit 202 of this embodiment. The downsizing processing unit 102 is included in the moving body 100 and performs downsizing processing. Through downsizing processing, the amount of data of the transmitted image data VT is reduced.

[0096] The super-resolution processing unit 202, included in the external device 200, performs super-resolution processing. More specifically, the super-resolution processing unit 202 determines whether downscaling has been applied to image data received from the moving body 100. The received bitrate of image data that has undergone downscaling is significantly reduced. This reduction is significantly larger than the change in bitrate unrelated to downscaling. Therefore, the super-resolution processing unit 202 can determine whether downscaling has been applied to the image data based on the change in received bitrate. When downscaling has been applied to the received image data, the super-resolution processing unit 202 applies super-resolution technology to the image data. As a result, the image quality of the downscaled image data is improved. Because the image quality of the image data is improved, it is easier to accurately grasp the surrounding situation of the moving body 100.

[0097] 2-3. Combined processing

[0098] "Combining processing" is the process of combining two or more image data from multiple image data obtained by multiple cameras 110. For illustration, consider two image data, namely, the first image data and the second image data. The first image data is obtained by the main camera, and the second image data is obtained by the sub-camera. The combining processing combines (composites) the first image data and the second image data by displaying the second image data within the frame of the first image data.

[0099] Figure 9 and Figure 10 This is a schematic diagram used to illustrate an example of combined processing. Figure 9 This indicates the combination of image data sets before processing. Figure 10 This indicates a group of combined image data.

[0100] For example, a front camera C1 and a rear camera C6 form a pair. The front camera C1 is the main camera, and the rear camera C6 is the secondary camera. Figure 10 As shown, the image data is combined by displaying the image data of the second image data obtained by the rear camera C6 in the image data of the first image data obtained by the front camera C1.

[0101] More specifically, a portion of the image area of ​​the first image data obtained from the front camera C1 is deleted. Additionally, the image size of the second image data obtained from the rear camera C6 is adjusted (reduced) to match the image area deleted from the first image data. Then, the first and second image data are combined (synthesized) by inserting the adjusted image of the second image data into the image area deleted from the first image data. This generates composite image data that combines the first and second image data.

[0102] Similarly, the front left camera C2 and the left side camera C4 form a pair. The front left camera C2 is the main camera, and the left side camera C4 is the sub-camera. Additionally, the front right camera C3 and the right side camera C5 form a pair. The front right camera C3 is the main camera, and the right side camera C5 is the sub-camera.

[0103] exist Figure 9 and Figure 10 In the example shown, by combining the processing, the number of image data streams was reduced from 6 to 3. This helps reduce the amount of data. In addition, removing a portion of the image area of ​​the first image data and reducing the image size of the second image data also contribute to the reduction in data volume.

[0104] Figure 11 This is a schematic diagram showing the bonding processing unit 103 of this embodiment. The bonding processing unit 103 is included in the moving body 100 and performs bonding processing.

[0105] It should be noted that the combination process can also be called "picture-in-picture" processing. However, typical picture-in-picture processing is performed on the receiving side, while the combination process in this embodiment is performed on the transmitting side.

[0106] 2-4. Effects

[0107] As explained above, according to this embodiment, data reduction processing is performed in the mobile unit 100. That is, before multiple image data are transmitted from the mobile unit 100 to the external device 200, the amount of data in these multiple image data is reduced. Because the amount of transmitted data is reduced, communication latency and communication costs are suppressed. Furthermore, communication is stabilized.

[0108] 3. Combination of data reduction processing

[0109] It can also combine two or more of the above selection, reduction, and combination processes. Examples of various combinations are explained below.

[0110] 3-1.Example 1

[0111] Figure 12 This is a block diagram used to illustrate the first example of data reduction processing.

[0112] The mobile unit 100 includes multiple cameras C1 to Cn (n being an integer of 2 or more), a selection processing unit 101, a reduction processing unit 102, a combining processing unit 103, and an encoder 104. Multiple image data V1 to Vn are acquired by the multiple cameras C1 to Cn respectively. The selection processing unit 101 performs selection processing on the multiple image data V1 to Vn. Next, the reduction processing unit 102 performs reduction processing on the selected image data. That is, the reduction processing unit 102 performs reduction processing on the image data that was not omitted in the selection process. Next, the combining processing unit 103 performs combining processing on the reduced image data. As a result, transmitted image data VT with reduced data volume is obtained. Then, the encoder 104 encodes the transmitted image data VT. The mobile unit 100 transmits the transmitted image data VT to the external device 200.

[0113] External device 200 includes a decoder 201, a super-resolution processing unit 202, and a display device 210. External device 200 receives transmitted image data VT sent from the mobile body 100. Decoder 201 decodes the received image data. Super-resolution processing unit 202 applies super-resolution technology to the received image data that has undergone downscaling to improve image quality. Then, display device 210 displays the received image data.

[0114] Figure 13This illustrates an example of data reduction. Nine types of image data, V1 to V9, are obtained from nine cameras C1 to C9. Each image data has a resolution of 1080p. Through selection processing, six types of image data, V1 to V6, are selected, reducing the data size by two-thirds. Next, through downscaling, image data V1 to V6 are reduced to reduced image data VR1 to VR6. The resolution is reduced from 1080p to 360p, reducing the data size by approximately one-ninth. Finally, through combining processing, the six reduced image data VR1 to VR6 are combined into three composite image data sets VA, VB, and VC, reducing the data size by half. As a result, the total data size is reduced by approximately 97%.

[0115] As explained above, in Example 1, the data reduction process includes selection, shrinking, and combining. Furthermore, the selection, shrinking, and combining processes are performed in this order. Based on this processing order, the following effect is obtained.

[0116] After at least one image data point is omitted through selection processing, downsizing is performed. Since unnecessary image data is not downsized, the processing load of downsizing is reduced.

[0117] Furthermore, after at least one image data point is omitted through selection processing, the merging process is performed. Since unnecessary image data is not merged, the processing load of the merging process is reduced.

[0118] Furthermore, after the image data is reduced in size through the downsizing process, the combining process is performed. In other words, the combining process is performed after the image size has been reduced through the downsizing process. Therefore, the processing load of the required image processing is reduced in the combining process.

[0119] 3-2.Example 2

[0120] Figure 14 This is a block diagram illustrating the second example of data reduction processing. Explanations repeated in the first example are omitted as appropriate. In this second example, the order of the reduction and combination processing is reversed compared to the first example. That is, selection processing, combination processing, and reduction processing are performed sequentially. The combination processing unit 103 performs combination processing on the image data after selection processing. In other words, the combination processing unit 103 performs combination processing on the image data that was not omitted in the selection processing. Next, the reduction processing unit 102 performs reduction processing on the image data after combination processing. Based on this processing order, the following effect is obtained.

[0121] After at least one image data point is omitted through selection processing, the merging process is performed. Since unnecessary image data is not merged, the processing load of the merging process is reduced.

[0122] Furthermore, after at least one image data point is omitted through selection processing, downsizing is performed. Since unnecessary image data is not downsized, the processing load of downsizing is reduced.

[0123] 3-3.Example 3

[0124] Figure 15 This is a block diagram illustrating the third example of data reduction processing. Explanations repeated in examples 1 and 2 are omitted where appropriate. In this third example, the data reduction processing includes selection and reduction processing. The selection and reduction processes are performed sequentially. Based on this processing order, the following effect is obtained.

[0125] After at least one image data point is omitted through selection processing, downsizing is performed. Since unnecessary image data is not downsized, the processing load of downsizing is reduced.

[0126] 3-4.Example 4

[0127] Figure 16 This is a block diagram illustrating the fourth example of data reduction processing. Explanations repeated in examples 1 and 2 are omitted where appropriate. In example 4, the data reduction processing includes selection and combination processing. The selection and combination processing are performed sequentially. Based on this processing order, the following effect is obtained.

[0128] After at least one image data point is omitted through selection processing, the merging process is performed. Since unnecessary image data is not merged, the processing load of the merging process is reduced.

[0129] 3-5.Example 5

[0130] Figure 17 This is a block diagram illustrating the fifth example of data reduction processing. Explanations repeated in examples 1 and 2 are omitted where appropriate. In example 5, the data reduction processing includes shrinking and combining processes. The shrinking and combining processes are performed sequentially. Based on this processing order, the following effect is obtained.

[0131] After the image data is reduced in size through downscaling, the combining process is performed. In other words, the image size is reduced through downscaling before the combining process is performed. Therefore, the processing load of the image processing is reduced in the combining process.

[0132] 4. Structural Examples of Moving Bodies

[0133] Figure 18This is a block diagram illustrating a structural example of a mobile body 100. The mobile body 100 includes a camera 110 (multiple cameras C1 to Cn), a sensor group 120, a communication device 130, a driving device 140, and a control device 150. In this example, the mobile body 100 is a vehicle, robot, or other mobile body equipped with wheels.

[0134] Camera 110 acquires image data representing the surrounding conditions of the moving object 100.

[0135] Sensor group 120 includes state sensors that detect the state of the moving body 100. State sensors include velocity sensors, acceleration sensors, yaw rate sensors, and steering angle sensors. Additionally, sensor group 120 includes position sensors that detect the position and orientation of the moving body 100. Examples of position sensors include GPS (Global Positioning System) sensors. Furthermore, sensor group 120 may also include identification sensors other than camera 110. Identification sensors identify (detect) the surroundings of the moving body 100. Examples of identification sensors include LIDAR (Laser Imaging Detection and Ranging), radar, etc.

[0136] The communication device 130 communicates with the outside of the mobile body 100. For example, the communication device 130 communicates with an external device 200.

[0137] The driving device 140 includes a steering mechanism, a drive mechanism, and a braking mechanism. The steering mechanism steers the wheels of the moving body 100. For example, the steering mechanism includes a power steering (EPS) device. The drive mechanism is a power source that generates driving force. Examples of drive mechanisms include an engine, an electric motor, and a hub motor. The braking mechanism generates braking force.

[0138] Control device 150 controls moving body 100. Control device 150 includes one or more processors 151 (hereinafter simply referred to as processor 151) and one or more memories 152 (hereinafter simply referred to as memory 152). Processor 151 performs various processes. For example, processor 151 includes a CPU (Central Processing Unit). Memory 152 stores various information. Examples of memory 152 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The control device 150 performs its functions by executing a control program 170, which is a computer program, through processor 151. Control program 170 is stored in memory 152. Alternatively, control program 170 may also be recorded on a computer-readable recording medium.

[0139] Processor 151 uses camera 110 and sensor group 120 to acquire moving body information 160. Moving body information 160 includes image data obtained by camera 110. Additionally, moving body information 160 includes state information indicating the state of moving body 100 detected by state sensors. Furthermore, moving body information 160 includes position information indicating the position and orientation of moving body 100 detected by position sensors. Furthermore, moving body information 160 includes object information related to an object identified (detected) by an identification sensor. The object information indicates the relative position and relative velocity of the object relative to moving body 100.

[0140] Additionally, processor 151 controls the movement of mobile body 100. Movement control includes steering control, acceleration control, and deceleration control. Processor 151 executes movement control by controlling driving device 140. Processor 151 can also perform automatic driving control. In the case of automatic driving control, processor 151 generates a target trajectory for mobile body 100 based on mobile body information 160. The target trajectory includes target position and target speed. Then, processor 151 executes movement control in a manner that causes mobile body 100 to follow the target trajectory.

[0141] Furthermore, the processor 151 communicates with the external device 200 via the communication device 130. For example, the processor 151 may send at least a portion of the moving body information 160 to the external device 200 as needed.

[0142] Specifically, the processor 151 sends the image data obtained by the camera 110 to the external device 200. At this time, the processor 151 performs the aforementioned data reduction processing. Then, the processor 151 sends the transmitted image data VT obtained through the data reduction processing to the external device 200.

[0143] In the case of remote assistance of the mobile body 100, the processor 151 receives operator instructions from the remote operator terminal 200A. Then, the processor 151 executes driving control according to the operator instructions.

[0144] 5. Example of the structure of a remote operator terminal

[0145] Figure 19 This is a block diagram illustrating a structural example of a remote operator terminal 200A, which is an example of an external device 200. The remote operator terminal 200A includes a display device 210, an input device 220, a communication device 230, and an information processing device 250.

[0146] Display device 210 displays various information. Examples of display devices 210 include liquid crystal displays, organic EL displays, head-mounted displays, and touch panels.

[0147] Input device 220 is an interface for receiving input from a remote operator. Examples of input devices 220 include touch panels, keyboards, and mice. Furthermore, in the case of remote assistance for remote driving, input device 220 includes driving control components for the remote operator to perform driving operations (steering, acceleration, and deceleration).

[0148] The communication device 230 communicates with the outside world. For example, the communication device 230 communicates with the mobile body 100.

[0149] Information processing device 250 performs various information processing operations. Information processing device 250 includes one or more processors 251 (hereinafter simply referred to as processor 251) and one or more memories 252 (hereinafter simply referred to as memory 252). Processor 251 performs various processing operations. For example, processor 251 includes a CPU. Memory 252 stores various information. Examples of memory 252 include volatile memory, non-volatile memory, HDD, SSD, etc. The functions of information processing device 250 are realized by executing a control program 270, which is a computer program, through processor 251. Control program 270 is stored in memory 252. Control program 270 may also be recorded on a computer-readable recording medium.

[0150] The processor 251 performs remote assistance processing for the actions of the remotely assisted mobile body 100. The remote assistance processing includes "information provision processing" and "operator instruction notification processing".

[0151] The information provision process is as follows. Processor 251 receives mobile body information 260 required for remote assistance from mobile body 100 via communication device 230. Mobile body information 260 includes at least a portion of mobile body information 160. Specifically, mobile body information 260 includes transmitted image data VT sent from mobile body 100. Processor 251 performs the aforementioned super-resolution processing as needed to improve the image quality of the transmitted image data VT. Then, processor 251 displays the mobile body information 260 to the remote operator on display device 210.

[0152] The remote operator observes the moving body information 260 displayed on the display device 210 to understand the surrounding conditions and status of the moving body 100. The remote operator remotely assists the movement of the moving body 100. Examples of remote assistance provided by the remote operator include recognition assistance, judgment assistance, and remote driving. The remote operator inputs operator instructions using the input device 220.

[0153] The operator instruction notification process is as follows. Processor 251 receives operator instructions input by a remote operator from input device 220. Then, processor 251 transmits the operator instructions to mobile body 100 via communication device 230.

Claims

1. An image transmission method for transmitting image data from a moving body to an external device, wherein, The image transmission method includes the following processing: Processing of acquiring multiple image data obtained by multiple cameras mounted on the mobile body; Data reduction processing performed by the mobile body to obtain transmitted image data by reducing the amount of data in the plurality of image data; as well as The process of transmitting the image data from the moving body to the external device after the data reduction processing. The data reduction process includes: In the selection process, according to the predetermined movement direction of the moving body, another part of the image data is omitted from the transmitted image data, thereby selecting a portion of the image data from the plurality of image data. and at least one of the reduction processing and the combination processing, In this reduction process, at least one of the plurality of image data is reduced in size. In this combination process, the first image data and the second image data are combined in such a way that the image of the second image data is displayed in the image of the first image data.

2. The image transmission method according to claim 1, wherein, In the case where the data reduction process includes the selection process and the combination process... The selection process is performed before the combination process. The image data selected by the selection process is then subjected to the combination process.

3. The image transmission method according to claim 2, wherein, The data reduction process also includes the shrinking process. The selection process is performed before the shrinking process. The image data selected by the selection process is then subjected to the downsizing process.

4. The image transmission method according to claim 3, wherein, The shrinking process is performed before the combining process.

5. The image transmission method according to claim 1, wherein, In the case where the data reduction process includes the selection process and the shrinking process... The selection process is performed before the shrinking process. The image data selected by the selection process is then subjected to the downsizing process.

6. The image transmission method according to any one of claims 3 to 5, wherein, It also includes the following processing: applying super-resolution technology to the image data that has undergone the downsizing process in the external device.

7. The image transmission method according to claim 1, wherein, The mobile object is the target of remote assistance provided by the remote operator. The external device is the remote operator terminal on the remote operator side.

8. An image transmission system for transmitting image data from a moving body to an external device, wherein, The image transmission system has one or more processors. The one or more processors are configured as follows: Acquire multiple image data obtained by multiple cameras mounted on the mobile body. The mobile body performs data reduction processing to acquire and transmit image data by reducing the amount of data in the plurality of image data. After the data reduction process, the transmitted image data is sent from the mobile body to the external device. The data reduction process includes: In the selection process, according to the predetermined movement direction of the moving body, another part of the image data is omitted from the transmitted image data, thereby selecting a portion of the image data from the plurality of image data. And at least one of reduction processing and combination processing, wherein in the reduction processing, at least one of the plurality of image data is reduced, and in the combination processing, the first image data and the second image data are combined in such a way that the image of the second image data is displayed in the image of the first image data.

9. A control device for controlling a moving body, wherein, The control device has one or more processors. The one or more processors are configured as follows: Acquire multiple image data obtained by multiple cameras mounted on the mobile body. The mobile body performs data reduction processing to obtain the transmitted image data by reducing the amount of data in the plurality of image data. After the data reduction processing, the transmitted image data is sent to an external device. The data reduction process includes: In the selection process, according to the predetermined movement direction of the moving body, another part of the image data is omitted from the transmitted image data, thereby selecting a portion of the image data from the plurality of image data. And at least one of reduction processing and combination processing, wherein in the reduction processing, at least one of the plurality of image data is reduced, and in the combination processing, the first image data and the second image data are combined in such a way that the image of the second image data is displayed in the image of the first image data.

Images