Image transmission system and image transmission method

Abstract

The invention relates to an image transmission system and an image transmission method. A video transmission system transmits a video from a moving body, which is a subject to remote driving by a remote driver, to a terminal on the remote driver side. A video transmission system detects or predicts a positive operation in which the degree of driving operation by a remote driver exceeds a first threshold value. Furthermore, for a first period after the positive operation has been detected or predicted, the video transmission system improves the quality of the video transmitted to the terminal compared to a period other than the first period. As a result, when a remote driver performs remote driving on a moving body, the accuracy of remote driving can be ensured while suppressing communication costs.

Description

Technical Field

[0001] The present disclosure relates to a technology for transmitting an image (video) from a moving object that is a target of remote driving by a remote driver to a terminal on the remote driver's side. Background Art

[0002] Patent document 1 discloses a technology for remotely monitoring an autonomous vehicle. The autonomous vehicle sends an image captured by a camera to a monitoring and control server. The monitoring and control server receives the image sent from the autonomous vehicle. If a key frame of the image is not received within a predetermined time, the monitoring and control server sends an emergency control signal to the autonomous vehicle. The autonomous vehicle limits its speed or makes an emergency stop according to the emergency control signal.

[0003] Patent document 2 discloses a technology for remotely monitoring an autonomous vehicle. An autonomous driving control device mounted on an autonomous vehicle sends detection data detected by a detection unit such as a camera to a remote monitoring center. In addition, the autonomous driving control device calculates the current danger level of the autonomous driving vehicle. The higher the danger level of the autonomous driving vehicle, the more the autonomous driving control device increases the amount of detection data sent to the remote monitoring center.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-003403

[0007] Patent Document 2: International Publication No. 2018 / 155159 Summary of the invention

[0008] Problems to be solved by the invention

[0009] Consider remote driving of a mobile object by a remote driver. Images required for remote driving are transmitted from the mobile object to the remote driver terminal. From the perspective of the accuracy of the remote driver's remote driving, it is preferred to provide high-quality images to the remote driver. However, as the image quality improves, the amount of transmitted data increases, and the communication cost increases.

[0010] One object of the present disclosure is to provide a technology capable of ensuring the accuracy of remote driving while suppressing communication costs in remote driving of a mobile object by a remote driver.

[0011] Technical solutions to solve problems

[0012] The first aspect relates to an image transmission system that transmits an image from a mobile object that is the object of remote driving by a remote driver to a terminal on the remote driver side. The image transmission system includes one or more processors. The one or more processors perform an active operation determination process in which an active operation in which the remote driver performs a driving operation to a degree exceeding a first threshold is detected or predicted. Furthermore, the one or more processors perform an image quality increase process in which, for a first period after the active operation is detected or predicted, the quality of the image sent to the terminal is improved compared to periods other than the first period.

[0013] The second aspect relates to an image transmission method for transmitting an image from a mobile object that is a subject of remote driving by a remote driver to a terminal on the remote driver side. The image transmission method includes: a positive operation determination process for detecting or predicting a positive operation in which the remote driver performs a driving operation to a degree exceeding a first threshold; and an image quality improvement process for improving the quality of an image sent to the terminal during a first period after the positive operation is detected or predicted, compared to periods other than the first period.

[0014] Effects of the Invention

[0015] According to the present disclosure, the image quality of the image transmitted from the mobile body to the terminal on the remote driver side is not uniformly set, but dynamically changed. Specifically, the image quality increases in the first period after the active operation of the remote driver is detected or predicted. In the case of active operation, the remote driver observes the conditions around the mobile body particularly carefully. Since the image quality increases at this time, the accuracy of remote driving by the remote driver is improved.

[0016] The image quality during the period other than the first period is set lower than the image quality during the first period. During the period other than the first period, the remote pilot does not actively operate, so the accuracy of remote driving can be ensured even if the image quality is not increased to a higher level than required. As long as the image of normal quality is provided, the remote pilot can continue the remote driving of the mobile body with sufficiently high accuracy. In addition, since the image quality is set relatively low, the amount of transmission data can be suppressed, and the communication cost can be suppressed.

[0017] Thus, according to the present disclosure, it is possible to ensure the accuracy of remote driving while suppressing the communication cost. In other words, it is possible to achieve both ensuring the accuracy of remote driving and suppressing the communication cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram showing an example of the configuration of a remote driving system.

[0019] Figure 2This is a conceptual diagram showing the relationship between the remote pilot's active operation and image quality.

[0020] Figure 3 This is a conceptual diagram for explaining an example of key frame control.

[0021] Figure 4 This is a conceptual diagram for explaining an example of image quality control taking vehicle speed into consideration.

[0022] Figure 5 is a timing diagram showing an example of image quality control taking active operation into consideration.

[0023] Figure 6 This is a conceptual diagram for explaining an example of detection and release of active operation.

[0024] Figure 7 This is a block diagram showing an example of the functional structure of a video transmission system.

[0025] Figure 8 is a flow chart generally showing image quality control taking active operations into account.

[0026] Fig. 9 This is a block diagram showing an example of the configuration of a vehicle.

[0027] Fig.10 This is a block diagram showing a configuration example of a remote driver terminal.

[0028] Fig.11 This is a block diagram showing a first example of the active operation determination unit.

[0029] Fig.12 It is a block diagram showing a second example of the active operation determination unit.

[0030] Fig.13 It is a block diagram showing a third example of the active operation determination unit.

[0031] Fig.14 It is a block diagram showing a fourth example of the active operation determination unit.

[0032] Fig.15 It is a block diagram showing a fifth example of the active operation determination unit.

[0033] Description of symbols

[0034] 1 remote driving system; 1X image transmission system; 10 active operation determination unit; 20 image quality determination unit; 21 frame rate setting unit; 22 key frame rate setting unit; 23 key frame insertion unit; 100 vehicle; 200 remote driver terminal; 300 management device; OPE remote operation information; VCL vehicle information. DETAILED DESCRIPTION

[0035] Embodiments of the present disclosure will be described with reference to the drawings.

[0036] 1. Overview of Remote Driving System

[0037] Consider remote driving of a mobile object. Examples of mobile objects include vehicles and robots. Vehicles may be self-driving vehicles. Examples of robots include logistics robots and work robots. As an example, in the following description, consider a case where the mobile object to be remotely driven is a vehicle. In general, the expression "vehicle" in the following description is replaced by "mobile object".

[0038] Figure 1 It is a schematic diagram showing an example of the configuration of the remote driving system 1 involved in the present embodiment. The remote driving system 1 includes a vehicle 100, a remote driver terminal 200, and a management device 300. The vehicle 100 is the object of remote driving by the remote driver O (remote operator). The remote driver terminal 200 is a terminal device used by the remote driver O when remotely driving the vehicle 100. The remote driver terminal 200 can also be referred to as a remote cockpit. The management device 300 manages the remote driving system 1. Typically, the management device 300 is a management server in the cloud. The management server can also be composed of multiple servers that perform distributed processing.

[0039] The vehicle 100, the remote driver terminal 200, and the management device 300 can communicate with each other via the communication network. The vehicle 100 and the remote driver terminal 200 can communicate with each other via the management device 300. In addition, the vehicle 100 and the remote driver terminal 200 may communicate directly without going through the management device 300.

[0040] The vehicle 100 is equipped with various sensors including a camera C. The camera C captures the conditions around the vehicle 100. The camera C obtains an image VID showing the conditions around the vehicle 100. The vehicle information VCL is information obtained by various sensors, including the image VID captured by the camera C. The vehicle 100 transmits the vehicle information VCL to the remote driver terminal 200.

[0041] The remote driver terminal 200 receives the vehicle information VCL sent from the vehicle 100. The remote driver terminal 200 presents the vehicle information VCL to the remote driver O. Specifically, the remote driver terminal 200 includes a display device 220, on which an image VID and the like are displayed. The remote driver O views the displayed information, identifies the conditions around the vehicle 100, and performs remote driving of the vehicle 100. The remote operation information OPE is information related to the driving operation (steering operation, acceleration operation, deceleration operation) of the remote driver O. For example, the remote operation information OPE includes the driving operation amount (steering operation amount, acceleration operation amount, deceleration operation amount) input by the remote driver O. The remote operation information OPE can be said to be information reflecting the degree of the driving operation of the remote driver O. The remote driver terminal 200 sends the remote operation information OPE to the vehicle 100.

[0042] The vehicle 100 receives the remote operation information OPE transmitted from the remote driver terminal 200. The vehicle 100 performs vehicle driving control according to the received remote operation information OPE. In this way, remote driving of the vehicle 100 is realized.

[0043] 2. Image quality control considering active operations

[0044] As described above, the image VID required for remote driving is transmitted from the vehicle 100 to the remote driver terminal 200. The remote driver O views the image VID and recognizes the situation around the vehicle 100. From the perspective of the accuracy of remote driving of the remote driver O, it is preferable to provide the remote driver O with a high-quality image VID. However, as the image quality improves, the amount of transmission data increases, and the communication cost increases. Therefore, this embodiment proposes a technology that can ensure the accuracy of remote driving while suppressing the communication cost.

[0045] 2-1. Active operation

[0046] First, refer to Figure 2 , the "active operation" of the remote driver O is explained. Active operation refers to the driving operation actively performed by the remote driver O. The driving operation is a steering operation (steering wheel operation) to turn the vehicle 100, an acceleration operation (accelerator operation) to accelerate the vehicle 100, or a deceleration operation (brake operation) to decelerate the vehicle 100. For example, when turning the vehicle 100 at a curve or an intersection, the remote driver O actively performs a steering operation. As another example, when stopping the vehicle 100 at an intersection or in front of a traffic light, the remote driver O actively performs a deceleration operation. These are active operations. On the contrary, when the vehicle 100 is traveling at a constant speed on a straight road, the remote driver O does not particularly perform active operations.

[0047] In order to more clearly define the active operation of the remote driver O, the term "operation degree" is introduced. The operation degree refers to the degree of the driving operation of the remote driver O. For example, the operation degree is proportional (directly proportional) to the steering amount of the steering operation performed by the remote driver O. As another example, the operation degree is proportional to the steering speed of the steering operation performed by the remote driver O. As another example, the operation degree is proportional to the required acceleration required (requested) by the remote driver O. The required acceleration is proportional to the acceleration operation amount of the acceleration operation performed by the remote driver O. As another example, the operation degree is proportional to the required deceleration required by the remote driver O. The required deceleration is proportional to the deceleration operation amount of the deceleration operation performed by the remote driver O.

[0048] The active operation of the remote driver O is a driving operation in which the degree of operation exceeds the first threshold value TH1. Figure 2 In the example, the operation level P0 is below the first threshold TH1, and the operation level P1 exceeds the first threshold TH1. The driving operation at the operation level P1 is an active operation. On the other hand, the driving operation at the operation level P0 is not an active operation. For example, the operation level when the vehicle 100 is traveling at a constant speed on a straight road is likely to be P0.

[0049] Active operation can also be divided into multiple levels. As an example, consider a second threshold TH2 that is higher than the first threshold TH1, and a third threshold TH3 that is higher than the second threshold TH2. The operation level P1 exceeds the first threshold TH1 and is below the second threshold TH2. The driving operation of such an operation level P1 is a weak active operation. The operation level P2 exceeds the second threshold TH2 and is below the third threshold TH3. The driving operation of such an operation level P2 is a medium-level active operation. The operation level P3 exceeds the third threshold TH3. The driving operation of such an operation level P3 is a strong active operation.

[0050] In addition, the “habit” of remote driving may differ among remote drivers O. Even in the same scene, the operation level P may differ among remote drivers O. In consideration of the personality of the remote drivers O, each threshold value including the first threshold value TH1 may be set for each remote driver O.

[0051] The active operation of the remote driver O can be detected based on the remote operation information OPE described above. In addition, as described later, the active operation can also be "predicted" to be performed before the active operation is actually performed by the remote driver O. The specific method for detecting or predicting the active operation will be described in the following section 5.

[0052] 2-2. Image quality control

[0053] When the remote driver O performs remote driving of the vehicle 100, the remote driver O carefully observes the conditions around the vehicle 100. In particular, when performing active operation, the remote driver O should carefully observe the conditions around the vehicle 100. Therefore, when the remote driver O performs active operation, it is preferable to provide the remote driver O with a high-quality image VID. As a result, the accuracy of remote driving is improved.

[0054] Even when not actively operating, the remote driver O must of course carefully observe the situation around the vehicle 100. For example, even when the vehicle 100 is traveling at a constant speed on a straight road, the remote driver O must of course carefully observe the situation around the vehicle 100. However, even if the image quality is not increased to a higher level than necessary, as long as the image VID of normal quality is provided, the remote driver O can continue the remote driving of the vehicle 100 with sufficiently high accuracy. By setting the image quality to normal quality, the amount of transmission data can be reduced, and the communication cost can be reduced.

[0055] From the above viewpoints, according to the present embodiment, the quality of the video VID transmitted from the vehicle 100 to the remote driver terminal 200 is dynamically controlled in consideration of the active operation of the remote driver O. Hereinafter, such control of the quality of the video VID is referred to as "video quality control".

[0056] Refer again Figure 2 , the image quality control considering active operation is described. The default image quality Q0 is the normal image quality of the image VID. In the case of the operation degree P0 below the first threshold TH1, the image quality is set to the default image quality Q0. That is, the operation degree P0 below the first threshold TH1 is associated with the default image quality Q0. Moreover, the operation degree P1 exceeding the first threshold TH1 is associated with an image quality Q1 higher than the default image quality Q0.

[0057] According to the present embodiment, when no active operation of the remote driver O is detected or predicted, the image quality is set to the default image quality Q0. Furthermore, in conjunction with the detection or prediction of the active operation of the remote driver O, the image quality is increased to an image quality Q1 higher than the default image quality Q0. In other words, the image quality is increased to an image quality Q1 higher than the default image quality Q0, triggered by the detection or prediction of the active operation of the remote driver O.

[0058] In the case where the positive operation is divided into a plurality of grades (levels), the image quality is also divided into a plurality of grades (levels). For example, in the case where a weak positive operation corresponding to the operation degree P1 is detected or predicted, the image quality is set to an image quality Q1 that is higher than the default image quality Q0. In the case where a medium-level positive operation corresponding to the operation degree P2 is detected or predicted, the image quality is set to an image quality Q2 that is higher than the image quality Q1. In the case where a strong positive operation corresponding to the operation degree P3 is detected or predicted, the image quality is set to an image quality Q3 that is higher than the image quality Q2.

[0059] As processing for improving image quality, various examples are conceivable.

[0060] The first process for improving the image quality is to increase the frame rate of the image VID. The default frame rate is a frame rate corresponding to the default image quality Q0. When an active operation is detected or predicted, the frame rate of the image VID is set higher than the default frame rate.

[0061] The second process for improving the image quality is to increase the key frame rate of the image VID, in other words, to reduce the key frame interval of the image VID. As is well known, the image VID is composed of a key frame (I frame) and other P frames and B frames. The key frame retains all the information of the image, and the entire image can be reproduced based on the key frame alone. On the other hand, the P frame and the B frame only retain the information of the difference with the key frame, and the P frame and the B frame alone cannot reproduce the entire image. If the P frame and the B frame continue, the deviation from the actual image may increase. Therefore, sending a key frame that can reproduce the entire image by a single body helps to improve the image quality. The default key frame rate is a key frame rate equivalent to the default image quality Q0. In conjunction with the detection or prediction of an active operation, the key frame rate of the image VID is set to be higher than the default key frame rate.

[0062] The third process for improving the image quality is to temporarily insert a key frame into the image VID regardless of the key frame rate setting of the image VID. In response to the detection or prediction of the active operation, the key frame is immediately inserted into the image VID. For the same reason as in the case of the second process described above, the image quality can also be improved by this.

[0063] The fourth process for improving the image quality is to increase the resolution of the image VID. The default resolution is a resolution equivalent to the default image quality Q0. In conjunction with the detection or prediction of an active operation, the resolution of the image VID is set higher than the default resolution.

[0064] The multiple processing candidates for improving image quality include two or more of the above-mentioned first to fourth processings. At least one of these multiple processing candidates is selected and executed. It is also possible to select two or more of the multiple processing candidates and execute them together. In other words, it is also possible to combine two or more processings.

[0065] Figure 3 This is a conceptual diagram for explaining an example of key frame control, and shows a key frame structure of a video VID. The horizontal axis represents time.

[0066] Figure 3 (A) in the figure shows the case where the second process is performed. The default value of the key frame interval is "5". At time ta, an active operation is detected or predicted. If an active operation is detected or predicted, the key frame interval is changed to "3" which is smaller than the default value. However, the key frame interval actually becomes "3" after time tb after time ta. That is, there is some time difference from the detection or prediction of an active operation to the actual improvement of image quality.

[0067] on the other hand, Figure 3 (B) in FIG. 1 shows a combination of the second process and the third process. If an active operation is detected or predicted at time ta, a temporary key frame is inserted immediately. As a result, the image quality is immediately improved without delay. In this way, by combining the third process with the second process, the delay until the image quality is improved can be suppressed.

[0068] The first threshold TH1 as a criterion for determining the active operation may be set separately for each of the plurality of processing candidates. The first threshold TH1 may be different for each of the plurality of processing candidates. For example, the first threshold TH1 for the second processing may be set lower than the first threshold TH1 for the first processing. In this case, the second processing is executed with priority over the first processing.

[0069] It is also possible to determine the executed processing candidate from among the multiple processing candidates based on the "type" of the active operation. For example, when the active operation is an acceleration operation, the vehicle speed increases. When the vehicle speed increases, the image VID appears to flow faster. In particular, the flow of the image VID in the peripheral field of view becomes faster. In this case, in order to express the flow speed of the image VID, it is effective to increase the frame rate of the image VID. That is, it is also possible to execute at least the first processing when the active operation is an acceleration operation.

[0070] As another example, when the active operation is a steering operation, the remote driver O not only pays attention to the front, but also increases attention to the left and right directions. That is, the remote driver O pays attention to the image VID throughout the entire screen. In this case, it is more effective to increase the overall fineness of the image VID than to express the flow speed of the image VID. For this reason, it is effective to increase the key frame rate of the image VID. That is, it is also possible to perform at least the second process when the active operation is a steering operation.

[0071] Figure 4 1 is a conceptual diagram for explaining an example of image quality control taking the vehicle speed of the vehicle 100 into consideration. Figure 4 In the example shown, a first process of controlling the frame rate and a second process of controlling the key frame rate are executed. The horizontal axis represents the vehicle speed, and the vertical axis represents the frame rate or the key frame rate.

[0072] When driving at high speed, the image VID appears to flow faster. In particular, the image VID in the peripheral field of view flows faster. In this case, in order to express the flow speed of the image VID, it is effective to increase the frame rate of the image VID. Therefore, the frame rate of the image VID can also be increased as the vehicle speed increases. Figure 4 In the example shown, during the period from Va to Vb, the frame rate increases as the vehicle speed increases. Figure 4 The mapping shown here sets the frame rate in the image quality control.

[0073] On the other hand, when driving at a low speed, the remote driver O not only pays attention to the front, but also increases his attention to the left and right directions. That is, the remote driver O pays attention to the image VID throughout the entire screen. For example, when the vehicle 100 is driving at a low speed in a parking lot, the remote driver O pays attention to the image VID throughout the entire screen. In this case, it is more effective to increase the overall fineness of the image VID than to express the flow speed of the image VID. For this reason, it is effective to increase the key frame rate of the image VID. That is to say, the key frame rate of the image VID can also be increased as the vehicle speed decreases. Figure 4 In the example shown, during the period from Vc to Vd, the key frame rate increases as the vehicle speed decreases. Figure 4 The mapping shown here sets the key frame rate in image quality control.

[0074] 2-3. Time chart

[0075] Figure 5 This is a time chart showing an example of image quality control taking active operation into consideration. The horizontal axis represents time and the vertical axis represents image quality.

[0076] Figure 5(A) in FIG. 1 shows a case where an active operation of the remote driver O is detected. Before time t1, the image quality is the default image quality Q0. At time t1, an active operation is detected. In conjunction with the detection of the active operation, the image quality increases to an image quality Q1 (first image quality) higher than the default image quality Q0.

[0077] The release condition is a condition for restoring the image quality from the image quality Q1 (first image quality) to the default image quality Q0. For example, the release condition is that the active operation of the remote driver O is no longer detected. As another example, the release condition is that a certain time has passed since the active operation was detected. At time t2 after time t1, the release condition is satisfied. When the release condition is satisfied, the image quality is restored from the image quality Q1 to the default image quality Q0.

[0078] like Figure 5 As shown, the image quality may also be gradually changed from the image quality Q1 to the default image quality Q0. That is, the time for the image quality to be restored from the image quality Q1 to the default image quality Q0 may also be set longer than the time for the image quality to be increased from the default image quality Q0 to the image quality Q1. In this case, the remote driver O is less likely to notice the decrease in image quality. As a result, the discomfort (discomfort) felt by the remote driver O can be reduced.

[0079] exist Figure 5 In the example shown in (A) in FIG, during the period from time t1 to time t3, the image quality becomes higher than the default image quality Q0. Hereinafter, the period during which the image quality becomes higher than the default image quality Q0 is referred to as the "first period". The first period is limited. The default image quality Q0 is the image quality during the period other than the first period. During the first period after the active operation of the remote driver O is detected, the image quality becomes higher than the default image quality Q0.

[0080] Figure 5 (B) in FIG. 1 shows a case where an active operation of the remote driver O is predicted. An active operation is predicted at time tp before time t1 when the active operation is actually performed. In conjunction with the prediction of the active operation, the image quality is increased to an image quality Q1 (first image quality) higher than the default image quality Q0. Figure 5 The same is true for (A) in . Figure 5 In the example shown in (B) in FIG. 1 , the period from time tp to time t3 is the first period. In the first period after the active operation by the remote driver O is predicted, the image quality becomes higher than the default image quality Q0.

[0081] Figure 6: is a conceptual diagram for explaining an example of detection and cancellation of active operation. The horizontal axis represents the degree of operation of the remote driver O. When the degree of operation exceeds the first threshold TH1, it is determined that the remote driver O has performed active operation. The cancellation threshold THd is a threshold for determining that the remote driver O is no longer performing active operation. The cancellation condition is that the degree of operation becomes less than the cancellation threshold THd. Figure 6 As shown, the release threshold THd may be set lower than the first threshold TH1. That is, the detection and release of the active operation may also have a hysteresis characteristic. Such a hysteresis characteristic can suppress the fluctuation of the image quality.

[0082] 2-4. Image transmission system

[0083] Figure 7 1 is a block diagram showing a functional configuration example of a video transmission system 1X for transmitting a video VID from the vehicle 100 to the remote driver terminal 200 . The video transmission system 1X is a part of the remote driving system 1 .

[0084] The video transmission system 1X includes a camera C and an encoder mounted on a vehicle 100 . The camera C acquires a video VID. The encoder encodes the video VID. The vehicle 100 transmits the encoded video VID to the remote driver terminal 200 .

[0085] The video transmission system 1 includes a decoder on the remote driver terminal 200 side and a display device 220. The remote driver terminal 200 receives the video VID transmitted from the vehicle 100. The decoder decodes the received video VID. The display device 220 displays the decoded video VID.

[0086] The image transmission system 1X further includes an active operation determination unit 10 and an image quality determination unit 20. The active operation determination unit 10 detects or predicts active operation of the remote pilot O. A specific example of the active operation determination unit 10 will be described in detail in Section 5 below.

[0087] The image quality determination unit 20 obtains the determination result of the active operation determination unit 10. Furthermore, the image quality determination unit 20 determines the control information of the image quality based on the determination result of the active operation determination unit 10. For example, the control information of the image quality is a set value of the frame rate of the image VID. As another example, the control information of the image quality is a set value of the key frame rate of the image VID. As another example, the control information of the image quality is an instruction to insert a temporary key frame into the image VID.

[0088] like Figure 7As shown, the image quality determination unit 20 includes a frame rate setting unit 21, a key frame rate setting unit 22, and a key frame insertion unit 23. The frame rate setting unit 21 sets the frame rate of the image VID based on the determination result of the active operation determination unit 10. The key frame rate setting unit 22 sets the key frame rate of the image VID based on the determination result of the active operation determination unit 10. The key frame insertion unit 23 generates an instruction to insert a key frame into the image VID based on the determination result of the active operation determination unit 10.

[0089] The video transmission system 1X controls the encoder mounted on the vehicle 100 according to the video quality control information. In this way, the video quality control according to the present embodiment is realized.

[0090] In addition, as described above, the vehicle 100, the remote driver terminal 200, and the management device 300 can communicate with each other via a communication network. As a result, information required for image quality control can be shared between the vehicle 100, the remote driver terminal 200, and the management device 300. Since the information required for image quality control can be shared, the active operation determination unit 10 and the image quality determination unit 20 can also be set in any one of the vehicle 100, the remote driver terminal 200, and the management device 300. The active operation determination unit 10 and the image quality determination unit 20 can also be set in separate devices. That is, the active operation determination unit 10 and the image quality determination unit 20 can also be dispersed in the vehicle 100, the remote driver terminal 200, and the management device 300. Generally speaking, the active operation determination unit 10 and the image quality determination unit 20 are implemented by one or more processors.

[0091] Figure 8 is a flow chart generally showing image quality control taking active operations into account.

[0092] In step S10, the image transmission system 1X performs "positive operation determination processing". Specifically, the image transmission system 1X detects or predicts the positive operation of the remote driver O. When a positive operation is detected or predicted (step S10: Yes), the process proceeds to step S20. On the other hand, when a positive operation is not detected or predicted (step S10: No), the process proceeds to step S40.

[0093] In step S20, the video transmission system 1X performs "video quality enhancement processing." Specifically, the video transmission system 1X sets the video quality of the video VID to be higher than the default video quality Q0. Then, the process proceeds to step S30.

[0094] In step S30, the video transmission system 1X determines whether the cancellation condition is satisfied. If the cancellation condition is not satisfied (step S30: No), the process returns to step S20. On the other hand, if the cancellation condition is satisfied (step S30: Yes), the process proceeds to step S40.

[0095] In step S40 , the image transmission system 1X sets the image quality of the image VID to a default image quality Q0 .

[0096] 2-5. Effect

[0097] As described above, according to the present embodiment, the image quality of the image VID transmitted from the vehicle 100 to the remote driver terminal 200 is not uniformly set, but is dynamically changed. Specifically, the image quality increases in the first period after the active operation of the remote driver O is detected or predicted. When the active operation is performed, the remote driver O observes the surrounding conditions of the vehicle 100 particularly carefully. Since the image quality increases at this time, the accuracy of the remote driving of the remote driver O is improved.

[0098] The image quality of the image VID during the period other than the first period is set lower than the image quality during the first period. During the period other than the first period, the remote driver O does not actively operate, so the accuracy of remote driving can be ensured even if the image quality is not increased to a higher level than required. As long as the image VID of normal quality is provided, the remote driver O can continue the remote driving of the vehicle 100 with sufficiently high accuracy. In addition, since the image quality is set relatively low, the amount of transmission data can be suppressed, and the communication cost can also be suppressed.

[0099] Thus, according to the present embodiment, it is possible to ensure the accuracy of remote driving while suppressing the communication cost. In other words, according to the present embodiment, it is possible to achieve both ensuring the accuracy of remote driving and suppressing the communication cost.

[0100] 3. Vehicle example

[0101] 3-1. Configuration example

[0102] Fig. 9 1 is a block diagram showing a configuration example of a vehicle 100 . The vehicle 100 includes a communication device 110 , a sensor group 120 , a travel device 130 , and a control device 150 .

[0103] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote driver terminal 200 and the management device 300. The communication device 110 may also include an encoder for encoding the image VID (see Figure 7 ).

[0104] The sensor group 120 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the conditions around the vehicle 100. As identification sensors, camera C, LIDAR (Laser Imaging Detection and Ranging), radar, and the like can be exemplified. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0105] The travel device 130 includes a steering device, a drive device, and a brake device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS: Electric Power Steering) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, an in-wheel electric motor, etc. The brake device generates a braking force.

[0106] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter, referred to as the processor 160) and one or more storage devices 170 (hereinafter, referred to as the storage device 170). The processor 160 performs various processes. As the processor 160, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. can be exemplified. The storage device 170 stores various information. As the storage device 170, a volatile memory, a non-volatile memory, a HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. can be exemplified.

[0107] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by the cooperation between the processor 160 executing the vehicle control program PROG1 and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0108] 3-2. Driving environment information

[0109] The control device 150 acquires driving environment information ENV indicating the driving environment of the vehicle 100 . The driving environment information ENV is stored in the storage device 170 .

[0110] The driving environment information ENV includes map information MAP. The map information MAP includes a general navigation map. The map information MAP may also indicate lane configuration and road shape. The map information MAP may also include location information of intersections, stop lines, traffic lights, road signs, etc. The control device 150 obtains the map information MAP of the required area from the map database. The map database may be stored in the storage device 170 or in a map management device outside the vehicle 100. In the latter case, the control device 150 communicates with the map management device via the communication device 110 to obtain the required map information MAP.

[0111] In addition, the driving environment information ENV includes surrounding condition information SUR indicating the recognition result of the recognition sensor. For example, the surrounding condition information SUR includes an image VID captured by the camera C. In addition, the surrounding condition information SUR may also include object information related to objects around the vehicle 100. As objects around the vehicle 100, pedestrians, other vehicles (the vehicle in front, parked vehicles, etc.), white lines, stop lines, traffic lights, road signs, roadside structures, etc. can be exemplified. The object information indicates the relative position and relative speed of the object relative to the vehicle 100. For example, by analyzing the image VID obtained by the camera, the object can be identified and the relative position of the object can be calculated. In addition, based on the point cloud information obtained by LIDAR, the object can be identified and the relative position and relative speed of the object can be obtained.

[0112] The driving environment information ENV includes vehicle state information STA indicating the detection result of the vehicle state sensor. The vehicle state information STA indicates the vehicle speed, acceleration (front and rear acceleration, lateral acceleration), yaw rate, steering angle, and the like of the vehicle 100 .

[0113] Furthermore, the driving environment information ENV includes vehicle position information POS indicating the position and moving direction (orientation) of the vehicle 100. The vehicle position information POS is obtained by a position sensor. High-precision vehicle position information POS can also be obtained by self-position estimation processing (Localization) using map information MAP and surrounding situation information SUR (object information).

[0114] 3-3. Vehicle driving control

[0115] The control device 150 performs vehicle travel control for controlling the travel of the vehicle 100. The vehicle travel control includes steering control, drive control, and brake control. The control device 150 performs vehicle travel control by controlling the travel device 130 (steering device, drive device, and brake device).

[0116] The control device 150 may also perform automatic driving control based on the driving environment information ENV. Automatic driving means automatically performing at least a part of the steering, acceleration, and deceleration of the vehicle 100 independently of the driver's operation. In more detail, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory (trajectory) required for the vehicle 100 to travel according to the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Moreover, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0117] 3-4. Processing related to remote driving

[0118] Hereinafter, a case where remote driving of the vehicle 100 is performed will be described. The control device 150 communicates with the remote driver terminal 200 via the communication device 110 .

[0119] The control device 150 sends the vehicle information VCL to the remote driver terminal 200. The vehicle information VCL is information required for the remote driver O to perform remote driving, and includes at least a part of the driving environment information ENV mentioned above. In particular, the vehicle information VCL includes the image VID captured by the camera C. The vehicle information VCL may also include other surrounding condition information SUR. The vehicle information VCL may also include vehicle status information STA. The vehicle information VCL may also include vehicle position information POS.

[0120] In addition, the control device 150 receives remote operation information OPE from the remote driver terminal 200. The remote operation information OPE is information related to remote driving of the remote driver O. For example, the remote operation information OPE includes a driving operation amount input by the remote driver O. The control device 150 performs vehicle driving control according to the received remote operation information OPE.

[0121] The control device 150 may also include Figure 7 The active operation determination unit 10 and the video quality determination unit 20 shown in the figure both have functions. The video quality determination unit 20 determines the video quality control information based on the determination result of the active operation determination unit 10. The control device 150 controls the encoder according to the video quality control information.

[0122] The control device 150 may not have the function of the active operation determination unit 10 but may have the function of the image quality determination unit 20 . In this case, the control device 150 obtains information on the determination result of the external active operation determination unit 10 via the communication device 110 .

[0123] The active operation determination unit 10 and the image quality determination unit 20 may be located outside the vehicle 100. In this case, the control device 150 obtains the image quality control information determined by the image quality determination unit 20 via the communication device 110. The control device 150 controls the encoder according to the image quality control information.

[0124] 4. Example of remote driver terminal configuration

[0125] Fig.10 2 is a block diagram showing a configuration example of the remote driver terminal 200 . The remote driver terminal 200 includes a communication device 210 , a display device 220 , an input device 230 , and a control device 250 .

[0126] The communication device 210 communicates with the vehicle 100 and the management device 300. The communication device 110 may also include a decoder for decoding the video VID (see Figure 7 ).

[0127] The display device 220 displays various information to the remote driver O who performs remote driving. In other words, the display device 220 displays various information to present various information to the remote driver O. The display device 220 may be a touch panel.

[0128] The input device 230 receives input from the remote driver O. For example, the input device 230 includes remote driving components that the remote driver O operates when remotely driving the vehicle 100. The remote driving components include a steering wheel, an accelerator pedal, a brake pedal, a direction indicator, and the like.

[0129] The control device 250 controls the remote driver terminal 200. The control device 250 includes one or more processors 260 (hereinafter, referred to as the processor 260) and one or more storage devices 270 (hereinafter, referred to as the storage device 270). The processor 260 performs various processes. As the processor 260, a CPU, a GPU, an ASIC, an FPGA, etc. can be exemplified. The storage device 270 stores various information. As the storage device 270, a volatile memory, a non-volatile memory, an HDD, an SSD, etc. can be exemplified.

[0130] The remote driving control program PROG2 is a computer program executed by the processor 260. The functions of the control device 250 can also be realized by the cooperation between the processor 260 and the storage device 270 that executes the remote driving control program PROG2. The remote driving control program PROG2 is stored in the storage device 270. Alternatively, the remote driving control program PROG2 can also be recorded in a computer-readable recording medium. The remote driving control program PROG2 can also be provided via a network.

[0131] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives the vehicle information VCL transmitted from the vehicle 100. The control device 250 displays the vehicle information VCL including the image VID on the display device 220, thereby presenting the vehicle information VCL to the remote driver O. The remote driver O can recognize the state of the vehicle 100 and / or the surrounding conditions based on the vehicle information VCL displayed on the display device 220.

[0132] The remote driver O operates the remote driving component of the input device 230. The operation amount of the remote driving component is detected by a sensor provided in the remote driving component. The control device 250 generates remote operation information OPE reflecting the operation amount of the remote driving component by the remote driver O. The remote operation information OPE can also be said to be information reflecting the degree of driving operation by the remote driver O. In addition, the control device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210.

[0133] The control device 250 may also include Figure 7 In this case, the control device 250 transmits information on the determination result of the active operation determination unit 10 to the vehicle 100 via the communication device 210 .

[0134] The control device 250 may also include Figure 7 The active operation determination unit 10 and the image quality determination unit 20 shown in FIG. 2 both have functions. The image quality determination unit 20 determines the control information of the image quality based on the determination result of the active operation determination unit 10. The control device 250 transmits the control information of the image quality to the vehicle 100 via the communication device 210.

[0135] 5. Example of actively operating the judgment unit

[0136] Various examples of the active operation determination unit 10 that detects or predicts an active operation by the remote driver O will be described below.

[0137] 5-1.Example 1

[0138] Fig.111 is a block diagram showing a first example of the active operation determination unit 10. The active operation determination unit 10 includes an active operation detection unit 11 and a determination unit 15. The active operation detection unit 11 detects an active operation of the remote driver O. Fig.11 In the illustrated example, the active operation detection unit 11 includes a first detection unit 11 - 1 and a second detection unit 11 - 2 .

[0139] The first detection unit 11-1 detects the active operation of the remote driver O based on the remote operation information OPE. As described above, the remote operation information OPE reflects the degree of the driving operation (operation degree) of the remote driver O. For example, the operation degree is proportional to the steering amount of the remote driver O. As another example, the operation degree is proportional to the steering speed of the remote driver O. As another example, the operation degree is proportional to the required acceleration required by the remote driver O. As another example, the operation degree is proportional to the required deceleration required by the remote driver O. The first detection unit 11-1 detects the active operation by determining whether the operation degree of the remote driver O exceeds the first threshold TH1 based on the remote operation information OPE. The first threshold TH1 is different for each operation degree. The first detection unit 11-1 can also detect the active operation based on a mapping representing the correspondence between the degree of driving operation and the active operation.

[0140] The second detection unit 11-2 detects the active operation of the remote driver O based on the vehicle state information STA. The remote driving of the vehicle 100 performed by the remote driver O is reflected in the behavior of the vehicle 100. Therefore, it is possible to detect the active operation of the remote driver O based on the vehicle state information STA. For example, when the front and rear acceleration of the vehicle 100 exceeds the first threshold TH1, the second detection unit 11-2 detects the active operation. As another example, when the yaw rate or lateral acceleration of the vehicle 100 exceeds the first threshold TH1, the second detection unit 11-2 detects the active operation. The first threshold TH1 is different for each vehicle state. The second detection unit 11-2 can also detect the active operation based on a mapping indicating the correspondence between the vehicle state information STA and the active operation.

[0141] The first threshold TH1 or the map may be set individually for each remote driver O. The active operation may be divided into a plurality of levels (see Figure 2 ).

[0142] The first detection unit 11-1 and the second detection unit 11-2 each output a detection result to the determination unit 15. The detection result includes whether or not an active operation is detected, the content (type) of the detected active operation, and the like.

[0143] The determination unit 15 receives the detection result of the first detection unit 11-1 and the detection result of the second detection unit 11-2. When at least one of the first detection unit 11-1 and the second detection unit 11-2 detects an active operation, the determination unit 15 determines that an active operation is performed by the remote pilot O. The determination unit 15 notifies the image quality determination unit 20 of the determination result. The determination unit 15 may also notify the image quality determination unit 20 of the content (type) of the detected active operation.

[0144] Furthermore, when the remote driver O performs remote driving of the vehicle 100, the remote operation information OPE changes at an earlier timing than the vehicle state information STA. Therefore, basically, the first detection unit 11-1 detects the active operation earlier than the second detection unit 11-2.

[0145] 5-2.Example 2

[0146] Fig.12 2 is a block diagram showing a second example of the active operation determination unit 10. The active operation determination unit 10 includes a history database and an active operation prediction unit 12 in addition to the active operation detection unit 11 described in the first example.

[0147] The history database indicates the position where the active operation was detected in the past. When the active operation is detected by the active operation detection unit 11, the active operation determination unit 10 registers the position of the vehicle 100 at the time when the active operation is detected in the history database. The position of the vehicle 100 can be obtained based on the vehicle position information POS. The history database can also associate the position where the active operation was detected in the past with the moving direction of the vehicle 100 at this time. The moving direction of the vehicle 100 can be obtained based on the vehicle position information POS. The history database can also associate the content of the active operation with the position where the active operation is detected.

[0148] In a scenario where a certain remote driver O performs an active operation, there is a high possibility that the same remote driver O or another remote driver O will also perform an active operation. Thus, by effectively utilizing the history database, it is possible to “predict” the occurrence of active operations in the near future.

[0149] More specifically, the active operation prediction unit 12 acquires the vehicle position information POS. The vehicle position information POS indicates the current position and moving direction of the vehicle 100. The active operation prediction unit 12 predicts the occurrence of an active operation on the road ahead of the vehicle 100 based on the history record database and the current position and moving direction of the vehicle 100. In other words, the active operation prediction unit 12 predicts the occurrence of an active operation that is likely to be performed by the remote driver O in the near future based on the history record database and the current position and moving direction of the vehicle 100. The active operation prediction unit 12 outputs the prediction result to the determination unit 15. The prediction result includes whether an active operation is predicted, the content (type) of the predicted active operation, and the like.

[0150] The determination unit 15 receives the detection result of the active operation detection unit 11 and the prediction result of the active operation prediction unit 12, and integrates them. The determination unit 15 may also weight the detection result of the active operation detection unit 11 and the prediction result of the active operation prediction unit 12. When an active operation is detected or predicted, the determination unit 15 notifies the image quality determination unit 20 of this fact. The determination unit 15 may also notify the image quality determination unit 20 of the content (type) of the detected or predicted active operation.

[0151] 5-3.Example 3

[0152] Fig.13 2 is a block diagram showing a third example of the active operation determination unit 10. The active operation determination unit 10 includes an active operation prediction unit 13. The active operation prediction unit 13 comprehends the condition of the road ahead of the vehicle 100 based on the driving environment information ENV and predicts the occurrence of an active operation on the road ahead.

[0153] For example, when the road ahead of the vehicle 100 is greatly curved, it is predicted that the remote driver O will actively turn. The shape of the road ahead of the vehicle 100 can be obtained based on the map information MAP. As another example, when there is an intersection, a stop line, a stop sign, or a traffic light in front of the vehicle 100, it is predicted that the remote driver O will actively slow down. The position information of the intersection, stop line, traffic light, road sign, etc. is registered in the map information MAP. The current position and moving direction of the vehicle 100 can be obtained based on the vehicle position information POS. The active operation prediction unit 13 grasps the condition of the road ahead of the vehicle 100 based on the vehicle position information POS and the map information MAP. Moreover, the active operation prediction unit 13 predicts the occurrence of active operation on the road ahead based on the condition of the road ahead of the vehicle 100.

[0154] Instead of the map information MAP, the condition of the road ahead of the vehicle 100 may be grasped based on the surrounding condition information SUR indicating the recognition result of the recognition sensor. In this case, the active operation prediction unit 13 grasps the condition of the road ahead of the vehicle 100 based on the vehicle position information POS and the surrounding condition information SUR. Furthermore, the active operation prediction unit 13 predicts the occurrence of an active operation on the road ahead based on the condition of the road ahead of the vehicle 100.

[0155] The active operation prediction unit 13 may also consider the vehicle speed in addition to the condition of the road ahead of the vehicle 100. This is because the higher the vehicle speed, the higher the possibility that the remote driver O actively decelerates. In this case, the active operation prediction unit 13 also considers the vehicle state information STA and predicts the occurrence of an active operation on the road ahead of the vehicle 100.

[0156] The prediction processing performed by the active operation prediction unit 13 may use a prediction model generated in advance by machine learning such as deep learning. The input to the prediction model is the driving environment information ENV, and the output from the prediction model is the presence or absence of active operation and the content of the active operation.

[0157] The active operation prediction unit 13 outputs the prediction result to the determination unit 15. The prediction result includes whether an active operation is predicted, the content (type) of the predicted active operation, and the like.

[0158] The determination unit 15 receives the prediction result of the active operation prediction unit 13. When an active operation is predicted, the determination unit 15 notifies the image quality determination unit 20 of this fact. The determination unit 15 may also notify the image quality determination unit 20 of the content (type) of the predicted active operation.

[0159] 5-4.Example 4

[0160] Fig.14 2 is a block diagram showing a fourth example of the active operation determination unit 10. The active operation determination unit 10 includes an active operation prediction unit 14 and a determination unit 15. The active operation prediction unit 14 predicts an active operation of the remote driver O based on the vehicle state information STA.

[0161] For example, when a strong side wind blows and the vehicle 100 skids, the remote driver O is likely to actively perform a steering operation to stabilize the vehicle 100 in order to return the vehicle 100 to its original lateral position. Therefore, when the lateral acceleration of the vehicle 100 exceeds the first threshold TH1, the active operation prediction unit 14 predicts an active operation by the remote driver O. The determination unit 15 is the same as in the third example.

[0162] 5-5.Example 5

[0163] Two or more of the above-mentioned examples of the active operation determination unit 10 may be combined.

[0164] Fig.15 4 is a block diagram showing a fifth example of the active operation determination unit 10. Fig.15 In the example shown, the active operation determination unit 10 includes an active operation detection unit 11 and active operation prediction units 12 to 14. The determination unit 15 receives the detection result of the active operation detection unit 11 and the prediction result of the active operation prediction units 12 to 14 and integrates them. The determination unit 15 may also weight the detection result of the active operation detection unit 11 and the prediction result of the active operation prediction units 12 to 14. When an active operation is detected or predicted, the determination unit 15 notifies the image quality determination unit 20 of this. The determination unit 15 may also notify the image quality determination unit 20 of the content (type) of the detected or predicted active operation.

Claims

1. A video transmission system for transmitting a video from a mobile object to be remotely driven by a remote driver to a terminal on the remote driver's side, Having one or more processors, The one or more processors are configured to perform active operation determination processing and image quality enhancement processing, In the active operation determination process, an active operation in which the degree of the driving operation of the remote driver exceeds a first threshold is detected or predicted, In the image quality increasing process, for a first period after the active operation is detected or predicted, the quality of the image transmitted to the terminal is improved compared with periods other than the first period.

2. The image transmission system according to claim 1, The image quality enhancement process includes: Regardless of the setting of the key frame rate of the image, a key frame is inserted into the image in response to the detection or prediction of the active operation.

3. The image transmission system according to claim 1 or 2, The image quality enhancement process includes: The key frame rate of the video in the first period is set higher than the key frame rate in a period other than the first period.

4. The image transmission system according to claim 1, The plurality of process candidates include two or more of the first process, the second process, and the third process. The first process is a process of making the frame rate of the image in the first period higher than the frame rate in a period other than the first period. The second process is a process of making the key frame rate of the image in the first period higher than the key frame rate of the period other than the first period. The third process is a process of inserting a key frame into the image in response to detection or prediction of the active operation, regardless of the setting of the key frame rate of the image. The image quality enhancement process includes: At least any one of the plurality of processing candidates is executed.

5. The image transmission system according to claim 4, The first threshold is independently set for each of the plurality of process candidates.

6. The image transmission system according to claim 4, The image quality enhancement process includes: A candidate process to be executed among the plurality of candidate processes is determined according to the type of the active operation.

7. The image transmission system according to claim 6, The image quality enhancement process includes: When the active operation is an acceleration operation, at least the first process is executed; and When the active operation is a steering operation, at least the second process is executed.

8. The image transmission system according to claim 4, The image quality enhancement process includes: As the speed of the moving object increases, the frame rate of the image in the first period increases; and As the speed of the moving object decreases, the key frame rate of the image in the first period is increased.

9. The image transmission system according to claim 1, The default image quality is the quality of the image during a period other than the first period. The image quality enhancement process includes: In conjunction with detecting or predicting the active operation, increasing the quality of the image to a first image quality that is higher than the default image quality; and When a release condition is satisfied after the quality of the image is increased, the quality of the image is restored from the first image quality to the default image quality.

10. The image transmission system according to claim 9, The image quality enhancement process includes: The time for the quality of the image to be restored from the first image quality to the default image quality is set to be longer than the time for the quality of the image to be increased from the default image quality to the first image quality.

11. The image transmission system according to claim 9, The release condition is that the degree of the driving operation of the remote driver becomes equal to or less than a release threshold. The release threshold is lower than the first threshold.

12. The image transmission system according to claim 1, The active operation determination process includes: acquiring remote operation information reflecting the degree of the driving operation by the remote driver; and The active operation is detected by determining whether the degree of the driving operation exceeds the first threshold value based on the remote operation information.

13. The image transmission system according to claim 12, The active operation determination process includes: accessing a history database indicating locations where the positive operation was detected in the past; Obtaining information on the current position and moving direction of the moving object; and The active operation on a road ahead of the moving body is predicted based on the history database and the current position and the moving direction of the moving body.

14. The image transmission system according to claim 1 or 12, The active operation determination process includes: Obtaining information on the current position and moving direction of the moving object; Based on map information or a recognition result of a recognition sensor mounted on the mobile body, grasping the condition of the road ahead of the mobile body; and The active operation on the road ahead of the moving body is predicted based on the condition of the road ahead of the moving body.

15. A method for transmitting an image from a mobile object that is a subject of remote driving by a remote driver to a terminal on the remote driver's side, comprising: Active operation determination processing for detecting or predicting an active operation in which the degree of the driving operation of the remote driver exceeds a first threshold; and The image quality improvement process improves the quality of the image transmitted to the terminal during a first period after the active operation is detected or predicted, compared to periods other than the first period.

Images