Control device, control method, and control program

Abstract

To suppress transmission delay of video data transmitted from a mobile body in remotely operating the mobile body by a remote operator.SOLUTION: A mobile body is to be remotely operated by a remote operator, the mobile body including a plurality of cameras. A control device for controlling the mobile body acquires multiple pieces of video data captured by the plurality of cameras, respectively, as transmission data to be transmitted to a remote operator terminal on the side of the remote operator. The control device acquires information on a communication speed of a communication from the mobile body to the remote operator terminal. The control device determines whether data reduction is required for limiting the total amount of the multiple pieces of video data per second within the communication speed. When it is determined that the data reduction is required, the control device executes data amount adjustment processing of reducing the amount of data to be transmitted of at least one piece of the multiple pieces of video data such that the total amount of data can be limited within the communication speed.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator, and in particular to a technique for controlling transmission of video data from a mobile object that is a target of remote operation. [Background technology]

[0002] Patent Document 1 discloses an in-vehicle communication device. The in-vehicle communication device supports both a mobile communication method and a WiFi communication method. When an abnormality in the vehicle is detected, the in-vehicle communication device transmits numerical data and image data indicating the vehicle driving status to a designated server. At this time, the in-vehicle communication device transmits the numerical data by the mobile communication method and transmits the image data by the WiFi communication method.

[0003] Other known techniques relating to communication of images captured by a camera include those disclosed in Patent Documents 2 and 3. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-120443 A [Patent Document 2] JP 2005-222307 A [Patent Document 3] Japanese Patent Application Publication No. 10-112861 Summary of the Invention [Problem to be solved by the invention]

[0005] Consider the remote operation of a moving object (e.g., a vehicle or a robot) by a remote operator. When remotely operating a moving object, video data obtained by a camera mounted on the moving object is transmitted to a remote operator terminal on the remote operator's side. At this time, if the communication speed decreases, there is a risk that the transmission delay of the video data will increase. An increase in the transmission delay of the video data will lead to delays in the judgments and operations of the remote operator, which may cause a decrease in the accuracy of the remote operation.

[0006] One object of the present disclosure is to provide a technique capable of suppressing a transmission delay in video data transmitted from a mobile object when the mobile object is remotely controlled by a remote operator. [Means for solving the problem]

[0007] The first aspect relates to a control device that controls a moving object. The moving object is a target for remote control by a remote operator and is equipped with a number of cameras. The controller comprises one or more processors. The one or more processors acquire a plurality of pieces of video data obtained by the respective plurality of cameras as transmission target data to be transmitted to a remote operator terminal on the remote operator side. The one or more processors obtain communication rate information for communications from a mobile unit to a remote operator terminal. The one or more processors determine whether or not data reduction is necessary to keep the total data amount per second of the plurality of video data items at or below the communication speed. When it is determined that data reduction is necessary, the one or more processors execute a data amount adjustment process for reducing the transmission data amount of at least one of the multiple video data so that the total data amount is kept below the communication speed.

[0008] The second aspect relates to a control method for controlling a moving object. The moving object is a target for remote control by a remote operator and is equipped with a number of cameras. The control method is A process of acquiring a plurality of pieces of video data obtained by each of a plurality of cameras as transmission target data to be transmitted to a remote operator terminal on the remote operator side; Obtaining information on the communication speed of communication from the mobile object to the remote operator terminal; A process of determining whether data reduction is necessary to suppress a total data amount per second of the plurality of video data items to a communication speed or less; When it is determined that data reduction is necessary, a data amount adjustment process is performed to reduce the amount of transmission data of at least one of the plurality of video data so that the total data amount is kept below the communication speed. Includes.

[0009] A third aspect relates to a control program executed by a computer to control a moving object. The moving object is a target for remote control by a remote operator and is equipped with a number of cameras. The control program is A process of acquiring a plurality of pieces of video data obtained by each of a plurality of cameras as transmission target data to be transmitted to a remote operator terminal on the remote operator side; Obtaining information on the communication speed of communication from the mobile object to the remote operator terminal; A process of determining whether data reduction is necessary to suppress a total data amount per second of the plurality of video data items to a communication speed or less; When it is determined that data reduction is necessary, a data amount adjustment process is performed to reduce the amount of transmission data of at least one of the plurality of video data so that the total data amount is kept below the communication speed. to be executed by the computer. Effect of the Invention

[0010] According to the present disclosure, the communication speed of communication from a mobile body to a remote operator terminal is taken into consideration. Then, a data amount adjustment process is performed so that the total data amount per second of a plurality of video data is kept below the communication speed. This data amount adjustment process makes it possible to suppress a transmission delay of video data transmitted from a mobile body. Since the transmission delay of video data is suppressed, delays in judgment and operation by a remote operator are also suppressed. In other words, it is possible to suppress a decrease in accuracy of remote operation caused by a transmission delay. This is preferable from the viewpoint of improving the safety of remote operation of a mobile body. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram illustrating a configuration example of a remote control system according to an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram showing an example of a functional configuration related to a data amount adjustment process according to an embodiment of the present disclosure. FIG. [Diagram 3] 10 is a flowchart showing processing related to a data amount adjustment process according to an embodiment of the present disclosure. [Figure 4] FIG. 11 is a block diagram for explaining a data amount adjustment process taking into consideration camera priority according to an embodiment of the present disclosure. [Diagram 5] 1 is a conceptual diagram for explaining an example of dynamic setting of camera priorities according to an embodiment of the present disclosure. [Figure 6] 11 is a conceptual diagram for explaining an example of a data amount adjustment process taking into account camera priority according to an embodiment of the present disclosure. FIG. [Figure 7] 13 is a conceptual diagram for explaining another example of the data amount adjustment process taking into consideration the camera priority according to the embodiment of the present disclosure. FIG. [Figure 8] FIG. 11 is a conceptual diagram for explaining an example of setting a Fix level according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a block diagram showing an example of a functional configuration related to a Fix level according to an embodiment of the present disclosure. [Figure 10] FIG. 11 is a conceptual diagram for explaining an example of transition of a Fix level according to an embodiment of the present disclosure. [Figure 11] 1 is a block diagram illustrating an example configuration of a vehicle according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0013] 1. Overview of the remote control system Consider remote control (remote driving) of a moving object. Examples of moving objects that can be remotely controlled include vehicles, robots, flying objects, etc. A vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots, etc. Examples of flying objects include airplanes and drones, etc.

[0014] As an example, in the following description, a case will be considered in which the moving object to be remotely controlled is a vehicle. In general, the term "vehicle" in the following description will be replaced with "moving object."

[0015] FIG. 1 is a schematic diagram showing a configuration example of a remote operation system 1 according to the present embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is a target of remote operation. The remote operator terminal 200 is a terminal device used when a remote operator O remotely operates the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. The management of the remote operation system 1 includes, for example, assigning a remote operator O to the vehicle 100 that needs to be remotely operated. The management device 300 can communicate with the vehicle 100 and the remote operator terminal 200 via a communication network. Typically, the management device 300 is a management server on a cloud. The management server may be composed of multiple servers that perform distributed processing.

[0016] The vehicle 100 is equipped with various sensors including a camera C. The camera C captures an image of the surroundings of the vehicle 100 and acquires an image IMG showing the surroundings of the vehicle 100. The vehicle information VCL is information obtained by the various sensors and includes the image IMG obtained by the camera C. The vehicle 100 transmits the vehicle information VCL to the remote operator terminal 200 via the management device 300. That is, the vehicle 100 transmits the vehicle information VCL to the management device 300, and the management device 300 transfers the received vehicle information VCL to the remote operator terminal 200.

[0017] The remote operator terminal 200 receives the vehicle information VCL transmitted from the vehicle 100. The remote operator terminal 200 presents the vehicle information VCL to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device and displays an image IMG or the like on the display device. The remote operator O recognizes the surrounding situation of the vehicle 100 by looking at the displayed information, and remotely operates the vehicle 100. The remote operation information OPE is information related to remote operation by the remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100 via the management device 300. That is, the remote operator terminal 200 transmits the remote operation information OPE to the management device 300, and the management device 300 transfers the received remote operation information OPE to the vehicle 100.

[0018] The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle travel control in accordance with the received remote operation information OPE. In this manner, the remote operation of the vehicle 100 is realized.

[0019] 2. Overview of data volume adjustment process Hereinafter, video data (video stream data) S will be considered as an image IMG captured by a camera C mounted on the vehicle 100. When the vehicle 100 is equipped with multiple cameras C1 to Cn (n is an integer equal to or greater than 2), multiple video data S1 to Sn are obtained by each of the multiple cameras C1 to Cn. The vehicle 100 transmits the multiple video data S1 to Sn in parallel to the remote operator terminal 200. The remote operator terminal 200 receives the multiple video data S1 to Sn in parallel and displays the multiple video data S1 to Sn on a display device. A remote operator O looks at the displayed video data S to recognize the situation around the vehicle 100 and remotely controls the vehicle 100.

[0020] When the communication speed (communication band) from the vehicle 100 to the remote operator terminal 200 decreases, there is a risk of an increase in the transmission delay of the video data S1-Sn. An increase in the transmission delay of the video data S1-Sn may lead to delays in the judgments and operations of the remote operator O, and may cause a decrease in the accuracy of remote operation. Therefore, a technique capable of suppressing the transmission delay of the video data S1-Sn transmitted from the vehicle 100 is desired. According to this embodiment, in order to suppress the transmission delay of the video data S1-Sn transmitted from the vehicle 100, a "data amount adjustment process" described below is performed as necessary.

[0021] 2 is a block diagram showing an example of a functional configuration related to the data amount adjustment process according to the present embodiment. The vehicle 100 includes, as functional blocks, a communication unit 101, a video data acquisition unit 102, a communication state acquisition unit 103, a determination unit 104, and a data amount adjustment unit 105.

[0022] The communication unit 101 communicates with the remote operator terminal 200 via the management device 300 .

[0023] The video data acquisition unit 102 acquires a plurality of video data (video stream data) S1 to Sn obtained by a plurality of cameras C1 to Cn mounted on the vehicle 100. The plurality of video data S1 to Sn are data to be transmitted in parallel from the vehicle 100 to the remote operator terminal 200.

[0024] The communication status acquisition unit 103 monitors communication with the remote operator terminal 200 by the communication unit 101. Then, the communication status acquisition unit 103 acquires information on the communication speed (communication bandwidth) of the communication with the remote operator terminal 200 based on the communication result with the remote operator terminal 200 by the communication unit 101. In particular, the communication status acquisition unit 103 acquires information on the communication speed Z of the communication from the vehicle 100 to the remote operator terminal 200. The communication speed Z here is, for example, throughput. For example, the communication status acquisition unit 103 can measure the communication speed Z (throughput) based on the amount of transmission data transmitted to the remote operator terminal 200 and feedback from the remote operator terminal 200. Note that various methods for measuring the communication speed and throughput have been proposed, and the estimation method is not particularly limited in this embodiment.

[0025] The communication speed Z varies over time. When the communication speed Z drops, the transmission delay of the video data S1 to Sn may increase. In particular, when the communication speed Z falls below the total data amount Yt per second of the multiple video data S1 to Sn, the transmission delay increases.

[0026] The determination unit 104 determines whether or not the total data amount Y per second of the multiple video data S1 to Sn exceeds the communication speed Z. In other words, the determination unit 104 determines whether or not data reduction is necessary to keep the total data amount Y per second of the multiple video data S1 to Sn equal to or less than the communication speed Z.

[0027] For example, a default value Y0 of the total data amount Y per second of the multiple video data S1 to Sn is given as known information. The default value Y0_i of the data amount per second of each video data Si (i=1 to n) is given by "default value F0_i of data amount per frame" x "default value R0_i of frame rate." The sum of the default values ​​Y0_1 to Y0_n for the multiple video data S1 to Sn is the default value Y0 of the total data amount Y per second of the multiple video data S1 to Sn. The amount of data exceeding the communication speed Z of the default value Y0 of the total data amount Y is expressed by the following formula (1).

[0028] Formula (1): X [bps] = Y0 [bps] - Z [bps]

[0029] The determination unit 104 compares the default value Y0 of the total data amount Y with the fluctuating communication speed Z. Then, if the default value Y0 of the total data amount Y exceeds the communication speed Z, the determination unit 104 determines that data reduction is necessary. In other words, if the fluctuating communication speed Z falls below the default value Y0 of the total data amount Y, the determination unit 104 determines that data reduction is necessary. In further other words, if the data excess amount X expressed by the above formula (1) is a positive value, the determination unit 104 determines that data reduction is necessary.

[0030] The determination unit 104 outputs determination result information RES indicating the determination result. The determination result information RES may include the excess data amount X expressed by the above formula (1).

[0031] The data amount adjustment unit 105 receives the plurality of video data S1-Sn and the determination result information RES. When the determination result information RES indicates that data reduction is necessary, the data amount adjustment unit 105 performs a data amount adjustment process. Specifically, the data amount adjustment unit 105 reduces the transmission data amount of at least one of the plurality of video data S1-Sn so that the total data amount Y per second of the plurality of video data S1-Sn is kept below the communication speed Z. The total reduction in the transmission data amount at this time is equal to or greater than the excess data amount X expressed by the above formula (1).

[0032] There are various possible methods for reducing the amount of transmission data for certain video data Sj (j=1 to n). For example, the amount of data per frame of the video data Sj is set to a value smaller than the default value F0_j. For example, the resolution (number of pixels) of the video data Sj is set to a value lower than the default value. For example, if the default value for the resolution of the video data Sj is 1080p, the resolution is reduced to 360p. This reduces the amount of transmission data for the video data Sj to approximately 1 / 9.

[0033] In order to further reduce the amount of transmission data of the video data Sj, the transmission of the video data Sj may be stopped, that is, the amount of transmission data of the video data Sj may be set to zero.

[0034] The communication unit 101 transmits vehicle information VCL including a plurality of video data S1 to Sn to the remote operator terminal 200. When the data amount adjustment process is performed, the transmission data amount of a certain video data Sj may become zero.

[0035] FIG. 3 is a flowchart outlining the processing related to the data amount adjustment processing.

[0036] In step S102, the vehicle 100 acquires a plurality of pieces of video data S1 to Sn obtained by the plurality of cameras C1 to Cn, respectively, as data to be transmitted.

[0037] In step S103, the vehicle 100 acquires information on the communication speed Z of the communication from the vehicle 100 to the remote operator terminal 200 based on the result of the communication with the remote operator terminal 200.

[0038] In step S104, the vehicle 100 determines whether data reduction is necessary to keep the total data amount Y per second of the multiple video data S1 to Sn equal to or less than the communication speed Z. If data reduction is necessary (step S104; Yes), the process proceeds to step S105. On the other hand, if data reduction is not necessary (step S104; No), the process proceeds to step S106.

[0039] In step S105, the vehicle 100 executes a data amount adjustment process. Specifically, the vehicle 100 reduces the transmission data amount of at least one of the plurality of video data S1 to Sn so that the total data amount Y per second of the plurality of video data S1 to Sn is kept equal to or less than the communication speed Z.

[0040] In step S106, the vehicle 100 transmits to the remote operator terminal 200 the vehicle information VCL including the plurality of video data S1 to Sn.

[0041] As described above, according to this embodiment, the communication speed Z of the communication from the vehicle 100 to the remote operator terminal 200 is taken into consideration. Then, a data amount adjustment process is performed so that the total data amount Y per second of the multiple video data S1-Sn is kept below the communication speed Z. This data amount adjustment process makes it possible to suppress a transmission delay of the video data S1-Sn transmitted from the vehicle 100. Since the transmission delay of the video data S1-Sn is suppressed, delays in judgment and operation by the remote operator O are also suppressed. That is, according to this embodiment, it is possible to suppress a decrease in the accuracy of remote operation caused by a transmission delay. This is preferable from the viewpoint of improving the safety of the remote operation of the vehicle 100.

[0042] 3. Data volume adjustment process considering camera priority The data amount adjustment process may take into consideration the priority of the cameras C1 to Cn. Specifically, the data amount adjustment process may be performed so that the transmission data amount of the video data S obtained by the camera C with a higher priority is increased.

[0043] FIG. 4 is a block diagram for explaining the data amount adjustment process considering the priority of the cameras C1 to Cn. The data amount adjustment unit 105 includes a priority setting unit 105P. The priority setting unit 105P sets the priority of the multiple cameras C1 to Cn. More specifically, the priority setting unit 105P holds priority policy information POL indicating a setting policy of the priority of the multiple cameras C1 to Cn. The priority setting unit 105P sets the priority of the multiple cameras C1 to Cn based on the priority policy information POL. Then, the data amount adjustment unit 105 executes the data amount adjustment process so that the transmission data amount of the video data S obtained by the camera C with a higher priority is increased. In other words, the data amount adjustment unit 105 secures the transmission data amount as much as possible for the video data S obtained by the camera C with a higher priority while suppressing the total data amount Y of the video data S1 to Sn.

[0044] In general, the priority of the first camera Ci is higher than that of the second camera Cj, and the priority of the second camera Cj is lower than that of the first camera Ci. The first video data Si is video data S obtained by the first camera Ci with a higher priority. The second video data Sj is video data S obtained by the second camera Cj with a lower priority. The data amount adjustment unit 105 executes a data amount adjustment process so that the transmission data amount of the first video data Si is greater than the transmission data amount of the second video data Sj. For example, the data amount adjustment unit 105 lowers the resolution of the second video data Sj to less than the default value without reducing the transmission data amount of the first video data Si. As another example, the data amount adjustment unit 105 transmits the first video data Si and stops transmitting the second video data Sj. In this way, the data amount adjustment unit 105 executes a data amount adjustment process so that the transmission data amount of the first video data Si obtained by the first camera Ci with a higher priority is greater.

[0045] The priorities of the multiple cameras C1 to Cn may be fixed or may be dynamically changed.

[0046] Fig. 5 is a conceptual diagram for explaining an example of dynamic setting of the priorities of cameras C1 to Cn. In the example shown in Fig. 5, vehicle 100 is equipped with front camera C1, left front camera C2, right front camera C3, and rear camera C4. The priorities of cameras C1 to C4 depend on the traveling direction of vehicle 100. In other words, the priority of camera C having a viewing direction close to the traveling direction is set higher than the priority of camera C having a viewing direction farther from the traveling direction. Circled numbers in Fig. 5 indicate the priorities.

[0047] Specifically, when the vehicle 100 travels straight, the traveling direction of the vehicle 100 is forward. In this case, the front camera C1 has the highest priority, and the rear camera C4 has the lowest priority. The left front camera C2 and the right front camera C3 have lower priorities than the front camera C1 and higher priorities than the rear camera C4.

[0048] When the vehicle 100 turns left, the traveling direction of the vehicle 100 is to the left. In this case, the priority of the left front camera C2 is higher than the priority of the right front camera C3. The priority of the front camera C1 is lower than the left front camera C2 and higher than the right front camera C3. The priority of the rear camera C4 is the lowest.

[0049] When the vehicle 100 turns right, the traveling direction of the vehicle 100 is to the right. In this case, the priority of the right front camera C3 is higher than the priority of the left front camera C2. The priority of the front camera C1 is lower than the right front camera C3 and higher than the left front camera C2. The priority of the rear camera C4 is the lowest.

[0050] FIG. 6 is a conceptual diagram for explaining an example of a data amount adjustment process taking into consideration the priorities of the cameras C1 to C4.

[0051] When the vehicle 100 travels straight, the data amount adjustment unit 105 maintains the resolution of the video data S1 obtained by the front camera C1, which has the highest priority, at the default value. In other words, the data amount adjustment unit 105 maintains the transmission data amount of the video data S1 obtained by the front camera C1, which has the highest priority, without reducing it. On the other hand, the data amount adjustment unit 105 reduces the resolution of the video data S2 to S4 obtained by the cameras C2 to C4, which have relatively low priorities, below the default value. For example, the resolution is reduced from the default value of 1080p to 360p.

[0052] When the vehicle 100 turns left, the data amount adjustment unit 105 maintains the resolution of the video data S1, S2 obtained by the cameras C1, C2, which have a relatively high priority, at the default value. In other words, the data amount adjustment unit 105 maintains the transmission data amount of the video data S1, S2 obtained by the cameras C1, C2, which have a relatively high priority, without reducing it. On the other hand, the data amount adjustment unit 105 reduces the resolution of the video data S3 obtained by the right front camera C3, which has a relatively low priority, below the default value. Furthermore, the data amount adjustment unit 105 stops transmission of the video data S4 obtained by the rear camera C4, which has the lowest priority.

[0053] When the vehicle 100 turns right, the data amount adjustment unit 105 maintains the resolution of the video data S1, S3 obtained by the cameras C1, C3, which have a relatively high priority, at the default value. In other words, the data amount adjustment unit 105 maintains the transmission data amount of the video data S1, S3 obtained by the cameras C1, C3, which have a relatively high priority, without reducing it. On the other hand, the data amount adjustment unit 105 reduces the resolution of the video data S2 obtained by the left front camera C2, which has a relatively low priority, below the default value. Furthermore, the data amount adjustment unit 105 stops transmission of the video data S4 obtained by the rear camera C4, which has the lowest priority.

[0054] 7 is a conceptual diagram for explaining another example of the data amount adjustment process taking into consideration the priorities of the cameras C1 to C4. In the example shown in FIG. 7, the amount of data reduction is greater than in the example shown in FIG.

[0055] When the vehicle 100 travels straight, the data amount adjustment unit 105 maintains the resolution of the video data S1 obtained by the front camera C1 with the highest priority at the default value. In other words, the data amount adjustment unit 105 maintains the transmission data amount of the video data S1 obtained by the front camera C1 with the highest priority without reducing it. On the other hand, the data amount adjustment unit 105 stops transmission of the video data S2 to S4 obtained by the cameras C2 to C4 with relatively low priorities.

[0056] When the vehicle 100 turns left, the data amount adjustment unit 105 reduces the resolution of the video data S1 and S2 obtained by the cameras C1 and C2, which have a relatively high priority, below the default value. Furthermore, the data amount adjustment unit 105 stops transmission of the video data S3 and S4 obtained by the cameras C3 and C4, which have a relatively low priority.

[0057] When the vehicle 100 turns right, the data amount adjustment unit 105 reduces the resolution of the video data S1 and S3 obtained by the cameras C1 and C3, which have a relatively high priority, below the default value. Furthermore, the data amount adjustment unit 105 stops transmission of the video data S2 and S4 obtained by the cameras C2 and C4, which have a relatively low priority.

[0058] The traveling direction of the vehicle 100 is obtained from vehicle state information such as the steering angle, blinker state, and shift position of the vehicle 100. Alternatively, the traveling direction of the vehicle 100 may be obtained based on a target route of the vehicle 100. The traveling direction information DIR is information indicating the traveling direction of the vehicle 100. The priority policy information POL indicates a dynamic setting policy of priority as exemplified in FIG. 5. The priority setting unit 105P obtains the traveling direction information DIR, and sets priorities of the multiple cameras C1 to Cn based on the traveling direction information DIR and the priority policy information POL (see FIG. 4). Then, the data amount adjustment unit 105 executes a data amount adjustment process taking into account the set priorities.

[0059] In this way, the data amount adjustment process is executed so that the amount of transmission data of the video data S obtained by the camera C with a higher priority is increased. That is, it is possible to ensure the maximum possible amount of transmission data for the video data S obtained by the camera C with a higher priority while suppressing the total data amount Y of the video data S1 to Sn. That is, it is possible to suppress a transmission delay of the video data S1 to Sn while ensuring the accuracy of the remote operation of the vehicle 100 by the remote operator O as much as possible.

[0060] 4. Data volume adjustment process based on FIX level As described above, the excess data amount X of the multiple video data S1 to Sn is expressed by formula (1). The determining unit 104 calculates the excess data amount X according to formula (1). As the excess data amount X increases, the amount of transmission data to be reduced in the data amount adjustment process also increases. Here, the amount of transmission data to be reduced, i.e., the content of the data amount adjustment process, may be switched in stages. For this purpose, the concept of "Fix level FL" is introduced.

[0061] FIG. 8 is a conceptual diagram for explaining an example of setting the Fix level. When the excess data amount X is a positive value, the communication speed Z is low and the need for data amount adjustment processing is high. On the other hand, when the excess data amount X is a negative value, the communication speed Z is high and the need for data amount adjustment processing is low. From this perspective, the excess data amount X is classified into a plurality of Fix levels FL.

[0062] For example, consider three thresholds X0, XN, and XP related to the excess data amount X. The threshold XN is a negative value, and the threshold XP is a positive value. The threshold X0 is greater than XN and less than XP (XP>X0>XN). The threshold X0 may be zero. If the excess data amount X is in the range of XN to X0, the fix level FL is set to a default value of "0". If the excess data amount X is in the range of X0 to XP, the fix level FL is set to "-1". If the excess data amount X is equal to or greater than XP, the fix level FL is set to "-2". On the other hand, if the excess data amount X is equal to or less than XN, the fix level FL is set to "+1".

[0063] When the fix level FL is the default value "0", the data amount adjustment process is not performed. When the fix level FL is "-1" or "-2", the data amount adjustment process is performed. In particular, the data reduction amount when the fix level FL is "-2" is set to be greater than when the fix level FL is "-1". For example, when the fix level FL is "-1", the data amount adjustment process as shown in FIG. 6 is performed, and when the fix level FL is "-2", the data amount adjustment process as shown in FIG. 7 is performed. In this way, the contents of the data amount adjustment process are switched stepwise according to the fix level FL.

[0064] 9 is a block diagram showing an example of a functional configuration related to the Fix level FL. The judgment unit 104 includes a level judgment unit 104A. The level judgment unit 104A judges to which of a plurality of Fix levels FL the excess data amount X belongs. Judgment result information RES indicating the judgment result by the judgment unit 104 includes the Fix level FL to which the excess data amount X belongs.

[0065] The data amount adjustment unit 105 switches the content of the data amount adjustment process according to the Fix level FL to which the excess data amount X belongs. For example, when the Fix level FL is "-1", the data amount adjustment unit 105 performs the data amount adjustment process as shown in Fig. 6. When the Fix level FL is "-2", the data amount adjustment unit 105 performs the data amount adjustment process as shown in Fig. 7.

[0066] When the Fix level is "+1", the communication speed Z has a considerable margin. Therefore, the data amount adjustment unit 105 may increase the resolution of at least one of the multiple video data S1 to Sn above the default value. For example, the data amount adjustment unit 105 increases the resolution of the video data S obtained by the camera C with the highest priority above the default value. This further improves the visibility of the video data S.

[0067] The determination unit 104 may further include a level transition adjustment unit 104B. The level transition adjustment unit 104B adjusts the slope (rate of change) of the Fix level FL when the Fix level FL to which the excess data amount X belongs transitions (changes).

[0068] 10 is a conceptual diagram for explaining transition examples of the Fix level FL. In the first example, the Fix level FL is greatly decreased. In this case, for safety reasons, the level transition adjustment unit 104B immediately decreases the Fix level FL.

[0069] In the second example, the amount of decrease in the fix level FL is relatively small. In this case, the level transition adjustment unit 104B may decrease the fix level FL slowly. When the fix level FL decreases slowly, the resolution of the video data S also decreases slowly.

[0070] In the third example, the fix level FL is increased. If the resolution of the video data S is suddenly improved, the remote operator O may feel uncomfortable. Therefore, the level transition adjustment unit 104B may increase the fix level FL slowly. When the fix level FL is increased slowly, the resolution of the video data S also increases slowly.

[0071] In this way, the content of the data amount adjustment process is switched according to the Fix level FL. Since the content of the data amount adjustment process is determined in advance, it is possible to perform the data amount adjustment process efficiently.

[0072] 5. Vehicle Examples 5-1. Configuration example 11 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0073] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300. This communication device 110 corresponds to the communication unit 101 shown in FIG.

[0074] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera C (C1 to Cn), a LIDAR (Laser Imaging Detection and Ranging), a radar, etc. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0075] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0076] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processor 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 executes various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various information required for processing by the processor 160. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The control device 150 may include one or more ECUs (Electronic Control Units).

[0077] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 are realized as a result of the processor 160 executing the vehicle control program PROG1. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0078] 5-2. Driving environment information The control device 150 uses the sensor group 120 to acquire driving environment information ENV indicating the driving environment of the vehicle 100. The driving environment information ENV is stored in the storage device 170.

[0079] The driving environment information ENV includes surrounding situation information indicating the recognition result by the recognition sensor. For example, the surrounding situation information includes images IMG (video data S1 to Sn) captured by the camera C. The surrounding situation information may include object information related to objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, traffic signals, signs, roadside structures, etc. The object information indicates the relative position and relative speed of the object with respect to the vehicle 100.

[0080] The driving environment information ENV also includes vehicle state information indicating the vehicle state detected by the vehicle state sensor.

[0081] Furthermore, the driving environment information ENV includes vehicle position information indicating the position and direction of the vehicle 100. The vehicle position information is obtained by a position sensor. Highly accurate vehicle position information may be obtained by a self-position estimation process (localization) using map information and surrounding situation information (object information).

[0082] 5-3.Vehicle driving control The control device 150 executes vehicle driving control for controlling the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes the vehicle driving control by controlling the driving device 130 (the steering device, the drive device, and the braking device).

[0083] The control device 150 may perform automatic driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0084] 5-4. Processing related to remote control The following describes a case where remote operation of the vehicle 100 is performed. The control device 150 communicates with the remote operator terminal 200 via the communication device 110.

[0085] The control device 150 transmits the vehicle information VCL to the remote operator terminal 200. The vehicle information VCL is information necessary for remote operation by the remote operator O, and includes at least a part of the driving environment information ENV described above. For example, the vehicle information VCL includes surrounding situation information (particularly, images IMG). The vehicle information VCL may further include vehicle state information and vehicle position information.

[0086] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle travel control according to the received remote operation information OPE.

[0087] Furthermore, the control device 150 has the functions of the video data acquisition unit 102, communication status acquisition unit 103, judgment unit 104, and data amount adjustment unit 105 described above (see Figs. 2, 4, and 8). The priority policy information POL is generated in advance and stored in the storage device 170. The traveling direction information DIR is obtained from vehicle status information (e.g., steering angle, blinker status, shift position). The traveling direction information DIR may be obtained based on the target route of the vehicle 100. The control device 150 executes the process shown in Fig. 3, and executes data amount adjustment processing as necessary. [Explanation of symbols]

[0088] 1 Remote Control System 100 vehicles 101 Communications Department 102 Video data acquisition unit 103 Communication status acquisition unit 104 Judgment section 104A Level Judgment Section 104B Level transition adjustment section 105 Data volume adjustment unit 105P Priority setting section 110 Communication equipment 120 Sensors 130 Running gear 150 Control device 160 processors 170 Storage device 200 Remote operator terminal 300 Management device C1~Cn Camera DIR Direction information FL Fix Level IMG Image OPE Remote Operation Information POL Priority Policy Information RES judgment result information S1~Sn video data VCL vehicle information

Claims

1. A control device that controls a moving object that is a target of remote operation by a remote operator and has a plurality of cameras, One or more processors; The one or more processors: A process of acquiring a plurality of pieces of video data obtained by each of the plurality of cameras as data to be transmitted to a remote operator terminal on the remote operator side; obtaining information on a communication speed of the communication from the mobile unit to the remote operator terminal; A process of calculating an excess amount of data from the communication speed of a default value of a total data amount per second of the plurality of video data; a data amount adjustment process for reducing a transmission data amount of at least one of the plurality of video data items so that the total data amount per second of the plurality of video data items is kept equal to or less than the communication speed when the data excess amount is a positive value; configured to run The plurality of levels of data overage amount include: a default level representing that the data excess amount is within a negative predetermined range; a first level representing that the data excess amount is within a first positive range; a second level indicating that the data excess amount is within a second positive range that is greater than the first range; and Including, The one or more processors further include: determining which of the plurality of levels the excess data amount belongs to; When the data excess amount transitions from the default level to the second level, the data amount adjustment process is executed so that the total data amount is suppressed to be equal to or less than the communication speed within a second period. When the data excess amount transitions from the default level to the first level, the data amount adjustment process is executed so that the total data amount is suppressed to be equal to or less than the communication speed within a first period that is longer than the second period. It was configured as Control device.

2. The control device according to claim 1 , The camera system further includes one or more storage devices for storing priority policy information indicating a priority setting policy for the plurality of cameras, The one or more processors execute the data amount adjustment process based on the priority policy information so that the transmission data amount of the video data obtained by the camera with a higher priority is increased. Control device.

3. The control device according to claim 2, the plurality of cameras includes a first camera and a second camera having a lower priority than the first camera, the plurality of video data include first video data obtained by the first camera and second video data obtained by the second camera, In the data amount adjustment process, the one or more processors reduce a resolution of the second video data below a default value without reducing the transmission data amount of the first video data. Control device.

4. The control device according to claim 2, the plurality of cameras includes a first camera and a second camera having a lower priority than the first camera, the plurality of video data include first video data obtained by the first camera and second video data obtained by the second camera, In the data amount adjustment process, the one or more processors transmit the first video data and stop transmitting the second video data. Control device.

5. The control device according to any one of claims 2 to 4, the priorities of the cameras depend on a traveling direction of the moving object; the priority of a camera having a line of sight closer to the direction of travel is higher than the priority of a camera having a line of sight further from the direction of travel; In the data amount adjustment process, the one or more processors acquire information including the traveling direction of the moving object, and set the priorities of the multiple cameras based on the traveling direction and the priority policy information. Control device.

6. A control device as claimed in claim 1, The one or more processors further include: When the data excess amount transitions from the second level to the first level, increasing the total data amount within a third period; When the excess amount of data transitions from the first level to the default level, the total amount of data is increased within a fourth period that is longer than the third period. It was configured as Control device.

7. A control method for controlling a moving object that is a target of remote operation by a remote operator and that is equipped with a plurality of cameras, comprising the steps of: A process of acquiring a plurality of pieces of video data obtained by each of the plurality of cameras as data to be transmitted to a remote operator terminal on the remote operator side; obtaining information on a communication speed of the communication from the mobile unit to the remote operator terminal; A process of calculating an excess amount of data from the communication speed of a default value of a total data amount per second of the plurality of video data; a data amount adjustment process for reducing a transmission data amount of at least one of the plurality of video data items so that the total data amount per second of the plurality of video data items is kept equal to or less than the communication speed when the data excess amount is a positive value; Including, The plurality of levels of data overage amount include: a default level representing that the data excess amount is within a negative predetermined range; a first level representing that the data excess amount is within a first positive range; a second level indicating that the data excess amount is within a second positive range that is greater than the first range; and Including, The control method further comprises: determining which of the plurality of levels the excess amount of data belongs to; When the data excess amount transitions from the default level to the second level, executing the data amount adjustment process so that the total data amount is suppressed to be equal to or less than the communication speed within a second period; When the excess data amount transitions from the default level to the first level, the data amount adjustment process is executed so that the total data amount is suppressed to be equal to or less than the communication speed within a first period that is longer than the second period. Includes Control methods.

8. A control program for controlling a moving object that is a target of remote operation by a remote operator and has a plurality of cameras, The control program is executed by a computer, A process of acquiring a plurality of pieces of video data obtained by each of the plurality of cameras as data to be transmitted to a remote operator terminal on the remote operator side; obtaining information on a communication speed of the communication from the mobile unit to the remote operator terminal; A process of calculating an excess amount of data from the communication speed of a default value of a total data amount per second of the plurality of video data; a data amount adjustment process for reducing a transmission data amount of at least one of the plurality of video data items so that the total data amount per second of the plurality of video data items is kept equal to or less than the communication speed when the data excess amount is a positive value; causing the computer to execute The plurality of levels of data overage amount include: a default level representing that the data excess amount is within a negative predetermined range; a first level representing that the data excess amount is within a first positive range; a second level indicating that the data excess amount is within a second positive range that is greater than the first range; and Including, The control program further comprises: determining which of the plurality of levels the excess amount of data belongs to; When the data excess amount transitions from the default level to the second level, executing the data amount adjustment process so that the total data amount is suppressed to be equal to or less than the communication speed within a second period; When the excess data amount transitions from the default level to the first level, the data amount adjustment process is executed so that the total data amount is suppressed to be equal to or less than the communication speed within a first period that is longer than the second period. causing the computer to execute Control program.

Images