Information processing system and information processing method

Abstract

To enable a moving body to continue its operation regardless of fluctuations in network conditions.SOLUTION: An information processing system includes a moving body control device 1 that controls a moving body V, and an information processing device 2. The moving body control device 1 transmits status data indicating the status of a network N to the information processing device 2, and transmits image data corresponding to image parameters received from the information processing device 2 to the information processing device 2. The information processing device 2 transmits to the moving body control device 1 image parameters determined based on the status of the network N indicated by the status data, and reference data based on the received image data. The moving body control device 1 changes a control method for the moving body V based on the reference data depending on the status of the network N.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system and an information processing method for controlling a moving object. [Background technology]

[0002] Conventionally, a system is known in which image data captured by an unmanned vehicle is transmitted to a cloud server via a network, and the cloud server controls the unmanned vehicle based on the results of analyzing the images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-023969 Summary of the Invention [Problem to be solved by the invention]

[0004] The bandwidth available for data transmission in a network fluctuates. When the bandwidth becomes insufficient, image data captured by a moving object cannot be transmitted to the cloud server properly, which causes a problem in that the cloud server cannot properly control the moving object, preventing it from continuing to operate.

[0005] The present invention has been made in consideration of these points, and has an object to enable a mobile object to continue operating regardless of fluctuations in the network state. [Means for solving the problem]

[0006] An information processing system of a first aspect of the present invention comprises a mobile body control device that controls a mobile body, and an information processing device that communicates with the mobile body control device via a network, wherein the mobile body control device has a data acquisition unit that acquires captured image data generated by an imaging unit provided on the mobile body, a status data creation unit that creates status data indicating a state of the network based on a reception status of reference data transmitted by the information processing device to the mobile body control device via the network, a transmission processing unit that transmits transmission image data based on the captured image data to the information processing device based on image parameters transmitted from the information processing device after transmitting the status data to the information processing device, and a mobile body control unit that controls the mobile body, wherein the information processing device has a data receiving unit that receives the status data and the transmission image data, a parameter determination unit that determines the image parameters to be transmitted to the mobile body control device based on the state of the network indicated by the status data, a reference data generation unit that generates the reference data corresponding to the state of the surroundings of the mobile body based on the transmission image data, and a data transmission unit that transmits the image parameters and the reference data to the mobile body control device, and the mobile body control unit changes a control method of the mobile body based on the reference data depending on the state of the network.

[0007] The data transmission unit may transmit the reference data indicating information regarding the position of other objects within a predetermined range of the moving body to the moving body control device, and the moving body control unit may control the moving body based on the position of the other objects indicated by the reference data.

[0008] The mobile object control unit may control the mobile object based on the captured image data while the reference data is not transmitted from the information processing device.

[0009] The information processing device may further have a memory unit that stores the state of the network and the value of an image parameter in association with each other, and the parameter determination unit may select the image parameter stored in the memory unit in association with the state of the network indicated by the state data.

[0010] The status data is data indicating the bandwidth of the network, and the data transmission unit may transmit the image parameters in such a way that the smaller the bandwidth indicated by the status data, the smaller the amount of data of the transmission image data that the mobile control device transmits to the information processing device.

[0011] The status data is data indicating an amount of transmission delay in the network, and the data transmission unit may transmit the image parameters with a smaller amount of data of the transmission image data that the mobile body control device transmits to the information processing device, the larger the amount of transmission delay indicated by the status data.

[0012] The status data may be the bandwidth of the network, and the data transmission unit may transmit transmission stop command data to the mobile control device to stop transmission of the transmission image data when the bandwidth indicated by the status data is below a bandwidth threshold.

[0013] The status data is data indicating an amount of transmission delay in the network, and the data transmission unit may transmit stop command data to the mobile control device to stop transmitting the transmission image data when the amount of transmission delay indicated by the status data is equal to or greater than a delay threshold.

[0014] The reference data includes time data indicating the time measured in the information processing device, and the status data creation unit may use the difference between the time indicated by the time data and the time measured in the mobile control device as the reception status, and create the status data indicating that the larger the difference, the smaller the bandwidth of the network or the larger the amount of transmission delay of the network.

[0015] The parameter determination unit may determine new image parameters corresponding to a larger amount of data than the image parameters previously determined when the network state satisfies a predetermined condition for a predetermined period of time or longer.

[0016] An information processing method of a second aspect of the present invention is an information processing method executed by a mobile body control device that controls a moving body and an information processing device that communicates with the mobile body control device via a network, and includes the steps of: the mobile body control device acquiring captured image data generated by an imaging unit provided in the moving body; the information processing device transmitting reference data to the mobile body control device via the network; the mobile body control device creating status data indicating the state of the network based on a reception status of the reference data; the information processing device determining image parameters of transmission image data to be transmitted from the mobile body control device to the information processing device based on the state of the network indicated by the status data; the information processing device transmitting the determined image parameters to the mobile body control device; the mobile body control device transmitting the transmission image data based on the captured image data with the image parameters to the information processing device; the information processing device transmitting reference data corresponding to the state of the surroundings of the moving body, generated based on the transmission image data, to the information processing device; and the mobile body control device changing a control method of the moving body based on the reference data depending on the state of the network. Effect of the Invention

[0017] According to the present invention, an advantage is achieved in that a mobile object can continue to operate regardless of fluctuations in the network state. [Brief description of the drawings]

[0018] [Figure 1] 1 is a diagram for explaining an overview of an information processing system S. [Diagram 2] 1 is a diagram showing a configuration of a mobile object control device 1. FIG. [Diagram 3] FIG. 2 is a diagram showing a configuration of an information processing device 2. [Figure 4] 4 is a flowchart showing a flow of processing in the mobile object control device 1. [Diagram 5] 13 is a flowchart showing the flow of processing by the mobile object control device 1 during standalone processing. [Figure 6] 13 is a flowchart showing the flow of processing of the mobile object control device 1 while executing cloud-based processing. [Figure 7] 13 is a flowchart showing a process flow in the information processing device 2. [Figure 8] 13 is a flowchart showing a process flow of the information processing device 2 executing stand-alone processing. [Figure 9] 13 is a flowchart showing a process flow of an information processing device 2 executing cloud-based processing. [Figure 10] 13 is a flowchart of a process in which the parameter determination unit 232 determines a parameter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] [Outline of Information Processing System S] FIG. 1 is a diagram for explaining an overview of an information processing system S. The information processing system S includes a mobile body control device 1 and an information processing device 2. The mobile body control device 1 is mounted on a mobile body V that travels based on the control of the mobile body control device 1. The mobile body V is, for example, a robot that performs work while moving, but the purpose and shape of the mobile body V are arbitrary. The mobile body control device 1 may be integrated with the mobile body V or may be separate from the mobile body V.

[0020] The mobile object control device 1 transmits and receives various data to and from the information processing device 2 via a network N. The network N includes, for example, the Internet, a wired communication line, or a wireless communication line. The mobile object control device 1 transmits and receives data via a wireless communication line L1, and the information processing device 2 transmits and receives data via a communication line L2. The communication line L2 is, for example, an optical line, but may also be a wireless communication line. The mobile object control device 1 and the information processing device 2 operate in synchronization using time stamps.

[0021] The mobile object control device 1 controls the traveling of the mobile object V. The mobile object control device 1 controls the mobile object V using, for example, captured image data (e.g., video data) showing an image of the surroundings captured by a camera possessed by the mobile object V, or sensor data showing the state of the surroundings of the mobile object V detected by a sensor possessed by the mobile object V. The mobile object control device 1 can also control the mobile object V using control data generated based on reference data received from the information processing device 2.

[0022] The reference data is data corresponding to the conditions around the moving body V, and is used by the moving body control device 1 to control the moving body V. The reference data is, for example, data indicating the type and position of an object present around the moving body V, or the speed or direction in which the moving body V should move, determined based on the conditions around the moving body V, and is used to prevent the moving body V from colliding with other objects. The reference data includes time data indicating the time measured by the information processing device 2. Note that, although the following description illustrates an example in which the moving body V outputs video data, the moving body V may also output still image data.

[0023] The information processing device 2 is, for example, a server on a cloud, and transmits and receives data between a plurality of mobile body control devices 1 via the network N. Based on the video data and sensor data transmitted from the mobile body control device 1, the information processing device 2 transmits to the mobile body control device 1 reference data including information for preventing the mobile body V from colliding with other mobile bodies V, and transmits to the mobile body control device 1 reference data including information for causing the mobile body V to execute various processes.

[0024] As shown in Fig. 1, video data transmitted from a moving body V to the moving body control device 1 reaches the information processing device 2 via wireless communication line L1 and communication line L2. The information processing device 2 generates reference data based on the video data to prevent the moving body V from colliding with other moving bodies or objects, and transmits the generated reference data. The reference data transmitted by the information processing device 2 reaches the moving body control device 1 via communication line L2 and wireless communication line L1. The moving body control device 1 controls the moving body V based on the reference data received from the information processing device 2.

[0025] The mobile object control device 1 and the information processing device 2 are characterized by switching processing depending on the state of the network N. Details will be described later, but for example, the mobile object control device 1 and the information processing device 2 operate as follows when the communication quality of the network N is good, medium, or poor.

[0026] (1) When network N is in good condition The mobile body control device 1 transmits high-quality video data with a high resolution (e.g., 1080p or higher) and a high frame rate (e.g., 30 to 60 fps) and sensor data output by various sensors to the information processing device 2. Based on the high-quality video data and sensor data received from the mobile body control device 1, the information processing device 2 transmits reference data to the mobile body control device 1 for the mobile body V to avoid collision with other mobile bodies and for the mobile body V to execute various processes.

[0027] (2) When the condition of network N is medium The moving body control device 1 transmits low-quality video data with a low resolution (e.g., equal to or higher than 720p and lower than 1080p) and a low frame rate (e.g., lower than 30 fps) to the information processing device 2. Based on the low-quality video data received from the moving body control device 1, the information processing device 2 transmits reference data to the moving body control device 1 for ensuring a minimum level of safety for the moving body V to avoid collision with other moving bodies.

[0028] (3) When the network N status is poor The mobile object control device 1 analyzes low-quality video data with low resolution (for example, less than 720p) and low frame rate (for example, less than 30 fps), and controls the mobile object V so that the mobile object V does not collide with other mobile objects. The mobile object control device 1 does not transmit the video data to the information processing device 2, and the information processing device 2 does not transmit reference data based on the video data to the mobile object control device 1.

[0029] The configurations and operations of the mobile body control device 1 and the information processing device 2 will be described in detail below. In the following description, the process in which the mobile body control device 1 controls the mobile body V based on the reference data transmitted by the information processing device 2 to the mobile body control device 1, as in (1) or (2) above, is referred to as cloud-type processing. Also, the process in which the information processing device 2 does not transmit reference data based on video data to the mobile body control device 1, as in (3) above, and the mobile body control device 1 controls the mobile body V based on video data generated by an imaging unit (e.g., a camera) provided on the mobile body V, is referred to as standalone processing.

[0030] [Configuration of mobile object control device 1] 2 is a diagram showing the configuration of the mobile object control device 1. The mobile object control device 1 has a first communication unit 11, a second communication unit 12, a storage unit 13, and a control unit 14. The control unit 14 has a data acquisition unit 141, a mobile object control unit 142, a status data creation unit 143, and a transmission processing unit 144.

[0031] The first communication unit 11 has a communication interface for transmitting and receiving data to and from the moving body V. The first communication unit 11 receives, for example, video data generated by a camera of the moving body V and sensor data generated by a sensor of the moving body V. The first communication unit 11 inputs the received video data and sensor data to the data acquisition unit 141. In addition, the first communication unit 11 acquires reference data for controlling the moving body V from the moving body control unit 142, and transmits control data generated based on the acquired reference data to the moving body V.

[0032] The second communication unit 12 has a communication interface for transmitting and receiving data to and from the information processing device 2 via the network N. The second communication unit 12 transmits, for example, video data received from a moving object V. In addition, the second communication unit 12 receives reference data received from the information processing device 2 via the network N.

[0033] The storage unit 13 includes storage media such as a Read Only Memory (ROM) and a Random Access Memory (RAM), etc. The storage unit 13 stores a program executed by the control unit 14.

[0034] The control unit 14 has, for example, a CPU (Central Processing Unit). The control unit 14 executes the programs stored in the storage unit 13 to function as a data acquisition unit 141, a mobile object control unit 142, a status data creation unit 143, and a transmission processing unit 144.

[0035] The data acquisition unit 141 acquires video data, which is captured image data generated by a camera provided on the moving object V, via the first communication unit 11. When the moving object V transmits sensor data, the data acquisition unit 141 also acquires the sensor data via the first communication unit 11. The data acquisition unit 141 inputs the acquired video data and sensor data to the moving object control unit 142 and the transmission processing unit 144.

[0036] The mobile object control unit 142 controls the mobile object V based on the reference data received from the information processing device 2. The mobile object control unit 142 changes the control method of the mobile object V depending on the state of the network N. For example, when the state of the network N is good and cloud-based processing is executed, the mobile object control unit 142 controls the mobile object V by transmitting control data based on data indicating the speed and direction of the mobile object V indicated by the reference data to the mobile object V. When the state of the network N is not good and standalone processing is executed, the mobile object control unit 142 stops or decelerates the mobile object V when the reference data indicates that there is another object within a predetermined distance from the mobile object V, and maintains the speed and direction of the mobile object V when there is no other object within the predetermined distance.

[0037] While executing the standalone processing, the moving object control unit 142 may control the moving object V by transmitting control data indicating the speed or traveling direction of the moving object V to the moving object V based on at least one of the video data and the sensor data input from the data acquisition unit 141. In other words, while reference data including data indicating instructions on the speed and direction of the moving object V is not transmitted from the information processing device 2, the moving object control unit 142 may control the moving object V based on the video data or the sensor data input from the data acquisition unit 141.

[0038] The status data creation unit 143 creates status data indicating the status of the network N. The status of the network N is represented by, for example, the bandwidth, transmission delay time, amount of jitter, or packet loss rate of the network N. The status data creation unit 143 creates the status data by specifying the bandwidth or amount of transmission delay using ping or iperf.

[0039] The status data creation unit 143 may create status data indicating the status of the network N based on the reception status of the reference data transmitted by the information processing device 2 to the mobile object control device 1 via the network N. For example, the status data creation unit 143 uses the difference between the transmission time indicated by the time data included in the reference data and the time measured by the mobile object control device 1 as the reception status, and creates status data indicating that the larger this difference is, the smaller the bandwidth of the network N is or the larger the transmission delay amount of the network N is. Specifically, the status data creation unit 143 identifies a transmission delay time based on the difference between the transmission time included in the reference data and the time when the second communication unit 12 received the reference data, and creates status data indicating the identified transmission delay time. The status data creation unit 143 may create status data indicating the identified jitter amount based on the amount of fluctuation in this time difference.

[0040] The status data creating unit 143 may specify a bandwidth based on the amount of data received per unit time and create status data indicating the specified bandwidth. The status data creating unit 143 may specify a packet loss rate by specifying reference data that was transmitted but not received based on a serial number included in the reference data, and create status data indicating the specified packet loss rate.

[0041] The transmission processing unit 144 creates transmission image data based on the video data acquired from the moving body V based on the image parameters transmitted from the information processing device 2 after transmitting the status data to the information processing device 2, and transmits the created transmission image data to the information processing device 2. The transmission image data is video data created by the transmission processing unit 144 by changing at least one of the number of pixels and the frame rate of the video data acquired from the moving body V based on the image parameters transmitted from the information processing device 2.

[0042] [Configuration of information processing device 2] 3 is a diagram showing a configuration of the information processing device 2. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 has a data receiving unit 231, a parameter determining unit 232, a reference data generating unit 233, and a data transmitting unit 234.

[0043] The communication unit 21 has a communication interface for transmitting and receiving data to and from the mobile body control device 1. The communication unit 21 receives, for example, video data from the mobile body control device 1 and inputs the received data to the data receiving unit 231. The communication unit 21 transmits the image parameters and reference data input from the data transmitting unit 234 to the mobile body control device 1.

[0044] The storage unit 22 has storage media such as a ROM, a RAM, and an SSD. The storage unit 22 stores a program executed by the control unit 23. The storage unit 22 also stores data required to determine image parameters to be notified to the mobile object control device 1. The storage unit 22 stores, for example, the state of the network N and the value of the image parameter in association with each other.

[0045] Specifically, the storage unit 22 stores parameter determination data in which the bandwidth, transmission delay time, amount of jitter or packet loss rate of the network N are associated with image parameters such as the number of pixels per frame (resolution) and frame rate of video data to be transmitted to the mobile object control device 1. In the parameter determination data, a combination of the resolution and frame rate may be associated with at least one of the allowable bandwidth, amount of transmission delay and amount of jitter.

[0046] The storage unit 22 may store a plurality of parameter determination data each having different image parameters in association with a user who uses the mobile body control device 1. The storage unit 22 may store a plurality of parameter determination data each having different image parameters in association with a type of the mobile body control device 1.

[0047] The control unit 23 includes, for example, a CPU. The control unit 23 executes the programs stored in the storage unit 22 to function as a data receiving unit 231, a parameter determining unit 232, a reference data generating unit 233, and a data transmitting unit 234.

[0048] The data receiving unit 231 receives various data transmitted from the mobile object control device 1 via the communication unit 21. The data receiving unit 231 receives, for example, status data and transmission image data (i.e., video data) transmitted by the mobile object control device 1. The data receiving unit 231 inputs the status data to the parameter determining unit 232 and inputs the video data to the reference data generating unit 233.

[0049] The parameter determination unit 232 determines the image parameters to be transmitted to the mobile object control device 1 based on the state of the network N indicated by the state data. The parameter determination unit 232 inputs the determined image parameters to the data transmission unit 234.

[0050] The parameter determination unit 232 determines image parameters so that the amount of video data transmitted by the mobile object control device 1 becomes smaller, for example, as the bandwidth indicated by the status data becomes smaller, the transmission delay time becomes larger, or the packet loss rate becomes larger. Specifically, the parameter determination unit 232 determines image parameters so that at least one of the number of pixels per frame and the frame rate becomes smaller as the status of the network N becomes worse, by selecting the number of pixels per frame and the frame rate corresponding to the status of the network N by referring to the parameter determination data stored in the storage unit 22. When the status of the network N is so bad that it is necessary to have the mobile object control device 1 stop transmitting the video data, the parameter determination unit 232 determines to stop transmitting the video data.

[0051] When the state of network N continues to satisfy a predetermined condition for a predetermined time or more, parameter determination unit 232 determines new image parameters corresponding to a larger amount of data than previously determined image parameters. For example, parameter determination unit 232 sets a counter to 0 when the image parameters are determined, and increments the counter value at predetermined time intervals when the bandwidth of network N is larger than the bandwidth required to transmit video data based on the image parameters. When the counter value reaches a predetermined value (for example, a value corresponding to 10 seconds), parameter determination unit 232 changes to a new image parameter.

[0052] When a plurality of parameter determination data corresponding to a user are stored in the storage unit 22, the parameter determination unit 232 may determine image parameters by referring to the parameter determination data associated with the user of the mobile body control device 1 that transmitted the status data. The parameter determination unit 232 notifies the determined image parameters to the mobile body control device 1 of the user corresponding to the referenced parameter determination data via the data transmission unit 234. By operating the parameter determination unit 232 in this manner, when the usage fee for communication services differs depending on the user of the mobile body control device 1, it is possible to transmit higher quality image data to the mobile body control device 1 of a user who charges a higher usage fee, and to execute cloud-type processing as much as possible.

[0053] When multiple parameter determination data corresponding to the type of the mobile body control device 1 are stored in the storage unit 22, the parameter determination unit 232 determines image parameters by referring to the parameter determination data associated with the type of the mobile body control device 1 that transmitted the status data. The parameter determination unit 232 identifies the type of the mobile body control device 1 by referring to the data indicating the type of the mobile body control device 1 stored in the storage unit 22, and selects the number of pixels and frame rate corresponding to the status data in the parameter determination data associated with the identified type. By operating the parameter determination unit 232 in this manner, a type of mobile body control device 1 that has difficulty in performing standalone processing can continue transmitting high-quality video data, making it easier for the mobile body control device 1 and the information processing device 2 to continue cloud-type processing.

[0054] The reference data generating unit 233 generates reference data for controlling the moving body V based on the video data transmitted from the moving body control device 1. While the cloud-based processing is being executed, the reference data generating unit 233 identifies objects within a predetermined distance from the moving body V, for example, by analyzing the video data. The reference data generating unit 233 may identify the traveling direction of the moving body V and other objects based on changes in the image shown by the video data, and identify objects that are expected to be located in the traveling direction of the identified moving body V.

[0055] The reference data generating unit 233 generates reference data indicating the direction of the identified object relative to a reference direction (e.g., the forward direction) of the moving body V and the distance from the moving body V to the object. The reference data generating unit 233 may generate reference data indicating the speed of the moving body V required to avoid a collision between the moving body V and the object and the direction in which the moving body V should proceed. The reference data generating unit 233 inputs the generated reference data to the data transmitting unit 234.

[0056] While the stand-alone process is being executed and no video data is being transmitted from the mobile object control device 1, the reference data generating unit 233 transmits reference data that is not based on the video data and that is obtained, for example, from an external device, and indicates the environment around the mobile object V. This reference data includes, for example, information indicating a traffic jam state or information indicating weather conditions.

[0057] The data transmission unit 234 transmits parameter information including the image parameters input from the parameter determination unit 232 and the reference data input from the reference data generation unit 233 to the mobile object control device 1. The data transmission unit 234 transmits the image parameters to the mobile object control device 1 when the state of the network N has changed based on the state data, for example.

[0058] The smaller the bandwidth of the network N indicated by the status data, the smaller the image parameters that the mobile body control device 1 transmits to the information processing device 2 are. The larger the transmission delay amount of the network N indicated by the status data, the smaller the image parameters that the mobile body control device 1 transmits to the information processing device 2 are. By operating the data transmission unit 234 in this manner, it is possible to prevent the network N from becoming congested and a state in which many mobile body control devices 1 are unable to transmit video data.

[0059] When the parameter determination unit 232 decides to cause the mobile object control device 1 to stop transmitting the video data, the data transmission unit 234 transmits parameter information including data instructing the mobile object control device 1 to stop transmitting the video data (hereinafter, may be referred to as a "transmission stop command") to the mobile object control device 1. That is, when the bandwidth indicated by the status data is equal to or less than the bandwidth threshold, the data transmission unit 234 transmits transmission stop command data for stopping the transmission of the video data to the mobile object control device. When the transmission delay amount indicated by the status data is equal to or more than the delay threshold, the data transmission unit 234 may transmit stop command data for stopping the transmission of the video data to the mobile object control device 1. By operating the data transmission unit 234 in this manner, it is possible to prevent the mobile object control device 1 from creating video data even though it is unable to transmit the video data, and therefore it is possible to reduce the power consumption of the mobile object control device 1.

[0060] Furthermore, the data transmission unit 234 transmits the reference data input from the reference data generation unit 233 to the mobile body control device 1. The data transmission unit 234 transmits, for example, reference data indicating information on other mobile bodies within a predetermined range from the mobile body V to the mobile body control device 1.

[0061] During execution of cloud-based processing, the data transmission unit 234 transmits reference data generated by the reference data generation unit 233 analyzing the video data, so that even if the image analysis processing capability of the mobile body control device 1 is low, the mobile body control device 1 can control the mobile body V so that the mobile body V does not collide with other objects. Even during execution of standalone processing, the data transmission unit 234 transmits reference data not based on video data to the mobile body control device 1, so that the mobile body control device 1 can determine the content of the operation to be performed by the mobile body V based on traffic congestion information, weather information, and the like indicated by the reference data. Also, even during standalone processing, the status data creation unit 143 can identify the status of the network N based on the reference data.

[0062] [Processing flow in information processing system S] Fig. 4 to Fig. 10 are flowcharts showing the process flow in the information processing system S. Fig. 4 is a flowchart showing the process flow in the mobile body control device 1. Fig. 5 is a flowchart showing the process flow in the mobile body control device 1 while executing stand-alone processing. Fig. 6 is a flowchart showing the process flow in the mobile body control device 1 while executing cloud-based processing.

[0063] Fig. 7 is a flowchart showing the process flow in the information processing device 2. Fig. 8 is a flowchart showing the process flow in the information processing device 2 executing stand-alone processing. Fig. 9 is a flowchart showing the process flow in the information processing device 2 executing cloud-based processing. Fig. 10 is a flowchart showing the process in which the parameter determination unit 232 determines parameters.

[0064] [Processing flow in the mobile object control device 1] First, the flow of processing in the mobile object control device 1 will be described with reference to Fig. 4. The status data creation unit 143 identifies the bandwidth of the network N based on data (e.g., reference data) received from the information processing device 2 (S11). The status data creation unit 143 transmits status data indicating the identified bandwidth to the information processing device 2 (S12). In parallel with these processes, the data acquisition unit 141 receives parameter information transmitted by the information processing device 2 (S13).

[0065] If the parameter information includes a command to stop transmitting the video data, the transmission processing unit 144 determines that the video data cannot be transmitted (NO in S14), and stops transmitting the video data (S15). Then, the mobile control unit 142 transitions to stand-alone processing (S16). If the parameter information does not include a command to stop transmitting the video data, the transmission processing unit 144 determines that the video data can be transmitted (YES in S14), and executes transmission of the video data (S17). Then, the mobile control unit 142 transitions to cloud-based processing (S18).

[0066] Although details will be described later, when the conditions for ending the execution of the cloud-type processing are satisfied, the mobile body control device 1 executes the processing of S15 and transitions to stand-alone type processing (S16). Although not shown in Fig. 4, if the power supply of the mobile body control device 1 is turned off or an instruction to end the operation is received while the mobile body control device 1 is executing the processing shown in Fig. 4, the mobile body control device 1 ends the processing shown in Fig. 4.

[0067] Next, the flow of the standalone processing in the mobile object control device 1 will be described with reference to Fig. 5. The mobile object control unit 142 receives reference data via the second communication unit 12 (S161), and executes various processes based on the reference data (S162). The mobile object control unit 142 executes processes such as stopping the moving object V or reducing the speed of the moving object V, for example.

[0068] Furthermore, the status data creation unit 143 measures the bandwidth and transmission delay time of the network N based on the reference data (S163). The status data creation unit 143 transmits status data indicating the bandwidth and transmission delay time to the information processing device 2 (S164).

[0069] When the second communication unit 12 receives a command to resume transmission of video data during the execution of the standalone process (YES in S165), the transmission processing unit 144 generates video data as transmission image data based on the image parameters received from the information processing device 2 (S166). After that, the control unit 14 proceeds to S17 shown in Fig. 4 and resumes transmission of the video data.

[0070] Next, the flow of cloud-based processing in the mobile object control device 1 will be described with reference to Fig. 6. During execution of the cloud-based processing, the mobile object control unit 142 executes various processes such as generating control data for controlling the speed and direction of travel of the mobile object V based on the reference data and transmitting the generated control data to the mobile object V (S181). In addition, the status data creation unit 143 measures the bandwidth and transmission delay time of the network N based on the reference data (S182). The status data creation unit 143 transmits status data indicating the bandwidth and transmission delay time to the information processing device 2 (S183).

[0071] If the second communication unit 12 receives a command to stop transmitting video data while performing cloud-based processing (YES in S184), the control unit 14 proceeds to S15 shown in FIG. 4 and starts standalone processing.

[0072] If the second communication unit 12 has not received a command to stop transmitting the video data (NO in S184) and the information processing device 2 has received a command to change the image parameters (YES in S185), the transmission processing unit 144 changes the image parameters used when creating the video data to be transmitted to the image parameters indicated by the parameter information received from the information processing device 2 (S186). The transmission processing unit 144 transmits the video data created based on the changed image parameters (S187). If the mobile object control device 1 has not received a command to change the image parameters from the information processing device 2 (NO in S185), the transmission processing unit 144 continues transmitting the video data without changing the image parameters (S187).

[0073] [Processing flow in information processing device 2] Next, the flow of processing in the information processing device 2 will be described with reference to Fig. 7. When the data receiving unit 231 receives status data from the mobile object control device 1 (S21), the parameter determining unit 232 determines image parameters based on the bandwidth and transmission delay time indicated by the status data (S22). The details of the image parameter determination process will be described later.

[0074] When the parameter determination unit 232 judges that the state of the network N is bad and the mobile object control device 1 cannot transmit the video data (NO in S23), the data transmission unit 234 transmits a command to stop transmitting the video data to the mobile object control device 1 (S24). Then, the process proceeds to stand-alone processing (S25).

[0075] When the parameter determination unit 232 determines that the state of the network N has improved and that the mobile object control device 1 can transmit image data (YES in S23), the data transmission unit 234 transmits a transmission start command to the mobile object control device 1 (S26). The data transmission unit 234 also transmits parameter information (S27). Thereafter, the data reception unit 231 starts receiving the image data (S28). Here, the data reception unit 231 initializes the value of a counter used to determine whether or not to change the image parameters (S29), and transitions to cloud-based processing (S30).

[0076] Although details will be described later, when a condition for ending the execution of the cloud-based processing is satisfied, the information processing device 2 executes the process of S24 and transitions to stand-alone processing (S25). Although not shown in FIG. 7, if the information processing device 2 is powered off or receives an instruction to end operation while the information processing device 2 is executing the process shown in FIG. 7, the information processing device 2 ends the process shown in FIG. 7.

[0077] Next, the flow of processing of the information processing device 2 during execution of the stand-alone processing will be described with reference to Fig. 8. During the stand-alone processing, the data transmission unit 234 transmits reference data not based on video data to the mobile object control device 1 (S251). The data transmission unit 234 does not need to transmit reference data during execution of the stand-alone processing. The data reception unit 231 receives status data from the mobile object control device 1 (S252). The parameter determination unit 232 determines image parameters based on the received status data (S253).

[0078] When the parameter determination unit 232 determines that the mobile object control device 1 is capable of transmitting video data (YES in S254), the control unit 23 proceeds to S26 shown in Fig. 7 to enable the mobile object control device 1 to execute cloud-based processing. When the parameter determination unit 232 determines that the mobile object control device 1 is not capable of transmitting video data (NO in S254), the control unit 23 repeats S251 to S254.

[0079] Next, a process flow of the information processing device 2 during execution of cloud-based processing will be described with reference to Fig. 9. During execution of cloud-based processing, the control unit 23 executes various processes (S301). For example, the control unit 23 generates reference data by analyzing video data.

[0080] When the data receiving unit 231 receives status data from the mobile object control device 1 (S302), the parameter determining unit 232 identifies the amount of transmission delay and the amount of jitter of the network N based on the status data (S303). The parameter determining unit 232 may calculate the amount of transmission delay and the amount of jitter based on the reception time of the status data or a fluctuation in the reception time.

[0081] The parameter determination unit 232 judges whether or not the identified transmission delay amount and jitter amount satisfy the condition for transmitting the video data with the image parameters notified to the mobile object control device 1 (S304). When the parameter determination unit 232 judges that the condition is not violated (NO in S304), it increments the counter value by 1 (S305).

[0082] Thereafter, when the parameter determination unit 232 determines that the condition has not been violated for a certain period of time or more (YES in S306), that is, when the counter value reaches or exceeds the threshold value, it determines to change the image parameters to those with a data amount included in the video data that is one step larger (S307). The data transmission unit 234 transmits parameter information including the changed image parameters (S308), and initializes the counter (S309).

[0083] If it is determined that the identified transmission delay amount and jitter amount violate the conditions for transmitting video data with the image parameters notified to the mobile object control device 1 (YES in S304), the bandwidth of the network N is identified based on the status data (S310). If the parameter determination unit 232 determines that the bandwidth of the network N is a bottleneck (YES in S311), it executes a process of changing the image parameters and executes a process of determining new image parameters suitable for the identified bandwidth (S312). If the parameter determination unit 232 determines that the bandwidth of the network N is not a bottleneck (NO in S311), the control unit 23 returns to S301.

[0084] If the parameter determination unit 232 determines that there are no image parameters that can be transmitted to the mobile object control device 1 (YES in S313), that is, if the parameter determination unit 232 determines that it is necessary to stop transmission of video data, the process proceeds to S24 shown in Fig. 7, where the data transmission unit 234 transmits a transmission stop command. If the parameter determination unit 232 determines that there are image parameters that can be transmitted to the mobile object control device 1 (NO in S313), the data transmission unit 234 transmits parameter information including the image parameters determined in S312 to the mobile object control device 1 (S314), and initializes a counter (S315).

[0085] 10 is a flowchart showing a process flow in which the parameter determination unit 232 determines image parameters. The parameter determination unit 232 specifies the bandwidth Bn of the network N based on the state data (S221). Next, the parameter determination unit 232 specifies the bandwidth Bf corresponding to the current image parameters by referring to the parameter determination data (S222). The current image parameters are the initial values ​​of the image parameters or the image parameters used by the mobile object control device 1.

[0086] If the identified bandwidth Bn is greater than the bandwidth Bf (YES in S223), the parameter determination unit 232 determines the image parameters corresponding to the bandwidth Bf that corresponds to the current image parameters as the image parameters to be notified to the mobile object control device 1 (S224). If the identified bandwidth Bn is equal to or less than the bandwidth Bf (NO in S223), the parameter determination unit 232 determines to reduce the amount of video data by one step (S225), and searches for image parameters corresponding to the reduced data amount (S226).

[0087] If there are no usable image parameters (YES in S227), the parameter determination unit 232 determines that the video data cannot be transmitted to the mobile object control device 1 (S228). If there are usable image parameters (NO in S227), the parameter determination unit 232 returns to S222, identifies the bandwidth Bf corresponding to the newly selected image parameters, and repeats the processes from S223 to S227.

[0088] [Effects of Information Processing System S] As described above, the mobile object control device 1 transmits video data corresponding to the image parameters notified from the information processing device 2 and status data indicating the status of the network N to the information processing device 2. The information processing device 2 has a parameter determination unit 232 that determines the image parameters to be transmitted to the mobile object control device 1 based on the network status indicated by the status data, and a reference data generation unit 233 that generates reference data corresponding to the surrounding status of the mobile object V based on the transmitted image data. The mobile object control unit 142 changes the control content of the mobile object V based on the reference data depending on the status of the network N.

[0089] With the mobile object control device 1 and the information processing device 2 configured in this manner, while the state of the network N is good, the reference data generating unit 233 creates reference data including information for controlling the mobile object V based on the high-quality video data transmitted by the mobile object control device 1 to the information processing device 2, and the mobile object control unit 142 can control the mobile object V with high accuracy based on the reference data. On the other hand, when the state of the network N is not good, the mobile object control unit 142 does not control the mobile object V using the reference data as it is, but controls the mobile object V based on the video data or sensor data acquired by the data acquisition unit 141, or controls the mobile object V based on information on surrounding objects indicated by the reference data. As a result, the mobile object V can continue to operate regardless of fluctuations in the state of the network.

[0090] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0091] 1. Mobile control device 2. Information processing equipment 11 First Communications Department 12 2nd Communications Department 13 Storage section 14 Control section 21 Communications Department 22 Memory section 23 Control Unit 141 Data Acquisition Section 142 Mobile control unit 143 Status Data Creation Department 144 Transmission processing unit 231 Data receiving unit 232 Parameter Determination Unit 233 Reference Data Generation Unit 234 Data transmission unit

Claims

1. A mobile body control device that controls a mobile body, and an information processing device that communicates with the mobile body control device via a network, The mobile object control device includes: a data acquisition unit that acquires captured image data generated by an imaging unit provided in the moving object; a status data creation unit that creates status data indicating a status of the network based on a reception status of reference data transmitted by the information processing device to the mobile object control device via the network; a transmission processing unit that transmits, to the information processing device, transmission image data based on the captured image data, based on image parameters transmitted from the information processing device after transmitting the status data to the information processing device; A moving body control unit that controls the moving body; having The information processing device includes: a data receiving unit for receiving the status data and the transmission image data; a parameter determination unit that determines the image parameters to be transmitted to the mobile object control device based on the state of the network indicated by the state data; a reference data generating unit that generates the reference data corresponding to a surrounding state of the moving object based on the transmitted image data; a data transmission unit that transmits the image parameters and the reference data to the mobile object control device; having The mobile object control unit is an information processing system that changes a control method of the mobile object based on the reference data depending on the state of the network.

2. the data transmission unit transmits to the mobile object control device the reference data indicating information regarding positions of other objects within a predetermined range from the mobile object; The moving object control unit controls the moving object based on the position of the other object indicated by the reference data. The information processing system according to claim 1 .

3. the mobile object control unit controls the mobile object based on the captured image data while the reference data is not transmitted from the information processing device. The information processing system according to claim 2 .

4. the information processing device further includes a storage unit that stores the network state and a value of an image parameter in association with each other; the parameter determination unit selects the image parameter stored in the storage unit in association with the state of the network indicated by the state data. The information processing system according to claim 1 .

5. the status data being data indicative of a bandwidth of the network; the data transmission unit transmits the image parameters with a smaller data amount of the transmission image data transmitted by the mobile object control device to the information processing device as the bandwidth indicated by the status data becomes smaller. The information processing system according to claim 1 .

6. the status data is data indicating a transmission delay amount of the network, the data transmission unit transmits the image parameters with a smaller data amount of the transmission image data transmitted by the mobile object control device to the information processing device as the transmission delay amount indicated by the status data increases. The information processing system according to claim 1 .

7. the state data being the bandwidth of the network; The data transmission unit transmits, to the mobile object control device, a transmission stop command data for stopping the transmission of the transmission image data when the bandwidth indicated by the status data is equal to or less than a bandwidth threshold value. The information processing system according to claim 1 .

8. the status data is data indicating a transmission delay amount of the network, The data transmission unit transmits, to the mobile object control device, stop command data for stopping transmission of the transmission image data when the transmission delay amount indicated by the status data is equal to or greater than a delay threshold value. The information processing system according to claim 1 .

9. the reference data includes time data indicating a time measured by the information processing device, The status data creation unit uses a difference between the time indicated by the time data and the time measured by the mobile object control device as the reception status, and creates the status data indicating that the larger the difference is, the smaller the bandwidth of the network is, or the larger the transmission delay amount of the network is. The information processing system according to claim 1 .

10. the parameter determination unit determines new image parameters corresponding to a larger amount of data than the image parameters previously determined, when the time during which the network state satisfies a predetermined condition continues for a predetermined time or longer. The information processing system according to claim 1 .

11. An information processing method executed by a mobile body control device that controls a mobile body and an information processing device that communicates with the mobile body control device via a network, comprising: A step in which the mobile object control device acquires captured image data generated by an imaging unit provided in the mobile object; a step of the information processing device transmitting reference data to the mobile object control device via the network; creating status data indicating a status of the network based on a reception status of the reference data by the mobile object control device; a step of determining, by the information processing device, image parameters of transmission image data to be transmitted from the mobile object control device to the information processing device based on the state of the network indicated by the state data; a step of the information processing device transmitting the determined image parameters to the mobile object control device; a step of the mobile object control device transmitting the transmission image data based on the captured image data using the image parameters to the information processing device; a step of transmitting, by the information processing device, reference data corresponding to a surrounding state of the moving object, generated based on the transmission image data, to the information processing device; a step of the mobile object control device changing a control method of the mobile object based on the reference data depending on a state of the network; An information processing method comprising the steps of:

Images