Platform systems, control circuits, storage media, and communication methods

Abstract

A platform system (20) comprises: a device group (70) that is equipped with a sensing function and outputs information obtained by the sensing function; an information processing base unit (50) that processes information acquired from the device group (70) in response to a request from an application (10); a network processing base unit (60) that monitors the communication status when acquiring information from the device group (70) and controls communication with the device group (70); an orchestrator (40) that monitors the operation of the device group (70), the information processing base unit (50), and the network processing base unit (60) and performs coordination control; and a service enabler (30) that instructs the information processing base unit (50) and the orchestrator (40) to operate on the basis of the request from the application (10).

Description

Technical Field

[0001] The present disclosure relates to a platform system, a control circuit, a storage medium, and a communication method for providing data regarding a requested service.

Background Art

[0002] Conventionally, as means for transmitting data for a service requested by a user, there are networks such as a wired network and a wireless network. In order to construct such a network, it is necessary for a person having knowledge of each network to construct it. However, for a network, different service requirements are required, such as low latency being required for control signaling depending on the service requested by the user and high bandwidth being required for video transmission. Therefore, there has been a demand for a system that allows a person who lacks knowledge of networks but is well-versed in services to construct a network without being aware of the construction conditions. Patent Document 1 discloses a technology regarding a network requirement generation system that analyzes service requirements input by a user to generate network requirements and creates network setting contents for a control device for constructing a network from the network requirements. It has been shown that the network requirement generation system described in Patent Document 1 can also adjust the bandwidth of the network based on the result of analyzing the network situation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in recent years, in response to the increasing sophistication of services demanded by users, it has become necessary to secure communication resources such as network bandwidth, as well as to integrate with computing resources in cyberspace (Cyber ​​System) for data processing and other tasks to achieve even lower latency, and with sensing functions in the real world (Physical System). However, networks built using the conventional technologies described above do not take into account the integration with computing resources and sensing functions. Therefore, especially in situations where communication resources and computing resources change in real time, even if one resource can be secured, the other cannot, resulting in the inability to provide services at the expected service level. Furthermore, in order to ensure the service level in the above situation, it has become necessary to distribute numerous sensing functions, which has led to the problem of increased costs for building sensing infrastructure.

[0005] This disclosure is made in view of the above, and aims to realize a platform system that can improve the probability of providing services by guiding sensing functions to a range in which each resource can provide a desired service level, in situations in which at least one of the communication resources and computing resources changes. [Means for solving the problem]

[0006] To solve the aforementioned problems and achieve the objectives, the platform system relating to this disclosure is characterized by comprising: a group of devices equipped with sensing functions that output information obtained by the sensing functions; an information processing infrastructure unit that processes information acquired from the group of devices in response to requests from applications; a network processing infrastructure unit that monitors the communication status when acquiring information from the group of devices and controls communication with the group of devices; an orchestrator that monitors the operation of the group of devices, the information processing infrastructure unit, and the network processing infrastructure unit and performs coordinated control; and a service enabler that instructs the information processing infrastructure unit and the orchestrator to perform actions based on requests from applications. [Effects of the Invention]

[0007] The platform system described herein has the effect of improving the probability of providing services by guiding the sensing function to a range in which each resource can provide the desired service level, even when at least one of the communication resources and computing resources changes. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an example configuration of a cyber-physical system according to the embodiment. [Figure 2] A diagram showing an example of the device configuration of a cyber-physical system according to an embodiment. [Figure 3] A sequence diagram showing the operation of the cyber-physical system according to the embodiment, up to the point where the drone can transmit data according to the status of the communication line. [Figure 4] Sequence diagram showing the operation up to the start of image processing of images acquired from a camera in the cyber-physical system according to the embodiment. [Figure 5] A flowchart illustrating the operation of the orchestrator in the cyber-physical system according to the embodiment, from acquiring data according to the network status to starting image processing. [Figure 6] Sequence diagram showing the operation when an orchestrator makes resource changes in a cyber-physical system according to the embodiment. [Figure 7] A flowchart illustrating the operation of an orchestrator when it makes resource changes in a cyber-physical system according to an embodiment. [Figure 8] Sequence diagram showing the operation of the cyber-physical system according to the embodiment, from the time it is set up to the time when it can use the drone in response to instructions from the application. [Figure 9] A sequence diagram showing the detailed operation of the shooting preparation process in step S603 shown in Figure 8 in the cyber-physical system according to the embodiment. [Figure 10] A sequence diagram showing the detailed operation of the training process in step S604 shown in Figure 8 in the cyber-physical system according to the embodiment. [Figure 11] A sequence diagram showing the detailed operation of the monitoring process in step S607 shown in Figure 8 in the cyber-physical system according to the embodiment. [Figure 12] Sequence diagram showing the operation of notifying the application of the estimated shooting time by the drone in the cyber-physical system according to the embodiment. [Figure 13] Sequence diagram showing the operation in which an application acquires spatial object images in a cyber-physical system according to an embodiment. [Figure 14] Sequence diagram showing the operation to check the communication status between the drone and the HMD (Head Mounted Display), which is the application terminal of the application, in the cyber-physical system according to the embodiment. [Figure 15] Sequence diagram showing the operation of a child drone transmitting data via a parent drone in a cyber-physical system according to an embodiment. [Figure 16] Sequence diagram showing the operation in which a parent drone synthesizes data acquired from a child drone and transmits the data in a cyber-physical system according to an embodiment. [Figure 17] Sequence diagram showing the operation when an orchestrator instructs a drone to change operating conditions such as camera shooting conditions in a cyber-physical system according to an embodiment. [Figure 18] A sequence diagram showing the operation of the HMD, which is the application terminal for the application, in the cyber-physical system according to the embodiment, until the HMD acquires 3D (Dimensions) images. [Figure 19] A sequence diagram showing the operation in which the information processing infrastructure unit performs calculations and notifies the application of the estimated shooting time by the drone in the cyber-physical system according to the embodiment. [Figure 20]Sequence diagram showing the operation of resource monitoring in the information processing infrastructure unit in the cyber physical system according to the embodiment [Figure 21] Sequence diagram showing the operation until the HMD, which is an application terminal of the application in the cyber physical system according to the embodiment, requests and acquires map information [Figure 22] Sequence diagram showing the operation when the user using the HMD, which is an application terminal of the application in the cyber physical system according to the embodiment, wants to change the viewing direction of the object or target location [Figure 23] Diagram showing a configuration example of the processing circuit when the processing circuit realizing the platform system according to the embodiment is composed of a processor and a memory [Figure 24] Diagram showing an example of the processing circuit when the processing circuit realizing the platform system according to the embodiment is composed of dedicated hardware

Mode for Carrying Out the Invention

[0009] Hereinafter, the platform system, control circuit, storage medium, and communication method according to the embodiment of the present disclosure will be described in detail based on the drawings.

[0010] Embodiment FIG. 1 is a block diagram showing a configuration example of the cyber physical system 1 according to the present embodiment. The cyber physical system 1 includes an application 10 and a platform system 20. The cyber physical system 1 is a system that requests a service from the application 10 to the platform system 20, and the platform system 20 provides data for the service requested from the application 10 to the application 10. The cyber physical system 1 is a system in which a CPS (Cyber Physical System) service is available by the application 10 which is a CPS application.

[0011] Application 10 requests services from the platform system 20, specifically the service enabler 30 provided by the platform system 20, by accepting operations from the user. In this embodiment, the services requested by Application 10 specifically include the acquisition of images taken by a drone, which is part of the device group 70 provided by the platform system 20, and which are images corresponding to the orientation of the head-mounted display (hereinafter referred to as HMD), which is the application terminal worn by the user. The application terminal may be a smartphone, mobile pad, or other terminal other than the HMD mentioned above. Furthermore, while Application 10 will be explained using a real-time remote monitoring application as an example, allowing the user to view a desired location from a desired viewpoint using the HMD, Application 10 may also be an application that operates a drone equipped with a sensor terminal. In the following explanation, "application" may be referred to as "app".

[0012] The platform system 20 provides data for services requested by the application 10. The platform system 20 comprises a service enabler 30, an orchestrator 40, an information processing infrastructure unit 50, a network processing infrastructure unit 60, and a group of devices 70.

[0013] The service enabler 30 instructs the information processing infrastructure unit 50 and the orchestrator 40 to operate based on requests from the application 10. The service enabler 30 receives service requests, or commands, from the application 10 via an API (Application Programming Interface), breaks down the received service requests, or commands, according to the destinations that manage the functions that provide the services, instructs the orchestrator 40, and instructs the information processing infrastructure unit 50 via the API. An API is an interface that handles the coordination between software, programs, etc., with different functions. The service enabler 30 also obtains data corresponding to the instructions from the application 10 from the information processing infrastructure unit 50 via the API. In the cyber-physical system 1, the application 10 can also perform the functions of the service enabler 30. The service enabler 30 issues instructions to the platform application group 51 of the information processing infrastructure unit 50, that is, to each function in the information processing infrastructure unit 50, so that the functions are linked in a certain order. The service enabler 30, for example, in the video processing unit 51A of the information processing infrastructure unit 50, includes a spatial reproduction unit 51D and a spatial presentation unit 51C. First, it instructs the spatial reproduction unit 51D to perform 3D spatial reproduction, and then instructs the spatial presentation unit 51C to process the map and display it on the HMD for spatial presentation.

[0014] The orchestrator 40 monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60, and controls their coordination with each function. For example, the orchestrator 40 controls and monitors the multi-layer relationship between the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60. The orchestrator 40 manages and operates the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70, and centrally executes function control, function settings, and inter-function coordination. For example, when the orchestrator 40 receives a command from the service enabler 30 to request the establishment of a communication path in order to acquire camera images, it takes action triggered by the acquisition. Also, when the orchestrator 40 controls the drone and takes camera images, it sends control messages to the network processing infrastructure unit 60 and makes parallel requests for camera image transmission and control information transmission. When the transmission request is completed, the orchestrator 40 sends a communication completion message to the service enabler 30 when communication for camera images is complete. The orchestrator 40 comprises an application service control unit 41, a data linkage unit 42, an application linkage unit 43, an E2E (End to End) slice control unit 44, and an E2E resource management unit 45.

[0015] The application service control unit 41 controls the operation of the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70. The data linkage unit 42 links data between the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70, such as configuration data and stored data. The application linkage unit 43 links applications 10, i.e., functions based on requested services, to the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70. The E2E slice control unit 44 performs slice control in the network processing infrastructure unit 60. The E2E resource management unit 45 performs resource management in the network processing infrastructure unit 60.

[0016] The information processing infrastructure unit 50 processes information acquired from the device group 70 in response to requests from the application 10. The information processing infrastructure unit 50 provides computing power to the user and manages computing power. In this embodiment, specifically, as will be described later, image information captured by the shooting function is output from the device group 70 as the aforementioned information, so the information processing infrastructure unit 50 performs video processing using the image information to display it on a display device corresponding to the application 10. The information processing infrastructure unit 50 also performs processing to synthesize multiple pieces of information acquired from the device group 70 as processing in response to requests from the application 10. The information processing infrastructure unit 50 comprises a platform application group 51, a resource monitoring unit 52, an edge processing unit 53, and an information security unit 54.

[0017] The platform application group 51 acquires images from the camera 74, which will be described later, provided by the device group 70, generates 3D images, acquires information such as the camera's shooting position and field of view direction from the HMD, which is the application terminal for running application 10, and delivers 3D image data based on the camera's shooting position and field of view direction from the HMD to the HMD, which is the application terminal for application 10. The camera's shooting position and field of view direction may be the position and orientation of the drone on which the camera 74 is mounted. The platform application group 51 includes a video processing unit 51A, a remote control unit 51E, a sensory sharing unit 51F, a data linkage unit 51G, and a sensor information analysis and prediction unit 51H.

[0018] The video processing unit 51A generates video based on requests from the application 10, synthesizes video, and provides video data to the application 10. Specifically, the video processing unit 51A provides video data to the HMD of the display device, which is the application terminal. The video processing unit 51A comprises an augmented space unit 51B, a spatial presentation unit 51C, and a spatial reproduction unit 51D. The augmented space unit 51B generates an AR (Augmented Reality) space, or augmented reality space, from data constructed by video processing. The spatial presentation unit 51C generates a VR (Virtual Reality) space, or a virtual reality space, based on model data generated by computation processing. The spatial reproduction unit 51D generates video to be provided to the application 10 based on the data generated by the augmented space unit 51B and the spatial presentation unit 51C. Specifically, the spatial reproduction unit 51D generates video corresponding to the viewpoint direction of the HMD of the display device, which is the application terminal.

[0019] The remote control unit 51E performs operations on the device group 70. For example, the remote control unit 51E gives instructions to the drones constituting the device group 70 to move to a shooting position in response to a request from the application 10. The sensory sharing unit 51F provides the user with reproducible sensations. In this embodiment, the sensory sharing unit 51F generates and provides data on the senses of sight, touch, and hearing, among the five senses. The data linkage unit 51G holds data from each sensor acquired from the device group 70 and provides the relevant data to the sensor information analysis and prediction unit 51H in response to a request from the sensor information analysis and prediction unit 51H. Based on the data acquired from the data linkage unit 51G, the sensor information analysis and prediction unit 51H uses its computing power to perform information analysis, learning, prediction, etc.

[0020] The resource monitoring unit 52 monitors the computing power status of each server and edge server that constitute the information processing infrastructure unit 50. The edge processing unit 53 is located near the device group 70 and performs computational processing and data management on data from the device group 70. The information security unit 54 protects against data corruption, protects against information leakage to parties other than the intended recipient, and maintains the data disclosure environment to the intended recipient.

[0021] The network processing infrastructure unit 60 monitors the communication status when acquiring information from the device group 70 and controls communication with the device group 70. The network processing infrastructure unit 60 manages and provides network capabilities to the user. The network processing infrastructure unit 60 comprises a resource monitoring unit 61, a network operation management unit 62, a network security unit 63, a slicing unit 64, and a deterministic network unit 65. In Figure 1, the network is abbreviated as NW (Network). The same applies to subsequent figures.

[0022] The resource monitoring unit 61 monitors the communication bandwidth, latency, packet loss rate, etc., of the communication devices constituting the network processing infrastructure unit 60 and the mobility nodes constituting the device group 70. The network operation management unit 62 monitors to determine whether the communication environment guaranteed by the SLA (Service Level Agreement) is being provided to each connection. The network security unit 63 provides a VPN (Virtual Private Network) function for data confidentiality. The slicing unit 64 virtually divides, i.e., slices, the network according to the communication characteristics that make up the network, and provides a logical network according to the SLA. The deterministic network unit 65 provides a network that guarantees reachability and latency fluctuations within an allowable latency time within a specified range.

[0023] The device group 70 is equipped with a sensing function and outputs information obtained by the sensing function. The device group 70 is a mobility node with a communication function, such as a smartphone, drone, or heavy machinery. The device group 70 may include multiple types of mobility nodes, or multiple of one type of mobility node. In this embodiment, the device group 70 is specifically an unmanned aerial vehicle, i.e., a drone, and the device group 70, i.e., the drone, is equipped with a shooting function as a sensing function. In this case, the device group 70, i.e., the drone, outputs image information captured by the shooting function as the aforementioned information. The device group 70 comprises a clustering unit 71, a spatiotemporal synchronization unit 72, a sensing unit 73, a security unit 75, and an actuator unit 76.

[0024] The clustering unit 71 forms a cluster with mobility nodes and performs cooperative operation between them. For example, if the mobility nodes are drones, the cooperative operation would be formation operation. The spatiotemporal synchronization unit 72 manages the spatial position of the mobility nodes constituting the device group 70, the synchronization status with respect to an arbitrary reference time, etc. The sensing unit 73 is equipped with sensors and acquires data generated from the sensors. In this embodiment, a camera 74 is assumed as the sensor, but it is not limited to this, and sensors that measure temperature, illuminance, gas, position, airborne particles, etc. may also be used. Furthermore, the sensing unit 73 may be equipped with multiple types of sensors. The images obtained by the camera 74 may be still images or moving images. The security unit 75 protects the data acquired from the sensors equipped in the sensing unit 73 and performs encryption processing of data transmitted to the information processing infrastructure unit 50. The actuator unit 76 converts energy such as electricity, gas, and oil into operation. The actuator unit 76 includes, for example, a motor for rotating the propellers when the mobility node is a drone, or a motor for controlling the shooting direction of the camera 74.

[0025] Here, we will explain the correspondence between each component shown in the block diagram of the cyber-physical system 1 in Figure 1 and the device configuration in the actual system. Figure 2 is a diagram showing an example of the device configuration of the cyber-physical system 1 according to this embodiment.

[0026] The application 10 shown in Figure 1 is running on the HMD 110, which is the application terminal used by the user in Figure 2, and the HMD 110 displays an image from the desired viewpoint to the user. At the same time, the HMD 110 presents a cyber space in which a 3D spatial model and virtual objects are synthesized through a synthesis process that constructs a real-time augmented space by the platform system 20.

[0027] The service enabler 30 shown in Figure 1 is implemented on the CPS server 151 in Figure 2.

[0028] The orchestrator 40 shown in Figure 1 is implemented on the CPS server 151 in Figure 2. The CPS server 151 controls communication resources, computing resources, and the device group 70 to provide the requested services to the application 10. Some functions of the orchestrator 40 may be implemented on other servers 150, such as edge servers 154 and 155, which are closer to the drones 170, which are the device group 70 that are operated by the application 10.

[0029] The information processing infrastructure unit 50 shown in Figure 1 is implemented in the CPS server 151, edge servers 154, 155, etc., in Figure 2. The functions of the information processing infrastructure unit 50 shown in Figure 1 may be implemented in a single server 150, or they may be distributed and implemented across multiple servers 150. Furthermore, functions may be transferred between servers 150 depending on the computing resource status of each server 150.

[0030] The network processing infrastructure unit 60 shown in Figure 1 is implemented in base stations installed on each line 160, such as public lines like the satellite line 162 and terrestrial line 163 shown in Figure 2, wired line 161, and private lines like L5G (Local 5th Generation) 164, 165, or in a base station that manages all of the lines 160 shown in Figure 2. The functions of the network processing infrastructure unit 60 shown in Figure 1 may be implemented in one base station or distributed and implemented across multiple base stations. The satellite line 162 is, for example, a communication line 160 for geostationary satellite communication, low Earth orbit satellite communication, etc. The terrestrial line 163 is, for example, a communication line 160 for cellular communication, etc.

[0031] The device group 70 shown in Figure 1 is assumed in Figure 2 to be a drone 170 in physical space, as described above. The drone 170 takes pictures with the camera 74 according to the user's instructions, and also moves, rotates, changes altitude, etc. according to the user's instructions, changing the location and direction of the camera 74's shooting. Although the description in Figure 2 is simplified, the device group 70 is assumed to consist of multiple drones 170.

[0032] In the cyber-physical system 1, the application terminal, HMD 110, instructs the drone 170, either directly or via the CPS server 151, on the shooting location, shooting direction, etc., so that the user can check the situation of a disaster site or other location captured by the drone 170. The drone 170 transmits the video footage captured from multiple viewpoints instructed by the user to the CPS server 151. The CPS server 151 generates an image in cyberspace by combining a map, which is a virtual object showing the terrain of the remote area to be photographed by the drone 170, with data that models fallen trees and other objects in 3D space, created based on the actual video footage captured by the drone 170. The CPS server 151 transmits the generated cyberspace data to the application terminal, HMD 110, via a satellite link 162 or a terrestrial link 163. As a result, the user can check the situation of the disaster site and other locations from the desired viewpoint through the cyberspace displayed on the HMD 110. In the cyber-physical system 1, the CPS server 151 displays a real-time augmented space to the HMD 110, allowing users to share the situation at a remote disaster site in real time and space. Note that some of the processing performed by the CPS server 151 may be carried out by edge servers 154, 155, or other servers closer to the drone 170, in order to reduce transmission delay time during data communication, or if the CPS server 151 cannot secure sufficient computing resources.

[0033] In this embodiment, the orchestrator 40 shown in Figure 1 monitors resource information for each server 150 and each line 160, and provides data for services requested by the application 10 by using the server 150 that can secure computing resources and the line 160 that can secure communication resources. The orchestrator 40 obtains computing resources from the information processing infrastructure unit 50 necessary to process tasks within the processing time requested by the user, obtains communication status such as bandwidth used for data transmission between the network processing infrastructure unit 60 and the device group 70, and communication resources necessary to process data transmission within the transmission time requested by the user from the network processing infrastructure unit 60, obtains location information from the device group 70, and monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 to perform coordinated control.

[0034] The platform system 20 acquires sensing information from disaster sites using multiple drones 170 equipped with pre-registered cameras 74, in order to understand the disaster situation in an area specified by the user, which is a CPS application, application 10. Furthermore, the platform system 20 performs a series of processes, from computer processing to generate 3D images so that the user can understand the disaster situation in three dimensions, to distributing them to the HMD 110, which is a display device used by the user, by efficiently considering communication and computing resources, thereby providing the necessary services.

[0035] For example, consider a scenario in disaster prevention application 10 where the management and control of computing resources such as image processing and communication resources are performed independently. Computing resources include the number and processing speed of CPUs (Central Processing Units), memory, and storage capacity. Communication resources include bandwidth used for communication, time occupancy, QoS (Quality of Service), RTT (Round Trip Time), and throughput. In such a case, even if the necessary communication resources are secured on the network, if the necessary computing resources are not secured on the network, the calculation process will take a long time, making real-time calculation processing impossible. Furthermore, remote control requires reliable periodic communication with relatively small volumes of data rather than large volumes of data, and computational processing that is small in scale but highly real-time. This made it difficult to accommodate the service levels of various applications 10.

[0036] Therefore, in this embodiment, the platform system 20, as an application 10 for disaster prevention, processes video footage obtained from a drone 170 equipped with a camera 74 to allow the user to understand the disaster situation, and displays it on the HMD 110 so that the user can view the target object or target location at the disaster site from the direction intended by the user, according to the service level of the application 10, i.e., real-time performance, operability, etc. In order to perform processing according to the service level of the application 10, the platform system 20 divides its functions into an information processing infrastructure unit 50, a network processing infrastructure unit 60, and an orchestrator 40, and the orchestrator 40 secures and controls communication resources and computing resources according to the service level of the application 10, i.e., real-time performance, operability, etc.

[0037] In this embodiment, the orchestrator 40 only needs to periodically or periodically monitor communication resources on the satellite link 162 and the terrestrial link 163, and computing resources such as available edge servers 154, 155 and a cloud (not shown), and be able to grasp and control the resource status in a simple manner. A simple method, in the case of communication resources, is, for example, a method of grasping the status using commands such as Ping for measuring communication delay time and Traceroute for determining the communication path, but is not limited to these.

[0038] In this embodiment, the service enabler 30 receives, for example, a request to use an application 10 related to a disaster prevention CPS, and breaks it down into processing units for controlling the information processing infrastructure unit 50 and the network processing infrastructure unit 60, such as a 3D image creation function, a display function, and processing functions related to drone flight, and transmits these to the orchestrator 40. This determines the requirements for real-time performance, operability, etc., in the platform system 20.

[0039] The orchestrator 40 monitors the operation and processing time of each sub-function in the information processing infrastructure 50 and the network processing infrastructure 60 to ensure that the necessary functions and performance are met. Based on this information, it performs control and appropriately monitors and controls the ever-changing communication and computing resources to ensure or maintain the service level for each application 10. The operation of each sub-function in the information processing infrastructure 50 and the network processing infrastructure 60 refers to the operation of each configuration of the information processing infrastructure 50 and the network processing infrastructure 60 as shown in Figure 1. In the platform system 20, the status of the orchestrator 40 is fed back to the service enabler 30, and the service enabler 30's requirements are reset according to the status of communication and computing resource availability, such as real-time performance and operability.

[0040] In this way, the platform system 20 can efficiently configure and coordinate the information processing infrastructure unit 50 and the network processing infrastructure unit 60 using the orchestrator 40 according to the service level of the application 10, thereby improving the real-time capabilities and operability of the CPS.

[0041] The specific operation of Cyber-Physical System 1 will be explained using sequence diagrams or flowcharts.

[0042] Figure 3 is a sequence diagram showing the operation in the cyber-physical system 1 according to this embodiment, up to the point when the drone 170 is able to transmit data according to the state of the line 160. When application 10 receives a service execution request from a user, it requests the orchestrator 40 to execute the service via a service enabler 30 (not shown) (step S101). The network processing infrastructure unit 60 acquires communication speed, communication quality, communication delay information, etc. from the satellite line 162 (step S102). Similarly, the network processing infrastructure unit 60 acquires communication speed, communication quality, communication delay information, etc. from the terrestrial line 163 (step S103). The network processing infrastructure unit 60 may acquire the above information from the base station of each line 160, or it may acquire the above information from a base station that manages each line 160.

[0043] The orchestrator 40 requests information obtained from the satellite link 162 and the terrestrial link 163 from the network processing infrastructure unit 60 (step S104). In response to the request from the orchestrator 40, the network processing infrastructure unit 60 transmits information on the communication line status obtained from the satellite link 162 and the terrestrial link 163 to the orchestrator 40 (step S105). The orchestrator 40 selects a communication line according to the sequence shown in Figure 5, which will be described later, and requests the network processing infrastructure unit 60 to select line 160 (step S106). Note that the "selection" of a communication line described here is not limited to just one line, but multiple lines may be selected, and protocols that use multiple lines simultaneously, such as MultiPath TCP (Transmission Control Protocol), may be used. The detailed operation of the orchestrator 40 will be described later. The network processing infrastructure unit 60 requests the drone 170 to select line 160 (step S107). The drone 170 transmits data to the satellite link 162, the terrestrial link 163, or both the satellite link 162 and the terrestrial link 163, based on a request from the network processing infrastructure unit 60 to select a link 160 (step S108).

[0044] The platform system 20, by performing the operations shown in the sequence diagram in Figure 3, can select a reliable communication line 160 for communication with the drone 170 by switching between the satellite line 162 and the terrestrial line 163, or by using both the satellite line 162 and the terrestrial line 163, depending on the communication line status during a disaster. Communication line status refers to, for example, communication speed, communication quality, communication delay, and whether periodic communication is possible.

[0045] Figure 4 is a sequence diagram showing the operation of the cyber-physical system 1 according to this embodiment up to the start of image processing of images acquired from the camera 74. When application 10 receives a service execution request from a user, it requests the orchestrator 40 to execute the service via a service enabler 30 (not shown) (step S201). The orchestrator 40 controls the operation of the information processing infrastructure unit 50 (step S202). The detailed operation of the orchestrator 40 will be described later. Based on the control of the orchestrator 40, the information processing infrastructure unit 50 creates a container for camera image processing in order to execute the service requested by application 10 (step S203). In this embodiment, a container is used as the virtualization technology, but a virtual machine, which is a similar technology, may also be used. The information processing infrastructure unit 50 deploys the camera image processing container to the server 150, which is the computer that actually performs the camera image processing (step S204). The server 150, which is the computer, starts the camera image processing (step S205). Note that this example assumes that there are n servers 150, which are computers that perform camera image processing, but the number of servers 150 can be as few as one.

[0046] The server 150, which is the computer that actually processes the camera images, is a server 150 that can be installed in multiple areas, including the edges. The platform system 20 can divide and distribute the computer processing required for 3D image generation among the servers 150, including the edge servers 154 and 155.

[0047] Figure 5 is a flowchart illustrating the operation of the orchestrator 40 in the cyber-physical system 1 according to this embodiment, from acquiring data according to the status of the line 160 to starting image processing. The orchestrator 40 receives a service request from the application 10 via a service enabler 30 (not shown) (step S301). The orchestrator 40 understands the computing power and the computing processes being executed for each server 150, which are computers (step S302). The orchestrator 40 derives the communication resources and computing resources required for the service requested by the application 10 (step S303). The orchestrator 40 understands the network communication line status to the drone 170 and HMD 110 (step S304). The orchestrator 40 determines the placement of the 3D image generation and distribution processes and the network route (step S305). In other words, when there are multiple communication paths for the network processing infrastructure unit 60 to acquire information from the device group 70, the orchestrator 40 determines the communication path for the network processing infrastructure unit 60 to acquire information from the device group 70 based on computing resources and communication resources. The orchestrator 40 gives control instructions to the information processing infrastructure unit 50 (step S306). In the flowchart shown in Figure 5, the operations from step S301 to step S305 correspond to the operations of the orchestrator 40 in the sequence diagram shown in Figure 3, and the operation in step S306 corresponds to the operations of the orchestrator 40 in the sequence diagram shown in Figure 4.

[0048] Figure 6 is a sequence diagram showing the operation of the orchestrator 40 when it makes resource changes in the cyber-physical system 1 according to this embodiment. The application 10 notifies the orchestrator 40 of quality degradation via a service enabler 30 (not shown) (step S401). Quality degradation includes, for example, the detection of dropped frames in video data displayed by the application. The orchestrator 40 requests information on computing resources from the information processing infrastructure unit 50 (step S402). The information processing infrastructure unit 50 transmits information on computing resources in response to the request from the orchestrator 40 (step S403). The orchestrator 40 requests information on communication resources from the network processing infrastructure unit 60 (step S404). The network processing infrastructure unit 60 transmits information on communication resources in response to the request from the orchestrator 40 (step S405). The orchestrator 40 makes resource changes based on the acquired information on computing resources and communication resources (step S406). The detailed operation of step S406 will be described later. The orchestrator 40 notifies application 10 of the change in service level (step S407). Upon receiving the notification, application 10 performs service level conflict resolution with other applications 10 (step S408). Application 10 resolves the conflict by either asking other applications 10 to lower their service level or by lowering its own service level.

[0049] Figure 7 is a flowchart showing the operation of the orchestrator 40 when it makes resource changes in the cyber-physical system 1 according to this embodiment. The flowchart in Figure 7 shows the details of the operation of step S406 in the sequence diagram shown in Figure 6. The orchestrator 40 grasps the computing power and computing processes being executed by each server 150 (step S501). The orchestrator 40 makes resource allocation changes considering priority (step S502). Resource allocation changes refer to increases or decreases in communication resources and computing resources. The orchestrator 40 makes communication path changes that involve changes in the computing process arrangement (step S503). If the orchestrator 40 detects a conflict in the application 10 due to the change (step S504: Yes), it notifies the relevant application 10 of the service level change (step S505). If the orchestrator 40 does not detect any conflicts in application 10 due to the change (step S504: No), and detects a resource shortage (step S506: Yes), it predicts a future resource shortage situation (step S507). After step S505, or after step S507, or if it does not detect a resource shortage (step S506: No), the orchestrator 40 implements the resource change (step S508).

[0050] In detail, the orchestrator 40 notifies the service enabler 30 if it cannot meet the service level required by application 10 based on computing and communication resources. Based on the notification from the orchestrator 40, the service enabler 30 requests a change in the service level from application 10.

[0051] As shown in Figures 6 and 7, when the orchestrator 40 receives a quality degradation notification from application 10, or detects a shortage of computing resources in the information processing infrastructure unit 50, or detects a shortage of communication resources in the network processing infrastructure unit 60, it performs actions such as changing resource allocation considering service priority, and changing communication paths that involve changing the computing processing arrangement. Changing resource allocation refers to increasing or decreasing communication resources and computing resources. If the cause of quality degradation or resource shortage is congestion by multiple applications 10, the orchestrator 40 notifies the relevant applications 10 of a change in service level and coordinates actions such as reducing the number of cameras 74 used, shrinking the drawing area or distribution area, and adjusting application 10 parameters such as service time or cycle. The orchestrator 40 may also predict future resource shortages based on its own service scheduling information and past resource usage history when detecting resource shortages.

[0052] Thus, when resources are insufficient or surplus, the cyber-physical system 1 mediates between the application 10, the orchestrator 40, and the information processing infrastructure unit 50, specifically by increasing or decreasing resources, increasing or decreasing processing, and changing priorities. For example, when communication resources are insufficient, the cyber-physical system 1 changes the processing method of the information processing infrastructure unit 50 so that data can be transmitted even with limited communication resources. Furthermore, when the processing of the information processing infrastructure unit 50 is large, the cyber-physical system 1 provides sufficient computing resources and communication resources for control. However, if necessary control information, such as maintaining periodicity, is required, it is necessary to change the priority of sending information via communication. In the cyber-physical system 1, the deterministic network unit 65 of the network processing infrastructure unit 60 appropriately controls such priorities.

[0053] In the following sections [1] through

[12] , the operation of the cyber-physical system 1 will be explained using the device configuration of the cyber-physical system 1 shown in Figure 2 as an example, focusing on the operation for understanding disaster situations for disaster prevention.

[0054] [1] In order to grasp the disaster situation in an identified disaster area, the user sets the flight routes of multiple drones 170 equipped with pre-registered cameras 74 via one of the lines 160 from application 10 so that the drones 170 go to an object or location to take camera images. The cyber-physical system 1 can set or instruct the drones 170, managed by an aircraft ID (IDentifier), on the location of the object or location to be photographed, using a wireless medium such as 5G (5th Generation), Wi-Fi (registered trademark), via the satellite line 162 or the terrestrial line 163 line 160. The position and orientation of the drones 170, as well as status information of the sensors and battery of the drones 170, can be obtained using a wireless medium at a location away from the drones 170, for example, the HMD 110 which is the application terminal of application 10.

[0055] [2] One or more drones 170 move toward an object or location to be observed for the disaster, and while moving around the area, acquire image data with the camera 74 based on instructions from a remote wireless medium, such as 5G or Wi-Fi. Time synchronization is to be performed in advance between one or more drones 170 and the ground-side communication line 160. For accurate time information, the drones 170 may use signals from a GPS (Global Positioning System) 200 (not shown) or time information supplied to the drones 170 from a base station. For GPS 200, other GNSS (Global Navigation Satellite System) may be used. In addition, the drones 170 acquire image data with the camera 74 along the flight route in order to check the movement status to the disaster area.

[0056] [3] In addition to images, the drone 170 also transmits aircraft information such as time, position, and orientation to the ground-side line 160 via the base station. If there is an edge server on the base station or on the parent drone 170a of multiple drones 170, the base station or parent drone 170a may add aircraft information such as time, position, and orientation to the images from the child drone 170b before transferring them to the line 160. The drone 170 can obtain information such as time, position, and orientation by using signals from the GPS 200, its own acceleration sensor, geomagnetic sensor, etc.

[0057] [4] As an example of edge processing that line 160 can provide, the edge processing unit 53 of the information processing infrastructure unit 50 generates a single image by combining the "simultaneous" video from multiple drones 170, that is, a single still image, and then transmits it to the next line 160. For example, in the case of a camera 74 with a frame rate of about 30fps, even if the frame period is constant, depending on the communication period, there may be a delay of up to 33ms just from frame synchronization, and a synchronization accuracy that is sufficiently shorter than that time is required.

[0058] [5] The orchestrator 40 determines whether to use satellite link 162 or terrestrial link 163 by aggregating information such as the status of link 160 along the path from application 10 to server 150, for example, the number of transfers, resource status, etc., using commands such as Ping for measuring communication delay time and Traceroute for understanding the communication path, thereby understanding the communication status over time and making a decision.

[0059] [6] If the drone 170 is located within the coverage area of ​​the base station associated with the ground-side line 160 and service is available, the application service control unit 41 of the orchestrator 40 selects and uses the ground-side line 160 route for routers and other devices connected to the edge servers 154 and 155, based on the results of resource monitoring. It is assumed that whether the drone 170 is located within the base station's coverage area and service is available will be determined by radio wave strength, the drone 170's location information, etc. If congestion occurs on the route of line 160 and the ground line 163 becomes unavailable, the application service control unit 41 of the orchestrator 40 selects a route that uses the satellite line 162.

[0060] [7] The cyber-physical system 1 selected a route and used it for transmitting information, but it transmits information to the server 150 of the target CPS server 151 or edge servers 154, 155, depending on the required computational processing, using either the satellite link 162 or the terrestrial link 163 route, or whichever route is more effective in terms of communication bandwidth, latency, etc. If the server 150 obtains information via both the satellite link 162 and the terrestrial link 163 route, it may use the information that arrived first, or it may use the information obtained from the route with a lower packet discard rate or the route with a lower RTT latency, depending on the data being handled.

[0061] [8] Server 150 of the CPS server 151 or edge servers 154, 155 performs calculation processing using information about the aircraft such as time, location, and orientation attached to the image. Server 150 of the CPS server 151 or edge servers 154, 155 creates a 3D image model for understanding the disaster situation using information such as time, location, and orientation attached to the image. In order to understand the situation in real time, for example, Server 150 of the CPS server 151 or edge servers 154, 155 creates a 3D image model at short time intervals that are in real time, and adjusts it to a size that is easy for the user to view so that it can be mapped, i.e., synthesized on an electronic map in cyberspace based on the location information of the disaster, and then outputs the synthesized 3D image from Server 150 to the router connected to Server 150. Server 150 of the CPS server 151 or edge servers 154, 155 may map 3D image models onto an electronic map as icon shapes, represent numerical data such as the number of houses in the display area using different colors or graphs with different markers, or represent areas with different colors depending on the level of danger (red or yellow, green for safe areas, etc.). The administrator will set the color identification based on the situation. Furthermore, Server 150 of the CPS server 151 or edge servers 154, 155 will synchronize the time of images obtained from multiple drones 170 and map them onto an electronic map to enable observation of multiple objects or locations. Specifically, in Server 150 of the CPS server 151 or edge servers 154, 155, the video processing unit 51A of the information processing infrastructure unit 50 performs the above processing.

[0062] [9] The information processing infrastructure unit 50 transfers the 3D image, which has been combined with a map in cyberspace, to a display device such as the HMD 110 of the user who is a remote operator. First, the server 150 of the CPS server 151 or edge servers 154, 155 outputs to the router connected to the server 150, and determines in advance whether to use the satellite link 162 or the terrestrial link 163 for transfer. To determine the route status, the information processing infrastructure unit 50 collects resource monitoring results of the link 160, such as QoS, RTT, throughput, etc., and makes a decision taking into account that there are differences in the transmission speeds of the uplink link 160 and the downlink link 160. The information processing infrastructure unit 50 may also transfer the 3D image to the user's HMD 110 using both the satellite link 162 and the terrestrial link 163.

[0063]

[10] The user sends control information to the target drone 170 to change its position so that the target object or location can be seen from the desired direction, and information on whether to turn the camera 74 on or off, in order to check the disaster situation in 3D images using the HMD 110. The control information includes, for example, the aircraft ID, position, and orientation. Such instructions are sent as control information to the server 150 of the CPS server 151 or edge servers 154, 155 via a similar route as in [9], and the slicing unit 64 of the network processing infrastructure unit 60 performs resource control at a lower level, such as deciding whether to use the satellite link 162 or the terrestrial link 163, in order to improve the response, i.e., reduce the delay.

[0064]

[11] The orchestrator 40 guides the drone 170 to return it to the recovery site, while obtaining control information of the drone 170, such as aircraft ID, position, and orientation, via satellite link 162 or ground link 163.

[0065]

[12] The platform system 20, assuming a database exists in normal times, stores information such as pre-disaster images, the position of the drone 170 when the images were taken, and the settings of the camera 74, as an application 10 that can be used to determine evacuation routes, for example, an application 10 that uses a 2D camera. Pre-disaster images are, for example, rough shots of overhead images or omnidirectional images of specific objects. This allows the platform system 20 to enable the drone 170 to take images after the disaster under the same conditions. The platform system 20 may also maintain conditions that allow it to process images as if they were taken under the same conditions, even if the shooting conditions are different. This allows the user to roughly check the extent of the disaster by comparing the situation before and after the disaster. It is assumed that base information is necessary when checking the detailed situation by operating the drone 170. The priority of shooting points is set by notifications, seismic intensity information, and pre-settings. Pre-settings include, for example, registering important locations. The platform system 20 can also be used to estimate earthquakes based on disaster situation information.

[0066] The operations described in [1] through

[12] above will be explained using sequence diagrams and other visual aids. Here, we will specifically explain the operation in which a user uses the HMD110 to check the disaster situation in area A, which is a disaster-stricken area, as captured by the drone 170.

[0067] Figure 8 is a sequence diagram showing the operation in the cyber-physical system 1 according to this embodiment until the drone 170 is ready for use in response to instructions from application 10. Application 10 instructs the service enabler 30 of the platform system 20 to take images with the drone 170 using the keywords "Area A" and "3D shooting" (step S601). The service enabler 30 instructs the application service control unit 41 of the orchestrator 40 to take images with the drone 170 using the keywords "Area A" and "3D shooting" (step S602). The application service control unit 41 of the orchestrator 40 performs shooting preparation processing with the drone 170, etc. (step S603). Details of the shooting preparation processing will be described later. Next, the application service control unit 41 of the orchestrator 40 performs training processing with the drone 170, etc. (step S604). Details of the training processing will be described later.

[0068] The application service control unit 41 of the orchestrator 40 responds to the instruction in step S602 from the service enabler 30 (step S605). The response includes the estimated time for acquiring the 3D images. The service enabler 30 responds to the instruction in step S601 from the application 10 (step S606). The response includes information such as whether acquisition is possible, the estimated acquisition time, training completion, communication quality of the satellite link 162, and communication quality of the terrestrial link 163. The application service control unit 41 of the orchestrator 40 performs monitoring processing of the link 160 as a periodic process (step S607). Details of the monitoring processing will be described later.

[0069] Figure 9 is a sequence diagram showing the detailed operation of the shooting preparation process in step S603 shown in Figure 8 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 gives a preparation instruction to the information processing infrastructure unit 50 in area A, using 3D shooting as the keyword (step S701). The information processing infrastructure unit 50 in area A instructs the drone 170 to prepare for shooting, using the aircraft ID or camera ID as the keyword (step S702). The drone 170 synchronizes its time with the GPS 200 (step S703). The drone 170 responds to the instruction in step S702 in the information processing infrastructure unit 50 in area A (step S704). The response includes information such as the aircraft ID or camera ID and time error. The information processing infrastructure unit 50 in area A responds to the instruction in step S701 in the application service control unit 41 of the orchestrator 40 (step S705). The response will include information such as whether the data can be obtained and the estimated time.

[0070] Figure 10 is a sequence diagram showing the detailed operation of the training process in step S604 shown in Figure 8 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 gives a preparation instruction to the information processing infrastructure unit 50 in area A with 3D shooting as the keyword (step S801). The information processing infrastructure unit 50 in area A gives a training instruction to the network processing infrastructure unit 60 in area A with aircraft ID or camera ID and 3D as the keywords (step S802). The network processing infrastructure unit 60 in area A gives a training instruction to the drone 170 with Path and 3D as the keywords (step S803).

[0071] Drone 170 transmits test data to the information processing infrastructure unit 50 located in area A (step S804). The test data includes, for example, data captured by Drone 170 for testing purposes. Drone 170 transmits drone information to the information processing infrastructure unit 50 located in area A (step S805). The drone information includes, for example, the battery level. Drone 170 provides a training response to the training instruction in step S803 to the network processing infrastructure unit 60 located in area A (step S806). The training response includes information such as training completion. The network processing infrastructure unit 60 located in area A provides a training completion response to the training instruction in step S802 to the information processing infrastructure unit 50 located in area A (step S807). The training completion response includes information such as training completion, communication quality value of satellite link 162, and communication quality value of ground link 163. The information processing infrastructure unit 50 in area A sends a response to the application service control unit 41 of the orchestrator 40 indicating that training is complete in response to the preparation instruction in step S801 (step S808). The response includes information such as training completion, the communication quality value of the satellite link 162, and the communication quality value of the terrestrial link 163.

[0072] Figure 11 is a sequence diagram showing the detailed operation of the monitoring process in step S607 shown in Figure 8 in the cyber-physical system 1 according to this embodiment. The network processing infrastructure unit 60 in area A instructs the lines 160, such as the satellite line 162 and the terrestrial line 163, to prepare the resource monitor (step S901). The lines 160, such as the satellite line 162 and the terrestrial line 163, provide a resource monitor response to the resource monitor preparation instruction in step S901 to the network processing infrastructure unit 60 in area A (step S902). The resource monitor response includes the bandwidth value, delay value, etc. of the line 160. Thereafter, the lines 160, such as the satellite line 162 and the terrestrial line 163, provide similar resource monitor responses to the network processing infrastructure unit 60 in area A. The application service control unit 41 of the orchestrator 40 instructs the network processing infrastructure unit 60 in area A to perform resource monitoring using area A as the keyword (step S903). The network processing infrastructure unit 60 in area A provides a resource monitor response to the resource monitor instruction in step S903 to the application service control unit 41 of the orchestrator 40 (step S904). The resource monitor response includes information such as the average bandwidth value and average delay value, based on resource monitor responses obtained multiple times by the network processing infrastructure unit 60 in area A from the lines 160, such as the satellite line 162 and the terrestrial line 163.

[0073] Figure 12 is a sequence diagram showing the operation in which the expected shooting time by the drone 170 is notified to the application 10 in the cyber-physical system 1 according to this embodiment. The application 10 gives instructions to the service enabler 30 using keywords such as area A, 3D shooting, and line specification (step S1001). The service enabler 30 gives instructions to the application service control unit 41 of the orchestrator 40 using keywords such as area A, 3D shooting, and line specification (step S1002). The application service control unit 41 of the orchestrator 40 gives instructions to the information processing infrastructure unit 50 located in area A using keywords such as area A, 3D shooting, and line specification (step S1003). The information processing infrastructure unit 50 located in area A responds to the instructions in step S1003 from the application service control unit 41 of the orchestrator 40 (step S1004). The response includes the expected shooting time, etc. The application service control unit 41 of the orchestrator 40 responds to the instruction in step S1002 from the service enabler 30 (step S1005). The response includes the estimated shooting time, etc. The service enabler 30 responds to the instruction in step S1001 from the application 10 (step S1006). The response includes the estimated shooting time, etc.

[0074] (1-1) Thus, in the cyber-physical system 1, application 10 instructs service enabler 30 that it wants to "use drone 170 to view the disaster situation in area A in 3D images."

[0075] (1-2) The service enabler 30 selects the network processing infrastructure unit 60 and the information processing infrastructure unit 50 closest to area A from the application service control unit 41 of the orchestrator 40. The information processing infrastructure unit 50 searches for the number of drones 170 in area A required for generating 3D images, calculates the arrival time of the drones 170, the estimated time for acquiring 3D images, etc., and notifies the application service control unit 41 of the orchestrator 40 of the presence or absence of the drones 170, and if present, the estimated time for acquiring 3D images by the drones 170, etc.

[0076] (1-3) The information processing platform unit 50 sets the aircraft ID or camera ID of multiple drones 170. The multiple drones 170 synchronize their time using GPS 200 or the like via the communication line 160 or the like. If there is a time difference among the multiple drones 170, the information processing platform unit 50 synchronizes the time of the multiple drones 170.

[0077] (1-4) During a disaster, communications may be disrupted, or there may be restrictions on communications, such as communication restrictions. Therefore, the application service control unit 41 of the orchestrator 40 monitors the availability of wireless communication at the site in area A using resource monitoring.

[0078] (1-5) In addition, if possible during a disaster, the application service control unit 41 of the orchestrator 40 requests the telecommunications company providing the public network to secure the communication resources necessary for image capture near the site in area A. Specifically, the application service control unit 41 of the orchestrator 40 secures the line 160 by requesting the telecommunications company of one of the available satellite lines 162 or terrestrial lines 163 from among the multiple public networks to set up the slice so that the target application 10 can be used preferentially.

[0079] (1-6) Before or during the flight of the drone 170, the network processing platform 60 transmits video from the camera 74 via the drone 170 as training for transmitting video from the camera 74, and checks the communication status, such as QoS, frame error rate, throughput, etc. The shooting time of the drone 170 depends on the remaining battery charge of the drone 170's flight battery, the remaining battery charge of the camera 74, and the remaining battery charge of the communication equipment. Therefore, the information processing platform 50 collects information on the remaining charge of each battery via the line 160 and calculates the continuous shooting time of the drone 170. The information processing platform 50 also considers the battery consumption of the communication equipment when communicating via the satellite line 162 as well as the ground line 163 as a means of communication.

[0080] (1-7) The application service control unit 41 of the orchestrator 40 selects one of the terrestrial line 163 and satellite line 162, or two lines 160 to broaden the service area coverage, as a communication means, based on the results of the communication quality confirmed by the network processing infrastructure unit 60 and the results such as the available shooting time calculated by the information processing infrastructure unit 50. The application service control unit 41 of the orchestrator 40 may combine different lines 160 of the terrestrial public network, or it may combine the terrestrial line 163 and the satellite line 162. The operations from (1-1) to (1-7) up to this point correspond to the operations in the sequence diagrams in Figures 8 to 11 described above.

[0081] (1-8) After the means of communication is determined, the power consumption of the drone 170, etc., is known, so first the information processing platform 50 considers the shooting time taking into account the remaining battery level mentioned above. The orchestrator 40 transmits the results of the shooting time consideration by the information processing platform 50 to the application 10 via the service enabler 30. This operation (1-8) corresponds to the operation in the sequence diagram of Figure 12 mentioned above. Also, the operations from (1-1) to (1-8) correspond to the operation in [1] mentioned above.

[0082] This allows the platform system 20 to configure the available lines 160 for the drone 170 along its flight path. If there are multiple drones 170, the platform system 20 can also set the priority of each drone 170 and assign a role to each drone 170. The role of each drone 170 could be, for example, the placement of each drone 170 when multiple drones 170 are flying in formation, or the shooting direction of each drone 170 when multiple drones 170 are photographing an object or location.

[0083] Figure 13 is a sequence diagram showing the operation in which application 10 acquires spatial object images in the cyber-physical system 1 according to this embodiment. Multiple drones 170 transmit data such as images taken by cameras 74 to the information processing infrastructure unit 50 located in area A (step S1101). The information processing infrastructure unit 50 performs video processing to generate a spatial representation image that can be displayed on the HMD 110 using the images taken by cameras 74 included in the data acquired from the drones 170 (step S1102). The spatial representation image is, for example, the aforementioned 3D image. The information processing infrastructure unit 50 transmits the generated spatial representation image to the HMD 110, which is the application terminal of application 10, via a service enabler 30 (not shown) (step S1103). The HMD 110, which is the application terminal of application 10, displays the acquired spatial representation image and performs object detection processing (step S1104). The information processing infrastructure unit 50 issues a movement instruction to the drone 170, including data on the movement position, using the aircraft ID or camera ID as a keyword (step S1105). The drone 170 issues a movement response to the information processing infrastructure unit 50, including data on the current position and the movement position, using the aircraft ID or camera ID as a keyword (step S1106). The information processing infrastructure unit 50 performs monitoring processing at the drone 170's position based on the current position information obtained from the drone 170 (step S1107).

[0084] Figure 14 is a sequence diagram showing the operation of checking the communication status between the drone 170 and the HMD 110, which is the application terminal of application 10, in the cyber-physical system 1 according to this embodiment. The information processing infrastructure unit 50 instructs the drone 170 to check the communication status using Ping and Traceroute, with the aircraft ID or camera ID as a keyword (step S1201). The drone 170 checks the communication status with the HMD 110, which is the application terminal of application 10, using Ping and Traceroute, that is, it executes Ping commands and Traceroute commands (step S1202). The drone 170 responds to the information processing infrastructure unit 50 for the instructions in step S1201 (step S1203). The response includes the results of the communication status check using Ping and Traceroute.

[0085] (2-1) In this way, in the cyber-physical system 1, when the drone 170 has started taking pictures, the information processing infrastructure unit 50 transmits the spatial representation video, which has been spatially reproduced through video processing, to the HMD 110, which is the application terminal of application 10. The HMD 110, which is the application terminal of application 10, performs object detection processing in the virtual space. In the object detection processing, for example, it detects the presence or absence of infrastructure such as fallen trees and utility poles, or collapsed buildings, river floods, and missing persons.

[0086] (2-2) The drone 170 changes the zoom of the camera 74, the attitude of the drone 170, the orientation of the camera 74, etc. The camera 74 is used differently depending on the application, depending on the type of camera itself, such as an infrared camera, and the type of lens, such as a wide-angle lens or a telephoto lens. In the cyber-physical system 1, if there are multiple drones 170, the combination of the camera 74 and lens type mounted on each drone 170 may be made different so that the drone 170 to be used can be selected based on the combination of camera 74 and lens. The edge processing unit 53 of the information processing infrastructure unit 50 controls the position of the drones 170 that have gathered in area A in order to photograph area A, which is a disaster area, and moves the drones 170 after determining the position of the drones 170. The position of the drones 170 may be determined using GPS 200, and the angle and orientation of the drones 170 may be determined using an acceleration sensor (not shown), which is one of the sensing units 73 mounted on the drone 170. When moving the drone 170, the information processing platform 50 controls the operation of the actuator unit 76, which is a motor that drives the propellers and other components mounted on the drone 170.

[0087] (2-3) In the cyber-physical system 1, the position information of the drone 170 is constantly shared through edge processing related to remote control, making it usable for flight route planning. The operations from (2-1) to (2-3) above correspond to the operations in the sequence diagram of Figure 13 mentioned above.

[0088] (2-4) In the cyber-physical system 1, clustering and spatiotemporal synchronization processing are performed while communicating between multiple drones 170 in a formation configuration. At this time, a high-precision time reference between the multiple drones 170 may be realized using the aforementioned GPS 200 or PTP (Precision Time Protocol). When multiple drones 170 are connected in a V2N2V (Vehicle to Network to Vehicle) network configuration, each drone 170 connects to the base station's edge servers 154, 155 via wireless communication, and the clustering unit 71 performs clustering processing and the spatiotemporal synchronization unit 72 performs spatiotemporal synchronization processing in the device group 70 included in the drone 170. In addition, in a V2V (Vehicle to Vehicle) network configuration, the drone 170 that becomes the parent drone 170a may have edge functionality, and the parent drone 170a may control the shooting position and shooting direction of the camera 74 of the child drone 170b in a subordinate manner. Each drone 170 selects a line 160 that best meets the required communication quality while checking the results of monitoring by the resource monitoring unit 61 of the network processing infrastructure unit 60 to see if a line 160 is available. As a result, depending on the drone 170, the use of a satellite line 162 or a public line 160 that is different from the usual is also possible.

[0089] (2-5) To determine whether to use satellite link 162 or terrestrial link 163, the orchestrator 40 gathers information such as the status of link 160 along the path from application 10 to server 150, for example, the number of transfers, resource status, etc., using commands such as Ping for measuring communication delay time and Traceroute for understanding the communication path, thereby understanding the communication status over time and making a decision.

[0090] (2-6) If the drone 170 is located within the coverage area of ​​the base station associated with the ground-side line 160 and service is available, the application service control unit 41 of the orchestrator 40 selects and uses the ground-side line 160 route for routers and other devices connected to the edge servers 154 and 155, based on the results of resource monitoring. It is assumed that whether the drone 170 is located within the base station's coverage area and service is available will be determined by the reception strength of radio waves transmitted from the base station and the drone 170's location information. If congestion occurs on the route of line 160 and the ground line 163 becomes unavailable, the application service control unit 41 of the orchestrator 40 selects a route that uses the satellite line 162. The operations from (2-4) to (2-6) up to this point correspond to the operations in the sequence diagram of Figure 14 mentioned above.

[0091] (3-1) The drone 170 has selected the route of line 160, but will use either the satellite line 162 or the terrestrial line 163 route, or whichever line 160 is available based on communication capacity, latency, etc., to transmit information to the target CPS server 151 or edge servers 154, 155 according to the required computational processing. If information arrives from both lines 160, the CPS server 151 or edge servers 154, 155 may use the information that arrived first, or may select the line 160 according to the application 10 being handled and use the information. Regarding the communication route from the drone 170 to the HMD 110, which is the application terminal of application 10, the drone 170 will make a choice to allocate resources according to application 10, or to change the operation of application 10 and the device group 70 according to the available resources. Regarding the latter, the uplink communication settings are changed based on the resolution × fps related to the function of the sensing unit 73, for example, as it relates to the amount of image data that can be transmitted, and the downlink communication settings are changed based on the degree of autonomy of the control of the drone 170 by the information processing infrastructure unit 50, for example, as it relates to the delay in the transfer of control signals to the drone 170. The delay in the transfer of control signals to the drone 170 can be determined because the resource monitoring unit 61 of the network processing infrastructure unit 60 constantly monitors the RTT. The operations from (2-1) to (2-6) and (3-1) correspond to the operations from [2] to [7] described above.

[0092] Figure 15 is a sequence diagram showing the operation of a child drone 170b transmitting data via a parent drone 170a in the cyber-physical system 1 according to this embodiment. The child drone 170b transmits data to the information processing infrastructure unit 50 (step S1301). However, it is assumed that the data transmitted from the child drone 170b did not reach the information processing infrastructure unit 50. The parent drone 170a transmits data to the information processing infrastructure unit 50 (step S1302). Since the data transmitted from the parent drone 170a reached the information processing infrastructure unit 50, the information processing infrastructure unit 50 gives a reception response to the parent drone 170a for the data transmission (step S1303).

[0093] The information processing infrastructure unit 50 determines that it has not received data from the child drone 170b and performs the following actions to acquire data from the child drone 170b via the parent drone 170a. First, the information processing infrastructure unit 50 issues a movement instruction to the child drone 170b, including the new location, using the aircraft ID or camera ID as a keyword (step S1304). The child drone 170b issues a movement response to the information processing infrastructure unit 50, including its current location and new location, using the aircraft ID or camera ID as a keyword (step S1305). The information processing infrastructure unit 50 issues a V2 line establishment instruction to the parent drone 170a and the child drone 170b, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords (step S1306). The parent drone 170a issues a V2 line establishment response to the information processing infrastructure unit 50, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords. Similarly, the child drone 170b sends a V2 line establishment response to the information processing infrastructure unit 50 using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords (step S1307).

[0094] The child drone 170b relays data to the parent drone 170a (step S1308). The parent drone 170a transmits the data acquired from the child drone 170b to the information processing infrastructure unit 50 (step S1309). Since the data from the child drone 170b transmitted from the parent drone 170a has reached the information processing infrastructure unit 50, the information processing infrastructure unit 50 sends a reception response to the parent drone 170a for the data transmission (step S1310). When the parent drone 170a receives the reception response from the information processing infrastructure unit 50 for the data transmission of the child drone 170b, it sends a reception response to the child drone 170b for the data transmission on behalf of the information processing infrastructure unit 50 (step S1311).

[0095] Thus, if the information processing platform 50 is unable to obtain the first information from the child drone 170b, which is the first device included in the device group 70, but can obtain the second information from the parent drone 170a, which is the second device included in the device group 70, and communication is possible between the child drone 170b and the parent drone 170a, the information processing platform 50 instructs the child drone 170b to transmit the first information to the parent drone 170a, and obtains both the first and second information from the parent drone 170a.

[0096] Figure 16 is a sequence diagram showing the operation in which the parent drone 170a synthesizes data acquired from the child drone 170b and transmits the data in the cyber-physical system 1 according to this embodiment. The video processing unit 51A of the information processing infrastructure unit 50 requests network resource information from the network processing infrastructure unit 60 using the aircraft ID or camera ID of the drone 170 as a keyword (step S1401). The network processing infrastructure unit 60 notifies the video processing unit 51A of the information processing infrastructure unit 50 of network resource information in response to the network resource information request, using the aircraft ID or camera ID of the drone 170 as a keyword (step S1402). The network resource information notification includes network resource information.

[0097] The video processing unit 51A of the information processing infrastructure unit 50 determines, based on network resource information obtained from the network processing infrastructure unit 60, that there are sufficient network resources between the information processing infrastructure unit 50 and the parent drone 170a, but not sufficient network resources between the information processing infrastructure unit 50 and the child drone 170b. Therefore, the video processing unit 51A of the information processing infrastructure unit 50 performs the following operations to synthesize data from the child drone 170b at the parent drone 170a before acquiring it. First, the video processing unit 51A of the information processing infrastructure unit 50 issues a video synthesis instruction to the parent drone 170a using the aircraft ID or camera ID and 2in1 as keywords (step S1403). The parent drone 170a issues a video synthesis response to the video synthesis instruction to the video processing unit 51A of the information processing infrastructure unit 50 using the aircraft ID or camera ID and 2in1 as keywords (step S1404). The video processing unit 51A of the information processing infrastructure unit 50 issues a V2 line establishment instruction to the parent drone 170a and the child drone 170b, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords (step S1405). The parent drone 170a responds to the video processing unit 51A of the information processing infrastructure unit 50 with a V2 line establishment response, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords. Similarly, the child drone 170b responds to the video processing unit 51A of the information processing infrastructure unit 50 with a V2 line establishment response, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as keywords (step S1406).

[0098] The child drone 170b transmits data to the parent drone 170a (step S1407). The parent drone 170a performs a 2-in-1 process to combine the image data acquired by its mounted camera 74 with the image data acquired from the child drone 170b (step S1408). The parent drone 170a transmits the image data obtained as a result of the 2-in-1 process to the information processing infrastructure unit 50 (step S1409). The information processing infrastructure unit 50 responds to the parent drone 170a for receiving the data transmission (step S1410). The parent drone 170a responds to the child drone 170b for receiving the data transmission (step S1411).

[0099] Thus, the information processing infrastructure unit 50 includes a child drone 170b, which is a first device that transmits first information to the device group 70, and a parent drone 170a, which is a second device that transmits second information. Communication with the parent drone 170a is possible, and if communication is possible between the child drone 170b and the parent drone 170a, the information processing infrastructure unit 50 instructs the child drone 170b to transmit first information to the parent drone 170a, and obtains combined information from the parent drone 170a, which is a combination of the first and second information.

[0100] Figure 17 is a sequence diagram showing the operation when the orchestrator 40 instructs the drone 170 to change operating conditions such as the shooting conditions of the camera 74 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues an encoder change instruction to the information processing infrastructure unit 50 in area A, using the drone 170's aircraft ID or camera ID as a keyword (step S1501). The encoder change instruction includes frame rate, resolution, bitrate, compression method, etc. The information processing infrastructure unit 50 in area A issues an encoder change instruction to the drone 170, using the drone 170's aircraft ID or camera ID as a keyword (step S1502). The drone 170 issues an encoder change response to the encoder change instruction to the information processing infrastructure unit 50 in area A, using the drone 170's aircraft ID or camera ID as a keyword (step S1503). The information processing infrastructure unit 50 in area A sends an encoder change response to the application service control unit 41 of the orchestrator 40, using the aircraft ID or camera ID of the drone 170 as a keyword (step S1504).

[0101] Figure 18 is a sequence diagram showing the operation of the HMD 110, which is the application terminal for application 10, in the cyber-physical system 1 according to this embodiment until it acquires 3D images. Multiple drones 170 transmit image data captured by cameras 74 to the video processing unit 51A of the information processing infrastructure unit 50 (step S1601). The data includes image data, time information at the time of capture, position and orientation of the camera 74, and information obtained by sensors such as acceleration sensors and lidars included in the sensing unit 73. The video processing unit 51A of the information processing infrastructure unit 50 performs synthesis processing to combine the image data acquired from the multiple drones 170 (step S1602). The HMD 110, which is the application terminal for application 10, gives an instruction to the application service control unit 41 of the orchestrator 40 to acquire images using "area A" and "image" as keywords (step S1603). The application service control unit 41 of the orchestrator 40 issues an instruction to the video processing unit 51A of the information processing infrastructure unit 50 to acquire video using "Area A" and "video" as keywords (step S1604).

[0102] The video processing unit 51A of the information processing infrastructure unit 50 transmits video to the HMD 110, which is the application terminal of application 10, via a service enabler 30 (not shown), without going through the application service control unit 41 of the orchestrator 40, in response to an acquisition instruction from the application service control unit 41 of the orchestrator 40 (step S1605). Alternatively, the video processing unit 51A of the information processing infrastructure unit 50 transmits video to the application service control unit 41 of the orchestrator 40 in response to an acquisition instruction from the application service control unit 41 of the orchestrator 40 (step S1606). The application service control unit 41 of the orchestrator 40 transmits video to the HMD 110, which is the application terminal of application 10, via a service enabler 30 (not shown), in response to an acquisition instruction from the HMD 110, which is the application terminal of application 10 (step S1607).

[0103] (4-1) In this way, in the cyber-physical system 1, if a drone 170 is unable to transmit video from camera 74, the drone 170 that is unable to transmit video from camera 74 buffers the video information and waits until it becomes possible to transmit video from camera 74. For example, as one solution, the remote control unit 51E of the information processing infrastructure unit 50 checks whether the radio wave conditions between the drone 170 and the line 160 can be improved, and in order to check whether the radio wave conditions between the drone 170 and the line 160 can be improved, it instructs the drone 170 to move its position and moves the drone 170 by a small distance of several times the wavelength of the radio signal. Alternatively, if communication cannot be expected to be restored by the above measures, in order to prevent the transmission of video from camera 74 from becoming impossible, the drone 170 that is unable to transmit video from camera 74 is designated as a child drone 170b, and a V2V communication path is set up between the parent drone 170a and the child drone 170b, so that the child drone 170b transmits video from camera 74 to the parent drone 170a, and the video from camera 74 is transmitted via the parent drone 170a. In this case, since the video from the parent drone 170a's own camera 74 is also transmitted, the transmission speed of video from the drone 170 increases for the cyber-physical system 1. The operation described in (4-1) up to this point corresponds to the operation shown in the sequence diagram of Figure 15 mentioned above.

[0104] (4-2) If the network processing infrastructure unit 60 determines that there is insufficient communication speed as a result of resource monitoring, the video-related edge processing on the parent drone 170a combines the images from the two cameras 74 of the parent drone 170a and the child drone 170b into a single image to reduce the image size. The video processing unit 51A of the information processing infrastructure unit 50 is responsible for suppressing the increase in the transmission speed of the communication terminal of the parent drone 170a. The operations in (4-2) up to this point correspond to the operations in the sequence diagram of Figure 16 mentioned above.

[0105] (4-3) If there is a drone 170 with insufficient communication speed and it is necessary to generate a 3D image, the orchestrator 40 changes the bitrate of the encoders of the other drones 170 to match the communication speed of the drone 170 determined by the resource monitoring unit 61 of the network processing infrastructure unit 60 that performs the operation shown in Figure 11.

[0106] (4-4) The network processing infrastructure unit 60 performs resource monitoring for each line 160, including received signal strength and QoS, and considers throughput. For example, the orchestrator 40 adjusts to the required rate by using a conversion function such as lowering the rate of the camera 74 required by the information processing infrastructure unit 50. Alternatively, the orchestrator 40 responds by adjusting all of the multiple cameras 74 to the minimum bitrate. Or, in the event of a disaster, if the line 160 itself does not have sufficient capacity, the orchestrator 40 creates capacity on the line 160 by stopping the transmission of video from the cameras 74 of multiple drones 170, and responds to the decrease in communication capacity by switching to the transmission of still images from the camera 74 of at least one drone 170 via the information processing infrastructure unit 50. The operations in (4-4) up to this point correspond to the operations in the sequence diagram of Figure 17 mentioned above.

[0107] (4-5) The location information of the drone 170 is constantly shared by the remote control-related edge processing unit 53 in the information processing infrastructure unit 50, and can be used to consider the flight route of the drone 170.

[0108] (4-6) For positioning one or more drones 170, in order to find the location of rubble and other objects as targets, the drones 170 add time information, the position and orientation of the camera 74, acceleration sensor information, LiDAR information, etc. to the video from the camera 74 at the network processing platform 60 and transmit it to the video processing unit 51A of the information processing platform 50. The HMD 110, which is the application terminal of application 10, acquires the video processing results, checks the local situation, and sets the position of the drones 170. When generating a 3D image using multiple drones 170, the position of the drones 170 or the orientation of the camera 74 is set so that the target object or target location is captured in the video from the camera 74. The target object or target location may be a characteristic object in the event of a disaster, or target location information indicated by latitude, longitude, altitude, etc., or marker information may be used.

[0109] (4-7) Alternatively, a user using the HMD110 may remotely adjust the position of each drone 170 so that the target object or location is visible to the camera 74 while monitoring the video feed from the 2D camera 74 of each drone 170. The operations described in (4-5) through (4-7) above correspond to the operations shown in the sequence diagram of Figure 18.

[0110] (4-8) If communication with a specific drone 170 fails, the network processing infrastructure unit 60 performs resource monitoring using Ping for measuring communication delay time and Traceroute for determining the communication path, etc., to check the communication status of the network processing infrastructure unit 60. Based on the results, the remote control processing unit of the information processing infrastructure unit 50 changes the position of the drone 170 in order to improve the communication status, according to instructions from the orchestrator 40.

[0111] (4-9) The network processing platform 60 also sends control information to the drone 170 to change the position of the drone 170, which is targeting the image from the camera 74, as well as the image captured by the camera 74, and gives instructions to the drone 170. The control information to change the position of the drone 170 includes, for example, the aircraft ID, the position of the drone 170, the orientation of the drone 170, and the image from the camera 74 of the remotely controlled drone 170. When the deterministic network unit 65 sends such instructions, the network processing platform 60 finely controls resources in the communication path to improve the response and periodicity of the communication as control information to the CPS server 151.

[0112] Figure 19 is a sequence diagram showing the operation in which the information processing infrastructure unit 50 performs calculation processing and notifies the application 10 of the estimated shooting time by the drone 170 in the cyber-physical system 1 according to this embodiment. The application 10 gives instructions to the service enabler 30 using keywords such as area A, 3D shooting, and line specification (step S1701). The service enabler 30 gives instructions to the application service control unit 41 of the orchestrator 40 using keywords such as area A, 3D shooting, and line specification (step S1702). The application service control unit 41 of the orchestrator 40 gives instructions to the video processing unit 51A of the information processing infrastructure unit 50 located in area A using keywords such as area A, 3D shooting, and line specification (step S1703).

[0113] The video processing unit 51A of the information processing infrastructure unit 50 located in area A performs calculation processing for 3D shooting based on keywords such as area A, 3D shooting, and line specification (step S1704). The video processing unit 51A of the information processing infrastructure unit 50 located in area A responds to the instruction in step S1703 to the application service control unit 41 of the orchestrator 40 (step S1705). The response includes the estimated shooting time, etc. The application service control unit 41 of the orchestrator 40 responds to the instruction in step S1702 to the service enabler 30 (step S1706). The response includes the estimated shooting time, etc. The service enabler 30 responds to the instruction in step S1701 to the application 10 (step S1707). The response includes the estimated shooting time, etc.

[0114] Figure 20 is a sequence diagram showing the operation of resource monitoring of the information processing infrastructure unit 50 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues a resource monitoring instruction to the network processing infrastructure unit 60 located in area A, using area A as the keyword (step S1801). The network processing infrastructure unit 60 located in area A issues a resource monitoring instruction to the video processing unit 51A of the first information processing infrastructure unit 50 located in area A (step S1802). The video processing unit 51A of the first information processing infrastructure unit 50 located in area A issues a preparation for resource monitoring to the video processing units 51A of the second to nth information processing infrastructure units 50 located in area A, using satellite link 162 or terrestrial link 163 as the keyword (step S1803). The video processing unit 51A of the second to nth information processing infrastructure unit 50 located in area A provides a resource monitor response to the resource monitor preparation instruction in step S1803 to the video processing unit 51A of the first information processing infrastructure unit 50 located in area A, using the satellite link 162 or terrestrial link 163 as a keyword (step S1804). The resource monitor response includes bandwidth values, delay values, etc., for the satellite link 162 or terrestrial link 163.

[0115] The video processing unit 51A of the first information processing infrastructure unit 50 located in area A provides a resource monitor response to the resource monitor instruction in step S1802 to the network processing infrastructure unit 60 located in area A (step S1805). The resource monitor response includes bandwidth values, delay values, etc., for the satellite link 162 or terrestrial link 163 in the video processing unit 51A of the first to nth information processing infrastructure units 50 located in area A. The network processing infrastructure unit 60 located in area A provides a resource monitor response to the resource monitor instruction in step S1801 to the application service control unit 41 of the orchestrator 40 (step S1806). The resource monitor response includes bandwidth values, delay times, etc., for the satellite link 162 or terrestrial link 163 in the video processing unit 51A of the first to nth information processing infrastructure units 50 located in area A.

[0116] The application service control unit 41 of the orchestrator 40 issues a resource monitor instruction to the video processing unit 51A of the first information processing infrastructure unit 50, using the video processing unit 51A of the first information processing infrastructure unit 50 as a keyword (step S1807). The video processing unit 51A of the first information processing infrastructure unit 50 issues a resource monitor response to the resource monitor instruction in step S1807 to the application service control unit 41 of the orchestrator 40 (step S1808). The resource monitor response includes identification information of the first information processing infrastructure unit 50, the computer usage rate of the video processing unit 51A of the first information processing infrastructure unit 50, and the memory capacity. Similar exchanges to steps S1807 and S1808 also take place between the application service control unit 41 of the orchestrator 40 and the video processing units 51A of the second to (n-1) information processing infrastructure units 50. The application service control unit 41 of the orchestrator 40 issues a resource monitor instruction to the video processing unit 51A of the nth information processing infrastructure unit 50, using the video processing unit 51A of the nth information processing infrastructure unit 50 as a keyword (step S1809). The video processing unit 51A of the nth information processing infrastructure unit 50 issues a resource monitor response to the resource monitor instruction in step S1809 to the application service control unit 41 of the orchestrator 40 (step S1810). The resource monitor response includes identification information of the nth information processing infrastructure unit 50, the computer usage rate of the video processing unit 51A of the nth information processing infrastructure unit 50, and the memory capacity.

[0117] Figure 21 is a sequence diagram showing the operation of the Cyber-Physical System 1 according to this embodiment, from the time the HMD 110, which is the application terminal of application 10, requests and acquires map information. The HMD 110, which is the application terminal of application 10, issues an acquisition instruction to the application service control unit 41 of the orchestrator 40, requesting a 3D image map using "Area A" as the keyword (step S1901). The application service control unit 41 of the orchestrator 40 issues an acquisition instruction to the video processing unit 51A of the information processing infrastructure unit 50 located in Area A, requesting a 3D image map using "Area A" as the keyword (step S1902). The video processing unit 51A of the information processing infrastructure unit 50 located in Area A performs the 3D model creation process (step S1903) and performs the map processing to map the created 3D model onto the map (step S1904). The video processing unit 51A of the information processing infrastructure unit 50 located in area A notifies the application service control unit 41 of the orchestrator 40 of the amount of communication resources required, using area A as the keyword (step S1905). The notification includes information such as that the resource target is the creation of a 3D image map, the type of application 10, and the specific amount of resources. The video processing unit 51A of the information processing infrastructure unit 50 located in area A transmits the 3D image map data created in steps S1903 and S1904 to the HMD 110, which is the application terminal of application 10 (step S1906).

[0118] (5-1) In this way, in the cyber-physical system 1, the service enabler 30 instructs the video processing unit 51A of the information processing infrastructure 50 to generate a 3D image for understanding the disaster situation, based on instructions from the application 10. Based on instructions from the service enabler 30, the video processing unit 51A of the information processing infrastructure 50 performs calculation processing using the video from the camera 74 obtained from the drone 170 and information added to it, such as time, position, and orientation of the drone 170. The operation up to (5-1) corresponds to the operation in the sequence diagram of Figure 19 mentioned above.

[0119] (5-2) The video processing unit 51A of the information processing infrastructure unit 50 performs resource monitoring so that a suitable computer for edge processing can be selected from among the multiple computers connected by the line 160. In addition, the application service control unit 41 of the orchestrator 40 performs processing to check in advance whether the target computation can be performed, for example, whether there is available computing power, and whether the computer is located in a place with low communication processing delay, as checked by the resource monitor of the network processing infrastructure unit 60. The operation of (5-2) up to this point corresponds to the operation in the sequence diagram of Figure 20 mentioned above.

[0120] (5-3) The video processing unit 51A of the information processing infrastructure unit 50 creates a 3D image model at short time intervals that are in real time, and maps the created 3D image model onto an electronic map in cyberspace based on the location information of the disaster. In other words, the video processing unit 51A of the information processing infrastructure unit 50 combines the 3D image model and the map information.

[0121] (5-4) The video processing unit 51A of the information processing infrastructure unit 50 further adjusts the synthesized map, i.e., the 3D image, to a size that is easy for the user to view. The 3D image whose size has been adjusted by the video processing unit 51A of the information processing infrastructure unit 50 is output from the CPS server 151 or the like to the router on line 160.

[0122] (5-5) The video processing unit 51A of the information processing infrastructure unit 50 may map the 3D image model as an icon shape on an electronic map, or it may represent it by color-coding it according to the degree of danger. For example, the color coding can be used to represent danger from high to low, for example, red for danger, yellow for caution, and green for safety. The administrator using application 10 can set the color coding based on the situation.

[0123] (5-6) Furthermore, the video processing unit 51A of the information processing infrastructure unit 50 can, in response to instructions from the application 10, map images obtained from multiple drones 170 at different shooting locations onto an electronic map in a time-synchronized manner, thereby enabling the HMD 110, which is the application terminal of the application 10, to simultaneously observe multiple objects or target locations. In addition, by displaying the positions of the drones 170 on the map, the user using the HMD 110, which is the application terminal of the application 10, can also grasp the position and other status of the drones 170 observing the disaster. The operations from (5-3) to (5-6) above correspond to the operations in the sequence diagram of Figure 21 mentioned above. Also, the operations from (5-1) to (5-6) correspond to the operations in [8] mentioned above.

[0124] Figure 22 is a sequence diagram showing the operation when a user using the HMD110, which is the application terminal for application 10, wants to change the direction in which they view an object or target location in the cyber-physical system 1 according to this embodiment. The HMD110, which is the application terminal for application 10, gives instructions to the service enabler 30 using "Area A" as a keyword, including information about a specified location that indicates the location or direction of a disaster site or other location that the user wants to check on a 3D map after receiving an operation from the user (step S2001). The service enabler 30 gives instructions to the application service control unit 41 of the orchestrator 40 using "Area A" as a keyword, including information about a specified location that indicates the location or direction of a disaster site or other location that the user wants to check on a 3D map (step S2002). The application service control unit 41 of the orchestrator 40 gives instructions to the video processing unit 51A of the information processing infrastructure unit 50 using "Area A" as a keyword, including information about a specified location that indicates the location or direction of a disaster site or other location that the user wants to check on a 3D map (step S2003). The video processing unit 51A of the information processing infrastructure unit 50 performs spatial reproduction processing and spatial presentation processing based on a specified position instructed by the application service control unit 41 of the orchestrator 40 (step S2004). The video processing unit 51A of the information processing infrastructure unit 50 generates a 3D image based on the position from the instructed specified position. The video processing unit 51A of the information processing infrastructure unit 50 transmits the generated 3D image data to the HMD 110, which is the application terminal of application 10 (step S2005).

[0125] (6-1) The information processing infrastructure unit 50 transmits the 3D image, which has been combined with a map in cyberspace, from the edge processing unit 53 of the information processing infrastructure unit 50, which is installed in a computer close to area A, to the network processing infrastructure unit 60 or the orchestrator 40, in order to transfer it to a display device such as an HMD 110 used by a remote operator, i.e., a user, who wants to understand the disaster situation in area A. At this time, the information processing infrastructure unit 50 presents the orchestrator 40 with the amount of communication resources required to transmit the 3D image via the line 160, as explained in the sequence diagram in Figure 21.

[0126] (6-2) The resource monitoring unit 61 of the network processing infrastructure unit 60 calculates the resources of the available network, i.e., the line 160, and considers a communication path that can secure communication resources "even in the event of a disaster" through the orchestrator 40. Considering a communication path means using the satellite line 162 or the terrestrial line 163, etc. If communication resources can be secured, the 3D image is transferred from the CPS server 151, etc., to the router of the line 160 on which communication resources have been secured. If communication resources cannot be secured, the video processing unit 51A of the information processing infrastructure unit 50 adjusts the communication speed by performing processing such as reducing the frame rate of the 3D image to be transmitted or reducing the image quality. Regarding the consideration of the communication path by the resource monitoring unit 61 of the network processing infrastructure unit 60, emphasis may be placed on the certainty of communication obtained through the deterministic network unit 65, as shown in the sequence diagram of Figure 11.

[0127] (6-3) On the other hand, the remote operator, i.e., the user using the HMD110, outputs the user's viewpoint direction as input information from the application 10 via the line 160 and the service enabler 30 to the video processing unit 51A or remote control unit 51E of the information processing infrastructure unit 50, so that the object or target location can be viewed from the direction the user wants to see it, in order to check the disaster situation in 3D images. The input information from the application 10 regarding the user's viewpoint direction may be information obtained by means of the user's body movements such as gestures and hand movements.

[0128] (6-4) Next, the video processing unit 51A of the information processing infrastructure unit 50 generates a 3D image from the direction the user wants to view it through spatial reproduction and spatial presentation processing, and transmits the generated 3D image data to the HMD 110 used by the user via the line 160. As a necessary process for this communication, the orchestrator 40 and the network processing infrastructure unit 60 cooperate to secure communication resources, which is done in the same way as described above. The operations from (6-1) to (6-4) up to this point correspond to the operations in the sequence diagram of Figure 22 mentioned above.

[0129] (6-5) However, the resource monitoring unit 61 of the network processing infrastructure unit 60 determines in advance whether to transmit the 3D image, which is the result of the video processing, via satellite link 162 or terrestrial link 163, by understanding the path conditions. However, there is a difference in communication speed between the uplink link 160 and the downlink link 160. Therefore, the determination of the path by the resource monitoring unit 61 of the network processing infrastructure unit 60 is performed using information that shows the status of the link 160 obtained from resource monitoring by the orchestrator 40, the network processing infrastructure unit 60, etc., so that appropriate communication paths can be selected for the uplink link 160 and the downlink link 160, by collecting network resource monitoring results such as QoS, RTT, and throughput, as shown in sequence diagrams such as Figure 3. The operations from (6-1) to (6-5) correspond to the operations from [9] to

[10] described above.

[0130] In this embodiment, the use of the cyber-physical system 1 during a disaster has been described, but in normal times, it can also be used to monitor infrastructure such as river conditions and bridge pier deterioration. In this embodiment, the cyber-physical system 1 requires both an application 10 for controlling the drone 170 and an application 10 for checking the debris situation. However, in order to collaborate with other companies, the cyber-physical system 1 can also be configured so that the remote control unit 51E of the information processing infrastructure unit 50 collaborates with a third-party server 150 that is equipped with both the application 10 for controlling the drone 170 and the application 10 for checking the debris situation. Furthermore, the cyber-physical system 1 can also collaborate with applications 10 that allow for communication delays.

[0131] Furthermore, the cyber-physical system 1 may retain information that allows for comparison between the situation before and after the disaster. For example, the cyber-physical system 1 retains pre-disaster images and information on shooting conditions such as the position of the drone 170 and the settings of the camera 74 when the pre-disaster images were taken. Pre-disaster images may include, for example, overhead images or omnidirectional images of specific objects. In order to retain pre-disaster images of many locations, the retained pre-disaster images may be of low quality. If the cyber-physical system 1 processes and retains pre-disaster images, it may also retain processing conditions. This allows the cyber-physical system 1 to instruct the drone 170 to take post-disaster images under the same shooting conditions as when the pre-disaster images were taken. In addition, users can roughly confirm the extent of the damage by comparing the pre-disaster and post-disaster images.

[0132] Next, the hardware configuration of the platform system 20 will be described. In the platform system 20, the device group 70 consists of mobility nodes such as drones 170. The service enabler 30, orchestrator 40, information processing infrastructure unit 50, and network processing infrastructure unit 60 are implemented by processing circuits. The processing circuits may be a processor and memory that execute programs stored in memory, or they may be dedicated hardware. Processing circuits are also called control circuits.

[0133] Figure 23 is a diagram showing an example configuration of a processing circuit 90 when the processing circuit realizing the platform system 20 according to this embodiment is composed of a processor 91 and a memory 92. The processing circuit 90 shown in Figure 23 is a control circuit and comprises a processor 91 and a memory 92. When the processing circuit 90 is composed of a processor 91 and a memory 92, each function of the processing circuit 90 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 92. In the processing circuit 90, each function is realized by the processor 91 reading and executing the program stored in the memory 92. That is, the processing circuit 90 includes a memory 92 for storing a program that will ultimately be executed as processing of the platform system 20. This program can also be said to be a program that causes the platform system 20 to execute each function realized by the processing circuit 90. This program may be provided by a storage medium on which the program is stored, or by other means such as a communication medium.

[0134] The above program can also be described as a program that causes the platform system 20 to execute the following steps: a first step in which the device group 70 is equipped with a sensing function and outputs information obtained by the sensing function; a second step in which the information processing infrastructure unit 50 processes the information obtained from the device group 70 in accordance with a request from the application 10; a third step in which the network processing infrastructure unit 60 monitors the communication status when obtaining information from the device group 70 and controls communication with the device group 70; a fourth step in which the orchestrator 40 monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 and performs coordinated control; and a fifth step in which the service enabler 30 instructs the information processing infrastructure unit 50 and the orchestrator 40 to perform an operation based on a request from the application 10.

[0135] Here, the processor 91 is, for example, a CPU (Central Processing Unit), processing unit, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor). The memory 92 is, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Registered Trademark) (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0136] Figure 24 shows an example of a processing circuit 93 when the processing circuit realizing the platform system 20 according to this embodiment is configured with dedicated hardware. The processing circuit 93 shown in Figure 24 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit may be partially implemented with dedicated hardware and partially implemented with software or firmware. In this way, the processing circuit can realize each of the above functions with dedicated hardware, software, firmware, or a combination thereof.

[0137] Furthermore, the control circuit, which is the processing circuit that realizes the platform system 20, may be configured to be located on multiple servers 150 and drones 170, rather than on a single server 150, as shown in Figure 2, such as the CPS server 151 and edge servers 154 and 155. Similarly, the storage medium on which the program for controlling the platform system 20 is stored may also be configured to be located on multiple servers 150 and drones 170, rather than on a single server 150, as shown in Figure 2, such as the CPS server 151 and edge servers 154 and 155.

[0138] As described above, according to this embodiment, in the platform system 20, the orchestrator 40 acquires computing resources from the information processing infrastructure unit 50 and communication resources from the network processing infrastructure unit 60, and monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 based on the acquired computing and communication resources, and performs coordinated control. As a result, even on systems in abnormal conditions such as disaster situations where computing and communication resources are constantly changing, the platform system 20 can provide operability, reliability, and low-latency real-time performance to the device group 70 by having the orchestrator 40 grasp and secure the resource status from an overview perspective. The platform system 20 can improve the probability of providing services by optimizing the sensing function in situations where at least one of the communication resources and computing resources is changing, and by guiding the sensing function to a range in which each resource can provide the desired service level.

[0139] The platform system 20 can provide high-quality video by offering, for example, operability, accuracy, and low latency real-time capabilities to the drone 170, which is part of the device group 70 that enables 3D shooting. The platform system 20 moves the drone 170, which is equipped with a sensing unit 73 such as a camera 74, to a shooting position more suitable for sensing and data transfer. For example, the platform system 20 can recognize collapsed houses from the entire disaster site captured by the camera 74 of the drone 170 in a wide angle, and within a range that satisfies certain communication conditions, it can approach the collapsed houses to collect information with the camera 74 of the drone 170 and evaluate hidden human and material disaster risks by examining the information to determine whether there are survivors and whether there are underlying factors that could cause future fires.

[0140] The platform system 20 selects a communication line that ensures reliable communication by switching between the satellite line 162 or the terrestrial line 163, which are more resistant to disasters, or by using both the satellite line 162 and the terrestrial line 163, depending on the communication line conditions during a disaster, such as communication speed, communication quality, communication delay, and the feasibility of periodic communication. In the platform system 20, the orchestrator 40 grasps the status of each sub-function in the information processing infrastructure unit 50 and the network processing infrastructure unit 60, such as the operation and processing time, in order to meet the required functions and performance, and performs control based on this grasped information, thereby appropriately grasping and controlling the computing resources and communication resources that change in real time.

[0141] In this embodiment, the example described is one in which the orchestrator 40 determines, or selects, one communication path when there are multiple communication paths for the network processing infrastructure 60 to acquire information from the device group 70, but the embodiment is not limited to this. The orchestrator 40 can also select multiple communication paths as communication paths for the network processing infrastructure 60 to acquire information from the device group 70. In this case, the drones 170 of the device group 70 output information encapsulating the same data simultaneously from multiple communication paths. The network processing infrastructure 60 acquires multiple pieces of information and reconstructs the encapsulated data.

[0142] The configurations shown in the above embodiments are merely examples, and can be combined with other known technologies. It is also possible to omit or modify parts of the configuration without departing from the gist of the invention. [Explanation of Symbols]

[0143] 1 Cyber-physical system, 10 Applications, 20 Platform system, 30 Service enabler, 40 Orchestrator, 41 Application service control unit, 42, 51G Data linkage unit, 43 Application linkage unit, 44 E2E slice control unit, 45 E2E resource management unit, 50 Information processing infrastructure unit, 51 Platform application group, 51A Video processing unit, 51B Extended space unit, 51C Spatial presentation unit, 51D Spatial reproduction unit, 51E Remote control unit, 51F Sensory sharing unit, 51H Sensor information analysis and prediction unit, 52, 61 Resource monitoring unit, 53 Edge processing unit, 54 Information security unit, 60 Network processing infrastructure unit, 62 Network operation management unit, 63 Network security unit, 64 Slicing unit, 65 Deterministic network unit, 70 Device group, 71 Clustering unit, 72 Spatiotemporal synchronization unit, 73 Sensing unit, 74 Camera, 75 Security unit, 76 Actuator unit, 90, 93 Processing circuit, 91 Processor, 92 Memory, 110 HMD, 150 Server, 151 CPS server, 154, 155 Edge server, 160 Line, 161 Wired line, 162 Satellite line, 163 Terrestrial line, 164, 165 L5G, 170 Drone, 170a Parent drone, 170b Child drone, 200 GPS.

Claims

1. A group of devices equipped with a sensing function and outputting information obtained by the said sensing function, An information processing infrastructure unit that processes the information acquired from the group of devices in response to a request from an application, A network processing infrastructure unit that monitors the communication status when acquiring the information from the group of devices and controls communication with the group of devices, An orchestrator that monitors and coordinates the operation of the aforementioned device group, the information processing infrastructure unit, and the network processing infrastructure unit, A service enabler that instructs the information processing infrastructure unit and the orchestrator to perform actions based on a request from the aforementioned application, A platform system characterized by comprising the following features.

2. The orchestrator acquires computing resources from the information processing infrastructure unit necessary to process tasks within the processing time requested by the user, acquires the communication status between the network processing infrastructure unit and the device group and communication resources necessary to process data transmission within the transmission time requested by the user from the network processing infrastructure unit, acquires location information from the device group, and monitors the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit to perform coordinated control. The platform system according to feature 1.

3. The network processing infrastructure unit has multiple communication paths for acquiring the information from the device group. The orchestrator determines the communication path for the network processing infrastructure to acquire the information from the device group, based on the computing resources and the communication resources. The platform system according to claim 2, characterized in that it is the same as described in claim 2.

4. If the orchestrator selects a plurality of communication paths as the communication path for the network processing infrastructure to acquire the information from the group of devices, The aforementioned group of devices outputs the information encapsulating the same data simultaneously from multiple communication paths. The network processing infrastructure unit acquires multiple pieces of information and restores the encapsulated data. The platform system according to feature 3.

5. The orchestrator, based on the computing resources and communication resources, notifies the service enabler if it cannot meet the service level required by the application. The service enabler, based on the notification from the orchestrator, requests the application to change the service level. The platform system according to claim 2, characterized in that it is the same as described in claim 2.

6. The aforementioned group of devices is an unmanned aerial vehicle, and the sensing function is a shooting function. The aforementioned unmanned aerial vehicle outputs image information captured by the aforementioned shooting function as the aforementioned information. The information processing infrastructure unit performs video processing using the image information for display on a display device according to the application. The platform system according to feature 1.

7. The information processing infrastructure unit performs a process to synthesize multiple pieces of information acquired from the device group as processing in response to a request from the application. The platform system according to feature 1.

8. The information processing infrastructure unit, if it is unable to obtain the first information from the first device included in the device group, but can obtain the second information from the second device included in the device group, and if communication is possible between the first device and the second device, instructs the first device to transmit the first information to the second device, and obtains the first information and the second information from the second device. A platform system according to any one of claims 1 to 7.

9. The information processing infrastructure unit includes a first device that transmits first information to the device group and a second device that transmits second information, and is capable of communicating with the second device. When communication is possible between the first device and the second device, it instructs the first device to transmit the first information to the second device and obtains composite information from the second device which is a combination of the first information and the second information. A platform system according to any one of claims 1 to 7.

10. A control circuit for controlling a platform system, The group of devices is equipped with a sensing function and outputs information obtained by the sensing function. The information processing infrastructure unit processes the information acquired from the device group in response to a request from the application. The network processing infrastructure unit monitors the communication status when acquiring the information from the device group and controls communication with the device group. The orchestrator monitors and coordinates the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit. The service enabler instructs the information processing infrastructure unit and the orchestrator to perform actions based on the request from the application. A control circuit characterized by causing the platform system to perform the above.

11. A storage medium in which a program for controlling a platform system is stored, The aforementioned program, The group of devices is equipped with a sensing function and outputs information obtained by the sensing function. The information processing infrastructure unit processes the information acquired from the device group in response to a request from the application. The network processing infrastructure unit monitors the communication status when acquiring the information from the device group and controls communication with the device group. The orchestrator monitors and coordinates the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit. The service enabler instructs the information processing infrastructure unit and the orchestrator to perform actions based on the request from the application. A storage medium characterized by causing the platform system to perform the above.

12. The first step involves a group of devices equipped with a sensing function and outputting information obtained by the sensing function, The second step involves the information processing infrastructure unit processing the information acquired from the device group in response to a request from the application, A third step involves the network processing infrastructure unit monitoring the communication status when acquiring the information from the device group and controlling communication with the device group. A fourth step involves the orchestrator monitoring the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit and performing coordinated control. A fifth step in which the service enabler instructs the information processing infrastructure unit and the orchestrator to perform an action based on a request from the application, A communication method characterized by including

Images