Platform system, control circuit, storage medium, and communication method

Abstract

A platform system (20) comprises: a device group (70) that is equipped with a sensing function and outputs information obtained by the sensing function; an information processing base unit (50) that processes information acquired from the device group (70) in response to a request from an application (10); a network processing base unit (60) that monitors the communication status when acquiring information from the device group (70) and controls communication with the device group (70); an orchestrator (40) that monitors the operation of the device group (70), the information processing base unit (50), and the network processing base unit (60) and performs coordination control; and a service enabler (30) that instructs the information processing base unit (50) and the orchestrator (40) to operate on the basis of the request from the application (10).

Description

Platform system, control circuit, storage medium and communication method

[0001] The present disclosure relates to a platform system, a control circuit, a storage medium, and a communication method for providing data about a requested service.

[0002] Conventionally, networks such as wired networks and wireless networks have been used as means for transmitting data for services requested by users. To build such networks, it is necessary for someone with knowledge of each network to perform the construction. However, networks have different service requirements depending on the service requested by the user, such as low latency for control signaling and high bandwidth for video transmission. Therefore, a system has been desired that allows someone with little knowledge of networks but who is familiar with the service to build a network without being aware of the construction conditions. Patent Document 1 discloses a technology for a network requirement generation system that analyzes service requirements input by a user to generate network requirements and creates network settings for a control device to build a network from the network requirements. It is also shown that the network requirement generation system described in Patent Document 1 can adjust the network bandwidth, etc., based on the results of analyzing the network status.

[0003] Japanese Patent Application Laid-Open No. 2020-140276

[0004] However, in recent years, in order to meet the increasing demands for advanced services from users, it has become necessary not only to secure communication resources such as network bandwidth, but also to link computer resources in cyberspace (cyber systems) for data processing and other tasks, and sensing functions in the real world (physical systems), in order to achieve even lower latency. However, networks built using the above-mentioned conventional technologies do not take into consideration the linkage between computer resources and sensing functions. Therefore, particularly in situations where communication resources and computer resources change in real time, there is a problem in that even if one resource can be secured, the other cannot be secured, making it impossible to provide services at the expected service level. Furthermore, in order to ensure service levels in such situations, it is necessary to distribute a large number of sensing functions, which also poses a problem of increasing the cost of building the sensing infrastructure.

[0005] The present disclosure has been made in consideration of the above, and aims to realize a platform system that can improve the probability of providing services in situations where at least one of communication resources and computer resources changes, by guiding sensing functions to a range where each resource can provide the desired service level.

[0006] In order to solve the above-mentioned problems and achieve the objectives, the platform system of the present disclosure is characterized by comprising: a device group equipped with a sensing function and outputting information obtained by the sensing function; an information processing infrastructure unit that processes information obtained from the device group in response to a request from an application; a network processing infrastructure unit that monitors the communication status when obtaining information from the device group and controls communication with the device group; an orchestrator that monitors the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit and performs collaborative control; and a service enabler that instructs the information processing infrastructure unit and the orchestrator to operate based on a request from an application.

[0007] The platform system of the present disclosure has the effect of improving the probability of providing services in situations where at least one of communication resources and computing resources changes, by guiding the sensing function to a range in which each resource can provide the desired service level.

[0008]

[0009] Hereinafter, a platform system, a control circuit, a storage medium, and a communication method according to embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] 1 is a block diagram showing an example configuration of a cyber-physical system 1 according to this embodiment. The cyber-physical system 1 includes an application 10 and a platform system 20. The cyber-physical system 1 is a system in which the application 10 requests a service from the platform system 20, and the platform system 20 provides data for the service requested by the application 10 to the application 10. The cyber-physical system 1 is a system in which a CPS (Cyber ​​Physical System) service can be used by the application 10, which is a CPS application.

[0011] The application 10 requests a service from the platform system 20, specifically, the service enabler 30 included in the platform system 20, by receiving an operation from a user. In this embodiment, the service requested by the application 10 specifically requests acquisition of an image captured by a drone, which is one of the device groups 70 included in the platform system 20, according to the orientation of a head-mounted display (hereinafter referred to as an HMD), which is an application terminal worn by a user. Note that the application terminal may be a device other than the HMD, such as a smartphone or mobile pad. Furthermore, the application 10 will be described using a real-time remote monitoring application as an example, allowing a user to view a desired location from a desired viewpoint using an HMD. However, the application 10 may also be, for example, an application for operating a drone equipped with a sensor terminal. Note that in the following description, "application" may also be referred to as "app."

[0012] The platform system 20 provides data for a service requested by the application 10. The platform system 20 includes a service enabler 30, an orchestrator 40, an information processing infrastructure unit 50, a network processing infrastructure unit 60, and a device group 70.

[0013] The service enabler 30 instructs the information processing infrastructure unit 50 and the orchestrator 40 to operate based on requests from the application 10. The service enabler 30 accepts service requests, i.e., commands, from the application 10 via an API (Application Programming Interface), breaks down the accepted service requests, i.e., commands, into destinations that manage the functions providing the service, and issues instructions to the orchestrator 40, which then issues instructions to the information processing infrastructure unit 50 via the API. An API is an interface that facilitates collaboration between software programs with different functions. The service enabler 30 also obtains data corresponding to instructions from the application 10 from the information processing infrastructure unit 50 via the API. In the cyber-physical system 1, the application 10 can also function as the service enabler 30. The service enabler 30 issues instructions to a platform application group 51 of the information processing infrastructure unit 50 (described later), i.e., each function in the information processing infrastructure unit 50, to link the functions in a certain order. The service enabler 30, for example, includes a spatial reproduction unit 51D and a spatial presentation unit 51C in the video processing unit 51A of the information processing infrastructure unit 50, and first issues an instruction to the spatial reproduction unit 51D to perform 3D spatial reproduction, and then issues an instruction to the spatial presentation unit 51C to process what is mapped on the map so that it can be displayed on the HMD and perform spatial presentation.

[0014] The orchestrator 40 monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 and controls the coordination of each function. The orchestrator 40, for example, controls and monitors the multi-layers between the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60. The orchestrator 40 manages and operates the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70, and centrally executes function control, function configuration, inter-function coordination, and the like. For example, when the orchestrator 40 receives a command from the service enabler 30 requesting the establishment of a communication path to acquire camera images, the orchestrator 40 operates in response to the command. Furthermore, when controlling the drone and capturing camera images, the orchestrator 40 transmits a control message to the network processing infrastructure unit 60 and concurrently requests the transmission of camera images and control information. Furthermore, upon completion of the transmission request, the orchestrator 40 transmits a communication end message to the service enabler 30 upon completion of the communication for the camera images. The orchestrator 40 includes an application service control unit 41 , a data linking unit 42 , an application linking unit 43 , an E2E (End to End) slice control unit 44 , and an E2E resource management unit 45 .

[0015] The application service control unit 41 controls the operations of the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70. The data linking unit 42 links data, such as setting data and stored data, among the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70. The application linking unit 43 links the information processing infrastructure unit 50, the network processing infrastructure unit 60, and the device group 70 with the application 10, i.e., functions based on the requested service. The E2E slice control unit 44 controls slices in the network processing infrastructure unit 60. The E2E resource management unit 45 manages resources in the network processing infrastructure unit 60.

[0016] The information processing infrastructure unit 50 processes information acquired from the device group 70 in response to a request from the application 10. The information processing infrastructure unit 50 provides computing power to the user and manages the computing power, etc. In this embodiment, specifically, as will be described later, the device group 70 outputs image information captured by a capture function as the above-mentioned information, and the information processing infrastructure unit 50 uses the image information to perform video processing for display on a display device corresponding to the application 10. The information processing infrastructure unit 50 also performs processing to combine multiple pieces of information acquired from the device group 70 in response to a request from the application 10. The information processing infrastructure unit 50 includes a platform application group 51, a resource monitoring unit 52, an edge processing unit 53, and an information security unit 54.

[0017] The platform application group 51 acquires an image from a camera 74 (described later) included in the device group 70 to generate a 3D image, acquires information such as the shooting position and viewing direction of the camera 74 from an HMD, which is an application terminal that executes the application 10, and distributes 3D image data based on the shooting position, viewing direction, etc. of the camera 74 from the HMD to the HMD, which is an application terminal of the application 10. The shooting position, viewing direction, etc. of the camera 74 may be the position, orientation, etc. of a drone on which the camera 74 is mounted. The platform application group 51 includes an image processing unit 51A, a remote control unit 51E, a sensory sharing unit 51F, a data linking unit 51G, and a sensor information analysis and prediction unit 51H.

[0018] The video processing unit 51A generates and synthesizes video images based on requests from the application 10, and provides video data to the application 10. Specifically, the video processing unit 51A provides video data to an HMD of a display device that is an application terminal. The video processing unit 51A includes an extended space unit 51B, a spatial presentation unit 51C, and a spatial reproduction unit 51D. The extended space unit 51B generates an AR (Augmented Reality) space, i.e., an augmented reality space, from data constructed by video processing. The spatial presentation unit 51C generates a VR (Virtual Reality) space, i.e., a virtual reality space, based on model data generated by computational processing. The spatial reproduction unit 51D generates video to be provided to the application 10 based on the data generated by the extended space unit 51B and the spatial presentation unit 51C. Specifically, the spatial reproduction unit 51D generates video corresponding to the viewing direction of the HMD of the display device that is an application terminal.

[0019] The remote control unit 51E operates the device group 70. For example, the remote control unit 51E instructs a drone constituting the device group 70 to move to a shooting position in response to a request from the application 10. The sense sharing unit 51F provides the user with a sense that can be reproduced by the user. In the present embodiment, the sense sharing unit 51F generates and provides data on the senses of sight, touch, and hearing, among the five senses. The data linking unit 51G holds data from each sensor acquired from the device group 70 and provides the corresponding data to the sensor information analysis prediction unit 51H in response to a request from the sensor information analysis prediction unit 51H. The sensor information analysis prediction unit 51H uses its computing power to perform information analysis, learning, prediction, and the like, based on the data acquired from the data linking unit 51G.

[0020] The resource monitoring unit 52 monitors the status of the computing power of each server, edge server, etc. that constitutes the information processing infrastructure unit 50. The edge processing unit 53 is located near the device group 70 and performs computational processing and data management on data from the device group 70. The information security unit 54 protects against data corruption, protects against information leaks to parties other than the disclosure recipient, and maintains the environment for disclosing data to the disclosure recipient.

[0021] The network processing infrastructure unit 60 monitors the communication status when acquiring information from the device group 70 and controls communication with the device group 70. The network processing infrastructure unit 60 manages and provides network capabilities to users. The network processing infrastructure unit 60 includes a resource monitoring unit 61, a network operations management unit 62, a network security unit 63, a slicing unit 64, and a deterministic network unit 65. Note that in FIG. 1, the network is abbreviated as NW (Network). This also applies to subsequent figures.

[0022] The resource monitoring unit 61 monitors the communication bandwidth, delay time, packet loss rate, etc. of the communication devices that make up the network processing infrastructure unit 60 and the mobility nodes that make up the device group 70. The network operation management unit 62 monitors to determine whether a communication environment guaranteed by an SLA (Service Level Agreement) is being provided for each connection. The network security unit 63 provides a VPN (Virtual Private Network) function for data concealment. The slicing unit 64 virtually divides, or slices, the network based on the communication characteristics that realize it, and provides a logical network in accordance with the SLA. The deterministic network unit 65 provides a network that guarantees reachability and delay fluctuation within a specified range of allowable delay time.

[0023] The device group 70 has a sensing function and outputs information obtained by the sensing function. The device group 70 is a mobility node with a communication function, such as a smartphone, a drone, or heavy machinery. The device group 70 may include multiple types of mobility nodes, or may include multiple mobility nodes of one type. In this embodiment, the device group 70 is specifically an unmanned aerial vehicle, i.e., a drone, and the device group 70, i.e., the drone, is described using an example in which the device group 70 is equipped with an imaging function as a sensing function. In this case, the device group 70, i.e., the drone, outputs image information captured by the imaging function as the aforementioned information. The device group 70 includes a clustering unit 71, a time-space synchronization unit 72, a sensing unit 73, a security unit 75, and an actuator unit 76.

[0024] The clustering unit 71 forms a cluster with mobility nodes and performs cooperative operations between the mobility nodes. For example, if the mobility nodes are drones, the cooperative operation is formation operation. The time-space synchronization unit 72 manages the spatial positions of the mobility nodes constituting the device group 70, their synchronization status relative to an arbitrary reference time, and the like. The sensing unit 73 includes a sensor and acquires data generated by the sensor. In this embodiment, a camera 74 is assumed as the sensor, but this is not limited thereto. Sensors that measure temperature, illuminance, gas, position, particles in the air, and the like may also be used. The sensing unit 73 may also include multiple types of sensors. Images captured by the camera 74 may be still images or moving images. The security unit 75 protects data acquired from the sensors included in the sensing unit 73 and encrypts data to be transmitted to the information processing infrastructure unit 50. The actuator unit 76 converts energy such as electricity, gas, and oil into motion. The actuator unit 76 is, for example, a motor that rotates the propellers when the mobility node is a drone, or a motor that controls the shooting direction of the camera 74, etc.

[0025] Here, the correspondence between each component shown in the block diagram of the cyber-physical system 1 shown in Fig. 1 and the device configuration in the actual system will be described. Fig. 2 is a diagram showing an example of the device configuration of the cyber-physical system 1 according to this embodiment.

[0026] 1 is executed on an HMD 110, which is an application terminal used by a user in FIG. 2, and an image from a desired viewpoint is displayed to the user by the HMD 110. At the same time, the HMD 110 presents a cyberspace in which a 3D spatial model and virtual objects are synthesized by a synthesis process that constructs a real-time augmented space using the platform system 20.

[0027] The service enabler 30 shown in FIG. 1 is implemented in the CPS server 151 in FIG.

[0028] 2, the orchestrator 40 shown in FIG. 1 is implemented in a CPS server 151. The CPS server 151 controls communication resources, computer resources, and a device group 70 to provide a requested service to the application 10. Note that some functions of the orchestrator 40 may be implemented in other servers 150, such as edge servers 154 and 155, which are close to a drone 170, which is one of the device group 70 to be operated by the application 10.

[0029] 2, the information processing infrastructure unit 50 shown in Fig. 1 is implemented in a CPS server 151, edge servers 154, 155, etc. The functions of the information processing infrastructure unit 50 shown in Fig. 1 may be implemented in one server 150, or may be distributed and implemented in multiple servers 150. Furthermore, functions may be transferred between the servers 150 depending on the status of the computer resources of each server 150.

[0030] The network processing infrastructure unit 60 shown in FIG. 1 is implemented in a base station installed in each line 160, such as a public line such as a satellite line 162 and a terrestrial line 163 shown in FIG. 2, a wired line 161, and private lines such as L5G (Local 5th Generation) 164 and 165, or in a base station that manages all of the lines 160 shown in FIG. 2. The functions of the network processing infrastructure unit 60 shown in FIG. 1 may be implemented in one base station or may be distributed and implemented in multiple base stations. The satellite line 162 is, for example, a line 160 for communication such as geostationary orbit satellite communication or low-earth orbit satellite communication. The terrestrial line 163 is, for example, a line 160 for communication such as cellular communication.

[0031] 1 is assumed to be a drone 170 in a physical space in FIG. 2 as described above. The drone 170 takes pictures with a camera 74 in response to instructions from a user, and also moves, rotates, changes altitude, etc. in response to instructions from the user to change the location and direction of photography taken by the camera 74. Although the illustration is simplified in FIG. 2, the device group 70 is assumed to be a plurality of drones 170.

[0032] In the cyber-physical system 1, the HMD 110, which is an application terminal, instructs the drone 170, directly or via the CPS server 151, etc., on the shooting location, shooting direction, etc., so that the user can check the situation of a disaster site or the like photographed by the drone 170. The drone 170 transmits images captured from multiple viewpoints specified by the user to the CPS server 151. The CPS server 151 generates an image in cyberspace that combines a map, which is a virtual object showing the terrain of a remote location to be photographed by the drone 170, with data that models fallen trees and the like in 3D space based on the actual images photographed by the drone 170. The CPS server 151 transmits the generated cyberspace data to the HMD 110, which is an application terminal, via a satellite line 162, a terrestrial line 163, etc. This allows the user to check the situation of a disaster site or the like from a desired viewpoint using the cyberspace displayed on the HMD 110. In the cyber-physical system 1, the CPS server 151 displays a real-time augmented space on the HMD 110, allowing users to share the situation at a remote disaster site or other location in real time. Note that some of the processing performed by the CPS server 151 may be performed by edge servers 154, 155 that are close to the drone 170 in order to reduce transmission delay time required for data communication or in situations where the CPS server 151 cannot secure computer resources.

[0033] 1 monitors resource information of each server 150 and each line 160, and provides data for a service requested by the application 10 by using a server 150 for which computer resources can be secured and a line 160 for which communication resources can be secured. The orchestrator 40 acquires, from the information processing infrastructure unit 50, computer resources required for the information processing infrastructure unit 50 to process a task within the processing time requested by the user, acquires, from the network processing infrastructure unit 60, communication conditions such as the bandwidth used for data transmission between the network processing infrastructure unit 60 and the device group 70 and round-trip delay time, and communication resources required for processing the data transmission within the transmission time requested by the user, acquires location information from the device group 70, and monitors the operations of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 to perform cooperative control.

[0034] The platform system 20 acquires sensing information of the disaster site using multiple drones 170 equipped with pre-registered cameras 74 in order to grasp the disaster situation in an area specified by the user via the application 10, which is a CPS application. The platform system 20 also performs computer processing to generate 3D images so that the user can grasp the disaster situation in three dimensions, and provides the necessary services by efficiently performing a series of processes, taking into consideration communication resources and computer resources, up to distributing the images to the HMD 110, which is the display device used by the user.

[0035] For example, assume that a disaster prevention application 10 independently manages and controls computer resources, such as image processing resources, and communication resources. Computer resources include the number and processing speed of central processing units (CPUs), memory and storage capacity, etc. Communication resources include bandwidth, time occupancy, quality of service (QoS), round trip time (RTT), throughput, etc., used in communication. In such a case, even if the necessary communication resources are available on the network, if the necessary computer resources are not available on the network, the computational processing takes time, making real-time computation impossible. Furthermore, remote control requires reliable periodic communication with a relatively small capacity rather than large-capacity communication, and requires computational processing with a small computational scale but high real-time performance, making it difficult to support the service levels of various applications 10.

[0036] Therefore, in this embodiment, the platform system 20, as the disaster prevention application 10, performs processing up to and including computer processing of images obtained from a drone 170 equipped with a camera 74 so that the user can grasp the disaster situation, and displaying the images on the HMD 110 so that the target object or target point at the disaster site can be viewed from the direction intended by the user, in accordance with the service level of the application 10, i.e., real-time performance, operability, etc. In order to perform processing in accordance with the service level of the application 10, the platform system 20 divides its functions into an information processing infrastructure unit 50, a network processing infrastructure unit 60, and an orchestrator 40, and the orchestrator 40 secures and controls communication resources and computer resources in accordance with the service level of the application 10, i.e., real-time performance, operability, etc.

[0037] In this embodiment, the orchestrator 40 only needs to periodically or periodically monitor the communication resources in the satellite link 162 and the terrestrial link 163, and the available edge servers 154 and 155, the cloud (not shown), and other computing resources, and grasp and control the resource status in a simple manner. In the case of communication resources, the simple method is, for example, a method of grasping the status using commands such as Ping for measuring communication delay time and Traceroute for grasping the communication path, but is not limited to these.

[0038] In this embodiment, the service enabler 30 receives a usage request for the application 10 related to, for example, a CPS for disaster prevention, breaks it down into processing units for controlling the information processing infrastructure unit 50 and the network processing infrastructure unit 60, such as a 3D image creation function, a display function, and a processing function related to drone flight, and transmits the request to the orchestrator 40. As a result, required values ​​for the platform system 20, such as real-time performance and operability, are determined.

[0039] The orchestrator 40 grasps the status of the operation and processing time of each sub-function in the information processing infrastructure unit 50 and the network processing infrastructure unit 60 so as to satisfy the required functions, required performance, etc., and performs control based on this grasped information, appropriately grasping and controlling the ever-changing communication resources and computer resources, thereby ensuring or maintaining the service level for each application 10. The operation of each sub-function in the information processing infrastructure unit 50 and the network processing infrastructure unit 60 refers to the operation of each component provided in the information processing infrastructure unit 50 and the network processing infrastructure unit 60 shown in FIG. 1. In the platform system 20, the status of the orchestrator 40 is fed back to the service enabler 30, and the required values ​​of the service enabler 30 are reset depending on the status of the securing of communication resources and computer resources, for example, real-time performance, operability, etc.

[0040] In this way, the platform system 20 is able to efficiently configure and link the information processing infrastructure unit 50 and the network processing infrastructure unit 60 using the orchestrator 40 according to the service level of the application 10, thereby enabling improvements in the real-time performance and operability of the CPS to be expected.

[0041] The specific operation of the cyber-physical system 1 will be explained using a sequence diagram or a flowchart.

[0042] 3 is a sequence diagram showing the operation of the cyber-physical system 1 according to the present embodiment until the drone 170 is able to transmit data depending on the state of the line 160. When the application 10 receives a service execution request from a user, the application 10 requests the orchestrator 40 to execute the service via the service enabler 30 (not shown) (step S101). The network processing infrastructure unit 60 acquires communication speed, communication quality, communication delay information, etc. from the satellite line 162 (step S102). Similarly, the network processing infrastructure unit 60 acquires communication speed, communication quality, communication delay information, etc. from the terrestrial line 163 (step S103). The network processing infrastructure unit 60 may acquire the above information from the base station of each line 160, or may acquire the above information from the base station that oversees each line 160.

[0043] The orchestrator 40 requests information acquired from the satellite line 162 and the terrestrial line 163 from the network processing infrastructure unit 60 (step S104). In response to the request from the orchestrator 40, the network processing infrastructure unit 60 transmits information on the communication line status acquired from the satellite line 162 and the terrestrial line 163 to the orchestrator 40 (step S105). The orchestrator 40 selects a communication line according to the sequence shown in FIG. 5 (described later) and requests the network processing infrastructure unit 60 to select a line 160 (step S106). Note that the "selection" of a communication line described here is not limited to a single line; multiple lines may be selected and a protocol that simultaneously uses multiple lines, such as MultiPath TCP (Transmission Control Protocol), may be used. Detailed operation of the orchestrator 40 will be described later. The network processing infrastructure unit 60 requests the drone 170 to select a line 160 (step S107). Based on a request to select the line 160 from the network processing infrastructure unit 60, the drone 170 transmits data to the satellite line 162, the terrestrial line 163, or the satellite line 162 and the terrestrial line 163 (step S108).

[0044] 3 , the platform system 20 can select the communication line 160 that ensures reliable communication with the drone 170 by switching between the satellite line 162 and the terrestrial line 163, or by using both the satellite line 162 and the terrestrial line 163, depending on the communication line status during a disaster. The communication line status includes, for example, communication speed, communication quality, communication delay, and whether periodic communication is possible.

[0045] FIG. 4 is a sequence diagram showing the operation up to the start of processing of an image acquired from the camera 74 in the cyber-physical system 1 according to this embodiment. When the application 10 receives a service execution request from a user, it requests the orchestrator 40 to execute the service via the service enabler 30 (not shown) (step S201). The orchestrator 40 controls the operation of the information processing infrastructure unit 50 (step S202). Detailed operation of the orchestrator 40 will be described later. Based on the control of the orchestrator 40, the information processing infrastructure unit 50 generates a container for camera image processing to execute the service requested by the application 10 (step S203). While containers are used as virtualization technology in this embodiment, a virtual machine, which is a similar technology, may also be used. The information processing infrastructure unit 50 deploys the camera image processing container to the server 150, which is the computer that actually performs the camera image processing (step S204). The server 150, which is the computer, starts the camera image processing (step S205). Here, it is assumed that there are n servers 150, which are computers that perform camera image processing, but the number of servers 150 may be one.

[0046] The server 150, which is a computer that actually processes the camera images, can be installed in multiple areas including the edge. The platform system 20 can divide and allocate the computer processing required for generating 3D images to each server 150, including the edge servers 154 and 155.

[0047] FIG. 5 is a flowchart illustrating the operation of the orchestrator 40 in the cyber-physical system 1 according to this embodiment, from acquiring data according to the state of the line 160 to starting image processing. The orchestrator 40 receives a service request from the application 10 via the service enabler 30 (not shown) (step S301). The orchestrator 40 determines the computing capacity and currently executing computing processes for each server 150 (step S302). The orchestrator 40 derives the communication resources and computing resources required for the service requested by the application 10 (step S303). The orchestrator 40 determines the state of the network communication lines leading to the drone 170 and the HMD 110 (step S304). The orchestrator 40 determines the layout of the 3D image generation process and distribution process and the network route (step S305). In other words, when there are multiple communication paths for the network processing infrastructure unit 60 to acquire information from the device group 70, the orchestrator 40 determines the communication path for the network processing infrastructure unit 60 to acquire information from the device group 70 based on computer resources and communication resources. The orchestrator 40 issues control instructions to the information processing infrastructure unit 50 (step S306). In the flowchart shown in Figure 5, the operations from step S301 to step S305 correspond to the operations of the orchestrator 40 in the sequence diagram shown in Figure 3, and the operation of step S306 corresponds to the operations of the orchestrator 40 in the sequence diagram shown in Figure 4.

[0048] FIG. 6 is a sequence diagram illustrating the operation of the orchestrator 40 when changing resources in the cyber-physical system 1 according to this embodiment. The application 10 notifies the orchestrator 40 of quality degradation via the service enabler 30 (not shown) (step S401). Quality degradation can be, for example, when frame dropping is detected in video data displayed by an application. The orchestrator 40 requests computer resource information from the information processing infrastructure unit 50 (step S402). The information processing infrastructure unit 50 transmits the computer resource information in response to the request from the orchestrator 40 (step S403). The orchestrator 40 requests communication resource information from the network processing infrastructure unit 60 (step S404). The network processing infrastructure unit 60 transmits the communication resource information in response to the request from the orchestrator 40 (step S405). The orchestrator 40 performs resource changes based on the acquired computer resource and communication resource information (step S406). Detailed operations of step S406 will be described later. The orchestrator 40 notifies the application 10 of the change in service level (step S407). The application 10 that has received the notification adjusts the conflict regarding the service level with the other application 10 (step S408). The application 10 that has received the notification resolves the conflict by either asking the other application 10 to lower its service level or by lowering its own service level.

[0049] FIG. 7 is a flowchart showing the operation of the orchestrator 40 when changing resources in the cyber-physical system 1 according to this embodiment. The flowchart in FIG. 7 shows details of the operation of step S406 in the sequence diagram shown in FIG. 6. The orchestrator 40 grasps the computing capacity and the computing processes being executed of each server 150 (step S501). The orchestrator 40 changes the resource allocation taking priority into consideration (step S502). The resource allocation change refers to an increase or decrease in communication resources and computing resources. The orchestrator 40 changes the communication path, which involves changing the computing process placement (step S503). If the orchestrator 40 detects a conflict between applications 10 due to the change (step S504: Yes), it notifies the related applications 10 of the change in service level (step S505). If the orchestrator 40 does not detect a conflict in the application 10 due to the change (step S504: No) but detects a resource shortage (step S506: Yes), the orchestrator 40 predicts a future resource shortage situation (step S507). After step S505, after step S507, or if the orchestrator 40 has not detected a resource shortage (step S506: No), the orchestrator 40 implements a resource change (step S508).

[0050] In particular, the orchestrator 40 notifies the service enabler 30 when it is unable to meet the service level requested by the application 10 based on the computer resources and communication resources. Based on the notification from the orchestrator 40, the service enabler 30 requests the application 10 to change the service level.

[0051] As shown in Figures 6 and 7, when the orchestrator 40 receives a quality degradation notification from an application 10, detects a shortage of computer resources in the information processing infrastructure unit 50, or detects a shortage of communication resources in the network processing infrastructure unit 60, it changes the resource allocation taking service priority into consideration and changes the communication path, which involves changing the computational processing layout. Changing the resource allocation refers to increasing or decreasing communication resources and computer resources. If the quality degradation or resource shortage is caused by congestion due to multiple applications 10, the orchestrator 40 notifies the relevant applications 10 of a change in service level and works together with the relevant applications 10 to reduce the number of cameras 74 used, shrink the drawing area or distribution area, and adjust application 10 parameters such as service time or period. When detecting a resource shortage, the orchestrator 40 may also predict future resource shortages based on its own service scheduling information and past resource usage history.

[0052] In this way, when there is a resource surplus or shortage, the cyber-physical system 1 mediates between the application 10, the orchestrator 40, and the information processing infrastructure unit 50, specifically increasing or decreasing resources, increasing or decreasing processing, and changing priorities. For example, when communication resources are insufficient, the cyber-physical system 1 can transmit data even with limited communication resources by changing the processing method of the information processing infrastructure unit 50. Furthermore, when the information processing infrastructure unit 50 is processing a large volume of data, the cyber-physical system 1 provides sufficient computer resources and communication resources for control. However, if necessary control information, such as maintaining periodicity, is required, the priority of sending information via communication must be changed. In the cyber-physical system 1, the deterministic network unit 65 of the network processing infrastructure unit 60 appropriately controls such priorities.

[0053] In the following, in [1] to

[12] , the operation of the cyber-physical system 1 will be explained, focusing on the device configuration of the cyber-physical system 1 shown in Figure 2, using as an example the operation for grasping a disaster situation for disaster prevention.

[0054] [1] To grasp the disaster situation after a disaster area has been identified, a user uses application 10 to set flight routes for multiple drones 170 equipped with pre-registered cameras 74 via one of the lines 160 so that the drones 170 fly to the target object or target location to capture camera images. In cyber-physical system 1, the drones 170 are managed by their aircraft IDs (identifiers). The drones 170 can be configured or instructed to capture the target object or target location using wireless media, such as 5G (5th Generation) or Wi-Fi (registered trademark), via satellite line 162 or terrestrial line 163 via line 160. The position and orientation of the drones 170, as well as status information about the sensors and batteries equipped on the drones 170, can be obtained via wireless media at a location remote from the drones 170, such as an HMD 110, which is an application terminal for application 10.

[0055] [2] One or more drones 170 move toward and around the target object or location where the disaster should be observed. While moving around the surrounding area, they acquire image data using their cameras 74 based on instructions from a remote location via a wireless medium, such as 5G or Wi-Fi. Time synchronization is established in advance between the one or more drones 170 and the ground-side line 160. For accurate time information, the drones 170 may use signals from a GPS (Global Positioning System) 200 (not shown) or time information provided to the drones 170 from a base station. The GPS 200 may also use other GNSS (Global Navigation Satellite Systems). The drones 170 acquire image data using their cameras 74 along their flight routes to check the traffic flow to the disaster area.

[0056] [3] In addition to the images, the drone 170 also transmits aircraft information such as time, position, and orientation via the base station to the ground-side line 160. If an edge server is located on the base station or on the parent drone 170a of multiple drones 170, the base station or parent drone 170a may add aircraft information such as time, position, and orientation to the images from the child drone 170b before transferring them to the line 160. The drone 170 can obtain information such as time, position, and orientation by using signals from the GPS 200, its own acceleration sensor, geomagnetic sensor, etc.

[0057] [4] As an example of edge processing that can be provided by the line 160, images from multiple drones 170 taken at the same time are combined into a single image, i.e., a single still image in which multiple images are pasted together, is generated by the edge processing unit 53 of the information processing infrastructure unit 50 and transferred to the next line 160. For example, in the case of a camera 74 of about 30 fps, even if the frame period is constant, there may be a delay of 33 ms in the worst case just due to frame synchronization depending on the communication period, and synchronization accuracy that is sufficiently shorter than that time is required.

[0058] [5] Whether to use the satellite line 162 or the terrestrial line 163 is determined by the orchestrator 40 by compiling information on the status of the line 160 on the route from the application 10 to the server 150, such as the number of transfers and resource status, using commands such as Ping to measure communication delay time and Traceroute to grasp the communication route, thereby grasping the communication status over time.

[0059] [6] If the drone 170 is located within the coverage area of ​​the base station associated with the ground line 160 and service is available, the application service control unit 41 of the orchestrator 40 selects and uses the route of the ground line 160 based on the results of resource monitoring for routers and the like connected to the edge servers 154 and 155. Whether the drone 170 is able to provide service within the coverage area of ​​the base station is assumed to be determined based on the strength of radio waves, the location information of the drone 170, and the like. If congestion or the like occurs on the route of the line 160 and the terrestrial line 163 is unavailable, the application service control unit 41 of the orchestrator 40 selects a route using the satellite line 162.

[0060] [7] The cyber-physical system 1 previously selected a route to use for transmitting information, but now it transmits information to the server 150 of the CPS server 151 or edge servers 154, 155 that should be targeted depending on the required computational processing, using both the satellite line 162 and the terrestrial line 163, or whichever route is more effective in terms of communication bandwidth, delay time, etc. When the server 150 obtains information via both the satellite line 162 and the terrestrial line 163, it may use the information that arrives first, or it may use the information obtained via a route with a low packet loss rate or a route with a short RTT delay depending on the data being handled.

[0061] [8] The CPS server 151 or the server 150 of the edge servers 154 and 155 performs calculations using aircraft information such as time, position, and orientation attached to the images. The CPS server 151 or the server 150 of the edge servers 154 and 155 creates a 3D image model for understanding the disaster situation using the information such as time, position, and orientation attached to the images. In order to understand the situation in real time, the CPS server 151 or the server 150 of the edge servers 154 and 155 creates a 3D image model, for example, at short, real-time intervals, and then adjusts the size so that the image can be mapped, i.e., synthesized, on an electronic map in cyberspace based on the location information for observing the disaster, and outputs the synthesized 3D image from the server 150 to a router connected to the server 150. The CPS server 151 or the server 150 of the edge servers 154 and 155 may map the 3D image model as an icon on an electronic map, or may color-code each numerical data, such as the number of houses in the display area, or represent it as a graph with different markers. The color coding may be used depending on the level of danger, such as red or yellow, or green for safety. The administrator determines the color coding based on the situation. Furthermore, the CPS server 151 or the server 150 of the edge servers 154 and 155 may time-synchronize multiple images obtained from multiple drones 170 and map them on an electronic map, allowing multiple objects or locations to be observed. Specifically, the image processing unit 51A of the information processing infrastructure unit 50 of the CPS server 151 or the server 150 of the edge servers 154 and 155 performs the above-described processing.

[0062] [9] To transfer a 3D image composited with a map in cyberspace to a display device such as the HMD 110 of a user who is a remote operator, the information processing infrastructure unit 50 first outputs the image from the server 150 of the CPS server 151 or the edge server 154, 155 to a router connected to the server 150, and then determines in advance whether to transfer the image via the satellite line 162 or the terrestrial line 163 by grasping the route status. To grasp the route status, the information processing infrastructure unit 50 collects resource monitoring results of the line 160, such as QoS, RTT, and throughput, and makes a decision taking into account the differences in transmission speeds between the uplink line 160 and the downlink line 160. Note that the information processing infrastructure unit 50 may transfer the 3D image to the user's HMD 110 via both the satellite line 162 and the terrestrial line 163.

[0063]

[10] To check the disaster situation using 3D images with the HMD 110, the user sends instructions such as control information to change the position of the target drone 170 so that the target object or target location can be viewed from the desired direction, and information to turn the camera 74 on and off. The control information includes, for example, the drone ID, position, and orientation. Such instructions are sent as control information to the CPS server 151 or the server 150 of the edge servers 154 and 155 via a route similar to that of [9], and the slicing unit 64 of the network processing infrastructure unit 60 performs resource control at a smaller level, such as determining whether to use the satellite line 162 or the terrestrial line 163, to improve response, i.e., reduce latency.

[0064]

[11] To return the drone 170 to the recovery site, the orchestrator 40 guides the drone 170 via the satellite link 162 or the terrestrial link 163 while obtaining control information of the drone 170, such as the aircraft ID, position, and orientation.

[0065]

[12] Assuming that a database for normal times is available, the platform system 20 stores an application 10 that can be used to determine evacuation routes, such as an application 10 that uses a 2D camera, including pre-disaster images, the position of the drone 170 when the images were captured, and information on the camera 74 settings. Pre-disaster images are, for example, roughly captured bird's-eye views or omnidirectional images of specific objects. This allows the platform system 20 to use the drone 170 to capture images after a disaster under the same conditions. The platform system 20 may also store conditions that allow images to be processed to appear as if they were captured under the same conditions, even if the capture conditions are different. This allows users to roughly check the damage situation by comparing the pre-disaster situation with the post-disaster situation. It is assumed that base information is required to check the detailed situation by operating the drone 170. The priority of the capture points is set by reports, seismic intensity information, pre-settings, etc. Pre-settings include, for example, registering important locations. The platform system 20 can also be used to estimate earthquakes based on information about the damage situation.

[0066] The operations described in [1] to

[12] above will be explained using sequence diagrams, etc. Specifically, the operation will be explained using an example in which a user uses the HMD 110 to check the disaster situation in District A, a disaster area photographed by the drone 170.

[0067] FIG. 8 is a sequence diagram showing the operation of the cyber-physical system 1 according to the present embodiment until the drone 170 is ready for use in response to an instruction from the application 10. The application 10 instructs the service enabler 30 of the platform system 20 to take photographs using the drone 170 using keywords "Area A" and "3D photography" (step S601). The service enabler 30 instructs the application service control unit 41 of the orchestrator 40 to take photographs using the drone 170 using keywords "Area A" and "3D photography" (step S602). The application service control unit 41 of the orchestrator 40 performs a photographing preparation process with the drone 170 and the like (step S603). Details of the photographing preparation process will be described later. Next, the application service control unit 41 of the orchestrator 40 performs a training process with the drone 170 and the like (step S604). Details of the training process will be described later.

[0068] The application service control unit 41 of the orchestrator 40 responds to the instruction in step S602 to the service enabler 30 (step S605). The response includes the estimated time for acquiring the 3D image. The service enabler 30 responds to the instruction in step S601 to the application 10 (step S606). The response includes information such as whether acquisition is possible, the estimated acquisition time, training completion, the communication quality of the satellite line 162, and the communication quality of the terrestrial line 163. The application service control unit 41 of the orchestrator 40 performs monitoring processing of the line 160 as a periodic process (step S607). Details of the monitoring processing will be described later.

[0069] FIG. 9 is a sequence diagram showing detailed operations of the imaging preparation process of step S603 shown in FIG. 8 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues a preparation instruction to the information processing infrastructure unit 50 in Area A using "3D imaging" as a keyword (step S701). The information processing infrastructure unit 50 in Area A issues an instruction to the drone 170 to prepare for imaging using the aircraft ID or camera ID as a keyword (step S702). The drone 170 synchronizes the time with the GPS 200 (step S703). The drone 170 responds to the instruction in step S702 to the information processing infrastructure unit 50 in Area A (step S704). The response includes information such as the aircraft ID or camera ID and the time error. The information processing infrastructure unit 50 in Area A responds to the instruction in step S701 to the application service control unit 41 of the orchestrator 40 (step S705). The response includes information such as whether acquisition is possible and the expected time.

[0070] 10 is a sequence diagram showing detailed operations of the training process of step S604 shown in FIG. 8 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues a preparation instruction to the information processing infrastructure unit 50 in Area A using "3D photography" as a keyword (step S801). The information processing infrastructure unit 50 in Area A issues a training instruction to the network processing infrastructure unit 60 in Area A using the aircraft ID or camera ID and "3D" as keywords (step S802). The network processing infrastructure unit 60 in Area A issues a training instruction to the drone 170 using "Path" and "3D" as keywords (step S803).

[0071] The drone 170 transmits test data to the information processing infrastructure unit 50 in area A (step S804). The test data includes, for example, data captured by the drone 170 for testing. The drone 170 transmits drone information to the information processing infrastructure unit 50 in area A (step S805). The drone information includes, for example, the remaining battery power. The drone 170 sends a training response to the network processing infrastructure unit 60 in area A in response to the training instruction sent in step S803 (step S806). The training response includes information such as training completion. The network processing infrastructure unit 60 in area A sends a training completion response to the information processing infrastructure unit 50 in area A in response to the training instruction sent in step S802 (step S807). The training completion response includes information such as training completion, the communication quality value of the satellite link 162, and the communication quality value of the terrestrial link 163. The information processing infrastructure unit 50 in area A responds to the application service control unit 41 of the orchestrator 40 in response to the preparation instruction in step S801, indicating that training has been completed (step S808). The response includes information such as the completion of training, the communication quality value of the satellite link 162, and the communication quality value of the terrestrial link 163.

[0072] FIG. 11 is a sequence diagram showing detailed operation of the monitoring process of step S607 shown in FIG. 8 in the cyber-physical system 1 according to this embodiment. The network processing infrastructure unit 60 in area A instructs the lines 160, such as the satellite line 162 and the terrestrial line 163, to prepare a resource monitor (step S901). The lines 160, such as the satellite line 162 and the terrestrial line 163, send a resource monitor response to the resource monitor preparation instruction sent in step S901 to the network processing infrastructure unit 60 in area A (step S902). The resource monitor response includes the bandwidth value and delay value of the line 160. Thereafter, the lines 160, such as the satellite line 162 and the terrestrial line 163, send similar resource monitor responses to the network processing infrastructure unit 60 in area A. The application service control unit 41 of the orchestrator 40 instructs the network processing infrastructure unit 60 in area A to perform resource monitoring using area A as a keyword (step S903). The network processing infrastructure unit 60 in area A sends a resource monitor response to the resource monitor instruction sent in step S903 to the application service control unit 41 of the orchestrator 40 (step S904). The resource monitor response includes information such as an average bandwidth value and an average delay value based on the resource monitor responses that the network processing infrastructure unit 60 in area A has acquired multiple times from the lines 160, such as the satellite line 162 and the terrestrial line 163.

[0073] FIG. 12 is a sequence diagram showing the operation of notifying the application 10 of the estimated shooting time by the drone 170 in the cyber-physical system 1 according to this embodiment. The application 10 issues an instruction to the service enabler 30 using keywords such as area A, 3D shooting, and line designation (step S1001). The service enabler 30 issues an instruction to the application service control unit 41 of the orchestrator 40 using keywords such as area A, 3D shooting, and line designation (step S1002). The application service control unit 41 of the orchestrator 40 issues an instruction to the information processing infrastructure unit 50 in area A using keywords such as area A, 3D shooting, and line designation (step S1003). The information processing infrastructure unit 50 in area A responds to the instruction in step S1003 to the application service control unit 41 of the orchestrator 40 (step S1004). The response includes the estimated shooting time and the like. The application service control unit 41 of the orchestrator 40 responds to the instruction in step S1002 to the service enabler 30 (step S1005). The response includes the estimated shooting time, etc. The service enabler 30 responds to the instruction in step S1001 to the application 10 (step S1006). The response includes the estimated shooting time, etc.

[0074] (1-1) In this way, in the cyber-physical system 1, the application 10 instructs the service enabler 30 that "I would like to use the drone 170 to view the disaster situation in area A in 3D images."

[0075] (1-2) The service enabler 30 selects the network processing infrastructure unit 60 and information processing infrastructure unit 50 that are closest to Area A for the application service control unit 41 of the orchestrator 40. The information processing infrastructure unit 50 searches for the number of drones 170 in Area A required to generate a 3D image, calculates the arrival time of the drones 170, the expected time for obtaining the 3D image, and the like, and notifies the application service control unit 41 of the orchestrator 40 of the presence or absence of the drones 170, and if present, the expected time for obtaining the 3D image by the drones 170, and the like.

[0076] (1-3) The aircraft IDs or camera IDs of the multiple drones 170 are set by the information processing infrastructure unit 50. The multiple drones 170 synchronize their times using GPS 200 or the like via the line 160 or the like. If there is a time discrepancy among the multiple drones 170, the information processing infrastructure unit 50 synchronizes the times of the multiple drones 170.

[0077] (1-4) In the event of a disaster, communication may be interrupted or there may be restrictions on communication itself, such as restrictions on communication. Therefore, the application service control unit 41 of the orchestrator 40 monitors the wireless availability status at the site in Area A through resource monitoring.

[0078] (1-5) Furthermore, if possible during a disaster, the application service control unit 41 of the orchestrator 40 requests the telecommunications company that provides the public line to secure the communication resources necessary for capturing images near the site in Area A. That is, the application service control unit 41 of the orchestrator 40 requests the telecommunications company of an available satellite line 162 or terrestrial line 163 from among multiple public lines to secure the line 160 in order to set a slice so that the target application 10 can use it preferentially.

[0079] (1-6) Before or during the flight of the drone 170, the network processing infrastructure unit 60 transmits the image from the camera 74 via the drone 170 as training for transmitting the image from the camera 74, and checks the communication status, such as QoS, frame error rate, and throughput. The available shooting time of the drone 170 is related to the remaining charge of the flight battery of the drone 170, the remaining charge of the battery of the camera 74, and the remaining charge of the battery for the communication device, so the information processing infrastructure unit 50 collects information on the remaining charge of each battery via the line 160 and calculates the available continuous shooting time of the drone 170. The information processing infrastructure unit 50 takes into account not only the terrestrial line 163 but also the consumption of the battery for the communication device when communicating via the satellite line 162 as a communication means.

[0080] (1-7) The application service control unit 41 of the orchestrator 40 selects one of the terrestrial line 163 and the satellite line 162 as a communication means, or selects two lines 160 to expand the coverage of the service area, based on the results of the communication quality confirmed by the network processing infrastructure unit 60 and the results of the available shooting time calculated by the information processing infrastructure unit 50. The application service control unit 41 of the orchestrator 40 may combine different terrestrial public lines 160, or may combine the terrestrial line 163 and the satellite line 162. The operations up to this point, from (1-1) to (1-7), correspond to the operations in the sequence diagrams of Figures 8 to 11 described above.

[0081] (1-8) After the communication means is determined, the power consumption of the drone 170 or the like is known, and therefore the information processing infrastructure unit 50 first considers the shooting time taking into account the remaining battery charge. The orchestrator 40 transmits the shooting time consideration result by the information processing infrastructure unit 50 to the application 10 via the service enabler 30. This operation (1-8) corresponds to the operation in the sequence diagram of FIG. 12 described above. Furthermore, the operations from (1-1) to (1-8) correspond to the operation [1] described above.

[0082] This allows the platform system 20 to set, for the drone 170, the lines 160 that are available on the flight route of the drone 170. When there are multiple drones 170, the platform system 20 can set a priority for each drone 170 and also set a role for each drone 170. The role of each drone 170 refers to, for example, the placement of each drone 170 when multiple drones 170 form a formation, the shooting direction of each drone 170 when multiple drones 170 photograph an object or target location, etc.

[0083] FIG. 13 is a sequence diagram showing the operation of the application 10 acquiring a spatial object image in the cyber-physical system 1 according to this embodiment. Multiple drones 170 transmit data, such as images captured by the camera 74, to the information processing infrastructure unit 50 in Area A (step S1101). The information processing infrastructure unit 50 performs image processing to generate a spatial representation image displayable on the HMD 110 using the images captured by the camera 74 included in the data acquired from the drones 170 (step S1102). The spatial representation image is, for example, the aforementioned 3D image. The information processing infrastructure unit 50 transmits the generated spatial representation image to the HMD 110, which is the application terminal of the application 10, via the service enabler 30 (not shown) (step S1103). The HMD 110, which is the application terminal of the application 10, displays the acquired spatial representation image and performs object detection processing (step S1104). The information processing infrastructure unit 50 issues a movement instruction to the drone 170, including data on the movement location, using the aircraft ID or camera ID as a keyword (step S1105). The drone 170 issues a movement response to the information processing infrastructure unit 50, including data on the current location and the movement location, using the aircraft ID or camera ID as a keyword (step S1106). The information processing infrastructure unit 50 performs monitoring processing at the position of the drone 170 based on the current location information acquired from the drone 170 (step S1107).

[0084] 14 is a sequence diagram showing an operation of checking the communication status between the drone 170 and the HMD 110, which is the application terminal of the application 10, in the cyber-physical system 1 according to the present embodiment. The information processing infrastructure unit 50 instructs the drone 170 to check the communication status using Ping and Traceroute, using the aircraft ID or camera ID as a keyword (step S1201). The drone 170 checks the communication status between the drone 170 and the HMD 110, which is the application terminal of the application 10, using Ping and Traceroute, i.e., executes a Ping command and a Traceroute command (step S1202). The drone 170 responds to the instruction in step S1201 to the information processing infrastructure unit 50 (step S1203). The response includes the results of checking the communication status using Ping and Traceroute.

[0085] (2-1) In this way, in the cyber-physical system 1, when the drone 170 starts capturing images, the information processing infrastructure unit 50 transmits a spatial representation image that is spatially reproduced through image processing to the HMD 110, which is the application terminal of the application 10. The HMD 110, which is the application terminal of the application 10, performs object detection processing in the virtual space. The object detection processing detects, for example, the presence or absence of infrastructure such as fallen trees and utility poles, collapsed buildings, river flooding, and victims.

[0086] (2-2) The drone 170 changes the zoom of the camera 74, the attitude of the drone 170, the direction of the camera 74, and so on. The camera 74 is used depending on the application, depending on the type of camera itself, such as an infrared camera, and the type of lens, such as a wide-angle lens or a telephoto lens. In the cyber-physical system 1, if multiple drones 170 exist, different combinations of cameras 74 and lenses may be mounted on the drones 170, allowing the drone 170 to be selected based on the combination of camera 74 and lens. The edge processing unit 53 of the information processing infrastructure unit 50 determines the position of the drone 170 and moves the drone 170 to control the position of the drones 170 that have gathered in Area A to photograph the disaster area. The position of the drone 170 may be determined using the GPS 200, and the angle, direction, etc. of the drone 170 may be determined using an acceleration sensor (not shown), which is one of the sensing units 73 mounted on the drone 170. When moving the drone 170, the information processing infrastructure unit 50 controls the operation of the actuator unit 76, which is a motor that drives a propeller or the like mounted on the drone 170, for example.

[0087] (2-3) In the cyber-physical system 1, the position information of the drone 170 is constantly shared by edge processing related to remote control, and can be used to consider flight routes. The operations from (2-1) to (2-3) up to this point correspond to the operations in the sequence diagram of Figure 13 mentioned above.

[0088] (2-4) In the cyber-physical system 1, clustering and spatiotemporal synchronization processes are performed while communicating between multiple drones 170 in a formation. The highly accurate time reference between the multiple drones 170 may be realized using the aforementioned GPS 200 or PTP (Precision Time Protocol). When multiple drones 170 are connected to each other via a V2N2V (Vehicle to Network to Vehicle) network configuration, each drone 170 connects to an edge server 154, 155 of a base station via wireless communication. In a device group 70 included in each drone 170, a clustering unit 71 performs clustering processing, and a spatiotemporal synchronization unit 72 performs spatiotemporal synchronization. In a V2V (Vehicle to Vehicle) network configuration, the parent drone 170a may have an edge function, and the parent drone 170a may control the shooting position and shooting direction of the camera 74 of the child drone 170b in a subordinate manner. Each drone 170 selects a line 160 that satisfies the required communication quality as much as possible while checking the results of monitoring by the resource monitoring unit 61 of the network processing infrastructure unit 60 to see if the line 160 has been secured. As a result, it is expected that some drones 170 will use a satellite line 162 or a line 160 that is a different public line from the usual one.

[0089] (2-5) Whether to use the satellite line 162 or the terrestrial line 163 is determined by the orchestrator 40, which aggregates information on the status of the line 160 on the route from the application 10 to the server 150, such as the number of transfers and resource status, using commands such as Ping to measure communication delay time and Traceroute to grasp the communication route, thereby grasping the communication status over time and making a decision.

[0090] (2-6) If the drone 170 is located within the coverage area of ​​the base station associated with the ground line 160 and service is available, the application service control unit 41 of the orchestrator 40 selects and uses the ground line 160 route based on the results of resource monitoring for routers connected to the edge servers 154 and 155. Whether the drone 170 is able to provide service within the base station's coverage area is assumed to be determined based on the reception strength of radio waves transmitted from the base station and the drone 170's location information. If congestion or other issues occur on the line 160 route and the ground line 163 is unavailable, the application service control unit 41 of the orchestrator 40 selects a route using the satellite line 162. The operations from (2-4) to (2-6) above correspond to the operations in the sequence diagram of Figure 14 described above.

[0091] (3-1) The drone 170 previously selected the line 160 route, but now transmits information to the CPS server 151 or edge servers 154, 155 that should be targeted for the required computational processing using both the satellite line 162 and the terrestrial line 163, or using whichever line 160 is available based on communication capacity, delay time, etc. When information arrives from both lines 160, the CPS server 151 or edge servers 154, 155 may use the information that arrives first, or may select a line 160 depending on the application 10 being handled and use the information. Regarding the communication route from the drone 170 to the HMD 110, which is the application terminal for the application 10, a selection is made to secure resources in accordance with the application 10, or to change the operation of the application 10 and the device group 70 in accordance with the available resources. Regarding the latter, since upstream communication relates to, for example, the amount of image data that can be transmitted, the setting is changed based on the resolution x fps related to the function of the sensing unit 73, and since downstream communication relates to, for example, the transfer delay of control signals to the drone 170, the setting is changed based on the degree of autonomy of control of the drone 170 by the information processing infrastructure unit 50. Note that the transfer delay of control signals to the drone 170 can be ascertained because the resource monitoring unit 61 of the network processing infrastructure unit 60 constantly monitors the RTT. The operations from (2-1) to (2-6) and (3-1) correspond to the operations [2] to [7] described above.

[0092] 15 is a sequence diagram showing the operation of the child drone 170b transmitting data via the parent drone 170a in the cyber-physical system 1 according to this embodiment. The child drone 170b transmits data to the information processing infrastructure unit 50 (step S1301). However, it is assumed that the data transmitted from the child drone 170b does not reach the information processing infrastructure unit 50. The parent drone 170a transmits data to the information processing infrastructure unit 50 (step S1302). Because the data transmitted from the parent drone 170a reaches the information processing infrastructure unit 50, the information processing infrastructure unit 50 transmits a reception response to the data transmission to the parent drone 170a (step S1303).

[0093] The information processing infrastructure unit 50 determines that data has not been received from the child drone 170b and performs the following operations to obtain data from the child drone 170b via the parent drone 170a. First, the information processing infrastructure unit 50 issues a movement instruction to the child drone 170b, including a movement location, using the aircraft ID or camera ID as a keyword (step S1304). The child drone 170b sends a movement response to the information processing infrastructure unit 50, including its current location and a movement location, using the aircraft ID or camera ID as a keyword (step S1305). The information processing infrastructure unit 50 issues a V2 circuit establishment instruction to the parent drone 170a and the child drone 170b, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as a keyword (step S1306). The parent drone 170a sends a V2 circuit establishment response to the information processing infrastructure unit 50, using the aircraft ID or camera ID of the parent drone 170a and the child drone 170b as a keyword. Similarly, the child drone 170b sends a V2 line establishment response to the information processing infrastructure unit 50 using the aircraft ID or camera ID of the parent drone 170a and child drone 170b as keywords (step S1307).

[0094] The child drone 170b relays data to the parent drone 170a (step S1308). The parent drone 170a transmits the data acquired from the child drone 170b to the information processing infrastructure unit 50 (step S1309). The data of the child drone 170b transmitted from the parent drone 170a reaches the information processing infrastructure unit 50, and the information processing infrastructure unit 50 sends a reception response to the data transmission to the parent drone 170a (step S1310). When the parent drone 170a receives the reception response to the data transmission of the child drone 170b from the information processing infrastructure unit 50, the parent drone 170a sends a reception response to the data transmission to the child drone 170b on behalf of the information processing infrastructure unit 50 (step S1311).

[0095] In this way, if the information processing infrastructure unit 50 cannot acquire the first information from the child drone 170b, which is the first device included in the device group 70, but can acquire the second information from the parent drone 170a, which is the second device included in the device group 70, and communication is possible between the child drone 170b, which is the first device, and the parent drone 170a, which is the second device, it instructs the child drone 170b, which is the first device, to send the first information to the parent drone 170a, which is the second device, and acquires the first information and the second information from the parent drone 170a, which is the second device.

[0096] 16 is a sequence diagram showing the operation of the parent drone 170a synthesizing and transmitting data acquired from the child drone 170b in the cyber-physical system 1 according to this embodiment. The image processing unit 51A of the information processing infrastructure unit 50 requests network resource information from the network processing infrastructure unit 60 using the drone 170's aircraft ID or camera ID as a keyword (step S1401). The network processing infrastructure unit 60 notifies the image processing unit 51A of the information processing infrastructure unit 50 of network resource information in response to the network resource information request using the drone 170's aircraft ID or camera ID as a keyword (step S1402). The network resource information notification includes the network resource information.

[0097] Based on the network resource information acquired from the network processing infrastructure unit 60, the image processing unit 51A of the information processing infrastructure unit 50 determines that the network resources between the information processing infrastructure unit 50 and the parent drone 170a are sufficient, but that the network resources between the information processing infrastructure unit 50 and the child drone 170b are insufficient. Therefore, the image processing unit 51A of the information processing infrastructure unit 50 performs the following operation to composite data from the child drone 170b in the parent drone 170a before acquiring the data. First, the image processing unit 51A of the information processing infrastructure unit 50 issues an image composite instruction to the parent drone 170a using the aircraft ID or camera ID and "2-in-1" as keywords (step S1403). The parent drone 170a then issues an image composite response to the image composite instruction to the image processing unit 51A of the information processing infrastructure unit 50 using the aircraft ID or camera ID and "2-in-1" as keywords (step S1404). The image processing unit 51A of the information processing infrastructure unit 50 issues a V2 line establishment instruction to the parent drone 170a and the child drone 170b, using the aircraft IDs or camera IDs of the parent drone 170a and the child drone 170b as keywords (step S1405). The parent drone 170a issues a V2 line establishment response to the image processing unit 51A of the information processing infrastructure unit 50, using the aircraft IDs or camera IDs of the parent drone 170a and the child drone 170b as keywords. Similarly, the child drone 170b issues a V2 line establishment response to the image processing unit 51A of the information processing infrastructure unit 50, using the aircraft IDs or camera IDs of the parent drone 170a and the child drone 170b as keywords (step S1406).

[0098] The child drone 170b transmits data to the parent drone 170a (step S1407). The parent drone 170a performs 2-in-1 processing to combine image data acquired by its own camera 74 with image data acquired from the child drone 170b (step S1408). The parent drone 170a transmits the image data obtained as a result of the 2-in-1 processing to the information processing infrastructure unit 50 (step S1409). The information processing infrastructure unit 50 sends a reception response to the data transmission to the parent drone 170a (step S1410). The parent drone 170a sends a reception response to the data transmission to the child drone 170b (step S1411).

[0099] In this way, the information processing infrastructure unit 50 includes a child drone 170b, which is a first device that transmits first information to the device group 70, and a parent drone 170a, which is a second device that transmits second information, and when communication with the parent drone 170a, which is the second device, is possible and communication is possible between the child drone 170b, which is the first device, and the parent drone 170a, which is the second device, it instructs the child drone 170b, which is the first device, to transmit the first information to the parent drone 170a, which is the second device, and obtains composite information from the parent drone 170a, which is a combination of the first information and the second information.

[0100] FIG. 17 is a sequence diagram showing the operation when the orchestrator 40 causes the drone 170 to change operating conditions, such as the shooting conditions of the camera 74, in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues an encoder change instruction to the information processing infrastructure unit 50 in Area A using the drone 170's aircraft ID or camera ID as a keyword (step S1501). The encoder change instruction includes the frame rate, resolution, bit rate, compression method, etc. The information processing infrastructure unit 50 in Area A issues an encoder change instruction to the drone 170 using the drone 170's aircraft ID or camera ID as a keyword (step S1502). The drone 170 issues an encoder change response to the encoder change instruction to the information processing infrastructure unit 50 in Area A using the drone 170's aircraft ID or camera ID as a keyword (step S1503). The information processing infrastructure unit 50 in area A sends an encoder change response to the encoder change instruction to the application service control unit 41 of the orchestrator 40 using the drone 170's aircraft ID or camera ID as a keyword (step S1504).

[0101] FIG. 18 is a sequence diagram showing the operation of the HMD 110, which is the application terminal of the application 10 in the cyber-physical system 1 according to this embodiment, until it acquires 3D video. The drones 170 transmit image data captured by the camera 74 to the image processing unit 51A of the information processing infrastructure unit 50 (step S1601). The data includes image data, time information at the time of capture, the position and orientation of the camera 74, and information obtained by sensors such as the acceleration sensor and LIDAR included in the sensing unit 73. The image processing unit 51A of the information processing infrastructure unit 50 performs a synthesis process to synthesize the image data acquired from the drones 170 (step S1602). The HMD 110, which is the application terminal of the application 10, instructs the application service control unit 41 of the orchestrator 40 to acquire video using keywords "Area A" and video (step S1603). The application service control unit 41 of the orchestrator 40 instructs the video processing unit 51A of the information processing infrastructure unit 50 to acquire the video using the keywords "Area A" and the video (step S1604).

[0102] In response to the acquisition instruction from the application service control unit 41 of the orchestrator 40, the video processing unit 51A of the information processing infrastructure unit 50 transmits the video to the HMD 110, which is the application terminal of the application 10, via the service enabler 30 (not shown) without going through the application service control unit 41 of the orchestrator 40 (step S1605). Alternatively, in response to the acquisition instruction from the application service control unit 41 of the orchestrator 40, the video processing unit 51A of the information processing infrastructure unit 50 transmits the video to the application service control unit 41 of the orchestrator 40 (step S1606). In response to the acquisition instruction from the HMD 110, which is the application terminal of the application 10, the application service control unit 41 of the orchestrator 40 transmits the video to the HMD 110, which is the application terminal of the application 10, via the service enabler 30 (not shown) (step S1607).

[0103] (4-1) In this manner, in the cyber-physical system 1, if a drone 170 is unable to transmit camera 74 video, the drone 170 that cannot transmit camera 74 video buffers the video information and waits until it can transmit camera 74 video. For example, as one solution, the remote control unit 51E of the information processing infrastructure unit 50 issues an instruction to the drone 170 to move its position to check whether the radio wave conditions between the drone 170 and the line 160 can be improved, and moves the drone 170 a very small distance, approximately several times the wavelength of the radio signal. Alternatively, if communication recovery is not expected with the above measures, in order to prevent the camera 74 video from being unable to be transmitted, the drone 170 that cannot transmit camera 74 video is designated as a child drone 170b, and a V2V communication route is set between the parent drone 170a and the child drone 170b, so that the child drone 170b transmits camera 74 video to the parent drone 170a, which then transmits camera 74 video via the parent drone 170a. In this case, the parent drone 170a also transmits the image from its own camera 74, which increases the transmission speed of the image from the drone 170 in the cyber-physical system 1. The operations up to this point in (4-1) correspond to the operations in the sequence diagram of FIG.

[0104] (4-2) If resource monitoring in the network processing infrastructure unit 60 indicates that there is insufficient communication speed, the parent drone 170a performs image-related edge processing to combine the images from the two cameras 74 of the parent drone 170a and the child drone 170b into a single image to reduce the image size. The process of suppressing an increase in the transmission speed of the parent drone 170a's communication terminal is performed by the image processing unit 51A of the information processing infrastructure unit 50. The operations in (4-2) up to this point correspond to the operations in the sequence diagram of FIG. 16 described above.

[0105] (4-3) When a drone 170 with an insufficient communication speed exists and a 3D image needs to be generated, the orchestrator 40 changes the bit rate of the encoder of the other drones 170 to match the communication speed of the drone 170 that has been set to the minimum communication speed by the resource monitoring unit 61 of the network processing infrastructure unit 60 that performs the operation of Figure 11.

[0106] (4-4) The network processing infrastructure unit 60 performs resource monitoring for each line 160, such as receiving signal strength and QoS, and considers throughput. For example, the orchestrator 40 adjusts the rate to the required rate using a conversion function, such as lowering the rate of the camera 74 required by the information processing infrastructure unit 50. Alternatively, the orchestrator 40 responds by adjusting all of the multiple cameras 74 to the minimum bit rate. Alternatively, if the line 160 itself does not have sufficient capacity during a disaster, the orchestrator 40 creates additional communication capacity for the line 160 by stopping video transmission from the cameras 74 from multiple drones 170, and then responds to the reduced communication capacity by switching to transmitting still images from the camera 74 from at least one drone 170 via the information processing infrastructure unit 50. The operations in (4-4) up to this point correspond to the operations in the sequence diagram of Figure 17 described above.

[0107] (4-5) The location information of the drone 170 is constantly shared by the edge processing unit 53 related to remote control in the information processing infrastructure unit 50, and can be used to consider the flight route of the drone 170.

[0108] (4-6) Regarding the positioning of one or more drones 170, in order to find the location of rubble or the like as a target, the drone 170 adds time information, the position and orientation of the camera 74, acceleration sensor information, lidar information, etc. to the image captured by the camera 74 via the network processing infrastructure unit 60, and transmits the added information to the image processing unit 51A of the information processing infrastructure unit 50. The HMD 110, which is the application terminal of the application 10, acquires the image processing results, checks the local situation, and sets the position of the drone 170. When generating a 3D image using multiple drones 170, the position of the drone 170 or the orientation of the camera 74 is set so that the target object or target point is reflected in the image captured by the camera 74. The target object or target point may be a characteristic object during a disaster, or target point information indicated by latitude, longitude, altitude, etc., or marker information may be used.

[0109] (4-7) Alternatively, a user using the HMD 110 may remotely adjust the position of each drone 170 while monitoring the image of the 2D camera 74 of each drone 170 so that the target object or target point is captured by the camera 74. The operations from (4-5) to (4-7) up to this point correspond to the operations in the sequence diagram of FIG. 18 described above.

[0110] (4-8) If communication with a specific drone 170 is not going well, the network processing infrastructure unit 60 changes the position of the drone 170 in response to instructions from the orchestrator 40 based on the results of resource monitoring using Ping to measure communication delay time to check the communication status of the network processing infrastructure unit 60, Traceroute to understand the communication path, etc., so that the communication conditions are improved in the remote control-related processing unit of the information processing infrastructure unit 50.

[0111] (4-9) Furthermore, the network processing infrastructure unit 60 sends control information for changing the position of the drone 170, which is the subject of the image from the camera 74, and image capture information from the camera 74, to the drone 170 to instruct it. The control information for changing the position of the drone 170 includes, for example, the drone ID, the position of the drone 170, the orientation of the drone 170, and an image from the camera 74 of the remotely controlled drone 170. When the deterministic network unit 65 sends such instructions to the network processing infrastructure unit 60, it meticulously controls resources to improve the response and periodicity of communication on the communication path as control information to the CPS server 151.

[0112] 19 is a sequence diagram showing the operation in which the information processing infrastructure unit 50 performs calculation processing in the cyber-physical system 1 according to this embodiment and notifies the application 10 of the estimated shooting time by the drone 170. The application 10 issues an instruction to the service enabler 30 using keywords such as area A, 3D shooting, and line designation (step S1701). The service enabler 30 issues an instruction to the application service control unit 41 of the orchestrator 40 using keywords such as area A, 3D shooting, and line designation (step S1702). The application service control unit 41 of the orchestrator 40 issues an instruction to the video processing unit 51A of the information processing infrastructure unit 50 in area A using keywords such as area A, 3D shooting, and line designation (step S1703).

[0113] The video processing unit 51A of the information processing infrastructure unit 50 in Area A performs calculation processing for processing 3D imaging based on the keywords specifying Area A, 3D imaging, line specification, etc. (step S1704). The video processing unit 51A of the information processing infrastructure unit 50 in Area A responds to the instruction in Step S1703 to the application service control unit 41 of the orchestrator 40 (step S1705). The response includes the estimated imaging time, etc. The application service control unit 41 of the orchestrator 40 responds to the instruction in Step S1702 to the service enabler 30 (step S1706). The response includes the estimated imaging time, etc. The service enabler 30 responds to the instruction in Step S1701 to the application 10 (step S1707). The response includes the estimated imaging time, etc.

[0114] 20 is a sequence diagram showing the operation of resource monitoring of the information processing infrastructure unit 50 in the cyber-physical system 1 according to this embodiment. The application service control unit 41 of the orchestrator 40 issues a resource monitoring instruction to the network processing infrastructure unit 60 in Area A, using Area A as a keyword (step S1801). The network processing infrastructure unit 60 in Area A issues a resource monitoring instruction to the video processing unit 51A of the first information processing infrastructure unit 50 in Area A (step S1802). The video processing unit 51A of the first information processing infrastructure unit 50 in Area A issues an instruction to the video processing units 51A of the second to nth information processing infrastructure units 50 in Area A to prepare for resource monitoring, using the satellite line 162 or the terrestrial line 163 as a keyword (step S1803). The video processing units 51A of the second to nth information processing infrastructure units 50 in area A send a resource monitor response to the resource monitor preparation instruction of step S1803 to the video processing unit 51A of the first information processing infrastructure unit 50 in area A using the satellite line 162 or the terrestrial line 163 as a keyword (step S1804). The resource monitor response includes the bandwidth value, delay value, etc. of the satellite line 162 or the terrestrial line 163.

[0115] The video processing unit 51A of the first information processing infrastructure unit 50 in area A sends a resource monitor response to the resource monitor instruction of step S1802 to the network processing infrastructure unit 60 in area A (step S1805). The resource monitor response includes bandwidth values, delay times, etc. for the satellite line 162 or terrestrial line 163 at the video processing units 51A of the first to nth information processing infrastructure units 50 in area A. The network processing infrastructure unit 60 in area A sends a resource monitor response to the resource monitor instruction of step S1801 to the application service control unit 41 of the orchestrator 40 (step S1806). The resource monitor response includes bandwidth values, delay times, etc. for the satellite line 162 or terrestrial line 163 at the video processing units 51A of the first to nth information processing infrastructure units 50 in area A.

[0116] The application service control unit 41 of the orchestrator 40 issues a resource monitoring instruction to the video processing unit 51A of the first information processing infrastructure unit 50, using the video processing unit 51A of the first information processing infrastructure unit 50 as a keyword (step S1807). The video processing unit 51A of the first information processing infrastructure unit 50 issues a resource monitoring response to the resource monitoring instruction of step S1807 to the application service control unit 41 of the orchestrator 40 (step S1808). The resource monitoring response includes identification information of the first information processing infrastructure unit 50, the computer usage rate of the video processing unit 51A of the first information processing infrastructure unit 50, memory capacity, etc. Similar exchanges as in steps S1807 and S1808 are also performed between the application service control unit 41 of the orchestrator 40 and the video processing units 51A of the second to (n-1)th information processing infrastructure units 50. The application service control unit 41 of the orchestrator 40 issues a resource monitor instruction to the video processing unit 51A of the nth information processing infrastructure unit 50, using the video processing unit 51A of the nth information processing infrastructure unit 50 as a keyword (step S1809). The video processing unit 51A of the nth information processing infrastructure unit 50 issues a resource monitor response to the resource monitor instruction of step S1809 to the application service control unit 41 of the orchestrator 40 (step S1810). The resource monitor response includes identification information of the nth information processing infrastructure unit 50, the computer usage rate of the video processing unit 51A of the nth information processing infrastructure unit 50, memory capacity, etc.

[0117] 21 is a sequence diagram showing the operation of the HMD 110, which is the application terminal of the application 10 in the cyber-physical system 1 according to this embodiment, requesting and acquiring map information. The HMD 110, which is the application terminal of the application 10, issues an acquisition instruction to the application service control unit 41 of the orchestrator 40 to request a 3D image map using "Area A" as a keyword (step S1901). The application service control unit 41 of the orchestrator 40 then issues an acquisition instruction to the video processing unit 51A of the information processing infrastructure unit 50 in Area A to request a 3D image map using "Area A" as a keyword (step S1902). The video processing unit 51A of the information processing infrastructure unit 50 in Area A creates a 3D model (step S1903) and performs a map process to map the created 3D model on a map (step S1904). The video processing unit 51A of the information processing infrastructure unit 50 in Area A then notifies the application service control unit 41 of the orchestrator 40 of the required communication resource amount using "Area A" as a keyword (step S1905). The notification content includes information such as that the resource is targeted for the creation of a 3D image map, the type of application 10, the specific amount of resources, etc. The video processing unit 51A of the information processing infrastructure unit 50 in area A transmits the data of the 3D image map created in steps S1903 and S1904 to the HMD 110, which is the application terminal of the application 10 (step S1906).

[0118] (5-1) In this way, in the cyber-physical system 1, the service enabler 30 instructs the image processing unit 51A of the information processing infrastructure unit 50 to generate a 3D image for grasping the disaster situation based on an instruction from the application 10. The image processing unit 51A of the information processing infrastructure unit 50 executes calculation processing based on an instruction from the service enabler 30, using the image from the camera 74 obtained from the drone 170 and information to which information about the drone 170, such as time, position, and orientation, has been added. The operations in (5-1) up to this point correspond to the operations in the sequence diagram of FIG. 19 described above.

[0119] (5-2) The video processing unit 51A of the information processing infrastructure unit 50 performs resource monitoring so that a computer suitable for edge processing can be selected from among multiple computers connected via the line 160. In addition, the application service control unit 41 of the orchestrator 40 performs processing to confirm in advance whether the target computational processing can be performed, for example, whether there is available computing power, and further, whether the computer is located in a location with minimal communication processing delays, using the resource monitor of the network processing infrastructure unit 60. The operations in (5-2) up to this point correspond to the operations in the sequence diagram of FIG. 20 described above.

[0120] (5-3) The video processing unit 51A of the information processing infrastructure unit 50 creates 3D image models at short time intervals that are real-time, and maps the created 3D image models on an electronic map in cyberspace based on disaster location information. In other words, the video processing unit 51A of the information processing infrastructure unit 50 combines the 3D image models with map information.

[0121] (5-4) The image processing unit 51A of the information processing infrastructure unit 50 further adjusts the size of the synthesized map, i.e., the 3D image, to a size that is easy for the user to view. The 3D image whose size has been adjusted by the image processing unit 51A of the information processing infrastructure unit 50 is output from the CPS server 151 or the like to a router or the like of the line 160.

[0122] (5-5) The video processing unit 51A of the information processing infrastructure unit 50 may map the 3D image model as an icon on an electronic map, or may express it in a color-coded manner according to the level of danger. The color-coded expression may be, for example, red for danger, yellow for caution, and green for safety, from the most dangerous to the safest. The color-coded expression may be set by an administrator using the application 10 based on the situation assessment.

[0123] (5-6) In addition, the image processing unit 51A of the information processing infrastructure unit 50 can, in response to instructions from the application 10, time-synchronize images acquired from multiple drones 170 captured at different locations and map them on an electronic map, allowing multiple objects or target locations to be simultaneously observed on the HMD 110, which is the application terminal for the application 10. Furthermore, by displaying the position of the drone 170 on a map, the user of the HMD 110, which is the application terminal for the application 10, can grasp the situation, such as the location of the drone 170 observing the disaster. The operations from (5-3) to (5-6) above correspond to the operations in the sequence diagram of Figure 21 described above. Furthermore, the operations from (5-1) to (5-6) correspond to the operation in [8] described above.

[0124] 22 is a sequence diagram showing the operation when a user of the HMD 110, which is an application terminal of the application 10 in the cyber-physical system 1 according to the present embodiment, wishes to change the viewing direction of an object or a target point. The HMD 110, which is an application terminal of the application 10, accepts a user operation and sends an instruction to the service enabler 30, using "Area A" as a keyword, including information on a designated location indicating the location or direction of a disaster site or other location the user wants to view on a 3D image map (step S2001). The service enabler 30 sends an instruction to the application service control unit 41 of the orchestrator 40, using "Area A" as a keyword, including information on a designated location indicating the location or direction of a disaster site or other location the user wants to view on a 3D image map (step S2002). The application service control unit 41 of the orchestrator 40 sends an instruction to the video processing unit 51A of the information processing infrastructure unit 50, using "Area A" as a keyword, including information on a designated location indicating the location or direction of a disaster site or other location the user wants to view on a 3D image map (step S2003). The video processing unit 51A of the information processing infrastructure unit 50 performs space reproduction processing and space presentation processing based on the specified position instructed by the application service control unit 41 of the orchestrator 40 (step S2004). The video processing unit 51A of the information processing infrastructure unit 50 generates a 3D image based on a position from the specified position. The video processing unit 51A of the information processing infrastructure unit 50 transmits data of the generated 3D image to the HMD 110, which is the application terminal of the application 10 (step S2005).

[0125] (6-1) The information processing infrastructure unit 50 transmits the 3D image composited with the map in cyberspace to a display device such as an HMD 110 used by a remote operator, i.e., a user, who wants to understand the disaster situation in Area A. To do so, the information processing infrastructure unit 50 transmits the 3D image composited with the map in cyberspace to the network processing infrastructure unit 60 or the orchestrator 40 from the edge processing unit 53 of the information processing infrastructure unit 50 implemented on a computer close to Area A. At this time, the information processing infrastructure unit 50 notifies the orchestrator 40 of the amount of communication resources required to transmit the 3D image via the line 160, as described in the sequence diagram of FIG. 21 .

[0126] (6-2) The resource monitoring unit 61 of the network processing infrastructure unit 60 calculates the available network, i.e., the resources of the line 160, and considers a communication route that can secure communication resources "even in the event of a disaster" through the orchestrator 40. Considering a communication route means using a satellite line 162 or a terrestrial line 163, for example. If communication resources can be secured, the 3D image is transferred from the CPS server 151 or the like to a router or the like of the line 160 where communication resources are secured. If communication resources cannot be secured, the video processing unit 51A of the information processing infrastructure unit 50 adjusts the communication speed by performing processing such as lowering the frame rate of the 3D image to be transmitted or lowering the image quality. When considering a communication route in the resource monitoring unit 61 of the network processing infrastructure unit 60, emphasis may be placed on the reliability of communication obtained through the deterministic network unit 65, as in the operation shown in the sequence diagram of FIG. 11 .

[0127] (6-3) On the other hand, in order to check the disaster situation in 3D images, the remote operator, i.e., the user using the HMD 110, outputs the user's viewpoint direction as input information from the application 10 to the video processing unit 51A or the remote control unit 51E of the information processing infrastructure unit 50 via the service enabler 30 through the line 160 so that the target object or target location can be viewed from the desired direction. The input information of the user's viewpoint direction from the application 10 may be information obtained by means of the user's body movements such as gestures and hand movements.

[0128] (6-4) Next, the video processing unit 51A of the information processing infrastructure unit 50 generates 3D video from the direction the user wants to see by spatial reproduction and spatial presentation processing, and transmits the generated 3D video data to the HMD 110 used by the user via the line 160. As a process required for communication at this time, the orchestrator 40 and the network processing infrastructure unit 60 cooperate to secure communication resources, which is performed in the same manner as described above. The operations from (6-1) to (6-4) up to this point correspond to the operations in the sequence diagram of FIG. 22 described above.

[0129] (6-5) However, the resource monitoring unit 61 of the network processing infrastructure unit 60 determines in advance whether to transfer the 3D image resulting from the video processing over the satellite line 162 or the terrestrial line 163 by grasping the route status. However, there are differences in communication speeds between the uplink line 160 and the downlink line 160. Therefore, as shown in the sequence diagram of FIG. 3, the resource monitoring unit 61 of the network processing infrastructure unit 60 determines the route using information indicating the status of the line 160 obtained from resource monitoring by the orchestrator 40, the network processing infrastructure unit 60, etc., so that the results of network resource monitoring such as QoS, RTT, and throughput can be collected and appropriate communication routes can be selected for the uplink line 160 and the downlink line 160. The operations from (6-1) to (6-5) correspond to the operations from [9] to

[10] described above.

[0130] While the present embodiment describes the use of the cyber-physical system 1 during a disaster, it can also be used in peacetime to monitor infrastructure such as the condition of rivers and the deterioration of bridge piers. Furthermore, in the present embodiment, the cyber-physical system 1 requires both an application 10 for controlling the drone 170 and an application 10 for checking the status of debris. However, in order to collaborate with other companies, the cyber-physical system 1 can also be configured such that the remote control unit 51E of the information processing infrastructure unit 50 collaborates with another company's server 150 that includes the application 10 for controlling the drone 170 and the application 10 for checking the status of debris. Furthermore, the cyber-physical system 1 can collaborate with applications 10 that allow for communication delays.

[0131] The cyber-physical system 1 may also store information that allows a comparison of the situation before and after a disaster. The cyber-physical system 1 stores, for example, pre-disaster images and information on shooting conditions, such as the position of the drone 170 when the pre-disaster images were captured and the settings of the camera 74. The pre-disaster images may be, for example, bird's-eye views or omnidirectional images of specific objects. Since pre-disaster images of many locations are stored, the stored pre-disaster images may have low image quality. When processing and storing pre-disaster images, the cyber-physical system 1 may further store processing conditions. This allows the cyber-physical system 1 to cause the drone 170 to capture post-disaster images under the same shooting conditions as when the pre-disaster images were captured. Furthermore, a user can check the general state of the disaster by comparing the pre-disaster images with the post-disaster images.

[0132] Next, the hardware configuration of the platform system 20 will be described. In the platform system 20, the device group 70 is a mobility node such as a drone 170. The service enabler 30, the orchestrator 40, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 are realized by processing circuits. The processing circuits may be a processor and memory that executes a program stored in a memory, or may be dedicated hardware. The processing circuits are also called control circuits.

[0133] FIG. 23 is a diagram illustrating an example of the configuration of a processing circuit 90 that realizes the platform system 20 according to this embodiment, where the processing circuit is configured with a processor 91 and a memory 92. The processing circuit 90 illustrated in FIG. 23 is a control circuit and includes a processor 91 and a memory 92. When the processing circuit 90 is configured with the processor 91 and the memory 92, each function of the processing circuit 90 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 92. The processing circuit 90 realizes each function by having the processor 91 read and execute the program stored in the memory 92. That is, the processing circuit 90 includes the memory 92 for storing a program that results in the processing of the platform system 20 being executed. This program can also be said to be a program that causes the platform system 20 to execute each function realized by the processing circuit 90. This program may be provided by a storage medium on which the program is stored, or by other means such as a communication medium.

[0134] The above program can also be said to be a program that causes the platform system 20 to execute the following steps: a first step in which the device group 70 is equipped with a sensing function and outputs information obtained by the sensing function; a second step in which the information processing infrastructure unit 50 processes the information obtained from the device group 70 in response to a request from the application 10; a third step in which the network processing infrastructure unit 60 monitors the communication status when obtaining information from the device group 70 and controls communication with the device group 70; a fourth step in which the orchestrator 40 monitors the operation of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60 and performs collaborative control; and a fifth step in which the service enabler 30 instructs the information processing infrastructure unit 50 and the orchestrator 40 to operate based on a request from the application 10.

[0135] Here, the processor 91 is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic unit, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor), etc. The memory 92 is, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, or a DVD (Digital Versatile Disc).

[0136] FIG. 24 is a diagram illustrating an example of a processing circuit 93 that implements the platform system 20 according to this embodiment and is configured with dedicated hardware. The processing circuit 93 illustrated in FIG. 24 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit may be partially implemented with dedicated hardware and partially implemented with software or firmware. In this way, the processing circuit can implement each of the above-described functions using dedicated hardware, software, firmware, or a combination thereof.

[0137] The control circuit, which is a processing circuit that realizes the platform system 20, may be configured to be arranged in multiple servers 150 and the drone 170, rather than in a single server 150, such as the CPS server 151 and edge servers 154 and 155 shown in Fig. 2. Similarly, the storage medium storing the program for controlling the platform system 20 may be configured to be provided in multiple servers 150 and the drone 170, rather than in a single server 150, such as the CPS server 151 and edge servers 154 and 155 shown in Fig. 2.

[0138] As described above, according to the present embodiment, in the platform system 20, the orchestrator 40 acquires computer resources from the information processing infrastructure unit 50 and communication resources from the network processing infrastructure unit 60. Based on the acquired computer resources and communication resources, the orchestrator 40 monitors and controls the operations of the device group 70, the information processing infrastructure unit 50, and the network processing infrastructure unit 60. As a result, even in an abnormal system state, such as a disaster situation, where computer resources and communication resources change from moment to moment, the platform system 20 can provide the device group 70 with operability, reliability, and real-time performance with low latency by using the orchestrator 40 to comprehensively grasp and ensure the resource status. In a situation where at least one of the communication resources and computer resources changes, the platform system 20 can improve the probability of providing services by optimizing the sensing function and guiding the sensing function to a range where each resource can provide the desired service level.

[0139] The platform system 20 can provide high-quality images by providing operability, accuracy, and real-time performance with low latency to, for example, a drone 170 that captures 3D images, which is one of the device groups 70. The platform system 20 moves the drone 170, which is equipped with a sensing unit 73 such as a camera 74, to a shooting position that is more suitable for sensing and data transfer. For example, the platform system 20 can recognize collapsed houses from the entire disaster site captured in a wide-angle view by the camera 74 of the drone 170, and can approach the collapsed houses within a range that satisfies certain communication conditions to determine whether there are any survivors and whether there are any underlying factors that could cause a future fire. The platform system 20 then collects information using the camera 74 of the drone 170 and re-analyzes the information to evaluate hidden risks of human disasters, material disasters, and the like.

[0140] The platform system 20 selects a communication line that ensures reliable communication by switching between the satellite line 162 and the terrestrial line 163, which are highly resistant to disasters, or by using both the satellite line 162 and the terrestrial line 163, depending on the communication line conditions during a disaster, such as communication speed, communication quality, communication delay, and whether periodic communication is possible. In the platform system 20, the orchestrator 40 grasps the status of the operation and processing time of each sub-function in the information processing infrastructure unit 50 and the network processing infrastructure unit 60 so as to satisfy the required functions and performance, and performs control based on this grasped information, thereby appropriately grasping and controlling computer resources and communication resources that change in real time.

[0141] Note that in the present embodiment, an example has been described in which the orchestrator 40 determines, or selects, one communication path when there are multiple communication paths for the network processing infrastructure unit 60 to acquire information from the device group 70, but this is not limiting. The orchestrator 40 can also select multiple communication paths as communication paths for the network processing infrastructure unit 60 to acquire information from the device group 70. In this case, the drones 170 of the device group 70 simultaneously output information in which the same data has been encapsulated from the multiple communication paths. The network processing infrastructure unit 60 acquires the multiple pieces of information and restores the encapsulated data.

[0142] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, and parts of the configurations may be omitted or modified without departing from the spirit of the invention.

[0143] 1 Cyber-physical system, 10 Application, 20 Platform system, 30 Service enabler, 40 Orchestrator, 41 Application service control unit, 42, 51G Data collaboration unit, 43 Application collaboration unit, 44 E2E slice control unit, 45 E2E resource management unit, 50 Information processing infrastructure unit, 51 Platform application group, 51A Video processing unit, 51B Augmented space unit, 51C Space presentation unit, 51D Space reproduction unit, 51E Remote control unit, 51F Sensor sharing unit, 51H Sensor information analysis and prediction unit, 52, 61 Resource monitoring unit, 53 Edge processing unit, 54 Information security unit, 60 Network processing infrastructure unit, 62 Network operation management unit, 63 Network security unit, 64 Slicing unit, 65 Deterministic network unit, 70 Device group, 71 Clustering unit, 72 Space-time synchronization unit, 73 Sensing unit, 74 Camera, 75 Security unit, 76 Actuator unit, 90, 93 Processing circuit, 91 Processor, 92 Memory, 110 HMD, 150 Server, 151 CPS server, 154, 155 Edge server, 160 Line, 161 Wired line, 162 Satellite line, 163 Terrestrial line, 164, 165 L5G, 170 Drone, 170a Parent drone, 170b Child drone, 200 GPS.

Claims

1. A group of devices equipped with a sensing function and outputting information obtained by the said sensing function, An information processing infrastructure unit that processes the information acquired from the group of devices in response to a request from an application, A network processing infrastructure unit that monitors the communication status when acquiring the information from the group of devices and controls communication with the group of devices, An orchestrator that monitors and coordinates the operation of the aforementioned device group, the information processing infrastructure unit, and the network processing infrastructure unit, A service enabler that instructs the information processing infrastructure unit and the orchestrator to perform actions based on a request from the aforementioned application, A platform system characterized by comprising the following features.

2. The orchestrator acquires computing resources from the information processing infrastructure unit necessary to process tasks within the processing time requested by the user, acquires the communication status between the network processing infrastructure unit and the device group and communication resources necessary to process data transmission within the transmission time requested by the user from the network processing infrastructure unit, acquires location information from the device group, and monitors the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit to perform coordinated control. The platform system according to feature 1.

3. The network processing infrastructure unit has multiple communication paths for acquiring the information from the group of devices. The orchestrator determines the communication path for the network processing infrastructure to acquire the information from the device group, based on the computing resources and the communication resources. The platform system according to claim 2, characterized in that it is the same as described in claim 2.

4. If the orchestrator selects a plurality of communication paths as the communication path for the network processing infrastructure to acquire the information from the group of devices, The aforementioned group of devices outputs the information encapsulating the same data simultaneously from multiple communication paths. The network processing infrastructure unit acquires multiple pieces of information and restores the encapsulated data. The platform system according to feature 3.

5. The orchestrator, based on the computing resources and communication resources, notifies the service enabler if it cannot meet the service level required by the application. The service enabler, based on the notification from the orchestrator, requests the application to change the service level. The platform system according to claim 2, characterized in that it is the same as described in claim 2.

6. The aforementioned group of devices is an unmanned aerial vehicle, and the sensing function is a shooting function. The aforementioned unmanned aerial vehicle outputs image information captured by the aforementioned shooting function as the aforementioned information. The information processing infrastructure unit performs video processing using the image information for display on a display device according to the application. The platform system according to feature 1.

7. The information processing infrastructure unit performs a process to synthesize multiple pieces of information acquired from the device group as processing in response to a request from the application. The platform system according to feature 1.

8. The information processing infrastructure unit, if it is unable to obtain the first information from the first device included in the device group, but can obtain the second information from the second device included in the device group, and if communication is possible between the first device and the second device, instructs the first device to transmit the first information to the second device, and obtains the first information and the second information from the second device. A platform system according to any one of claims 1 to 7.

9. The information processing infrastructure unit includes a first device that transmits first information to the device group and a second device that transmits second information, and is capable of communicating with the second device. When communication is possible between the first device and the second device, it instructs the first device to transmit the first information to the second device and obtains composite information from the second device which is a combination of the first information and the second information. A platform system according to any one of claims 1 to 7.

10. A control circuit for controlling a platform system, The group of devices is equipped with a sensing function and outputs information obtained by the sensing function. The information processing infrastructure unit processes the information acquired from the device group in response to a request from the application. The network processing infrastructure unit monitors the communication status when acquiring the information from the device group and controls communication with the device group. The orchestrator monitors and coordinates the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit. The service enabler instructs the information processing infrastructure unit and the orchestrator to perform actions based on the request from the application. A control circuit characterized by causing the platform system to perform the above.

11. A storage medium in which a program for controlling a platform system is stored, The aforementioned program, The group of devices is equipped with a sensing function and outputs information obtained by the sensing function. The information processing infrastructure unit processes the information acquired from the device group in response to a request from the application. The network processing infrastructure unit monitors the communication status when acquiring the information from the device group and controls communication with the device group. The orchestrator monitors and coordinates the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit. The service enabler instructs the information processing infrastructure unit and the orchestrator to perform actions based on the request from the application. A storage medium characterized by causing the platform system to perform the above.

12. The first step involves a group of devices equipped with a sensing function and outputting information obtained by the sensing function, The second step involves the information processing infrastructure unit processing the information acquired from the device group in response to a request from the application, A third step involves the network processing infrastructure unit monitoring the communication status when acquiring the information from the device group and controlling communication with the device group. A fourth step involves the orchestrator monitoring the operation of the device group, the information processing infrastructure unit, and the network processing infrastructure unit and performing coordinated control. A fifth step in which the service enabler instructs the information processing infrastructure unit and the orchestrator to perform an action based on a request from the application, A communication method characterized by including